Merge branch 'release-v1.24.0' of github.com:matrix-org/synapse into matrix-org-hotfixes

This commit is contained in:
Patrick Cloke 2020-12-04 09:03:12 -05:00
commit a41b1dc49f
62 changed files with 153 additions and 71 deletions

View file

@ -1,3 +1,90 @@
Synapse 1.24.0rc2 (2020-12-04)
==============================
Bugfixes
--------
- Fix a regression in v1.24.0rc1 which failed to allow SAML mapping providers which were unable to redirect users to an additional page. ([\#8878](https://github.com/matrix-org/synapse/issues/8878))
Internal Changes
----------------
- Add support for the `prometheus_client` newer than 0.9.0. Contributed by Jordan Bancino. ([\#8875](https://github.com/matrix-org/synapse/issues/8875))
Synapse 1.24.0rc1 (2020-12-02)
==============================
Features
--------
- Add admin API for logging in as a user. ([\#8617](https://github.com/matrix-org/synapse/issues/8617))
- Allow specification of the SAML IdP if the metadata returns multiple IdPs. ([\#8630](https://github.com/matrix-org/synapse/issues/8630))
- Add support for re-trying generation of a localpart for OpenID Connect mapping providers. ([\#8801](https://github.com/matrix-org/synapse/issues/8801), [\#8855](https://github.com/matrix-org/synapse/issues/8855))
- Allow the `Date` header through CORS. Contributed by Nicolas Chamo. ([\#8804](https://github.com/matrix-org/synapse/issues/8804))
- Add a config option, `push.group_by_unread_count`, which controls whether unread message counts in push notifications are defined as "the number of rooms with unread messages" or "total unread messages". ([\#8820](https://github.com/matrix-org/synapse/issues/8820))
- Add `force_purge` option to delete-room admin api. ([\#8843](https://github.com/matrix-org/synapse/issues/8843))
Bugfixes
--------
- Fix a bug where appservices may be sent an excessive amount of read receipts and presence. Broke in v1.22.0. ([\#8744](https://github.com/matrix-org/synapse/issues/8744))
- Fix a bug in some federation APIs which could lead to unexpected behaviour if different parameters were set in the URI and the request body. ([\#8776](https://github.com/matrix-org/synapse/issues/8776))
- Fix a bug where synctl could spawn duplicate copies of a worker. Contributed by Waylon Cude. ([\#8798](https://github.com/matrix-org/synapse/issues/8798))
- Allow per-room profiles to be used for the server notice user. ([\#8799](https://github.com/matrix-org/synapse/issues/8799))
- Fix a bug where logging could break after a call to SIGHUP. ([\#8817](https://github.com/matrix-org/synapse/issues/8817))
- Fix `register_new_matrix_user` failing with "Bad Request" when trailing slash is included in server URL. Contributed by @angdraug. ([\#8823](https://github.com/matrix-org/synapse/issues/8823))
- Fix a minor long-standing bug in login, where we would offer the `password` login type if a custom auth provider supported it, even if password login was disabled. ([\#8835](https://github.com/matrix-org/synapse/issues/8835))
- Fix a long-standing bug which caused Synapse to require unspecified parameters during user-interactive authentication. ([\#8848](https://github.com/matrix-org/synapse/issues/8848))
- Fix a bug introduced in v1.20.0 where the user-agent and IP address reported during user registration for CAS, OpenID Connect, and SAML were of the wrong form. ([\#8784](https://github.com/matrix-org/synapse/issues/8784))
Improved Documentation
----------------------
- Clarify the usecase for a msisdn delegate. Contributed by Adrian Wannenmacher. ([\#8734](https://github.com/matrix-org/synapse/issues/8734))
- Remove extraneous comma from JSON example in User Admin API docs. ([\#8771](https://github.com/matrix-org/synapse/issues/8771))
- Update `turn-howto.md` with troubleshooting notes. ([\#8779](https://github.com/matrix-org/synapse/issues/8779))
- Fix the example on how to set the `Content-Type` header in nginx for the Client Well-Known URI. ([\#8793](https://github.com/matrix-org/synapse/issues/8793))
- Improve the documentation for the admin API to list all media in a room with respect to encrypted events. ([\#8795](https://github.com/matrix-org/synapse/issues/8795))
- Update the formatting of the `push` section of the homeserver config file to better align with the [code style guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format). ([\#8818](https://github.com/matrix-org/synapse/issues/8818))
- Improve documentation how to configure prometheus for workers. ([\#8822](https://github.com/matrix-org/synapse/issues/8822))
- Update example prometheus console. ([\#8824](https://github.com/matrix-org/synapse/issues/8824))
Deprecations and Removals
-------------------------
- Remove old `/_matrix/client/*/admin` endpoints which were deprecated since Synapse 1.20.0. ([\#8785](https://github.com/matrix-org/synapse/issues/8785))
- Disable pretty printing JSON responses for curl. Users who want pretty-printed output should use [jq](https://stedolan.github.io/jq/) in combination with curl. Contributed by @tulir. ([\#8833](https://github.com/matrix-org/synapse/issues/8833))
Internal Changes
----------------
- Simplify the way the `HomeServer` object caches its internal attributes. ([\#8565](https://github.com/matrix-org/synapse/issues/8565), [\#8851](https://github.com/matrix-org/synapse/issues/8851))
- Add an example and documentation for clock skew to the SAML2 sample configuration to allow for clock/time difference between the homserver and IdP. Contributed by @localguru. ([\#8731](https://github.com/matrix-org/synapse/issues/8731))
- Generalise `RoomMemberHandler._locally_reject_invite` to apply to more flows than just invite. ([\#8751](https://github.com/matrix-org/synapse/issues/8751))
- Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events. ([\#8754](https://github.com/matrix-org/synapse/issues/8754))
- Refactor test utilities for injecting HTTP requests. ([\#8757](https://github.com/matrix-org/synapse/issues/8757), [\#8758](https://github.com/matrix-org/synapse/issues/8758), [\#8759](https://github.com/matrix-org/synapse/issues/8759), [\#8760](https://github.com/matrix-org/synapse/issues/8760), [\#8761](https://github.com/matrix-org/synapse/issues/8761), [\#8777](https://github.com/matrix-org/synapse/issues/8777))
- Consolidate logic between the OpenID Connect and SAML code. ([\#8765](https://github.com/matrix-org/synapse/issues/8765))
- Use `TYPE_CHECKING` instead of magic `MYPY` variable. ([\#8770](https://github.com/matrix-org/synapse/issues/8770))
- Add a commandline script to sign arbitrary json objects. ([\#8772](https://github.com/matrix-org/synapse/issues/8772))
- Minor log line improvements for the SSO mapping code used to generate Matrix IDs from SSO IDs. ([\#8773](https://github.com/matrix-org/synapse/issues/8773))
- Add additional error checking for OpenID Connect and SAML mapping providers. ([\#8774](https://github.com/matrix-org/synapse/issues/8774), [\#8800](https://github.com/matrix-org/synapse/issues/8800))
- Add type hints to HTTP abstractions. ([\#8806](https://github.com/matrix-org/synapse/issues/8806), [\#8812](https://github.com/matrix-org/synapse/issues/8812))
- Remove unnecessary function arguments and add typing to several membership replication classes. ([\#8809](https://github.com/matrix-org/synapse/issues/8809))
- Optimise the lookup for an invite from another homeserver when trying to reject it. ([\#8815](https://github.com/matrix-org/synapse/issues/8815))
- Add tests for `password_auth_provider`s. ([\#8819](https://github.com/matrix-org/synapse/issues/8819))
- Drop redundant database index on `event_json`. ([\#8845](https://github.com/matrix-org/synapse/issues/8845))
- Simplify `uk.half-shot.msc2778.login.application_service` login handler. ([\#8847](https://github.com/matrix-org/synapse/issues/8847))
- Refactor `password_auth_provider` support code. ([\#8849](https://github.com/matrix-org/synapse/issues/8849))
- Add missing `ordering` to background database updates. ([\#8850](https://github.com/matrix-org/synapse/issues/8850))
- Allow for specifying a room version when creating a room in unit tests via `RestHelper.create_room_as`. ([\#8854](https://github.com/matrix-org/synapse/issues/8854))
Synapse 1.23.0 (2020-11-18)
===========================

View file

@ -1 +0,0 @@
Simplify the way the `HomeServer` object caches its internal attributes.

View file

@ -1 +0,0 @@
Add admin API for logging in as a user.

View file

@ -1 +0,0 @@
Allow specification of the SAML IdP if the metadata returns multiple IdPs.

View file

@ -1 +0,0 @@
Add an example and documentation for clock skew to the SAML2 sample configuration to allow for clock/time difference between the homserver and IdP. Contributed by @localguru.

View file

@ -1 +0,0 @@
Clarify the usecase for an msisdn delegate. Contributed by Adrian Wannenmacher.

View file

@ -1 +0,0 @@
Fix a bug where appservices may be sent an excessive amount of read receipts and presence. Broke in v1.22.0.

View file

@ -1 +0,0 @@
Generalise `RoomMemberHandler._locally_reject_invite` to apply to more flows than just invite.

View file

@ -1 +0,0 @@
Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events.

View file

@ -1 +0,0 @@
Refactor test utilities for injecting HTTP requests.

View file

@ -1 +0,0 @@
Refactor test utilities for injecting HTTP requests.

View file

@ -1 +0,0 @@
Refactor test utilities for injecting HTTP requests.

View file

@ -1 +0,0 @@
Refactor test utilities for injecting HTTP requests.

View file

@ -1 +0,0 @@
Refactor test utilities for injecting HTTP requests.

View file

@ -1 +0,0 @@
Consolidate logic between the OpenID Connect and SAML code.

View file

@ -1 +0,0 @@
Use `TYPE_CHECKING` instead of magic `MYPY` variable.

View file

@ -1 +0,0 @@
Remove extraneous comma from JSON example in User Admin API docs.

View file

@ -1 +0,0 @@
Add a commandline script to sign arbitrary json objects.

View file

@ -1 +0,0 @@
Minor log line improvements for the SSO mapping code used to generate Matrix IDs from SSO IDs.

View file

@ -1 +0,0 @@
Add additional error checking for OpenID Connect and SAML mapping providers.

View file

@ -1 +0,0 @@
Fix a bug in some federation APIs which could lead to unexpected behaviour if different parameters were set in the URI and the request body.

View file

@ -1 +0,0 @@
Refactor test utilities for injecting HTTP requests.

View file

@ -1 +0,0 @@
Update `turn-howto.md` with troubleshooting notes.

View file

@ -1 +0,0 @@
Fix a bug introduced in v1.20.0 where the user-agent and IP address reported during user registration for CAS, OpenID Connect, and SAML were of the wrong form.

View file

@ -1 +0,0 @@
Remove old `/_matrix/client/*/admin` endpoints which was deprecated since Synapse 1.20.0.

View file

@ -1 +0,0 @@
Fix the example on how to set the `Content-Type` header in nginx for the Client Well-Known URI.

View file

@ -1 +0,0 @@
Improve the documentation for the admin API to list all media in a room with respect to encrypted events.

View file

@ -1 +0,0 @@
Fix a bug where synctl could spawn duplicate copies of a worker. Contributed by Waylon Cude.

View file

@ -1 +0,0 @@
Allow per-room profiles to be used for the server notice user.

View file

@ -1 +0,0 @@
Add additional error checking for OpenID Connect and SAML mapping providers.

View file

@ -1 +0,0 @@
Add support for re-trying generation of a localpart for OpenID Connect mapping providers.

View file

@ -1 +0,0 @@
Allow Date header through CORS. Contributed by Nicolas Chamo.

View file

@ -1 +0,0 @@
Add type hints to HTTP abstractions.

View file

@ -1 +0,0 @@
Remove unnecessary function arguments and add typing to several membership replication classes.

View file

@ -1 +0,0 @@
Add type hints to HTTP abstractions.

View file

@ -1 +0,0 @@
Optimise the lookup for an invite from another homeserver when trying to reject it.

View file

@ -1 +0,0 @@
Fix bug where logging could break after a call to SIGHUP.

View file

@ -1 +0,0 @@
Update the formatting of the `push` section of the homeserver config file to better align with the [code style guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).

View file

@ -1 +0,0 @@
Add tests for `password_auth_provider`s.

View file

@ -1 +0,0 @@
Add a config option, `push.group_by_unread_count`, which controls whether unread message counts in push notifications are defined as "the number of rooms with unread messages" or "total unread messages".

View file

@ -1 +0,0 @@
Improve documentation how to configure prometheus for workers.

View file

@ -1 +0,0 @@
Fix `register_new_matrix_user` failing with "Bad Request" when trailing slash is included in server URL. Contributed by @angdraug.

View file

@ -1 +0,0 @@
Update example prometheus console.

View file

@ -1 +0,0 @@
Disable pretty printing JSON responses for curl. Users who want pretty-printed output should use [jq](https://stedolan.github.io/jq/) in combination with curl. Contributed by @tulir.

View file

@ -1 +0,0 @@
Fix minor long-standing bug in login, where we would offer the `password` login type if a custom auth provider supported it, even if password login was disabled.

View file

@ -1 +0,0 @@
Add `force_purge` option to delete-room admin api.

View file

@ -1 +0,0 @@
Drop redundant database index on `event_json`.

View file

@ -1 +0,0 @@
Simplify `uk.half-shot.msc2778.login.application_service` login handler.

View file

@ -1 +0,0 @@
Fix a long-standing bug which caused Synapse to require unspecified parameters during user-interactive authentication.

View file

@ -1 +0,0 @@
Refactor `password_auth_provider` support code.

View file

@ -1 +0,0 @@
Add missing `ordering` to background database updates.

View file

@ -1 +0,0 @@
Simplify the way the `HomeServer` object caches its internal attributes.

View file

@ -1 +0,0 @@
Allow for specifying a room version when creating a room in unit tests via `RestHelper.create_room_as`.

View file

@ -1 +0,0 @@
Add support for re-trying generation of a localpart for OpenID Connect mapping providers.

View file

@ -37,7 +37,7 @@ RUN pip install --prefix="/install" --no-warn-script-location \
jaeger-client \
opentracing \
# Match the version constraints of Synapse
"prometheus_client>=0.4.0,<0.9.0" \
"prometheus_client>=0.4.0" \
psycopg2 \
pycparser \
pyrsistent \

View file

@ -168,6 +168,13 @@ A custom mapping provider must specify the following methods:
the value of `mxid_localpart`.
* `emails` - A list of emails for the new user. If not provided, will
default to an empty list.
Alternatively it can raise a `synapse.api.errors.RedirectException` to
redirect the user to another page. This is useful to prompt the user for
additional information, e.g. if you want them to provide their own username.
It is the responsibility of the mapping provider to either redirect back
to `client_redirect_url` (including any additional information) or to
complete registration using methods from the `ModuleApi`.
### Default SAML Mapping Provider

View file

@ -48,7 +48,7 @@ try:
except ImportError:
pass
__version__ = "1.23.0"
__version__ = "1.24.0rc2"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when

View file

@ -888,7 +888,7 @@ class OidcHandler(BaseHandler):
# continue to already be in use. Note that the error raised is
# arbitrary and will get turned into a MappingException.
if failures:
raise RuntimeError(
raise MappingException(
"Mapping provider does not support de-duplicating Matrix IDs"
)

View file

@ -17,6 +17,7 @@ from typing import TYPE_CHECKING, Awaitable, Callable, List, Optional
import attr
from synapse.api.errors import RedirectException
from synapse.handlers._base import BaseHandler
from synapse.http.server import respond_with_html
from synapse.types import UserID, contains_invalid_mxid_characters
@ -28,7 +29,9 @@ logger = logging.getLogger(__name__)
class MappingException(Exception):
"""Used to catch errors when mapping the UserInfo object
"""Used to catch errors when mapping an SSO response to user attributes.
Note that the msg that is raised is shown to end-users.
"""
@ -145,6 +148,14 @@ class SsoHandler(BaseHandler):
sso_to_matrix_id_mapper: A callable to generate the user attributes.
The only parameter is an integer which represents the amount of
times the returned mxid localpart mapping has failed.
It is expected that the mapper can raise two exceptions, which
will get passed through to the caller:
MappingException if there was a problem mapping the response
to the user.
RedirectException to redirect to an additional page (e.g.
to prompt the user for more information).
grandfather_existing_users: A callable which can return an previously
existing matrix ID. The SSO ID is then linked to the returned
matrix ID.
@ -154,8 +165,8 @@ class SsoHandler(BaseHandler):
Raises:
MappingException if there was a problem mapping the response to a user.
RedirectException: some mapping providers may raise this if they need
to redirect to an interstitial page.
RedirectException: if the mapping provider needs to redirect the user
to an additional page. (e.g. to prompt for more information)
"""
# first of all, check if we already have a mapping for this user
@ -179,10 +190,16 @@ class SsoHandler(BaseHandler):
for i in range(self._MAP_USERNAME_RETRIES):
try:
attributes = await sso_to_matrix_id_mapper(i)
except (RedirectException, MappingException):
# Mapping providers are allowed to issue a redirect (e.g. to ask
# the user for more information) and can issue a mapping exception
# if a name cannot be generated.
raise
except Exception as e:
# Any other exception is unexpected.
raise MappingException(
"Could not extract user attributes from SSO response: " + str(e)
)
"Could not extract user attributes from SSO response."
) from e
logger.debug(
"Retrieved user attributes from user mapping provider: %r (attempt %d)",

View file

@ -40,6 +40,10 @@ logger = logging.getLogger(__name__)
# Note that these both represent runtime dependencies (and the versions
# installed are checked at runtime).
#
# Also note that we replicate these constraints in the Synapse Dockerfile while
# pre-installing dependencies. If these constraints are updated here, the same
# change should be made in the Dockerfile.
#
# [1] https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers.
REQUIREMENTS = [
@ -69,14 +73,7 @@ REQUIREMENTS = [
"msgpack>=0.5.2",
"phonenumbers>=8.2.0",
# we use GaugeHistogramMetric, which was added in prom-client 0.4.0.
# prom-client has a history of breaking backwards compatibility between
# minor versions (https://github.com/prometheus/client_python/issues/317),
# so we also pin the minor version.
#
# Note that we replicate these constraints in the Synapse Dockerfile while
# pre-installing dependencies. If these constraints are updated here, the
# same change should be made in the Dockerfile.
"prometheus_client>=0.4.0,<0.9.0",
"prometheus_client>=0.4.0",
# we use attr.validators.deep_iterable, which arrived in 19.1.0 (Note:
# Fedora 31 only has 19.1, so if we want to upgrade we should wait until 33
# is out in November.)

View file

@ -705,8 +705,7 @@ class OidcHandlerTestCase(HomeserverTestCase):
MappingException,
)
self.assertEqual(
str(e.value),
"Could not extract user attributes from SSO response: Mapping provider does not support de-duplicating Matrix IDs",
str(e.value), "Mapping provider does not support de-duplicating Matrix IDs",
)
@override_config({"oidc_config": {"allow_existing_users": True}})

View file

@ -14,6 +14,7 @@
import attr
from synapse.api.errors import RedirectException
from synapse.handlers.sso import MappingException
from tests.unittest import HomeserverTestCase, override_config
@ -49,6 +50,13 @@ class TestMappingProvider:
return {"mxid_localpart": localpart, "displayname": None}
class TestRedirectMappingProvider(TestMappingProvider):
def saml_response_to_user_attributes(
self, saml_response, failures, client_redirect_url
):
raise RedirectException(b"https://custom-saml-redirect/")
class SamlHandlerTestCase(HomeserverTestCase):
def default_config(self):
config = super().default_config()
@ -166,3 +174,23 @@ class SamlHandlerTestCase(HomeserverTestCase):
self.assertEqual(
str(e.value), "Unable to generate a Matrix ID from the SSO response"
)
@override_config(
{
"saml2_config": {
"user_mapping_provider": {
"module": __name__ + ".TestRedirectMappingProvider"
},
}
}
)
def test_map_saml_response_redirect(self):
saml_response = FakeAuthnResponse({"uid": "test", "username": "test_user"})
redirect_url = ""
e = self.get_failure(
self.handler._map_saml_response_to_user(
saml_response, redirect_url, "user-agent", "10.10.10.10"
),
RedirectException,
)
self.assertEqual(e.value.location, b"https://custom-saml-redirect/")