Merge remote-tracking branch 'origin/release-v1.28.0' into matrix-org-hotfixes

This commit is contained in:
Erik Johnston 2021-02-19 10:06:01 +00:00
commit 5054eb291e
49 changed files with 148 additions and 87 deletions

View file

@ -14,7 +14,7 @@ jobs:
platforms: linux/amd64
- docker_build:
tag: -t matrixdotorg/synapse:${CIRCLE_TAG}
platforms: linux/amd64,linux/arm/v7,linux/arm64
platforms: linux/amd64,linux/arm64
dockerhubuploadlatest:
docker:
@ -27,7 +27,7 @@ jobs:
# until all of the platforms are built.
- docker_build:
tag: -t matrixdotorg/synapse:latest
platforms: linux/amd64,linux/arm/v7,linux/arm64
platforms: linux/amd64,linux/arm64
workflows:
build:

View file

@ -1,9 +1,98 @@
Synapse 1.28.0rc1 (2021-02-18)
==============================
Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).
This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.
Removal warning
---------------
The v1 list accounts API is deprecated and will be removed in a future release.
This API was undocumented and misleading. It can be replaced by the
[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
which has been available since Synapse 1.7.0 (2019-12-13).
Please check if you're using any scripts which use the admin API and replace
`GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.
Features
--------
- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150))
- Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301))
- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311))
- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376))
- Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400))
Bugfixes
--------
- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257))
- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296))
- Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393))
- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333))
- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361))
- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391))
- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395))
- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421))
- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425))
- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426))
- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428))
Updates to the Docker image
---------------------------
- Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433))
Improved Documentation
----------------------
- Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281))
- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291))
- Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308))
- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317))
- Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322))
- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390))
- Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404))
- Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407))
- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434))
Deprecations and Removals
-------------------------
- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429))
Internal Changes
----------------
- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003))
- Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123))
- Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240))
- Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299))
- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305))
- Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307))
- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326))
- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362))
- Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396))
- Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381))
- Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384))
- Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394))
- Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))
Synapse 1.27.0 (2021-02-16)
===========================
Note that this release includes a change in Synapse to use Redis as a cache ─ as well as a pub/sub mechanism ─ if Redis support is enabled for workers. No action is needed by server administrators, and we do not expect resource usage of the Redis instance to change dramatically.
This release also changes the callback URI for OpenID Connect (OIDC) identity providers. If your server is configured to use single sign-on via an OIDC/OAuth2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.
This release also changes the callback URI for OpenID Connect (OIDC) and SAML2 identity providers. If your server is configured to use single sign-on via an OIDC/OAuth2 or SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.
This release also changes escaping of variables in the HTML templates for SSO or email notifications. If you have customised these templates, please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.

View file

@ -88,20 +88,21 @@ for example:
Upgrading to v1.27.0
====================
Changes to callback URI for OAuth2 / OpenID Connect
---------------------------------------------------
Changes to callback URI for OAuth2 / OpenID Connect and SAML2
-------------------------------------------------------------
This version changes the URI used for callbacks from OAuth2 identity providers. If
your server is configured for single sign-on via an OpenID Connect or OAuth2 identity
provider, you will need to add ``[synapse public baseurl]/_synapse/client/oidc/callback``
to the list of permitted "redirect URIs" at the identity provider.
This version changes the URI used for callbacks from OAuth2 and SAML2 identity providers:
See `docs/openid.md <docs/openid.md>`_ for more information on setting up OpenID
Connect.
* If your server is configured for single sign-on via an OpenID Connect or OAuth2 identity
provider, you will need to add ``[synapse public baseurl]/_synapse/client/oidc/callback``
to the list of permitted "redirect URIs" at the identity provider.
(Note: a similar change is being made for SAML2; in this case the old URI
``[synapse public baseurl]/_matrix/saml2`` is being deprecated, but will continue to
work, so no immediate changes are required for existing installations.)
See `docs/openid.md <docs/openid.md>`_ for more information on setting up OpenID
Connect.
* If your server is configured for single sign-on via a SAML2 identity provider, you will
need to add ``[synapse public baseurl]/_synapse/client/saml2/authn_response`` as a permitted
"ACS location" (also known as "allowed callback URLs") at the identity provider.
Changes to HTML templates
-------------------------

View file

@ -1 +0,0 @@
Fix 'object name reserved for internal use' errors with recent versions of SQLite.

View file

@ -1 +0,0 @@
Add experimental support for running Synapse with PyPy.

View file

@ -1 +0,0 @@
New API /_synapse/admin/rooms/{roomId}/context/{eventId}.

View file

@ -1 +0,0 @@
Deny access to additional IP addresses by default.

View file

@ -1 +0,0 @@
Fix long-standing bug where sending email push would fail for rooms that the server had since left.

View file

@ -1 +0,0 @@
Reorganizing CHANGELOG.md.

View file

@ -1 +0,0 @@
Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable.

View file

@ -1 +0,0 @@
Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted.

View file

@ -1 +0,0 @@
Update the `Cursor` type hints to better match PEP 249.

View file

@ -1 +0,0 @@
Further improvements to the user experience of registration via single sign-on.

View file

@ -1 +0,0 @@
Further improvements to the user experience of registration via single sign-on.

View file

@ -1 +0,0 @@
Add debug logging for SRV lookups. Contributed by @Bubu.

View file

@ -1 +0,0 @@
Improve logging for OIDC login flow.

View file

@ -1 +0,0 @@
Correct name of Synapse's service file in TURN howto.

View file

@ -1 +0,0 @@
Add hook to spam checker modules that allow checking file uploads and remote downloads.

View file

@ -1 +0,0 @@
Fix the braces in the `oidc_providers` section of the sample config.

View file

@ -1 +0,0 @@
Assert a maximum length for some parameters for spec compliance.

View file

@ -1 +0,0 @@
Update installation instructions on Fedora.

View file

@ -1 +0,0 @@
Share the code for handling required attributes between the CAS and SAML handlers.

View file

@ -1 +0,0 @@
Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.".

View file

@ -1 +0,0 @@
Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`.

View file

@ -1 +0,0 @@
Clean up the code to load the metadata for OpenID Connect identity providers.

View file

@ -1 +0,0 @@
Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s.

View file

@ -1 +0,0 @@
Convert tests to use `HomeserverTestCase`.

View file

@ -1 +0,0 @@
Update the version of black used to 20.8b1.

View file

@ -1 +0,0 @@
Allow OIDC config to override discovered values.

View file

@ -1 +0,0 @@
Fix bug where Synapse would occaisonally stop reconnecting after the connection was lost.

View file

@ -1 +0,0 @@
Assert a maximum length for some parameters for spec compliance.

View file

@ -1 +0,0 @@
Remove some dead code from the acceptance of room invites path.

View file

@ -1 +0,0 @@
Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'".

View file

@ -1 +0,0 @@
Convert tests to use `HomeserverTestCase`.

View file

@ -1 +0,0 @@
Add the shadow-banning status to the display user admin API.

View file

@ -1 +0,0 @@
Update docs for using Gitea as OpenID provider.

View file

@ -1 +0,0 @@
Document that pusher instances are shardable.

View file

@ -1 +0,0 @@
Clean up an unused method in the presence handler code.

View file

@ -1 +0,0 @@
Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication.

View file

@ -40,12 +40,12 @@ the reverse proxy and the homeserver.
```
server {
listen 443 ssl;
listen [::]:443 ssl;
listen 443 ssl http2;
listen [::]:443 ssl http2;
# For the federation port
listen 8448 ssl default_server;
listen [::]:8448 ssl default_server;
listen 8448 ssl http2 default_server;
listen [::]:8448 ssl http2 default_server;
server_name matrix.example.com;

View file

@ -2228,8 +2228,8 @@ password_config:
#require_uppercase: true
ui_auth:
# The number of milliseconds to allow a user-interactive authentication
# session to be active.
# The amount of time to allow a user-interactive authentication session
# to be active.
#
# This defaults to 0, meaning the user is queried for their credentials
# before every action, but this can be overridden to allow a single
@ -2240,7 +2240,7 @@ ui_auth:
# Uncomment below to allow for credential validation to last for 15
# seconds.
#
#session_timeout: 15000
#session_timeout: "15s"
# Configuration for sending emails from Synapse.

View file

@ -48,7 +48,7 @@ try:
except ImportError:
pass
__version__ = "1.27.0"
__version__ = "1.28.0rc1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when

View file

@ -37,7 +37,9 @@ class AuthConfig(Config):
# User-interactive authentication
ui_auth = config.get("ui_auth") or {}
self.ui_auth_session_timeout = ui_auth.get("session_timeout", 0)
self.ui_auth_session_timeout = self.parse_duration(
ui_auth.get("session_timeout", 0)
)
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """\
@ -93,8 +95,8 @@ class AuthConfig(Config):
#require_uppercase: true
ui_auth:
# The number of milliseconds to allow a user-interactive authentication
# session to be active.
# The amount of time to allow a user-interactive authentication session
# to be active.
#
# This defaults to 0, meaning the user is queried for their credentials
# before every action, but this can be overridden to allow a single
@ -105,5 +107,5 @@ class AuthConfig(Config):
# Uncomment below to allow for credential validation to last for 15
# seconds.
#
#session_timeout: 15000
#session_timeout: "15s"
"""

View file

@ -143,6 +143,10 @@ class UserDirectoryHandler(StateDeltasHandler):
if self.pos is None:
self.pos = await self.store.get_user_directory_stream_pos()
# If still None then the initial background update hasn't happened yet.
if self.pos is None:
return None
# Loop round handling deltas until we're up to date
while True:
with Measure(self.clock, "user_dir_delta"):

View file

@ -56,7 +56,7 @@ from twisted.web.client import (
)
from twisted.web.http import PotentialDataLoss
from twisted.web.http_headers import Headers
from twisted.web.iweb import IAgent, IBodyProducer, IResponse
from twisted.web.iweb import UNKNOWN_LENGTH, IAgent, IBodyProducer, IResponse
from synapse.api.errors import Codes, HttpResponseException, SynapseError
from synapse.http import QuieterFileBodyProducer, RequestTimedOutError, redact_uri
@ -408,6 +408,9 @@ class SimpleHttpClient:
agent=self.agent,
data=body_producer,
headers=headers,
# Avoid buffering the body in treq since we do not reuse
# response bodies.
unbuffered=True,
**self._extra_treq_args,
) # type: defer.Deferred
@ -702,18 +705,6 @@ class SimpleHttpClient:
resp_headers = dict(response.headers.getAllRawHeaders())
if (
b"Content-Length" in resp_headers
and max_size
and int(resp_headers[b"Content-Length"][0]) > max_size
):
logger.warning("Requested URL is too large > %r bytes" % (max_size,))
raise SynapseError(
502,
"Requested file is too large > %r bytes" % (max_size,),
Codes.TOO_LARGE,
)
if response.code > 299:
logger.warning("Got %d when downloading %s" % (response.code, url))
raise SynapseError(502, "Got error %d" % (response.code,), Codes.UNKNOWN)
@ -780,7 +771,9 @@ class _ReadBodyWithMaxSizeProtocol(protocol.Protocol):
# in the meantime.
if self.max_size is not None and self.length >= self.max_size:
self.deferred.errback(BodyExceededMaxSize())
self.transport.loseConnection()
# Close the connection (forcefully) since all the data will get
# discarded anyway.
self.transport.abortConnection()
def connectionLost(self, reason: Failure) -> None:
# If the maximum size was already exceeded, there's nothing to do.
@ -814,6 +807,11 @@ def read_body_with_max_size(
Returns:
A Deferred which resolves to the length of the read body.
"""
# If the Content-Length header gives a size larger than the maximum allowed
# size, do not bother downloading the body.
if max_size is not None and response.length != UNKNOWN_LENGTH:
if response.length > max_size:
return defer.fail(BodyExceededMaxSize())
d = defer.Deferred()
response.deliverBody(_ReadBodyWithMaxSizeProtocol(stream, d, max_size))

View file

@ -54,11 +54,7 @@ def build_synapse_client_resource_tree(hs: "HomeServer") -> Mapping[str, Resourc
if hs.config.saml2_enabled:
from synapse.rest.synapse.client.saml2 import SAML2Resource
res = SAML2Resource(hs)
resources["/_synapse/client/saml2"] = res
# This is also mounted under '/_matrix' for backwards-compatibility.
resources["/_matrix/saml2"] = res
resources["/_synapse/client/saml2"] = SAML2Resource(hs)
return resources

View file

@ -707,7 +707,13 @@ class UserDirectoryStore(UserDirectoryBackgroundUpdateStore):
return {row["room_id"] for row in rows}
async def get_user_directory_stream_pos(self) -> int:
async def get_user_directory_stream_pos(self) -> Optional[int]:
"""
Get the stream ID of the user directory stream.
Returns:
The stream token or None if the initial background update hasn't happened yet.
"""
return await self.db_pool.simple_select_one_onecol(
table="user_directory_stream_pos",
keyvalues={},

View file

@ -18,6 +18,7 @@ from mock import Mock
from twisted.python.failure import Failure
from twisted.web.client import ResponseDone
from twisted.web.iweb import UNKNOWN_LENGTH
from synapse.http.client import BodyExceededMaxSize, read_body_with_max_size
@ -27,12 +28,12 @@ from tests.unittest import TestCase
class ReadBodyWithMaxSizeTests(TestCase):
def setUp(self):
"""Start reading the body, returns the response, result and proto"""
self.response = Mock()
response = Mock(length=UNKNOWN_LENGTH)
self.result = BytesIO()
self.deferred = read_body_with_max_size(self.response, self.result, 6)
self.deferred = read_body_with_max_size(response, self.result, 6)
# Fish the protocol out of the response.
self.protocol = self.response.deliverBody.call_args[0][0]
self.protocol = response.deliverBody.call_args[0][0]
self.protocol.transport = Mock()
def _cleanup_error(self):
@ -88,7 +89,7 @@ class ReadBodyWithMaxSizeTests(TestCase):
self.protocol.dataReceived(b"1234567890")
self.assertIsInstance(self.deferred.result, Failure)
self.assertIsInstance(self.deferred.result.value, BodyExceededMaxSize)
self.protocol.transport.loseConnection.assert_called_once()
self.protocol.transport.abortConnection.assert_called_once()
# More data might have come in.
self.protocol.dataReceived(b"1234567890")

View file

@ -343,7 +343,7 @@ class UIAuthTests(unittest.HomeserverTestCase):
},
)
@unittest.override_config({"ui_auth": {"session_timeout": 5 * 1000}})
@unittest.override_config({"ui_auth": {"session_timeout": "5s"}})
def test_can_reuse_session(self):
"""
The session can be reused if configured.