Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes

This commit is contained in:
Erik Johnston 2020-01-31 15:06:16 +00:00
commit c854d255e5
158 changed files with 1736 additions and 565 deletions

View file

@ -1,3 +1,96 @@
Synapse 1.9.1 (2020-01-28)
==========================
Bugfixes
--------
- Fix bug where setting `mau_limit_reserved_threepids` config would cause Synapse to refuse to start. ([\#6793](https://github.com/matrix-org/synapse/issues/6793))
Synapse 1.9.0 (2020-01-23)
==========================
**WARNING**: As of this release, Synapse no longer supports versions of SQLite before 3.11, and will refuse to start when configured to use an older version. Administrators are recommended to migrate their database to Postgres (see instructions [here](docs/postgres.md)).
If your Synapse deployment uses workers, note that the reverse-proxy configurations for the `synapse.app.media_repository`, `synapse.app.federation_reader` and `synapse.app.event_creator` workers have changed, with the addition of a few paths (see the updated configurations [here](docs/workers.md#available-worker-applications)). Existing configurations will continue to work.
Improved Documentation
----------------------
- Fix endpoint documentation for the List Rooms admin API. ([\#6770](https://github.com/matrix-org/synapse/issues/6770))
Synapse 1.9.0rc1 (2020-01-22)
=============================
Features
--------
- Allow admin to create or modify a user. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5742](https://github.com/matrix-org/synapse/issues/5742))
- Add new quarantine media admin APIs to quarantine by media ID or by user who uploaded the media. ([\#6681](https://github.com/matrix-org/synapse/issues/6681), [\#6756](https://github.com/matrix-org/synapse/issues/6756))
- Add `org.matrix.e2e_cross_signing` to `unstable_features` in `/versions` as per [MSC1756](https://github.com/matrix-org/matrix-doc/pull/1756). ([\#6712](https://github.com/matrix-org/synapse/issues/6712))
- Add a new admin API to list and filter rooms on the server. ([\#6720](https://github.com/matrix-org/synapse/issues/6720))
Bugfixes
--------
- Correctly proxy HTTP errors due to API calls to remote group servers. ([\#6654](https://github.com/matrix-org/synapse/issues/6654))
- Fix media repo admin APIs when using a media worker. ([\#6664](https://github.com/matrix-org/synapse/issues/6664))
- Fix "CRITICAL" errors being logged when a request is received for a uri containing non-ascii characters. ([\#6682](https://github.com/matrix-org/synapse/issues/6682))
- Fix a bug where we would assign a numeric user ID if somebody tried registering with an empty username. ([\#6690](https://github.com/matrix-org/synapse/issues/6690))
- Fix `purge_room` admin API. ([\#6711](https://github.com/matrix-org/synapse/issues/6711))
- Fix a bug causing Synapse to not always purge quiet rooms with a low `max_lifetime` in their message retention policies when running the automated purge jobs. ([\#6714](https://github.com/matrix-org/synapse/issues/6714))
- Fix the `synapse_port_db` not correctly running background updates. Thanks @tadzik for reporting. ([\#6718](https://github.com/matrix-org/synapse/issues/6718))
- Fix changing password via user admin API. ([\#6730](https://github.com/matrix-org/synapse/issues/6730))
- Fix `/events/:event_id` deprecated API. ([\#6731](https://github.com/matrix-org/synapse/issues/6731))
- Fix monthly active user limiting support for worker mode, fixes [#4639](https://github.com/matrix-org/synapse/issues/4639). ([\#6742](https://github.com/matrix-org/synapse/issues/6742))
- Fix bug when setting `account_validity` to an empty block in the config. Thanks to @Sorunome for reporting. ([\#6747](https://github.com/matrix-org/synapse/issues/6747))
- Fix `AttributeError: 'NoneType' object has no attribute 'get'` in `hash_password` when configuration has an empty `password_config`. Contributed by @ivilata. ([\#6753](https://github.com/matrix-org/synapse/issues/6753))
- Fix the `docker-compose.yaml` overriding the entire `/etc` folder of the container. Contributed by Fabian Meyer. ([\#6656](https://github.com/matrix-org/synapse/issues/6656))
Improved Documentation
----------------------
- Fix a typo in the configuration example for purge jobs in the sample configuration file. ([\#6621](https://github.com/matrix-org/synapse/issues/6621))
- Add complete documentation of the message retention policies support. ([\#6624](https://github.com/matrix-org/synapse/issues/6624), [\#6665](https://github.com/matrix-org/synapse/issues/6665))
- Add some helpful tips about changelog entries to the GitHub pull request template. ([\#6663](https://github.com/matrix-org/synapse/issues/6663))
- Clarify the `account_validity` and `email` sections of the sample configuration. ([\#6685](https://github.com/matrix-org/synapse/issues/6685))
- Add more endpoints to the documentation for Synapse workers. ([\#6698](https://github.com/matrix-org/synapse/issues/6698))
Deprecations and Removals
-------------------------
- Synapse no longer supports versions of SQLite before 3.11, and will refuse to start when configured to use an older version. Administrators are recommended to migrate their database to Postgres (see instructions [here](docs/postgres.md)). ([\#6675](https://github.com/matrix-org/synapse/issues/6675))
Internal Changes
----------------
- Add `local_current_membership` table for tracking local user membership state in rooms. ([\#6655](https://github.com/matrix-org/synapse/issues/6655), [\#6728](https://github.com/matrix-org/synapse/issues/6728))
- Port `synapse.replication.tcp` to async/await. ([\#6666](https://github.com/matrix-org/synapse/issues/6666))
- Fixup `synapse.replication` to pass mypy checks. ([\#6667](https://github.com/matrix-org/synapse/issues/6667))
- Allow `additional_resources` to implement `IResource` directly. ([\#6686](https://github.com/matrix-org/synapse/issues/6686))
- Allow REST endpoint implementations to raise a `RedirectException`, which will redirect the user's browser to a given location. ([\#6687](https://github.com/matrix-org/synapse/issues/6687))
- Updates and extensions to the module API. ([\#6688](https://github.com/matrix-org/synapse/issues/6688))
- Updates to the SAML mapping provider API. ([\#6689](https://github.com/matrix-org/synapse/issues/6689), [\#6723](https://github.com/matrix-org/synapse/issues/6723))
- Remove redundant `RegistrationError` class. ([\#6691](https://github.com/matrix-org/synapse/issues/6691))
- Don't block processing of incoming EDUs behind processing PDUs in the same transaction. ([\#6697](https://github.com/matrix-org/synapse/issues/6697))
- Remove duplicate check for the `session` query parameter on the `/auth/xxx/fallback/web` Client-Server endpoint. ([\#6702](https://github.com/matrix-org/synapse/issues/6702))
- Attempt to retry sending a transaction when we detect a remote server has come back online, rather than waiting for a transaction to be triggered by new data. ([\#6706](https://github.com/matrix-org/synapse/issues/6706))
- Add `StateMap` type alias to simplify types. ([\#6715](https://github.com/matrix-org/synapse/issues/6715))
- Add a `DeltaState` to track changes to be made to current state during event persistence. ([\#6716](https://github.com/matrix-org/synapse/issues/6716))
- Add more logging around message retention policies support. ([\#6717](https://github.com/matrix-org/synapse/issues/6717))
- When processing a SAML response, log the assertions for easier configuration. ([\#6724](https://github.com/matrix-org/synapse/issues/6724))
- Fixup `synapse.rest` to pass mypy. ([\#6732](https://github.com/matrix-org/synapse/issues/6732), [\#6764](https://github.com/matrix-org/synapse/issues/6764))
- Fixup `synapse.api` to pass mypy. ([\#6733](https://github.com/matrix-org/synapse/issues/6733))
- Allow streaming cache 'invalidate all' to workers. ([\#6749](https://github.com/matrix-org/synapse/issues/6749))
- Remove unused CI docker compose files. ([\#6754](https://github.com/matrix-org/synapse/issues/6754))
Synapse 1.8.0 (2020-01-09)
==========================

View file

@ -76,6 +76,15 @@ for example:
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
Upgrading to **<NEXT_VERSION>**
===============================
Synapse will now log a warning on start up if used with a PostgreSQL database
that has a non-recommended locale set.
See [docs/postgres.md](docs/postgres.md) for details.
Upgrading to v1.8.0
===================

View file

@ -1 +0,0 @@
Allow admin to create or modify a user. Contributed by Awesome Technologies Innovationslabor GmbH.

View file

@ -1 +0,0 @@
Fix a typo in the configuration example for purge jobs in the sample configuration file.

View file

@ -1 +0,0 @@
Add complete documentation of the message retention policies support.

View file

@ -1 +0,0 @@
Correctly proxy HTTP errors due to API calls to remote group servers.

View file

@ -1 +0,0 @@
Add `local_current_membership` table for tracking local user membership state in rooms.

View file

@ -1 +0,0 @@
No more overriding the entire /etc folder of the container in docker-compose.yaml. Contributed by Fabian Meyer.

View file

@ -1 +0,0 @@
Add some helpful tips about changelog entries to the github pull request template.

View file

@ -1 +0,0 @@
Fix media repo admin APIs when using a media worker.

View file

@ -1 +0,0 @@
Add complete documentation of the message retention policies support.

View file

@ -1 +0,0 @@
Port `synapse.replication.tcp` to async/await.

View file

@ -1 +0,0 @@
Fixup `synapse.replication` to pass mypy checks.

View file

@ -1 +0,0 @@
Synapse no longer supports versions of SQLite before 3.11, and will refuse to start when configured to use an older version. Administrators are recommended to migrate their database to Postgres (see instructions [here](docs/postgres.md)).

View file

@ -1 +0,0 @@
Add new quarantine media admin APIs to quarantine by media ID or by user who uploaded the media.

View file

@ -1,2 +0,0 @@
Fix "CRITICAL" errors being logged when a request is received for a uri containing non-ascii characters.

View file

@ -1 +0,0 @@
Clarify the `account_validity` and `email` sections of the sample configuration.

View file

@ -1 +0,0 @@
Allow additional_resources to implement IResource directly.

View file

@ -1 +0,0 @@
Allow REST endpoint implementations to raise a RedirectException, which will redirect the user's browser to a given location.

View file

@ -1 +0,0 @@
Updates and extensions to the module API.

View file

@ -1 +0,0 @@
Updates to the SAML mapping provider API.

View file

@ -1 +0,0 @@
Fix a bug where we would assign a numeric userid if somebody tried registering with an empty username.

View file

@ -1 +0,0 @@
Remove redundant RegistrationError class.

View file

@ -1 +0,0 @@
Don't block processing of incoming EDUs behind processing PDUs in the same transaction.

View file

@ -1 +0,0 @@
Add more endpoints to the documentation for Synapse workers.

View file

@ -1 +0,0 @@
Remove duplicate check for the `session` query parameter on the `/auth/xxx/fallback/web` Client-Server endpoint.

View file

@ -1 +0,0 @@
Attempt to retry sending a transaction when we detect a remote server has come back online, rather than waiting for a transaction to be triggered by new data.

View file

@ -1 +0,0 @@
Fix `purge_room` admin API.

View file

@ -1 +0,0 @@
Add org.matrix.e2e_cross_signing to unstable_features in /versions as per [MSC1756](https://github.com/matrix-org/matrix-doc/pull/1756).

View file

@ -1 +0,0 @@
Fix a bug causing Synapse to not always purge quiet rooms with a low `max_lifetime` in their message retention policies when running the automated purge jobs.

View file

@ -1 +0,0 @@
Add StateMap type alias to simplify types.

View file

@ -1 +0,0 @@
Add a `DeltaState` to track changes to be made to current state during event persistence.

View file

@ -1 +0,0 @@
Add more logging around message retention policies support.

View file

@ -1 +0,0 @@
Fix a bug causing the `synapse_port_db` script to return 0 in a specific error case.

View file

@ -1 +0,0 @@
Add a new admin API to list and filter rooms on the server.

View file

@ -1 +0,0 @@
Updates to the SAML mapping provider API.

View file

@ -1 +0,0 @@
When processing a SAML response, log the assertions for easier configuration.

View file

@ -1 +0,0 @@
Add `local_current_membership` table for tracking local user membership state in rooms.

1
changelog.d/6729.misc Normal file
View file

@ -0,0 +1 @@
Record room versions in the `rooms` table.

View file

@ -1 +0,0 @@
Fix changing password via user admin API.

View file

@ -1 +0,0 @@
Fix `/events/:event_id` deprecated API.

View file

@ -1 +0,0 @@
Fixup `synapse.rest` to pass mypy.

View file

@ -1 +0,0 @@
Fixup synapse.api to pass mypy.

1
changelog.d/6734.bugfix Normal file
View file

@ -0,0 +1 @@
Warn if postgres database has a non-C locale, as that can cause issues when upgrading locales (e.g. due to upgrading OS).

View file

@ -1 +0,0 @@
Fix monthly active user limiting support for worker mode, fixes [#4639](https://github.com/matrix-org/synapse/issues/4639).

View file

@ -1 +0,0 @@
Fix bug when setting `account_validity` to an empty block in the config. Thanks to @Sorunome for reporting.

1
changelog.d/6748.misc Normal file
View file

@ -0,0 +1 @@
Propagate cache invalidates from workers to other workers.

View file

@ -1 +0,0 @@
Allow streaming cache 'invalidate all' to workers.

1
changelog.d/6751.misc Normal file
View file

@ -0,0 +1 @@
Remove some unnecessary admin handler abstraction methods.

View file

@ -1 +0,0 @@
Fix `AttributeError: 'NoneType' object has no attribute 'get'` in `hash_password` when configuration has an empty `password_config`. Contributed by @ivilata.

View file

@ -1 +0,0 @@
Remove unused CI docker compose files.

View file

@ -1 +0,0 @@
Add new quarantine media admin APIs to quarantine by media ID or by user who uploaded the media.

1
changelog.d/6757.misc Normal file
View file

@ -0,0 +1 @@
Add some debugging for media storage providers.

1
changelog.d/6761.bugfix Normal file
View file

@ -0,0 +1 @@
Minor fixes to `PUT /_synapse/admin/v2/users` admin api.

View file

@ -1 +0,0 @@
Fixup `synapse.rest` to pass mypy.

1
changelog.d/6767.bugfix Normal file
View file

@ -0,0 +1 @@
Validate `client_secret` parameter using the regex provided by the Client-Server API, temporarily allowing `:` characters for older clients. The `:` character will be removed in a future release.

1
changelog.d/6771.bugfix Normal file
View file

@ -0,0 +1 @@
Fix persisting redaction events that have been redacted (or otherwise don't have a redacts key).

1
changelog.d/6775.doc Normal file
View file

@ -0,0 +1 @@
Clarify documentation related to `user_dir` and `federation_reader` workers.

1
changelog.d/6776.misc Normal file
View file

@ -0,0 +1 @@
Detect unknown remote devices and mark cache as stale.

1
changelog.d/6786.misc Normal file
View file

@ -0,0 +1 @@
Attempt to resync remote users' devices when detected as stale.

1
changelog.d/6787.feature Normal file
View file

@ -0,0 +1 @@
Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260).

1
changelog.d/6788.misc Normal file
View file

@ -0,0 +1 @@
Record room versions in the `rooms` table.

1
changelog.d/6790.feature Normal file
View file

@ -0,0 +1 @@
Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260).

1
changelog.d/6792.misc Normal file
View file

@ -0,0 +1 @@
Delete current state from the database when server leaves a room.

1
changelog.d/6794.feature Normal file
View file

@ -0,0 +1 @@
Implement updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260).

1
changelog.d/6795.bugfix Normal file
View file

@ -0,0 +1 @@
Fix outbound federation request metrics.

1
changelog.d/6796.bugfix Normal file
View file

@ -0,0 +1 @@
Fix bug where querying a remote user's device keys that weren't cached resulted in only returning a single device.

1
changelog.d/6797.misc Normal file
View file

@ -0,0 +1 @@
When a client asks for a remote user's device keys check if the local cache for that user has been marked as potentially stale.

1
changelog.d/6799.bugfix Normal file
View file

@ -0,0 +1 @@
Fix race in federation sender worker that delayed sending of device updates.

1
changelog.d/6800.bugfix Normal file
View file

@ -0,0 +1 @@
Fix race in federation sender worker that delayed sending of device updates.

1
changelog.d/6801.bugfix Normal file
View file

@ -0,0 +1 @@
Fix bug where Synapse didn't invalidate cache of remote users' devices when Synapse left a room.

1
changelog.d/6802.misc Normal file
View file

@ -0,0 +1 @@
Add background update to clean out left rooms from current state.

1
changelog.d/6803.misc Normal file
View file

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

1
changelog.d/6805.misc Normal file
View file

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

1
changelog.d/6806.misc Normal file
View file

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

1
changelog.d/6807.misc Normal file
View file

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

1
changelog.d/6810.misc Normal file
View file

@ -0,0 +1 @@
Record room versions in the `rooms` table.

1
changelog.d/6811.bugfix Normal file
View file

@ -0,0 +1 @@
Fix waking up other workers when remote server is detected to have come back online.

1
changelog.d/6816.misc Normal file
View file

@ -0,0 +1 @@
Add background update to clean out left rooms from current state.

1
changelog.d/6819.misc Normal file
View file

@ -0,0 +1 @@
Detect unknown remote devices and mark cache as stale.

1
changelog.d/6820.misc Normal file
View file

@ -0,0 +1 @@
Refactoring work in preparation for changing the event redaction algorithm.

12
debian/changelog vendored
View file

@ -1,3 +1,15 @@
matrix-synapse-py3 (1.9.1) stable; urgency=medium
* New synapse release 1.9.1.
-- Synapse Packaging team <packages@matrix.org> Tue, 28 Jan 2020 13:09:23 +0000
matrix-synapse-py3 (1.9.0) stable; urgency=medium
* New synapse release 1.9.0.
-- Synapse Packaging team <packages@matrix.org> Thu, 23 Jan 2020 12:56:31 +0000
matrix-synapse-py3 (1.8.0) stable; urgency=medium
[ Richard van der Hoff ]

View file

@ -46,7 +46,7 @@ The following fields are possible in the JSON response body:
A standard request with no filtering:
```
GET /_synapse/admin/rooms
GET /_synapse/admin/v1/rooms
{}
```
@ -78,7 +78,7 @@ Response:
Filtering by room name:
```
GET /_synapse/admin/rooms?search_term=TWIM
GET /_synapse/admin/v1/rooms?search_term=TWIM
{}
```
@ -103,7 +103,7 @@ Response:
Paginating through a list of rooms:
```
GET /_synapse/admin/rooms?order_by=size
GET /_synapse/admin/v1/rooms?order_by=size
{}
```
@ -139,7 +139,7 @@ To get the next batch of room results, we repeat our request, setting the `from`
parameter to the value of `next_token`.
```
GET /_synapse/admin/rooms?order_by=size&from=100
GET /_synapse/admin/v1/rooms?order_by=size&from=100
{}
```

View file

@ -32,7 +32,7 @@ Assuming your PostgreSQL database user is called `postgres`, first authenticate
su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
Then, create a user ``synapse_user`` with:
createuser --pwprompt synapse_user
@ -63,6 +63,24 @@ You may need to enable password authentication so `synapse_user` can
connect to the database. See
<https://www.postgresql.org/docs/11/auth-pg-hba-conf.html>.
### Fixing incorrect `COLLATE` or `CTYPE`
Synapse will refuse to set up a new database if it has the wrong values of
`COLLATE` and `CTYPE` set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to take a dump and recreate the database with
the correct `COLLATE` and `CTYPE` parameters (as per
[docs/postgres.md](docs/postgres.md)). It is also possible to change the
parameters on a live database and run a `REINDEX` on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
## Tuning Postgres
The default settings should be fine for most deployments. For larger

View file

@ -185,6 +185,19 @@ reverse-proxy configuration.
The `^/_matrix/federation/v1/send/` endpoint must only be handled by a single
instance.
Note that `federation` must be added to the listener resources in the worker config:
```yaml
worker_app: synapse.app.federation_reader
...
worker_listeners:
- type: http
port: <port>
resources:
- names:
- federation
```
### `synapse.app.federation_sender`
Handles sending federation traffic to other servers. Doesn't handle any
@ -265,6 +278,10 @@ the following regular expressions:
^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$
When using this worker you must also set `update_user_directory: False` in the
shared configuration file to stop the main synapse running background
jobs related to updating the user directory.
### `synapse.app.frontend_proxy`
Proxies some frequently-requested client endpoints to add caching and remove

View file

@ -36,7 +36,7 @@ try:
except ImportError:
pass
__version__ = "1.9.0.dev2"
__version__ = "1.9.1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when

View file

@ -33,6 +33,7 @@ from synapse.api.errors import (
MissingClientTokenError,
ResourceLimitError,
)
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
from synapse.config.server import is_threepid_reserved
from synapse.types import StateMap, UserID
from synapse.util.caches import CACHE_SIZE_FACTOR, register_cache
@ -77,15 +78,17 @@ class Auth(object):
self._account_validity = hs.config.account_validity
@defer.inlineCallbacks
def check_from_context(self, room_version, event, context, do_sig_check=True):
def check_from_context(self, room_version: str, event, context, do_sig_check=True):
prev_state_ids = yield context.get_prev_state_ids()
auth_events_ids = yield self.compute_auth_events(
event, prev_state_ids, for_verification=True
)
auth_events = yield self.store.get_events(auth_events_ids)
auth_events = {(e.type, e.state_key): e for e in itervalues(auth_events)}
room_version_obj = KNOWN_ROOM_VERSIONS[room_version]
event_auth.check(
room_version, event, auth_events=auth_events, do_sig_check=do_sig_check
room_version_obj, event, auth_events=auth_events, do_sig_check=do_sig_check
)
@defer.inlineCallbacks

View file

@ -402,11 +402,9 @@ class UnsupportedRoomVersionError(SynapseError):
"""The client's request to create a room used a room version that the server does
not support."""
def __init__(self):
def __init__(self, msg="Homeserver does not support this room version"):
super(UnsupportedRoomVersionError, self).__init__(
code=400,
msg="Homeserver does not support this room version",
errcode=Codes.UNSUPPORTED_ROOM_VERSION,
code=400, msg=msg, errcode=Codes.UNSUPPORTED_ROOM_VERSION,
)

View file

@ -57,6 +57,9 @@ class RoomVersion(object):
state_res = attr.ib() # int; one of the StateResolutionVersions
enforce_key_validity = attr.ib() # bool
# bool: before MSC2260, anyone was allowed to send an aliases event
special_case_aliases_auth = attr.ib(type=bool, default=False)
class RoomVersions(object):
V1 = RoomVersion(
@ -65,6 +68,7 @@ class RoomVersions(object):
EventFormatVersions.V1,
StateResolutionVersions.V1,
enforce_key_validity=False,
special_case_aliases_auth=True,
)
V2 = RoomVersion(
"2",
@ -72,6 +76,7 @@ class RoomVersions(object):
EventFormatVersions.V1,
StateResolutionVersions.V2,
enforce_key_validity=False,
special_case_aliases_auth=True,
)
V3 = RoomVersion(
"3",
@ -79,6 +84,7 @@ class RoomVersions(object):
EventFormatVersions.V2,
StateResolutionVersions.V2,
enforce_key_validity=False,
special_case_aliases_auth=True,
)
V4 = RoomVersion(
"4",
@ -86,6 +92,7 @@ class RoomVersions(object):
EventFormatVersions.V3,
StateResolutionVersions.V2,
enforce_key_validity=False,
special_case_aliases_auth=True,
)
V5 = RoomVersion(
"5",
@ -93,6 +100,14 @@ class RoomVersions(object):
EventFormatVersions.V3,
StateResolutionVersions.V2,
enforce_key_validity=True,
special_case_aliases_auth=True,
)
MSC2260_DEV = RoomVersion(
"org.matrix.msc2260",
RoomDisposition.UNSTABLE,
EventFormatVersions.V3,
StateResolutionVersions.V2,
enforce_key_validity=True,
)
@ -104,5 +119,6 @@ KNOWN_ROOM_VERSIONS = {
RoomVersions.V3,
RoomVersions.V4,
RoomVersions.V5,
RoomVersions.MSC2260_DEV,
)
} # type: Dict[str, RoomVersion]

View file

@ -38,7 +38,11 @@ from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
from synapse.replication.slave.storage.registration import SlavedRegistrationStore
from synapse.replication.slave.storage.transactions import SlavedTransactionStore
from synapse.replication.tcp.client import ReplicationClientHandler
from synapse.replication.tcp.streams._base import ReceiptsStream
from synapse.replication.tcp.streams._base import (
DeviceListsStream,
ReceiptsStream,
ToDeviceStream,
)
from synapse.server import HomeServer
from synapse.storage.database import Database
from synapse.types import ReadReceipt
@ -256,6 +260,20 @@ class FederationSenderHandler(object):
"process_receipts_for_federation", self._on_new_receipts, rows
)
# ... as well as device updates and messages
elif stream_name == DeviceListsStream.NAME:
hosts = set(row.destination for row in rows)
for host in hosts:
self.federation_sender.send_device_messages(host)
elif stream_name == ToDeviceStream.NAME:
# The to_device stream includes stuff to be pushed to both local
# clients and remote servers, so we ignore entities that start with
# '@' (since they'll be local users rather than destinations).
hosts = set(row.entity for row in rows if not row.entity.startswith("@"))
for host in hosts:
self.federation_sender.send_device_messages(host)
@defer.inlineCallbacks
def _on_new_receipts(self, rows):
"""

View file

@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
#
# Copyright 2014-2016 OpenMarket Ltd
# Copyright 2020 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -17,13 +18,17 @@
import collections.abc
import hashlib
import logging
from typing import Dict
from canonicaljson import encode_canonical_json
from signedjson.sign import sign_json
from signedjson.types import SigningKey
from unpaddedbase64 import decode_base64, encode_base64
from synapse.api.errors import Codes, SynapseError
from synapse.api.room_versions import RoomVersion
from synapse.events.utils import prune_event, prune_event_dict
from synapse.types import JsonDict
logger = logging.getLogger(__name__)
@ -112,18 +117,28 @@ def compute_event_reference_hash(event, hash_algorithm=hashlib.sha256):
return hashed.name, hashed.digest()
def compute_event_signature(event_dict, signature_name, signing_key):
def compute_event_signature(
room_version: RoomVersion,
event_dict: JsonDict,
signature_name: str,
signing_key: SigningKey,
) -> Dict[str, Dict[str, str]]:
"""Compute the signature of the event for the given name and key.
Args:
event_dict (dict): The event as a dict
signature_name (str): The name of the entity signing the event
room_version: the version of the room that this event is in.
(the room version determines the redaction algorithm and hence the
json to be signed)
event_dict: The event as a dict
signature_name: The name of the entity signing the event
(typically the server's hostname).
signing_key (syutil.crypto.SigningKey): The key to sign with
signing_key: The key to sign with
Returns:
dict[str, dict[str, str]]: Returns a dictionary in the same format of
an event's signatures field.
a dictionary in the same format of an event's signatures field.
"""
redact_json = prune_event_dict(event_dict)
redact_json.pop("age_ts", None)
@ -137,23 +152,26 @@ def compute_event_signature(event_dict, signature_name, signing_key):
def add_hashes_and_signatures(
event_dict, signature_name, signing_key, hash_algorithm=hashlib.sha256
room_version: RoomVersion,
event_dict: JsonDict,
signature_name: str,
signing_key: SigningKey,
):
"""Add content hash and sign the event
Args:
event_dict (dict): The event to add hashes to and sign
signature_name (str): The name of the entity signing the event
room_version: the version of the room this event is in
event_dict: The event to add hashes to and sign
signature_name: The name of the entity signing the event
(typically the server's hostname).
signing_key (syutil.crypto.SigningKey): The key to sign with
hash_algorithm: A hasher from `hashlib`, e.g. hashlib.sha256, to use
to hash the event
signing_key: The key to sign with
"""
name, digest = compute_content_hash(event_dict, hash_algorithm=hash_algorithm)
name, digest = compute_content_hash(event_dict, hash_algorithm=hashlib.sha256)
event_dict.setdefault("hashes", {})[name] = encode_base64(digest)
event_dict["signatures"] = compute_event_signature(
event_dict, signature_name=signature_name, signing_key=signing_key
room_version, event_dict, signature_name=signature_name, signing_key=signing_key
)

View file

@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2014 - 2016 OpenMarket Ltd
# Copyright 2020 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -23,17 +24,27 @@ from unpaddedbase64 import decode_base64
from synapse.api.constants import EventTypes, JoinRules, Membership
from synapse.api.errors import AuthError, EventSizeError, SynapseError
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, EventFormatVersions
from synapse.api.room_versions import (
KNOWN_ROOM_VERSIONS,
EventFormatVersions,
RoomVersion,
)
from synapse.types import UserID, get_domain_from_id
logger = logging.getLogger(__name__)
def check(room_version, event, auth_events, do_sig_check=True, do_size_check=True):
def check(
room_version_obj: RoomVersion,
event,
auth_events,
do_sig_check=True,
do_size_check=True,
):
""" Checks if this event is correctly authed.
Args:
room_version (str): the version of the room
room_version_obj: the version of the room
event: the event being checked.
auth_events (dict: event-key -> event): the existing room state.
@ -89,7 +100,12 @@ def check(room_version, event, auth_events, do_sig_check=True, do_size_check=Tru
if not event.signatures.get(event_id_domain):
raise AuthError(403, "Event not signed by sending server")
# Implementation of https://matrix.org/docs/spec/rooms/v1#authorization-rules
#
# 1. If type is m.room.create:
if event.type == EventTypes.Create:
# 1b. If the domain of the room_id does not match the domain of the sender,
# reject.
sender_domain = get_domain_from_id(event.sender)
room_id_domain = get_domain_from_id(event.room_id)
if room_id_domain != sender_domain:
@ -97,39 +113,49 @@ def check(room_version, event, auth_events, do_sig_check=True, do_size_check=Tru
403, "Creation event's room_id domain does not match sender's"
)
room_version = event.content.get("room_version", "1")
if room_version not in KNOWN_ROOM_VERSIONS:
# 1c. If content.room_version is present and is not a recognised version, reject
room_version_prop = event.content.get("room_version", "1")
if room_version_prop not in KNOWN_ROOM_VERSIONS:
raise AuthError(
403, "room appears to have unsupported version %s" % (room_version,)
403,
"room appears to have unsupported version %s" % (room_version_prop,),
)
# FIXME
logger.debug("Allowing! %s", event)
return
# 3. If event does not have a m.room.create in its auth_events, reject.
creation_event = auth_events.get((EventTypes.Create, ""), None)
if not creation_event:
raise AuthError(403, "No create event in auth events")
# additional check for m.federate
creating_domain = get_domain_from_id(event.room_id)
originating_domain = get_domain_from_id(event.sender)
if creating_domain != originating_domain:
if not _can_federate(event, auth_events):
raise AuthError(403, "This room has been marked as unfederatable.")
# FIXME: Temp hack
# 4. If type is m.room.aliases
if event.type == EventTypes.Aliases:
# 4a. If event has no state_key, reject
if not event.is_state():
raise AuthError(403, "Alias event must be a state event")
if not event.state_key:
raise AuthError(403, "Alias event must have non-empty state_key")
# 4b. If sender's domain doesn't matches [sic] state_key, reject
sender_domain = get_domain_from_id(event.sender)
if event.state_key != sender_domain:
raise AuthError(
403, "Alias event's state_key does not match sender's domain"
)
logger.debug("Allowing! %s", event)
return
# 4c. Otherwise, allow.
# This is removed by https://github.com/matrix-org/matrix-doc/pull/2260
if room_version_obj.special_case_aliases_auth:
logger.debug("Allowing! %s", event)
return
if logger.isEnabledFor(logging.DEBUG):
logger.debug("Auth events: %s", [a.event_id for a in auth_events.values()])
@ -160,7 +186,7 @@ def check(room_version, event, auth_events, do_sig_check=True, do_size_check=Tru
_check_power_levels(event, auth_events)
if event.type == EventTypes.Redaction:
check_redaction(room_version, event, auth_events)
check_redaction(room_version_obj, event, auth_events)
logger.debug("Allowing! %s", event)
@ -386,7 +412,7 @@ def _can_send_event(event, auth_events):
return True
def check_redaction(room_version, event, auth_events):
def check_redaction(room_version_obj: RoomVersion, event, auth_events):
"""Check whether the event sender is allowed to redact the target event.
Returns:
@ -406,11 +432,7 @@ def check_redaction(room_version, event, auth_events):
if user_level >= redact_level:
return False
v = KNOWN_ROOM_VERSIONS.get(room_version)
if not v:
raise RuntimeError("Unrecognized room version %r" % (room_version,))
if v.event_format == EventFormatVersions.V1:
if room_version_obj.event_format == EventFormatVersions.V1:
redacter_domain = get_domain_from_id(event.event_id)
redactee_domain = get_domain_from_id(event.redacts)
if redacter_domain == redactee_domain:

View file

@ -23,6 +23,7 @@ from unpaddedbase64 import encode_base64
from synapse.api.errors import UnsupportedRoomVersionError
from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, EventFormatVersions
from synapse.types import JsonDict
from synapse.util.caches import intern_dict
from synapse.util.frozenutils import freeze
@ -116,16 +117,32 @@ class _EventInternalMetadata(object):
return getattr(self, "redacted", False)
def _event_dict_property(key):
_SENTINEL = object()
def _event_dict_property(key, default=_SENTINEL):
"""Creates a new property for the given key that delegates access to
`self._event_dict`.
The default is used if the key is missing from the `_event_dict`, if given,
otherwise an AttributeError will be raised.
Note: If a default is given then `hasattr` will always return true.
"""
# We want to be able to use hasattr with the event dict properties.
# However, (on python3) hasattr expects AttributeError to be raised. Hence,
# we need to transform the KeyError into an AttributeError
def getter(self):
def getter_raises(self):
try:
return self._event_dict[key]
except KeyError:
raise AttributeError(key)
def getter_default(self):
return self._event_dict.get(key, default)
def setter(self, v):
try:
self._event_dict[key] = v
@ -138,7 +155,11 @@ def _event_dict_property(key):
except KeyError:
raise AttributeError(key)
return property(getter, setter, delete)
if default is _SENTINEL:
# No default given, so use the getter that raises
return property(getter_raises, setter, delete)
else:
return property(getter_default, setter, delete)
class EventBase(object):
@ -165,7 +186,7 @@ class EventBase(object):
origin = _event_dict_property("origin")
origin_server_ts = _event_dict_property("origin_server_ts")
prev_events = _event_dict_property("prev_events")
redacts = _event_dict_property("redacts")
redacts = _event_dict_property("redacts", None)
room_id = _event_dict_property("room_id")
sender = _event_dict_property("sender")
user_id = _event_dict_property("sender")
@ -177,7 +198,7 @@ class EventBase(object):
def is_state(self):
return hasattr(self, "state_key") and self.state_key is not None
def get_dict(self):
def get_dict(self) -> JsonDict:
d = dict(self._event_dict)
d.update({"signatures": self.signatures, "unsigned": dict(self.unsigned)})
@ -189,7 +210,7 @@ class EventBase(object):
def get_internal_metadata_dict(self):
return self.internal_metadata.get_dict()
def get_pdu_json(self, time_now=None):
def get_pdu_json(self, time_now=None) -> JsonDict:
pdu_json = self.get_dict()
if time_now is not None and "age_ts" in pdu_json["unsigned"]:

View file

@ -12,8 +12,10 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from typing import Optional
import attr
from nacl.signing import SigningKey
from twisted.internet import defer
@ -23,13 +25,18 @@ from synapse.api.room_versions import (
KNOWN_EVENT_FORMAT_VERSIONS,
KNOWN_ROOM_VERSIONS,
EventFormatVersions,
RoomVersion,
)
from synapse.crypto.event_signing import add_hashes_and_signatures
from synapse.types import EventID
from synapse.events import (
EventBase,
_EventInternalMetadata,
event_type_from_format_version,
)
from synapse.types import EventID, JsonDict
from synapse.util import Clock
from synapse.util.stringutils import random_string
from . import _EventInternalMetadata, event_type_from_format_version
@attr.s(slots=True, cmp=False, frozen=True)
class EventBuilder(object):
@ -40,7 +47,7 @@ class EventBuilder(object):
content/unsigned/internal_metadata fields are still mutable)
Attributes:
format_version (int): Event format version
room_version: Version of the target room
room_id (str)
type (str)
sender (str)
@ -63,7 +70,7 @@ class EventBuilder(object):
_hostname = attr.ib()
_signing_key = attr.ib()
format_version = attr.ib()
room_version = attr.ib(type=RoomVersion)
room_id = attr.ib()
type = attr.ib()
@ -108,7 +115,8 @@ class EventBuilder(object):
)
auth_ids = yield self._auth.compute_auth_events(self, state_ids)
if self.format_version == EventFormatVersions.V1:
format_version = self.room_version.event_format
if format_version == EventFormatVersions.V1:
auth_events = yield self._store.add_event_hashes(auth_ids)
prev_events = yield self._store.add_event_hashes(prev_event_ids)
else:
@ -148,7 +156,7 @@ class EventBuilder(object):
clock=self._clock,
hostname=self._hostname,
signing_key=self._signing_key,
format_version=self.format_version,
room_version=self.room_version,
event_dict=event_dict,
internal_metadata_dict=self.internal_metadata.get_dict(),
)
@ -201,7 +209,7 @@ class EventBuilderFactory(object):
clock=self.clock,
hostname=self.hostname,
signing_key=self.signing_key,
format_version=room_version.event_format,
room_version=room_version,
type=key_values["type"],
state_key=key_values.get("state_key"),
room_id=key_values["room_id"],
@ -214,29 +222,19 @@ class EventBuilderFactory(object):
def create_local_event_from_event_dict(
clock,
hostname,
signing_key,
format_version,
event_dict,
internal_metadata_dict=None,
):
clock: Clock,
hostname: str,
signing_key: SigningKey,
room_version: RoomVersion,
event_dict: JsonDict,
internal_metadata_dict: Optional[JsonDict] = None,
) -> EventBase:
"""Takes a fully formed event dict, ensuring that fields like `origin`
and `origin_server_ts` have correct values for a locally produced event,
then signs and hashes it.
Args:
clock (Clock)
hostname (str)
signing_key
format_version (int)
event_dict (dict)
internal_metadata_dict (dict|None)
Returns:
FrozenEvent
"""
format_version = room_version.event_format
if format_version not in KNOWN_EVENT_FORMAT_VERSIONS:
raise Exception("No event format defined for version %r" % (format_version,))
@ -257,7 +255,7 @@ def create_local_event_from_event_dict(
event_dict.setdefault("signatures", {})
add_hashes_and_signatures(event_dict, hostname, signing_key)
add_hashes_and_signatures(room_version, event_dict, hostname, signing_key)
return event_type_from_format_version(format_version)(
event_dict, internal_metadata_dict=internal_metadata_dict
)

View file

@ -12,8 +12,9 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import collections
import re
from typing import Mapping, Union
from six import string_types
@ -422,3 +423,37 @@ class EventClientSerializer(object):
return yieldable_gather_results(
self.serialize_event, events, time_now=time_now, **kwargs
)
def copy_power_levels_contents(
old_power_levels: Mapping[str, Union[int, Mapping[str, int]]]
):
"""Copy the content of a power_levels event, unfreezing frozendicts along the way
Raises:
TypeError if the input does not look like a valid power levels event content
"""
if not isinstance(old_power_levels, collections.Mapping):
raise TypeError("Not a valid power-levels content: %r" % (old_power_levels,))
power_levels = {}
for k, v in old_power_levels.items():
if isinstance(v, int):
power_levels[k] = v
continue
if isinstance(v, collections.Mapping):
power_levels[k] = h = {}
for k1, v1 in v.items():
# we should only have one level of nesting
if not isinstance(v1, int):
raise TypeError(
"Invalid power_levels value for %s.%s: %r" % (k, k1, v1)
)
h[k1] = v1
continue
raise TypeError("Invalid power_levels value for %s: %r" % (k, v))
return power_levels

View file

@ -17,6 +17,7 @@
import copy
import itertools
import logging
from typing import Dict, Iterable
from prometheus_client import Counter
@ -29,6 +30,7 @@ from synapse.api.errors import (
FederationDeniedError,
HttpResponseException,
SynapseError,
UnsupportedRoomVersionError,
)
from synapse.api.room_versions import (
KNOWN_ROOM_VERSIONS,
@ -196,7 +198,7 @@ class FederationClient(FederationBase):
logger.debug("backfill transaction_data=%r", transaction_data)
room_version = yield self.store.get_room_version(room_id)
room_version = yield self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version)
pdus = [
@ -334,7 +336,7 @@ class FederationClient(FederationBase):
def get_event_auth(self, destination, room_id, event_id):
res = yield self.transport_layer.get_event_auth(destination, room_id, event_id)
room_version = yield self.store.get_room_version(room_id)
room_version = yield self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version)
auth_chain = [
@ -385,6 +387,8 @@ class FederationClient(FederationBase):
return res
except InvalidResponseError as e:
logger.warning("Failed to %s via %s: %s", description, destination, e)
except UnsupportedRoomVersionError:
raise
except HttpResponseException as e:
if not 500 <= e.code < 600:
raise e.to_synapse_error()
@ -404,7 +408,13 @@ class FederationClient(FederationBase):
raise SynapseError(502, "Failed to %s via any server" % (description,))
def make_membership_event(
self, destinations, room_id, user_id, membership, content, params
self,
destinations: Iterable[str],
room_id: str,
user_id: str,
membership: str,
content: dict,
params: Dict[str, str],
):
"""
Creates an m.room.member event, with context, without participating in the room.
@ -417,21 +427,23 @@ class FederationClient(FederationBase):
Note that this does not append any events to any graphs.
Args:
destinations (Iterable[str]): Candidate homeservers which are probably
destinations: Candidate homeservers which are probably
participating in the room.
room_id (str): The room in which the event will happen.
user_id (str): The user whose membership is being evented.
membership (str): The "membership" property of the event. Must be
one of "join" or "leave".
content (dict): Any additional data to put into the content field
of the event.
params (dict[str, str|Iterable[str]]): Query parameters to include in the
request.
room_id: The room in which the event will happen.
user_id: The user whose membership is being evented.
membership: The "membership" property of the event. Must be one of
"join" or "leave".
content: Any additional data to put into the content field of the
event.
params: Query parameters to include in the request.
Return:
Deferred[tuple[str, FrozenEvent, int]]: resolves to a tuple of
`(origin, event, event_format)` where origin is the remote
homeserver which generated the event, and event_format is one of
`synapse.api.room_versions.EventFormatVersions`.
Deferred[Tuple[str, FrozenEvent, RoomVersion]]: resolves to a tuple of
`(origin, event, room_version)` where origin is the remote
homeserver which generated the event, and room_version is the
version of the room.
Fails with a `UnsupportedRoomVersionError` if remote responds with
a room version we don't understand.
Fails with a ``SynapseError`` if the chosen remote server
returns a 300/400 code.
@ -453,8 +465,10 @@ class FederationClient(FederationBase):
# Note: If not supplied, the room version may be either v1 or v2,
# however either way the event format version will be v1.
room_version = ret.get("room_version", RoomVersions.V1.identifier)
event_format = room_version_to_event_format(room_version)
room_version_id = ret.get("room_version", RoomVersions.V1.identifier)
room_version = KNOWN_ROOM_VERSIONS.get(room_version_id)
if not room_version:
raise UnsupportedRoomVersionError()
pdu_dict = ret.get("event", None)
if not isinstance(pdu_dict, dict):
@ -474,11 +488,11 @@ class FederationClient(FederationBase):
self._clock,
self.hostname,
self.signing_key,
format_version=event_format,
room_version=room_version,
event_dict=pdu_dict,
)
return (destination, ev, event_format)
return (destination, ev, room_version)
return self._try_destination_list(
"make_" + membership, destinations, send_request
@ -633,7 +647,7 @@ class FederationClient(FederationBase):
@defer.inlineCallbacks
def send_invite(self, destination, room_id, event_id, pdu):
room_version = yield self.store.get_room_version(room_id)
room_version = yield self.store.get_room_version_id(room_id)
content = yield self._do_send_invite(destination, pdu, room_version)
@ -641,7 +655,7 @@ class FederationClient(FederationBase):
logger.debug("Got response to send_invite: %s", pdu_dict)
room_version = yield self.store.get_room_version(room_id)
room_version = yield self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version)
pdu = event_from_pdu_json(pdu_dict, format_ver)
@ -843,7 +857,7 @@ class FederationClient(FederationBase):
timeout=timeout,
)
room_version = yield self.store.get_room_version(room_id)
room_version = yield self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version)
events = [

View file

@ -234,7 +234,7 @@ class FederationServer(FederationBase):
continue
try:
room_version = await self.store.get_room_version(room_id)
room_version = await self.store.get_room_version_id(room_id)
except NotFoundError:
logger.info("Ignoring PDU for unknown room_id: %s", room_id)
continue
@ -334,7 +334,7 @@ class FederationServer(FederationBase):
)
)
room_version = await self.store.get_room_version(room_id)
room_version = await self.store.get_room_version_id(room_id)
resp["room_version"] = room_version
return 200, resp
@ -385,7 +385,7 @@ class FederationServer(FederationBase):
origin_host, _ = parse_server_name(origin)
await self.check_server_matches_acl(origin_host, room_id)
room_version = await self.store.get_room_version(room_id)
room_version = await self.store.get_room_version_id(room_id)
if room_version not in supported_versions:
logger.warning(
"Room version %s not in %s", room_version, supported_versions
@ -410,14 +410,14 @@ class FederationServer(FederationBase):
origin_host, _ = parse_server_name(origin)
await self.check_server_matches_acl(origin_host, pdu.room_id)
pdu = await self._check_sigs_and_hash(room_version, pdu)
ret_pdu = await self.handler.on_invite_request(origin, pdu)
ret_pdu = await self.handler.on_invite_request(origin, pdu, room_version)
time_now = self._clock.time_msec()
return {"event": ret_pdu.get_pdu_json(time_now)}
async def on_send_join_request(self, origin, content, room_id):
logger.debug("on_send_join_request: content: %s", content)
room_version = await self.store.get_room_version(room_id)
room_version = await self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version)
pdu = event_from_pdu_json(content, format_ver)
@ -440,7 +440,7 @@ class FederationServer(FederationBase):
await self.check_server_matches_acl(origin_host, room_id)
pdu = await self.handler.on_make_leave_request(origin, room_id, user_id)
room_version = await self.store.get_room_version(room_id)
room_version = await self.store.get_room_version_id(room_id)
time_now = self._clock.time_msec()
return {"event": pdu.get_pdu_json(time_now), "room_version": room_version}
@ -448,7 +448,7 @@ class FederationServer(FederationBase):
async def on_send_leave_request(self, origin, content, room_id):
logger.debug("on_send_leave_request: content: %s", content)
room_version = await self.store.get_room_version(room_id)
room_version = await self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version)
pdu = event_from_pdu_json(content, format_ver)
@ -495,7 +495,7 @@ class FederationServer(FederationBase):
origin_host, _ = parse_server_name(origin)
await self.check_server_matches_acl(origin_host, room_id)
room_version = await self.store.get_room_version(room_id)
room_version = await self.store.get_room_version_id(room_id)
format_ver = room_version_to_event_format(room_version)
auth_chain = [
@ -664,7 +664,7 @@ class FederationServer(FederationBase):
logger.info("Accepting join PDU %s from %s", pdu.event_id, origin)
# We've already checked that we know the room version by this point
room_version = await self.store.get_room_version(pdu.room_id)
room_version = await self.store.get_room_version_id(pdu.room_id)
# Check signature.
try:

View file

@ -69,8 +69,6 @@ class FederationRemoteSendQueue(object):
self.edus = SortedDict() # stream position -> Edu
self.device_messages = SortedDict() # stream position -> destination
self.pos = 1
self.pos_time = SortedDict()
@ -92,7 +90,6 @@ class FederationRemoteSendQueue(object):
"keyed_edu",
"keyed_edu_changed",
"edus",
"device_messages",
"pos_time",
"presence_destinations",
]:
@ -171,12 +168,6 @@ class FederationRemoteSendQueue(object):
for key in keys[:i]:
del self.edus[key]
# Delete things out of device map
keys = self.device_messages.keys()
i = self.device_messages.bisect_left(position_to_delete)
for key in keys[:i]:
del self.device_messages[key]
def notify_new_events(self, current_id):
"""As per FederationSender"""
# We don't need to replicate this as it gets sent down a different
@ -249,9 +240,8 @@ class FederationRemoteSendQueue(object):
def send_device_messages(self, destination):
"""As per FederationSender"""
pos = self._next_pos()
self.device_messages[pos] = destination
self.notifier.on_new_replication_data()
# We don't need to replicate this as it gets sent down a different
# stream.
def get_current_token(self):
return self.pos - 1
@ -339,14 +329,6 @@ class FederationRemoteSendQueue(object):
for (pos, edu) in edus:
rows.append((pos, EduRow(edu)))
# Fetch changed device messages
i = self.device_messages.bisect_right(from_token)
j = self.device_messages.bisect_right(to_token) + 1
device_messages = {v: k for k, v in self.device_messages.items()[i:j]}
for (destination, pos) in iteritems(device_messages):
rows.append((pos, DeviceRow(destination=destination)))
# Sort rows based on pos
rows.sort()
@ -472,28 +454,9 @@ class EduRow(BaseFederationRow, namedtuple("EduRow", ("edu",))): # Edu
buff.edus.setdefault(self.edu.destination, []).append(self.edu)
class DeviceRow(BaseFederationRow, namedtuple("DeviceRow", ("destination",))): # str
"""Streams the fact that either a) there is pending to device messages for
users on the remote, or b) a local users device has changed and needs to
be sent to the remote.
"""
TypeId = "d"
@staticmethod
def from_data(data):
return DeviceRow(destination=data["destination"])
def to_data(self):
return {"destination": self.destination}
def add_to_buffer(self, buff):
buff.device_destinations.add(self.destination)
TypeToRow = {
Row.TypeId: Row
for Row in (PresenceRow, PresenceDestinationsRow, KeyedEduRow, EduRow, DeviceRow)
for Row in (PresenceRow, PresenceDestinationsRow, KeyedEduRow, EduRow,)
}
@ -504,7 +467,6 @@ ParsedFederationStreamData = namedtuple(
"presence_destinations", # list of tuples of UserPresenceState and destinations
"keyed_edus", # dict of destination -> { key -> Edu }
"edus", # dict of destination -> [Edu]
"device_destinations", # set of destinations
),
)
@ -523,11 +485,7 @@ def process_rows_for_federation(transaction_queue, rows):
# them into the appropriate collection and then send them off.
buff = ParsedFederationStreamData(
presence=[],
presence_destinations=[],
keyed_edus={},
edus={},
device_destinations=set(),
presence=[], presence_destinations=[], keyed_edus={}, edus={},
)
# Parse the rows in the stream and add to the buffer
@ -555,6 +513,3 @@ def process_rows_for_federation(transaction_queue, rows):
for destination, edu_list in iteritems(buff.edus):
for edu in edu_list:
transaction_queue.send_edu(edu, None)
for destination in buff.device_destinations:
transaction_queue.send_device_messages(destination)

View file

@ -15,6 +15,7 @@
# limitations under the License.
import logging
from typing import Any, Dict
from six.moves import urllib
@ -352,7 +353,9 @@ class TransportLayerClient(object):
else:
path = _create_v1_path("/publicRooms")
args = {"include_all_networks": "true" if include_all_networks else "false"}
args = {
"include_all_networks": "true" if include_all_networks else "false"
} # type: Dict[str, Any]
if third_party_instance_id:
args["third_party_instance_id"] = (third_party_instance_id,)
if limit:

View file

@ -18,6 +18,7 @@
import functools
import logging
import re
from typing import Optional, Tuple, Type
from twisted.internet.defer import maybeDeferred
@ -267,6 +268,8 @@ class BaseFederationServlet(object):
returned.
"""
PATH = "" # Overridden in subclasses, the regex to match against the path.
REQUIRE_AUTH = True
PREFIX = FEDERATION_V1_PREFIX # Allows specifying the API version
@ -347,9 +350,6 @@ class BaseFederationServlet(object):
return response
# Extra logic that functools.wraps() doesn't finish
new_func.__self__ = func.__self__
return new_func
def register(self, server):
@ -824,7 +824,7 @@ class PublicRoomList(BaseFederationServlet):
if not self.allow_access:
raise FederationDeniedError(origin)
limit = int(content.get("limit", 100))
limit = int(content.get("limit", 100)) # type: Optional[int]
since_token = content.get("since", None)
search_filter = content.get("filter", None)
@ -971,7 +971,7 @@ class FederationGroupsAddRoomsConfigServlet(BaseFederationServlet):
if get_domain_from_id(requester_user_id) != origin:
raise SynapseError(403, "requester_user_id doesn't match origin")
result = await self.groups_handler.update_room_in_group(
result = await self.handler.update_room_in_group(
group_id, requester_user_id, room_id, config_key, content
)
@ -1422,11 +1422,13 @@ FEDERATION_SERVLET_CLASSES = (
On3pidBindServlet,
FederationVersionServlet,
RoomComplexityServlet,
)
) # type: Tuple[Type[BaseFederationServlet], ...]
OPENID_SERVLET_CLASSES = (OpenIdUserInfo,)
OPENID_SERVLET_CLASSES = (
OpenIdUserInfo,
) # type: Tuple[Type[BaseFederationServlet], ...]
ROOM_LIST_CLASSES = (PublicRoomList,)
ROOM_LIST_CLASSES = (PublicRoomList,) # type: Tuple[Type[PublicRoomList], ...]
GROUP_SERVER_SERVLET_CLASSES = (
FederationGroupsProfileServlet,
@ -1447,17 +1449,19 @@ GROUP_SERVER_SERVLET_CLASSES = (
FederationGroupsAddRoomsServlet,
FederationGroupsAddRoomsConfigServlet,
FederationGroupsSettingJoinPolicyServlet,
)
) # type: Tuple[Type[BaseFederationServlet], ...]
GROUP_LOCAL_SERVLET_CLASSES = (
FederationGroupsLocalInviteServlet,
FederationGroupsRemoveLocalUserServlet,
FederationGroupsBulkPublicisedServlet,
)
) # type: Tuple[Type[BaseFederationServlet], ...]
GROUP_ATTESTATION_SERVLET_CLASSES = (FederationGroupsRenewAttestaionServlet,)
GROUP_ATTESTATION_SERVLET_CLASSES = (
FederationGroupsRenewAttestaionServlet,
) # type: Tuple[Type[BaseFederationServlet], ...]
DEFAULT_SERVLET_GROUPS = (
"federation",

Some files were not shown because too many files have changed in this diff Show more