Suggest using /etc/matrix-synapse/conf.d/ for configuration with Debian packages

CHANGES.md

@@ -1,97 +1,3 @@
-Synapse 1.47.0rc2 (2021-11-10)
-==============================
-
-This fixes an issue with publishing the Debian packages for 1.47.0rc1.
-It is otherwise identical to 1.47.0rc1.
-
-
-Synapse 1.47.0rc1 (2021-11-09)
-==============================
-
-Deprecations and Removals
--------------------------
-
-- The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information. ([\#11206](https://github.com/matrix-org/synapse/issues/11206))
-- Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). ([\#11213](https://github.com/matrix-org/synapse/issues/11213))
-
-
-Features
---------
-
-- Advertise support for Client-Server API r0.6.1. ([\#11097](https://github.com/matrix-org/synapse/issues/11097))
-- Add search by room ID and room alias to the List Room admin API. ([\#11099](https://github.com/matrix-org/synapse/issues/11099))
-- Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room. ([\#11126](https://github.com/matrix-org/synapse/issues/11126))
-- Add a module API method to update a user's membership in a room. ([\#11147](https://github.com/matrix-org/synapse/issues/11147))
-- Add metrics for thread pool usage. ([\#11178](https://github.com/matrix-org/synapse/issues/11178))
-- Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288). ([\#11187](https://github.com/matrix-org/synapse/issues/11187))
-- Add a module API method to retrieve the current state of a room. ([\#11204](https://github.com/matrix-org/synapse/issues/11204))
-- Calculate a default value for `public_baseurl` based on `server_name`. ([\#11210](https://github.com/matrix-org/synapse/issues/11210))
-- Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443. ([\#11211](https://github.com/matrix-org/synapse/issues/11211))
-- Add admin APIs to pause, start and check the status of background updates. ([\#11263](https://github.com/matrix-org/synapse/issues/11263))
-
-
-Bugfixes
---------
-
-- Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat. ([\#10097](https://github.com/matrix-org/synapse/issues/10097))
-- Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine. ([\#10969](https://github.com/matrix-org/synapse/issues/10969), [\#11212](https://github.com/matrix-org/synapse/issues/11212))
-- Do not accept events if a third-party rule `check_event_allowed` callback raises an exception. ([\#11033](https://github.com/matrix-org/synapse/issues/11033))
-- Fix long-standing bug where verification requests could fail in certain cases if a federation whitelist was in place but did not include your own homeserver. ([\#11129](https://github.com/matrix-org/synapse/issues/11129))
-- Allow an empty list of `state_events_at_start` to be sent when using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint and the author of the historical messages is already part of the current room state at the given `?prev_event_id`. ([\#11188](https://github.com/matrix-org/synapse/issues/11188))
-- Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p. ([\#11191](https://github.com/matrix-org/synapse/issues/11191))
-- Delete `to_device` messages for hidden devices that will never be read, reducing database size. ([\#11199](https://github.com/matrix-org/synapse/issues/11199))
-- Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error. ([\#11200](https://github.com/matrix-org/synapse/issues/11200))
-- Fix a long-standing bug which could result in serialization errors and potentially duplicate transaction data when sending ephemeral events to application services. Contributed by @Fizzadar at Beeper. ([\#11207](https://github.com/matrix-org/synapse/issues/11207))
-- Fix a bug introduced in Synapse 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats. ([\#11217](https://github.com/matrix-org/synapse/issues/11217))
-- Fix long-standing bug where cross signing keys were not included in the response to `/r0/keys/query` the first time a remote user was queried. ([\#11234](https://github.com/matrix-org/synapse/issues/11234))
-- Fix a long-standing bug where all requests that read events from the database could get stuck as a result of losing the database connection. ([\#11240](https://github.com/matrix-org/synapse/issues/11240))
-- Fix a bug preventing Synapse from being rolled back to an earlier version when using workers. ([\#11255](https://github.com/matrix-org/synapse/issues/11255), [\#11276](https://github.com/matrix-org/synapse/issues/11276))
-- Fix a bug introduced in Synapse 1.37.1 which caused a remote event being processed by a worker to not get processed on restart if the worker was killed. ([\#11262](https://github.com/matrix-org/synapse/issues/11262))
-- Only allow old Element/Riot Android clients to send read receipts without a request body. All other clients must include a request body as required by the specification. Contributed by @rogersheu. ([\#11157](https://github.com/matrix-org/synapse/issues/11157))
-
-
-Updates to the Docker image
----------------------------
-
-- Avoid changing user ID when started as a non-root user, and no explicit `UID` is set. ([\#11209](https://github.com/matrix-org/synapse/issues/11209))
-
-
-Improved Documentation
-----------------------
-
-- Improve example HAProxy config in the docs to properly handle HTTP `Host` headers with port information. This is required for federation over port 443 to work correctly. ([\#11128](https://github.com/matrix-org/synapse/issues/11128))
-- Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5. ([\#11151](https://github.com/matrix-org/synapse/issues/11151))
-- Clarify lack of support for Windows. ([\#11198](https://github.com/matrix-org/synapse/issues/11198))
-- Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper. ([\#11221](https://github.com/matrix-org/synapse/issues/11221))
-- Add documentation for using LemonLDAP as an OpenID Connect Identity Provider. Contributed by @l00ptr. ([\#11257](https://github.com/matrix-org/synapse/issues/11257))
-
-
-Internal Changes
-----------------
-
-- Add type annotations for the `log_function` decorator. ([\#10943](https://github.com/matrix-org/synapse/issues/10943))
-- Add type hints to `synapse.events`. ([\#11098](https://github.com/matrix-org/synapse/issues/11098))
-- Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code. ([\#11137](https://github.com/matrix-org/synapse/issues/11137))
-- Add type hints so that `synapse.http` passes `mypy` checks. ([\#11164](https://github.com/matrix-org/synapse/issues/11164))
-- Update scripts to pass Shellcheck lints. ([\#11166](https://github.com/matrix-org/synapse/issues/11166))
-- Add knock information in admin export. Contributed by Rafael Gonçalves. ([\#11171](https://github.com/matrix-org/synapse/issues/11171))
-- Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly. ([\#11179](https://github.com/matrix-org/synapse/issues/11179))
-- Refactor `Filter` to check different fields depending on the data type. ([\#11194](https://github.com/matrix-org/synapse/issues/11194))
-- Improve type hints for the relations datastore. ([\#11205](https://github.com/matrix-org/synapse/issues/11205))
-- Replace outdated links in the pull request checklist with links to the rendered documentation. ([\#11225](https://github.com/matrix-org/synapse/issues/11225))
-- Fix a bug in unit test `test_block_room_and_not_purge`. ([\#11226](https://github.com/matrix-org/synapse/issues/11226))
-- In `ObservableDeferred`, run observers in the order they were registered. ([\#11229](https://github.com/matrix-org/synapse/issues/11229))
-- Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`. ([\#11231](https://github.com/matrix-org/synapse/issues/11231))
-- Add `twine` and `towncrier` as dev dependencies, as they're used by the release script. ([\#11233](https://github.com/matrix-org/synapse/issues/11233))
-- Allow `stream_writers.typing` config to be a list of one worker. ([\#11237](https://github.com/matrix-org/synapse/issues/11237))
-- Remove debugging statement in tests. ([\#11239](https://github.com/matrix-org/synapse/issues/11239))
-- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers. ([\#11244](https://github.com/matrix-org/synapse/issues/11244))
-- Add an additional test for the `cachedList` method decorator. ([\#11246](https://github.com/matrix-org/synapse/issues/11246))
-- Make minor correction to the type of `auth_checkers` callbacks. ([\#11253](https://github.com/matrix-org/synapse/issues/11253))
-- Clean up trivial aspects of the Debian package build tooling. ([\#11269](https://github.com/matrix-org/synapse/issues/11269), [\#11273](https://github.com/matrix-org/synapse/issues/11273))
-- Blacklist new SyTest that checks that key uploads are valid pending the validation being implemented in Synapse. ([\#11270](https://github.com/matrix-org/synapse/issues/11270))
-
-
 Synapse 1.46.0 (2021-11-02)
 ===========================
 

changelog.d/10097.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat.

changelog.d/10943.misc

@@ -0,0 +1 @@
+Add type annotations for the `log_function` decorator.

changelog.d/10969.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.

changelog.d/11033.bugfix

@@ -0,0 +1 @@
+Do not accept events if a third-party rule module API callback raises an exception.

changelog.d/11097.feature

@@ -0,0 +1 @@
+Advertise support for Client-Server API r0.6.1.

changelog.d/11098.misc

@@ -0,0 +1 @@
+Add type hints to `synapse.events`.

changelog.d/11099.feature

@@ -0,0 +1 @@
+Add search by room ID and room alias to List Room admin API.

changelog.d/11126.feature

@@ -0,0 +1 @@
+Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room.

changelog.d/11128.doc

@@ -0,0 +1 @@
+Improve example HAProxy config in the docs to properly handle host headers with port information. This is required for federation over port 443 to work correctly.

changelog.d/11129.bugfix

@@ -0,0 +1 @@
+Fix long-standing bug where verification requests could fail in certain cases if whitelist was in place but did not include your own homeserver.

changelog.d/11137.misc

@@ -0,0 +1 @@
+Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code.

changelog.d/11147.feature

@@ -0,0 +1 @@
+Add a module API method to update a user's membership in a room.

changelog.d/11151.doc

@@ -0,0 +1 @@
+Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5.

changelog.d/11164.misc

@@ -0,0 +1 @@
+Add type hints so that `synapse.http` passes `mypy` checks.

changelog.d/11166.misc

@@ -0,0 +1 @@
+Update scripts to pass Shellcheck lints.

changelog.d/11171.misc

@@ -0,0 +1 @@
+Add knock information in admin export. Contributed by Rafael Gonçalves.

changelog.d/11178.feature

@@ -0,0 +1 @@
+Add metrics for thread pool usage.

changelog.d/11179.misc

@@ -0,0 +1 @@
+Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly.

changelog.d/11187.feature

@@ -0,0 +1 @@
+Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288).

changelog.d/11188.bugfix

@@ -0,0 +1 @@
+Allow an empty list of `state_events_at_start` to be sent when using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint and the author of the historical messages is already part of the current room state at the given `?prev_event_id`.

changelog.d/11191.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p.

changelog.d/11194.misc

@@ -0,0 +1 @@
+Refactor `Filter` to check different fields depending on the data type.

changelog.d/11198.doc

@@ -0,0 +1 @@
+Clarify lack of support for Windows.

changelog.d/11199.bugfix

@@ -0,0 +1 @@
+Delete `to_device` messages for hidden devices that will never be read, reducing database size.

changelog.d/11200.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error.

changelog.d/11204.feature

@@ -0,0 +1 @@
+Add a module API method to retrieve the current state of a room.

changelog.d/11205.misc

@@ -0,0 +1 @@
+Improve type hints for the relations datastore.

changelog.d/11206.removal

@@ -0,0 +1 @@
+The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information.

changelog.d/11207.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug which could result in serialization errors and potentially duplicate transaction data when sending ephemeral events to application services. Contributed by @Fizzadar at Beeper.

changelog.d/11209.docker

@@ -0,0 +1 @@
+Avoid changing userid when started as a non-root user, and no explicit `UID` is set.

changelog.d/11210.feature

@@ -0,0 +1 @@
+Calculate a default value for `public_baseurl` based on `server_name`.

changelog.d/11211.feature

@@ -0,0 +1 @@
+Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443.

changelog.d/11212.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.

changelog.d/11213.removal

@@ -0,0 +1 @@
+Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`).

changelog.d/11217.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats.

changelog.d/11221.doc

@@ -0,0 +1 @@
+Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper.

changelog.d/11225.misc

@@ -0,0 +1 @@
+Replace outdated links in the pull request checklist with links to the rendered documentation.

changelog.d/11226.misc

@@ -0,0 +1 @@
+Fix a bug in unit test `test_block_room_and_not_purge`.

changelog.d/11228.feature

@@ -1 +0,0 @@
-Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it.

changelog.d/11229.misc

@@ -0,0 +1 @@
+`ObservableDeferred`: run registered observers in order.

changelog.d/11231.misc

@@ -0,0 +1 @@
+Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`.

changelog.d/11233.misc

@@ -0,0 +1 @@
+Add `twine` and `towncrier` as dev dependencies, as they're used by the release script.

changelog.d/11236.feature

@@ -1 +0,0 @@
-Support filtering by relation senders & types per [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440).

changelog.d/11237.misc

@@ -0,0 +1 @@
+Allow `stream_writers.typing` config to be a list of one worker.

changelog.d/11239.misc

@@ -0,0 +1 @@
+Remove debugging statement in tests.

changelog.d/11240.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where all requests that read events from the database could get stuck as a result of losing the database connection.

changelog.d/11242.misc

@@ -1 +0,0 @@
-Split out federated PDU retrieval function into a non-cached version.

changelog.d/11244.misc

@@ -0,0 +1 @@
+Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers.

changelog.d/11246.misc

@@ -0,0 +1 @@
+Add an additional test for the `cachedList` method decorator.

changelog.d/11247.misc

@@ -1 +0,0 @@
-Clean up code relating to to-device messages and sending ephemeral events to application services.

changelog.d/11253.misc

@@ -0,0 +1 @@
+Make minor correction to the type of `auth_checkers` callbacks.

changelog.d/11255.bugfix

@@ -0,0 +1 @@
+Fix rolling back Synapse version when using workers.

changelog.d/11257.doc

@@ -0,0 +1 @@
+Add documentation for using LemonLDAP as an OpenID Connect Identity Provider. Contributed by @l00ptr.

changelog.d/11262.bugfix

@@ -0,0 +1 @@
+Fix a bug where if a remote event is being processed by a worker when it gets killed then it won't get processed on restart. Introduced in v1.37.1.

changelog.d/11263.feature

@@ -0,0 +1 @@
+Add some background update admin APIs.

changelog.d/11269.misc

@@ -0,0 +1 @@
+Clean up trivial aspects of the Debian package build tooling.

changelog.d/11270.misc

@@ -0,0 +1 @@
+Blacklist new SyTest that checks that key uploads are valid pending the validation being implemented in Synapse.

changelog.d/11273.misc

@@ -0,0 +1 @@
+Clean up trivial aspects of the Debian package build tooling.

changelog.d/11276.bugfix

@@ -0,0 +1 @@
+Fix rolling back Synapse version when using workers.

changelog.d/11278.misc

@@ -1 +0,0 @@
-Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`.

changelog.d/11281.doc

@@ -0,0 +1 @@
+Suggest users of the Debian packages add configuration to `/etc/matrix-synapse/conf.d/` to prevent, upon upgrade, being asked to choose between their configuration and the maintainer's.

changelog.d/11282.misc

@@ -1 +0,0 @@
-Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.

changelog.d/11285.misc

@@ -1 +0,0 @@
-Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.

changelog.d/11286.doc

@@ -1 +0,0 @@
-Fix typo in the word `available` and fix HTTP method (should be `GET`) for the `username_available` admin API. Contributed by Stanislav Motylkov.

changelog.d/11287.misc

@@ -1 +0,0 @@
-Add missing type hints to `synapse.app`.

changelog.d/11288.bugfix

@@ -1 +0,0 @@
-Fix a long-standing bug where uploading extremely thin images (e.g. 1000x1) would fail. Contributed by @Neeeflix.

changelog.d/11292.misc

@@ -1 +0,0 @@
-Remove unused parameters on `FederationEventHandler._check_event_auth`.

changelog.d/11297.misc

@@ -1 +0,0 @@
-Add type hints to `synapse._scripts`.

changelog.d/11298.doc

@@ -1 +0,0 @@
-Add Single Sign-On, SAML and CAS pages to the documentation.

debian/changelog

@@ -1,16 +1,12 @@
-matrix-synapse-py3 (1.47.0~rc2) stable; urgency=medium
+matrix-synapse-py3 (1.47.0+nmu1) UNRELEASED; urgency=medium
 
-  [ Dan Callahan ]
   * Update scripts to pass Shellcheck lints.
   * Remove unused Vagrant scripts from debian/ directory.
   * Allow building Debian packages for any architecture, not just amd64.
   * Preinstall the "wheel" package when building virtualenvs.
   * Do not error if /etc/default/matrix-synapse is missing.
 
-  [ Synapse Packaging team ]
-  * New synapse release 1.47.0~rc2.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 10 Nov 2021 09:41:01 +0000
+ -- Dan Callahan <danc@element.io>  Fri, 22 Oct 2021 22:20:31 +0000
 
 matrix-synapse-py3 (1.46.0) stable; urgency=medium
 

docs/SUMMARY.md

@@ -23,10 +23,10 @@
     - [Structured Logging](structured_logging.md)
     - [Templates](templates.md)
     - [User Authentication](usage/configuration/user_authentication/README.md)
-      - [Single-Sign On](usage/configuration/user_authentication/single_sign_on/README.md)
+      - [Single-Sign On]()
         - [OpenID Connect](openid.md)
-        - [SAML](usage/configuration/user_authentication/single_sign_on/saml.md)
-        - [CAS](usage/configuration/user_authentication/single_sign_on/cas.md)
+        - [SAML]()
+        - [CAS]()
         - [SSO Mapping Providers](sso_mapping_providers.md)
       - [Password Auth Providers](password_auth_providers.md)
       - [JSON Web Tokens](jwt.md)

docs/admin_api/rooms.md

@@ -396,17 +396,13 @@ The new room will be created with the user specified by the `new_room_user_id` p
 as room administrator and will contain a message explaining what happened. Users invited
 to the new room will have power level `-10` by default, and thus be unable to speak.
 
-If `block` is `true`, users will be prevented from joining the old room.
-This option can also be used to pre-emptively block a room, even if it's unknown
-to this homeserver. In this case, the room will be blocked, and no further action
-will be taken. If `block` is `false`, attempting to delete an unknown room is
-invalid and will be rejected as a bad request.
+If `block` is `True` it prevents new joins to the old room.
 
 This API will remove all trace of the old room from your database after removing
 all local users. If `purge` is `true` (the default), all traces of the old room will
 be removed from your database after removing all local users. If you do not want
 this to happen, set `purge` to `false`.
-Depending on the amount of history being purged, a call to the API may take
+Depending on the amount of history being purged a call to the API may take
 several minutes or longer.
 
 The local server will only have the power to move local user and room aliases to
@@ -468,9 +464,8 @@ The following JSON body parameters are available:
               `new_room_user_id` in the new room. Ideally this will clearly convey why the
                original room was shut down. Defaults to `Sharing illegal content on this server
                is not permitted and rooms in violation will be blocked.`
-* `block` - Optional. If set to `true`, this room will be added to a blocking list,
-            preventing future attempts to join the room. Rooms can be blocked
-            even if they're not yet known to the homeserver. Defaults to `false`.
+* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing
+            future attempts to join the room. Defaults to `false`.
 * `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
             Defaults to `true`.
 * `force_purge` - Optional, and ignored unless `purge` is `true`. If set to `true`, it
@@ -488,8 +483,7 @@ The following fields are returned in the JSON response body:
 * `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
 * `local_aliases` - An array of strings representing the local aliases that were migrated from
                     the old room to the new.
-* `new_room_id` - A string representing the room ID of the new room, or `null` if
-                  no such room was created.
+* `new_room_id` - A string representing the room ID of the new room.
 
 
 ## Undoing room deletions

docs/admin_api/user_admin_api.md

@@ -1107,7 +1107,7 @@ This endpoint will work even if registration is disabled on the server, unlike
 The API is:
 
 ```
-GET /_synapse/admin/v1/username_available?username=$localpart
+POST /_synapse/admin/v1/username_availabile?username=$localpart
 ```
 
 The request and response format is the same as the [/_matrix/client/r0/register/available](https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available) API.

docs/setup/installation.md

@@ -76,6 +76,12 @@ The fingerprint of the repository signing key (as shown by `gpg
 /usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
 `AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
 
+When installing with Debian packages, you might prefer to place files in
+`/etc/matrix-synapse/conf.d/` to override your configuration without editing
+the main configuration file at `/etc/matrix-synapse/homeserver.yaml`.
+By doing that, you won't be asked if you want to replace your configuration
+file when you upgrade the Debian package to a later version.
+
 ##### Downstream Debian packages
 
 We do not recommend using the packages from the default Debian `buster`

docs/usage/configuration/user_authentication/single_sign_on/README.md

@@ -1,5 +0,0 @@
-# Single Sign-On
-
-Synapse supports single sign-on through the SAML, Open ID Connect or CAS protocols. 
-LDAP and other login methods are supported through first and third-party password
-auth provider modules.

docs/usage/configuration/user_authentication/single_sign_on/cas.md

@@ -1,8 +0,0 @@
-# CAS
-
-Synapse supports authenticating users via the [Central Authentication
-Service protocol](https://en.wikipedia.org/wiki/Central_Authentication_Service)
-(CAS) natively.
-
-Please see the `cas_config` and `sso` sections of the [Synapse configuration
-file](../../../configuration/homeserver_sample_config.md) for more details.

docs/usage/configuration/user_authentication/single_sign_on/saml.md

@@ -1,8 +0,0 @@
-# SAML
-
-Synapse supports authenticating users via the [Security Assertion
-Markup Language](https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language)
-(SAML) protocol natively.
-
-Please see the `saml2_config` and `sso` sections of the [Synapse configuration
-file](../../../configuration/homeserver_sample_config.md) for more details.

mypy.ini

@@ -10,162 +10,86 @@ warn_unreachable = True
 local_partial_types = True
 no_implicit_optional = True
 
+# To find all folders that pass mypy you run:
+#
+#   find synapse/* -type d -not -name __pycache__ -exec bash -c "mypy '{}' > /dev/null"  \; -print
+
 files =
   scripts-dev/sign_json,
-  setup.py,
-  synapse/,
-  tests/
-
-# Note: Better exclusion syntax coming in mypy > 0.910
-# https://github.com/python/mypy/pull/11329
-#
-# For now, set the (?x) flag enable "verbose" regexes
-# https://docs.python.org/3/library/re.html#re.X
-exclude = (?x)
-  ^(
-   |synapse/storage/databases/__init__.py
-   |synapse/storage/databases/main/__init__.py
-   |synapse/storage/databases/main/account_data.py
-   |synapse/storage/databases/main/cache.py
-   |synapse/storage/databases/main/censor_events.py
-   |synapse/storage/databases/main/deviceinbox.py
-   |synapse/storage/databases/main/devices.py
-   |synapse/storage/databases/main/directory.py
-   |synapse/storage/databases/main/e2e_room_keys.py
-   |synapse/storage/databases/main/end_to_end_keys.py
-   |synapse/storage/databases/main/event_federation.py
-   |synapse/storage/databases/main/event_push_actions.py
-   |synapse/storage/databases/main/events_bg_updates.py
-   |synapse/storage/databases/main/events_forward_extremities.py
-   |synapse/storage/databases/main/events_worker.py
-   |synapse/storage/databases/main/filtering.py
-   |synapse/storage/databases/main/group_server.py
-   |synapse/storage/databases/main/lock.py
-   |synapse/storage/databases/main/media_repository.py
-   |synapse/storage/databases/main/metrics.py
-   |synapse/storage/databases/main/monthly_active_users.py
-   |synapse/storage/databases/main/openid.py
-   |synapse/storage/databases/main/presence.py
-   |synapse/storage/databases/main/profile.py
-   |synapse/storage/databases/main/purge_events.py
-   |synapse/storage/databases/main/push_rule.py
-   |synapse/storage/databases/main/receipts.py
-   |synapse/storage/databases/main/rejections.py
-   |synapse/storage/databases/main/room.py
-   |synapse/storage/databases/main/room_batch.py
-   |synapse/storage/databases/main/roommember.py
-   |synapse/storage/databases/main/search.py
-   |synapse/storage/databases/main/signatures.py
-   |synapse/storage/databases/main/state.py
-   |synapse/storage/databases/main/state_deltas.py
-   |synapse/storage/databases/main/stats.py
-   |synapse/storage/databases/main/tags.py
-   |synapse/storage/databases/main/transactions.py
-   |synapse/storage/databases/main/user_directory.py
-   |synapse/storage/databases/main/user_erasure_store.py
-   |synapse/storage/schema/
-
-   |tests/api/test_auth.py
-   |tests/api/test_ratelimiting.py
-   |tests/app/test_openid_listener.py
-   |tests/appservice/test_scheduler.py
-   |tests/config/test_cache.py
-   |tests/config/test_tls.py
-   |tests/crypto/test_keyring.py
-   |tests/events/test_presence_router.py
-   |tests/events/test_utils.py
-   |tests/federation/test_federation_catch_up.py
-   |tests/federation/test_federation_sender.py
-   |tests/federation/test_federation_server.py
-   |tests/federation/transport/test_knocking.py
-   |tests/federation/transport/test_server.py
-   |tests/handlers/test_cas.py
-   |tests/handlers/test_directory.py
-   |tests/handlers/test_e2e_keys.py
-   |tests/handlers/test_federation.py
-   |tests/handlers/test_oidc.py
-   |tests/handlers/test_presence.py
-   |tests/handlers/test_profile.py
-   |tests/handlers/test_saml.py
-   |tests/handlers/test_typing.py
-   |tests/http/federation/test_matrix_federation_agent.py
-   |tests/http/federation/test_srv_resolver.py
-   |tests/http/test_fedclient.py
-   |tests/http/test_proxyagent.py
-   |tests/http/test_servlet.py
-   |tests/http/test_site.py
-   |tests/logging/__init__.py
-   |tests/logging/test_terse_json.py
-   |tests/module_api/test_api.py
-   |tests/push/test_email.py
-   |tests/push/test_http.py
-   |tests/push/test_presentable_names.py
-   |tests/push/test_push_rule_evaluator.py
-   |tests/rest/admin/test_admin.py
-   |tests/rest/admin/test_device.py
-   |tests/rest/admin/test_media.py
-   |tests/rest/admin/test_server_notice.py
-   |tests/rest/admin/test_user.py
-   |tests/rest/admin/test_username_available.py
-   |tests/rest/client/test_account.py
-   |tests/rest/client/test_events.py
-   |tests/rest/client/test_filter.py
-   |tests/rest/client/test_groups.py
-   |tests/rest/client/test_register.py
-   |tests/rest/client/test_report_event.py
-   |tests/rest/client/test_rooms.py
-   |tests/rest/client/test_third_party_rules.py
-   |tests/rest/client/test_transactions.py
-   |tests/rest/client/test_typing.py
-   |tests/rest/client/utils.py
-   |tests/rest/key/v2/test_remote_key_resource.py
-   |tests/rest/media/v1/test_base.py
-   |tests/rest/media/v1/test_media_storage.py
-   |tests/rest/media/v1/test_url_preview.py
-   |tests/scripts/test_new_matrix_user.py
-   |tests/server.py
-   |tests/server_notices/test_resource_limits_server_notices.py
-   |tests/state/test_v2.py
-   |tests/storage/test_account_data.py
-   |tests/storage/test_appservice.py
-   |tests/storage/test_background_update.py
-   |tests/storage/test_base.py
-   |tests/storage/test_client_ips.py
-   |tests/storage/test_database.py
-   |tests/storage/test_event_federation.py
-   |tests/storage/test_id_generators.py
-   |tests/storage/test_roommember.py
-   |tests/test_metrics.py
-   |tests/test_phone_home.py
-   |tests/test_server.py
-   |tests/test_state.py
-   |tests/test_terms_auth.py
-   |tests/test_visibility.py
-   |tests/unittest.py
-   |tests/util/caches/test_cached_call.py
-   |tests/util/caches/test_deferred_cache.py
-   |tests/util/caches/test_descriptors.py
-   |tests/util/caches/test_response_cache.py
-   |tests/util/caches/test_ttlcache.py
-   |tests/util/test_async_helpers.py
-   |tests/util/test_batching_queue.py
-   |tests/util/test_dict_cache.py
-   |tests/util/test_expiring_cache.py
-   |tests/util/test_file_consumer.py
-   |tests/util/test_linearizer.py
-   |tests/util/test_logcontext.py
-   |tests/util/test_lrucache.py
-   |tests/util/test_rwlock.py
-   |tests/util/test_wheel_timer.py
-   |tests/utils.py
-   )$
+  synapse/__init__.py,
+  synapse/api,
+  synapse/appservice,
+  synapse/config,
+  synapse/crypto,
+  synapse/event_auth.py,
+  synapse/events,
+  synapse/federation,
+  synapse/groups,
+  synapse/handlers,
+  synapse/http,
+  synapse/logging,
+  synapse/metrics,
+  synapse/module_api,
+  synapse/notifier.py,
+  synapse/push,
+  synapse/replication,
+  synapse/rest,
+  synapse/server.py,
+  synapse/server_notices,
+  synapse/spam_checker_api,
+  synapse/state,
+  synapse/storage/__init__.py,
+  synapse/storage/_base.py,
+  synapse/storage/background_updates.py,
+  synapse/storage/databases/main/appservice.py,
+  synapse/storage/databases/main/client_ips.py,
+  synapse/storage/databases/main/events.py,
+  synapse/storage/databases/main/keys.py,
+  synapse/storage/databases/main/pusher.py,
+  synapse/storage/databases/main/registration.py,
+  synapse/storage/databases/main/relations.py,
+  synapse/storage/databases/main/session.py,
+  synapse/storage/databases/main/stream.py,
+  synapse/storage/databases/main/ui_auth.py,
+  synapse/storage/databases/state,
+  synapse/storage/database.py,
+  synapse/storage/engines,
+  synapse/storage/keys.py,
+  synapse/storage/persist_events.py,
+  synapse/storage/prepare_database.py,
+  synapse/storage/purge_events.py,
+  synapse/storage/push_rule.py,
+  synapse/storage/relations.py,
+  synapse/storage/roommember.py,
+  synapse/storage/state.py,
+  synapse/storage/types.py,
+  synapse/storage/util,
+  synapse/streams,
+  synapse/types.py,
+  synapse/util,
+  synapse/visibility.py,
+  tests/replication,
+  tests/test_event_auth.py,
+  tests/test_utils,
+  tests/handlers/test_password_providers.py,
+  tests/handlers/test_room.py,
+  tests/handlers/test_room_summary.py,
+  tests/handlers/test_send_email.py,
+  tests/handlers/test_sync.py,
+  tests/handlers/test_user_directory.py,
+  tests/rest/client/test_login.py,
+  tests/rest/client/test_auth.py,
+  tests/rest/client/test_relations.py,
+  tests/rest/media/v1/test_filepath.py,
+  tests/rest/media/v1/test_oembed.py,
+  tests/storage/test_state.py,
+  tests/storage/test_user_directory.py,
+  tests/util/test_itertools.py,
+  tests/util/test_stream_change_cache.py
 
 [mypy-synapse.api.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.app.*]
-disallow_untyped_defs = True
-
 [mypy-synapse.crypto.*]
 disallow_untyped_defs = True
 
@@ -348,9 +272,6 @@ ignore_missing_imports = True
 [mypy-opentracing]
 ignore_missing_imports = True
 
-[mypy-parameterized.*]
-ignore_missing_imports = True
-
 [mypy-phonenumbers.*]
 ignore_missing_imports = True
 

setup.py

@@ -17,7 +17,6 @@
 # limitations under the License.
 import glob
 import os
-from typing import Any, Dict
 
 from setuptools import Command, find_packages, setup
 
@@ -50,6 +49,8 @@ here = os.path.abspath(os.path.dirname(__file__))
 # [1]: http://tox.readthedocs.io/en/2.5.0/example/basic.html#integration-with-setup-py-test-command
 # [2]: https://pypi.python.org/pypi/setuptools_trial
 class TestCommand(Command):
+    user_options = []
+
     def initialize_options(self):
         pass
 
@@ -74,7 +75,7 @@ def read_file(path_segments):
 
 def exec_file(path_segments):
     """Execute a single python file to get the variables defined in it"""
-    result: Dict[str, Any] = {}
+    result = {}
     code = read_file(path_segments)
     exec(code, result)
     return result
@@ -110,7 +111,6 @@ CONDITIONAL_REQUIREMENTS["mypy"] = [
     "types-Pillow>=8.3.4",
     "types-pyOpenSSL>=20.0.7",
     "types-PyYAML>=5.4.10",
-    "types-requests>=2.26.0",
     "types-setuptools>=57.4.0",
 ]
 

synapse/__init__.py

@@ -47,7 +47,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.47.0rc2"
+__version__ = "1.46.0"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/_scripts/register_new_matrix_user.py

@@ -1,6 +1,5 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2018 New Vector
-# Copyright 2021 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,23 +19,22 @@ import hashlib
 import hmac
 import logging
 import sys
-from typing import Callable, Optional
 
 import requests as _requests
 import yaml
 
 
 def request_registration(
-    user: str,
-    password: str,
-    server_location: str,
-    shared_secret: str,
-    admin: bool = False,
-    user_type: Optional[str] = None,
+    user,
+    password,
+    server_location,
+    shared_secret,
+    admin=False,
+    user_type=None,
     requests=_requests,
-    _print: Callable[[str], None] = print,
-    exit: Callable[[int], None] = sys.exit,
-) -> None:
+    _print=print,
+    exit=sys.exit,
+):
 
     url = "%s/_synapse/admin/v1/register" % (server_location.rstrip("/"),)
 
@@ -67,13 +65,13 @@ def request_registration(
         mac.update(b"\x00")
         mac.update(user_type.encode("utf8"))
 
-    hex_mac = mac.hexdigest()
+    mac = mac.hexdigest()
 
     data = {
         "nonce": nonce,
         "username": user,
         "password": password,
-        "mac": hex_mac,
+        "mac": mac,
         "admin": admin,
         "user_type": user_type,
     }
@@ -93,17 +91,10 @@ def request_registration(
     _print("Success!")
 
 
-def register_new_user(
-    user: str,
-    password: str,
-    server_location: str,
-    shared_secret: str,
-    admin: Optional[bool],
-    user_type: Optional[str],
-) -> None:
+def register_new_user(user, password, server_location, shared_secret, admin, user_type):
     if not user:
         try:
-            default_user: Optional[str] = getpass.getuser()
+            default_user = getpass.getuser()
         except Exception:
             default_user = None
 
@@ -132,8 +123,8 @@ def register_new_user(
             sys.exit(1)
 
     if admin is None:
-        admin_inp = input("Make admin [no]: ")
-        if admin_inp in ("y", "yes", "true"):
+        admin = input("Make admin [no]: ")
+        if admin in ("y", "yes", "true"):
             admin = True
         else:
             admin = False
@@ -143,7 +134,7 @@ def register_new_user(
     )
 
 
-def main() -> None:
+def main():
 
     logging.captureWarnings(True)
 

synapse/_scripts/review_recent_signups.py

@@ -92,7 +92,7 @@ def get_recent_users(txn: LoggingTransaction, since_ms: int) -> List[UserInfo]:
     return user_infos
 
 
-def main() -> None:
+def main():
     parser = argparse.ArgumentParser()
     parser.add_argument(
         "-c",
@@ -142,8 +142,7 @@ def main() -> None:
     engine = create_engine(database_config.config)
 
     with make_conn(database_config, engine, "review_recent_signups") as db_conn:
-        # This generates a type of Cursor, not LoggingTransaction.
-        user_infos = get_recent_users(db_conn.cursor(), since_ms)  # type: ignore[arg-type]
+        user_infos = get_recent_users(db_conn.cursor(), since_ms)
 
     for user_info in user_infos:
         if exclude_users_with_email and user_info.emails:

synapse/api/filtering.py

@@ -1,7 +1,7 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
 # Copyright 2018-2019 New Vector Ltd
-# Copyright 2019-2021 The Matrix.org Foundation C.I.C.
+# Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -86,9 +86,6 @@ ROOM_EVENT_FILTER_SCHEMA = {
         # cf https://github.com/matrix-org/matrix-doc/pull/2326
         "org.matrix.labels": {"type": "array", "items": {"type": "string"}},
         "org.matrix.not_labels": {"type": "array", "items": {"type": "string"}},
-        # MSC3440, filtering by event relations.
-        "io.element.relation_senders": {"type": "array", "items": {"type": "string"}},
-        "io.element.relation_types": {"type": "array", "items": {"type": "string"}},
     },
 }
 
@@ -149,16 +146,14 @@ def matrix_user_id_validator(user_id_str: str) -> UserID:
 
 class Filtering:
     def __init__(self, hs: "HomeServer"):
-        self._hs = hs
+        super().__init__()
         self.store = hs.get_datastore()
 
-        self.DEFAULT_FILTER_COLLECTION = FilterCollection(hs, {})
-
     async def get_user_filter(
         self, user_localpart: str, filter_id: Union[int, str]
     ) -> "FilterCollection":
         result = await self.store.get_user_filter(user_localpart, filter_id)
-        return FilterCollection(self._hs, result)
+        return FilterCollection(result)
 
     def add_user_filter(
         self, user_localpart: str, user_filter: JsonDict
@@ -196,22 +191,21 @@ FilterEvent = TypeVar("FilterEvent", EventBase, UserPresenceState, JsonDict)
 
 
 class FilterCollection:
-    def __init__(self, hs: "HomeServer", filter_json: JsonDict):
+    def __init__(self, filter_json: JsonDict):
         self._filter_json = filter_json
 
         room_filter_json = self._filter_json.get("room", {})
 
         self._room_filter = Filter(
-            hs,
-            {k: v for k, v in room_filter_json.items() if k in ("rooms", "not_rooms")},
+            {k: v for k, v in room_filter_json.items() if k in ("rooms", "not_rooms")}
         )
 
-        self._room_timeline_filter = Filter(hs, room_filter_json.get("timeline", {}))
-        self._room_state_filter = Filter(hs, room_filter_json.get("state", {}))
-        self._room_ephemeral_filter = Filter(hs, room_filter_json.get("ephemeral", {}))
-        self._room_account_data = Filter(hs, room_filter_json.get("account_data", {}))
-        self._presence_filter = Filter(hs, filter_json.get("presence", {}))
-        self._account_data = Filter(hs, filter_json.get("account_data", {}))
+        self._room_timeline_filter = Filter(room_filter_json.get("timeline", {}))
+        self._room_state_filter = Filter(room_filter_json.get("state", {}))
+        self._room_ephemeral_filter = Filter(room_filter_json.get("ephemeral", {}))
+        self._room_account_data = Filter(room_filter_json.get("account_data", {}))
+        self._presence_filter = Filter(filter_json.get("presence", {}))
+        self._account_data = Filter(filter_json.get("account_data", {}))
 
         self.include_leave = filter_json.get("room", {}).get("include_leave", False)
         self.event_fields = filter_json.get("event_fields", [])
@@ -238,37 +232,25 @@ class FilterCollection:
     def include_redundant_members(self) -> bool:
         return self._room_state_filter.include_redundant_members
 
-    async def filter_presence(
+    def filter_presence(
         self, events: Iterable[UserPresenceState]
     ) -> List[UserPresenceState]:
-        return await self._presence_filter.filter(events)
+        return self._presence_filter.filter(events)
 
-    async def filter_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return await self._account_data.filter(events)
+    def filter_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._account_data.filter(events)
 
-    async def filter_room_state(self, events: Iterable[EventBase]) -> List[EventBase]:
-        return await self._room_state_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_state(self, events: Iterable[EventBase]) -> List[EventBase]:
+        return self._room_state_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_timeline(
-        self, events: Iterable[EventBase]
-    ) -> List[EventBase]:
-        return await self._room_timeline_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_timeline(self, events: Iterable[EventBase]) -> List[EventBase]:
+        return self._room_timeline_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_ephemeral(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return await self._room_ephemeral_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_ephemeral(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._room_ephemeral_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_account_data(
-        self, events: Iterable[JsonDict]
-    ) -> List[JsonDict]:
-        return await self._room_account_data.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._room_account_data.filter(self._room_filter.filter(events))
 
     def blocks_all_presence(self) -> bool:
         return (
@@ -292,9 +274,7 @@ class FilterCollection:
 
 
 class Filter:
-    def __init__(self, hs: "HomeServer", filter_json: JsonDict):
-        self._hs = hs
-        self._store = hs.get_datastore()
+    def __init__(self, filter_json: JsonDict):
         self.filter_json = filter_json
 
         self.limit = filter_json.get("limit", 10)
@@ -317,20 +297,6 @@ class Filter:
         self.labels = filter_json.get("org.matrix.labels", None)
         self.not_labels = filter_json.get("org.matrix.not_labels", [])
 
-        # Ideally these would be rejected at the endpoint if they were provided
-        # and not supported, but that would involve modifying the JSON schema
-        # based on the homeserver configuration.
-        if hs.config.experimental.msc3440_enabled:
-            self.relation_senders = self.filter_json.get(
-                "io.element.relation_senders", None
-            )
-            self.relation_types = self.filter_json.get(
-                "io.element.relation_types", None
-            )
-        else:
-            self.relation_senders = None
-            self.relation_types = None
-
     def filters_all_types(self) -> bool:
         return "*" in self.not_types
 
@@ -340,7 +306,7 @@ class Filter:
     def filters_all_rooms(self) -> bool:
         return "*" in self.not_rooms
 
-    def _check(self, event: FilterEvent) -> bool:
+    def check(self, event: FilterEvent) -> bool:
         """Checks whether the filter matches the given event.
 
         Args:
@@ -454,30 +420,8 @@ class Filter:
 
         return room_ids
 
-    async def _check_event_relations(
-        self, events: Iterable[FilterEvent]
-    ) -> List[FilterEvent]:
-        # The event IDs to check, mypy doesn't understand the ifinstance check.
-        event_ids = [event.event_id for event in events if isinstance(event, EventBase)]  # type: ignore[attr-defined]
-        event_ids_to_keep = set(
-            await self._store.events_have_relations(
-                event_ids, self.relation_senders, self.relation_types
-            )
-        )
-
-        return [
-            event
-            for event in events
-            if not isinstance(event, EventBase) or event.event_id in event_ids_to_keep
-        ]
-
-    async def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
-        result = [event for event in events if self._check(event)]
-
-        if self.relation_senders or self.relation_types:
-            return await self._check_event_relations(result)
-
-        return result
+    def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
+        return list(filter(self.check, events))
 
     def with_room_ids(self, room_ids: Iterable[str]) -> "Filter":
         """Returns a new filter with the given room IDs appended.
@@ -489,7 +433,7 @@ class Filter:
             filter: A new filter including the given rooms and the old
                     filter's rooms.
         """
-        newFilter = Filter(self._hs, self.filter_json)
+        newFilter = Filter(self.filter_json)
         newFilter.rooms += room_ids
         return newFilter
 
@@ -500,3 +444,6 @@ def _matches_wildcard(actual_value: Optional[str], filter_value: str) -> bool:
         return actual_value.startswith(type_prefix)
     else:
         return actual_value == filter_value
+
+
+DEFAULT_FILTER_COLLECTION = FilterCollection({})

synapse/app/__init__.py

@@ -13,7 +13,6 @@
 # limitations under the License.
 import logging
 import sys
-from typing import Container
 
 from synapse import python_dependencies  # noqa: E402
 
@@ -28,9 +27,7 @@ except python_dependencies.DependencyException as e:
     sys.exit(1)
 
 
-def check_bind_error(
-    e: Exception, address: str, bind_addresses: Container[str]
-) -> None:
+def check_bind_error(e, address, bind_addresses):
     """
     This method checks an exception occurred while binding on 0.0.0.0.
     If :: is specified in the bind addresses a warning is shown.
@@ -41,9 +38,9 @@ def check_bind_error(
     When binding on 0.0.0.0 after :: this can safely be ignored.
 
     Args:
-        e: Exception that was caught.
-        address: Address on which binding was attempted.
-        bind_addresses: Addresses on which the service listens.
+        e (Exception): Exception that was caught.
+        address (str): Address on which binding was attempted.
+        bind_addresses (list): Addresses on which the service listens.
     """
     if address == "0.0.0.0" and "::" in bind_addresses:
         logger.warning(

synapse/app/_base.py

@@ -22,27 +22,13 @@ import socket
 import sys
 import traceback
 import warnings
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Awaitable,
-    Callable,
-    Collection,
-    Dict,
-    Iterable,
-    List,
-    NoReturn,
-    Tuple,
-    cast,
-)
+from typing import TYPE_CHECKING, Awaitable, Callable, Iterable
 
 from cryptography.utils import CryptographyDeprecationWarning
+from typing_extensions import NoReturn
 
 import twisted
-from twisted.internet import defer, error, reactor as _reactor
-from twisted.internet.interfaces import IOpenSSLContextFactory, IReactorSSL, IReactorTCP
-from twisted.internet.protocol import ServerFactory
-from twisted.internet.tcp import Port
+from twisted.internet import defer, error, reactor
 from twisted.logger import LoggingFile, LogLevel
 from twisted.protocols.tls import TLSMemoryBIOFactory
 from twisted.python.threadpool import ThreadPool
@@ -62,7 +48,6 @@ from synapse.logging.context import PreserveLoggingContext
 from synapse.metrics import register_threadpool
 from synapse.metrics.background_process_metrics import wrap_as_background_process
 from synapse.metrics.jemalloc import setup_jemalloc_stats
-from synapse.types import ISynapseReactor
 from synapse.util.caches.lrucache import setup_expire_lru_cache_entries
 from synapse.util.daemonize import daemonize_process
 from synapse.util.gai_resolver import GAIResolver
@@ -72,44 +57,33 @@ from synapse.util.versionstring import get_version_string
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
-# Twisted injects the global reactor to make it easier to import, this confuses
-# mypy which thinks it is a module. Tell it that it a more proper type.
-reactor = cast(ISynapseReactor, _reactor)
-
-
 logger = logging.getLogger(__name__)
 
 # list of tuples of function, args list, kwargs dict
-_sighup_callbacks: List[
-    Tuple[Callable[..., None], Tuple[Any, ...], Dict[str, Any]]
-] = []
+_sighup_callbacks = []
 
 
-def register_sighup(func: Callable[..., None], *args: Any, **kwargs: Any) -> None:
+def register_sighup(func, *args, **kwargs):
     """
     Register a function to be called when a SIGHUP occurs.
 
     Args:
-        func: Function to be called when sent a SIGHUP signal.
+        func (function): Function to be called when sent a SIGHUP signal.
         *args, **kwargs: args and kwargs to be passed to the target function.
     """
     _sighup_callbacks.append((func, args, kwargs))
 
 
-def start_worker_reactor(
-    appname: str,
-    config: HomeServerConfig,
-    run_command: Callable[[], None] = reactor.run,
-) -> None:
+def start_worker_reactor(appname, config, run_command=reactor.run):
     """Run the reactor in the main process
 
     Daemonizes if necessary, and then configures some resources, before starting
     the reactor. Pulls configuration from the 'worker' settings in 'config'.
 
     Args:
-        appname: application name which will be sent to syslog
-        config: config object
-        run_command: callable that actually runs the reactor
+        appname (str): application name which will be sent to syslog
+        config (synapse.config.Config): config object
+        run_command (Callable[]): callable that actually runs the reactor
     """
 
     logger = logging.getLogger(config.worker.worker_app)
@@ -127,32 +101,32 @@ def start_worker_reactor(
 
 
 def start_reactor(
-    appname: str,
-    soft_file_limit: int,
-    gc_thresholds: Tuple[int, int, int],
-    pid_file: str,
-    daemonize: bool,
-    print_pidfile: bool,
-    logger: logging.Logger,
-    run_command: Callable[[], None] = reactor.run,
-) -> None:
+    appname,
+    soft_file_limit,
+    gc_thresholds,
+    pid_file,
+    daemonize,
+    print_pidfile,
+    logger,
+    run_command=reactor.run,
+):
     """Run the reactor in the main process
 
     Daemonizes if necessary, and then configures some resources, before starting
     the reactor
 
     Args:
-        appname: application name which will be sent to syslog
-        soft_file_limit:
+        appname (str): application name which will be sent to syslog
+        soft_file_limit (int):
         gc_thresholds:
-        pid_file: name of pid file to write to if daemonize is True
-        daemonize: true to run the reactor in a background process
-        print_pidfile: whether to print the pid file, if daemonize is True
-        logger: logger instance to pass to Daemonize
-        run_command: callable that actually runs the reactor
+        pid_file (str): name of pid file to write to if daemonize is True
+        daemonize (bool): true to run the reactor in a background process
+        print_pidfile (bool): whether to print the pid file, if daemonize is True
+        logger (logging.Logger): logger instance to pass to Daemonize
+        run_command (Callable[]): callable that actually runs the reactor
     """
 
-    def run() -> None:
+    def run():
         logger.info("Running")
         setup_jemalloc_stats()
         change_resource_limit(soft_file_limit)
@@ -211,7 +185,7 @@ def redirect_stdio_to_logs() -> None:
     print("Redirected stdout/stderr to logs")
 
 
-def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> None:
+def register_start(cb: Callable[..., Awaitable], *args, **kwargs) -> None:
     """Register a callback with the reactor, to be called once it is running
 
     This can be used to initialise parts of the system which require an asynchronous
@@ -221,7 +195,7 @@ def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> N
     will exit.
     """
 
-    async def wrapper() -> None:
+    async def wrapper():
         try:
             await cb(*args, **kwargs)
         except Exception:
@@ -250,7 +224,7 @@ def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> N
     reactor.callWhenRunning(lambda: defer.ensureDeferred(wrapper()))
 
 
-def listen_metrics(bind_addresses: Iterable[str], port: int) -> None:
+def listen_metrics(bind_addresses, port):
     """
     Start Prometheus metrics server.
     """
@@ -262,11 +236,11 @@ def listen_metrics(bind_addresses: Iterable[str], port: int) -> None:
 
 
 def listen_manhole(
-    bind_addresses: Collection[str],
+    bind_addresses: Iterable[str],
     port: int,
     manhole_settings: ManholeConfig,
     manhole_globals: dict,
-) -> None:
+):
     # twisted.conch.manhole 21.1.0 uses "int_from_bytes", which produces a confusing
     # warning. It's fixed by https://github.com/twisted/twisted/pull/1522), so
     # suppress the warning for now.
@@ -285,18 +259,12 @@ def listen_manhole(
     )
 
 
-def listen_tcp(
-    bind_addresses: Collection[str],
-    port: int,
-    factory: ServerFactory,
-    reactor: IReactorTCP = reactor,
-    backlog: int = 50,
-) -> List[Port]:
+def listen_tcp(bind_addresses, port, factory, reactor=reactor, backlog=50):
     """
     Create a TCP socket for a port and several addresses
 
     Returns:
-        list of twisted.internet.tcp.Port listening for TCP connections
+        list[twisted.internet.tcp.Port]: listening for TCP connections
     """
     r = []
     for address in bind_addresses:
@@ -305,19 +273,12 @@ def listen_tcp(
         except error.CannotListenError as e:
             check_bind_error(e, address, bind_addresses)
 
-    # IReactorTCP returns an object implementing IListeningPort from listenTCP,
-    # but we know it will be a Port instance.
-    return r  # type: ignore[return-value]
+    return r
 
 
 def listen_ssl(
-    bind_addresses: Collection[str],
-    port: int,
-    factory: ServerFactory,
-    context_factory: IOpenSSLContextFactory,
-    reactor: IReactorSSL = reactor,
-    backlog: int = 50,
-) -> List[Port]:
+    bind_addresses, port, factory, context_factory, reactor=reactor, backlog=50
+):
     """
     Create an TLS-over-TCP socket for a port and several addresses
 
@@ -333,13 +294,10 @@ def listen_ssl(
         except error.CannotListenError as e:
             check_bind_error(e, address, bind_addresses)
 
-    # IReactorSSL incorrectly declares that an int is returned from listenSSL,
-    # it actually returns an object implementing IListeningPort, but we know it
-    # will be a Port instance.
-    return r  # type: ignore[return-value]
+    return r
 
 
-def refresh_certificate(hs: "HomeServer") -> None:
+def refresh_certificate(hs: "HomeServer"):
     """
     Refresh the TLS certificates that Synapse is using by re-reading them from
     disk and updating the TLS context factories to use them.
@@ -371,7 +329,7 @@ def refresh_certificate(hs: "HomeServer") -> None:
         logger.info("Context factories updated.")
 
 
-async def start(hs: "HomeServer") -> None:
+async def start(hs: "HomeServer"):
     """
     Start a Synapse server or worker.
 
@@ -402,7 +360,7 @@ async def start(hs: "HomeServer") -> None:
     if hasattr(signal, "SIGHUP"):
 
         @wrap_as_background_process("sighup")
-        def handle_sighup(*args: Any, **kwargs: Any) -> None:
+        def handle_sighup(*args, **kwargs):
             # Tell systemd our state, if we're using it. This will silently fail if
             # we're not using systemd.
             sdnotify(b"RELOADING=1")
@@ -415,7 +373,7 @@ async def start(hs: "HomeServer") -> None:
         # We defer running the sighup handlers until next reactor tick. This
         # is so that we're in a sane state, e.g. flushing the logs may fail
         # if the sighup happens in the middle of writing a log entry.
-        def run_sighup(*args: Any, **kwargs: Any) -> None:
+        def run_sighup(*args, **kwargs):
             # `callFromThread` should be "signal safe" as well as thread
             # safe.
             reactor.callFromThread(handle_sighup, *args, **kwargs)
@@ -478,8 +436,12 @@ async def start(hs: "HomeServer") -> None:
         atexit.register(gc.freeze)
 
 
-def setup_sentry(hs: "HomeServer") -> None:
-    """Enable sentry integration, if enabled in configuration"""
+def setup_sentry(hs: "HomeServer"):
+    """Enable sentry integration, if enabled in configuration
+
+    Args:
+        hs
+    """
 
     if not hs.config.metrics.sentry_enabled:
         return
@@ -504,7 +466,7 @@ def setup_sentry(hs: "HomeServer") -> None:
         scope.set_tag("worker_name", name)
 
 
-def setup_sdnotify(hs: "HomeServer") -> None:
+def setup_sdnotify(hs: "HomeServer"):
     """Adds process state hooks to tell systemd what we are up to."""
 
     # Tell systemd our state, if we're using it. This will silently fail if
@@ -519,7 +481,7 @@ def setup_sdnotify(hs: "HomeServer") -> None:
 sdnotify_sockaddr = os.getenv("NOTIFY_SOCKET")
 
 
-def sdnotify(state: bytes) -> None:
+def sdnotify(state):
     """
     Send a notification to systemd, if the NOTIFY_SOCKET env var is set.
 
@@ -528,7 +490,7 @@ def sdnotify(state: bytes) -> None:
     package which many OSes don't include as a matter of principle.
 
     Args:
-        state: notification to send
+        state (bytes): notification to send
     """
     if not isinstance(state, bytes):
         raise TypeError("sdnotify should be called with a bytes")

synapse/app/admin_cmd.py

@@ -17,7 +17,6 @@ import logging
 import os
 import sys
 import tempfile
-from typing import List, Optional
 
 from twisted.internet import defer, task
 
@@ -26,7 +25,6 @@ from synapse.app import _base
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
-from synapse.events import EventBase
 from synapse.handlers.admin import ExfiltrationWriter
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
@@ -42,7 +40,6 @@ from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.server import HomeServer
 from synapse.storage.databases.main.room import RoomWorkerStore
-from synapse.types import StateMap
 from synapse.util.logcontext import LoggingContext
 from synapse.util.versionstring import get_version_string
 
@@ -68,11 +65,16 @@ class AdminCmdSlavedStore(
 
 
 class AdminCmdServer(HomeServer):
-    DATASTORE_CLASS = AdminCmdSlavedStore  # type: ignore
+    DATASTORE_CLASS = AdminCmdSlavedStore
 
 
-async def export_data_command(hs: HomeServer, args: argparse.Namespace) -> None:
-    """Export data for a user."""
+async def export_data_command(hs: HomeServer, args):
+    """Export data for a user.
+
+    Args:
+        hs
+        args (argparse.Namespace)
+    """
 
     user_id = args.user_id
     directory = args.output_directory
@@ -90,12 +92,12 @@ class FileExfiltrationWriter(ExfiltrationWriter):
     Note: This writes to disk on the main reactor thread.
 
     Args:
-        user_id: The user whose data is being exfiltrated.
-        directory: The directory to write the data to, if None then will write
-            to a temporary directory.
+        user_id (str): The user whose data is being exfiltrated.
+        directory (str|None): The directory to write the data to, if None then
+            will write to a temporary directory.
     """
 
-    def __init__(self, user_id: str, directory: Optional[str] = None):
+    def __init__(self, user_id, directory=None):
         self.user_id = user_id
 
         if directory:
@@ -109,7 +111,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
         if list(os.listdir(self.base_directory)):
             raise Exception("Directory must be empty")
 
-    def write_events(self, room_id: str, events: List[EventBase]) -> None:
+    def write_events(self, room_id, events):
         room_directory = os.path.join(self.base_directory, "rooms", room_id)
         os.makedirs(room_directory, exist_ok=True)
         events_file = os.path.join(room_directory, "events")
@@ -118,9 +120,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in events:
                 print(json.dumps(event.get_pdu_json()), file=f)
 
-    def write_state(
-        self, room_id: str, event_id: str, state: StateMap[EventBase]
-    ) -> None:
+    def write_state(self, room_id, event_id, state):
         room_directory = os.path.join(self.base_directory, "rooms", room_id)
         state_directory = os.path.join(room_directory, "state")
         os.makedirs(state_directory, exist_ok=True)
@@ -131,9 +131,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event.get_pdu_json()), file=f)
 
-    def write_invite(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
-    ) -> None:
+    def write_invite(self, room_id, event, state):
         self.write_events(room_id, [event])
 
         # We write the invite state somewhere else as they aren't full events
@@ -147,9 +145,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event), file=f)
 
-    def write_knock(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
-    ) -> None:
+    def write_knock(self, room_id, event, state):
         self.write_events(room_id, [event])
 
         # We write the knock state somewhere else as they aren't full events
@@ -163,11 +159,11 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event), file=f)
 
-    def finished(self) -> str:
+    def finished(self):
         return self.base_directory
 
 
-def start(config_options: List[str]) -> None:
+def start(config_options):
     parser = argparse.ArgumentParser(description="Synapse Admin Command")
     HomeServerConfig.add_arguments_to_parser(parser)
 
@@ -235,7 +231,7 @@ def start(config_options: List[str]) -> None:
     # We also make sure that `_base.start` gets run before we actually run the
     # command.
 
-    async def run() -> None:
+    async def run():
         with LoggingContext("command"):
             await _base.start(ss)
             await args.func(ss, args)

synapse/app/generic_worker.py

@@ -14,10 +14,11 @@
 # limitations under the License.
 import logging
 import sys
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, Optional
 
 from twisted.internet import address
-from twisted.web.resource import Resource
+from twisted.web.resource import IResource
+from twisted.web.server import Request
 
 import synapse
 import synapse.events
@@ -43,7 +44,7 @@ from synapse.config.server import ListenerConfig
 from synapse.federation.transport.server import TransportLayerServer
 from synapse.http.server import JsonResource, OptionsResource
 from synapse.http.servlet import RestServlet, parse_json_object_from_request
-from synapse.http.site import SynapseRequest, SynapseSite
+from synapse.http.site import SynapseSite
 from synapse.logging.context import LoggingContext
 from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
@@ -118,7 +119,6 @@ from synapse.storage.databases.main.stats import StatsStore
 from synapse.storage.databases.main.transactions import TransactionWorkerStore
 from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
 from synapse.storage.databases.main.user_directory import UserDirectoryStore
-from synapse.types import JsonDict
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.versionstring import get_version_string
 
@@ -143,9 +143,7 @@ class KeyUploadServlet(RestServlet):
         self.http_client = hs.get_simple_http_client()
         self.main_uri = hs.config.worker.worker_main_http_uri
 
-    async def on_POST(
-        self, request: SynapseRequest, device_id: Optional[str]
-    ) -> Tuple[int, JsonDict]:
+    async def on_POST(self, request: Request, device_id: Optional[str]):
         requester = await self.auth.get_user_by_req(request, allow_guest=True)
         user_id = requester.user.to_string()
         body = parse_json_object_from_request(request)
@@ -189,8 +187,9 @@ class KeyUploadServlet(RestServlet):
                 # If the header exists, add to the comma-separated list of the first
                 # instance of the header. Otherwise, generate a new header.
                 if x_forwarded_for:
-                    x_forwarded_for = [x_forwarded_for[0] + b", " + previous_host]
-                    x_forwarded_for.extend(x_forwarded_for[1:])
+                    x_forwarded_for = [
+                        x_forwarded_for[0] + b", " + previous_host
+                    ] + x_forwarded_for[1:]
                 else:
                     x_forwarded_for = [previous_host]
             headers[b"X-Forwarded-For"] = x_forwarded_for
@@ -254,16 +253,13 @@ class GenericWorkerSlavedStore(
     SessionStore,
     BaseSlavedStore,
 ):
-    # Properties that multiple storage classes define. Tell mypy what the
-    # expected type is.
-    server_name: str
-    config: HomeServerConfig
+    pass
 
 
 class GenericWorkerServer(HomeServer):
-    DATASTORE_CLASS = GenericWorkerSlavedStore  # type: ignore
+    DATASTORE_CLASS = GenericWorkerSlavedStore
 
-    def _listen_http(self, listener_config: ListenerConfig) -> None:
+    def _listen_http(self, listener_config: ListenerConfig):
         port = listener_config.port
         bind_addresses = listener_config.bind_addresses
 
@@ -271,10 +267,10 @@ class GenericWorkerServer(HomeServer):
 
         site_tag = listener_config.http_options.tag
         if site_tag is None:
-            site_tag = str(port)
+            site_tag = port
 
         # We always include a health resource.
-        resources: Dict[str, Resource] = {"/health": HealthResource()}
+        resources: Dict[str, IResource] = {"/health": HealthResource()}
 
         for res in listener_config.http_options.resources:
             for name in res.names:
@@ -390,7 +386,7 @@ class GenericWorkerServer(HomeServer):
 
         logger.info("Synapse worker now listening on port %d", port)
 
-    def start_listening(self) -> None:
+    def start_listening(self):
         for listener in self.config.worker.worker_listeners:
             if listener.type == "http":
                 self._listen_http(listener)
@@ -415,7 +411,7 @@ class GenericWorkerServer(HomeServer):
         self.get_tcp_replication().start_replication(self)
 
 
-def start(config_options: List[str]) -> None:
+def start(config_options):
     try:
         config = HomeServerConfig.load_config("Synapse worker", config_options)
     except ConfigError as e:

synapse/app/homeserver.py

@@ -16,10 +16,10 @@
 import logging
 import os
 import sys
-from typing import Dict, Iterable, Iterator, List
+from typing import Iterator
 
-from twisted.internet.tcp import Port
-from twisted.web.resource import EncodingResourceWrapper, Resource
+from twisted.internet import reactor
+from twisted.web.resource import EncodingResourceWrapper, IResource
 from twisted.web.server import GzipEncoderFactory
 from twisted.web.static import File
 
@@ -76,27 +76,23 @@ from synapse.util.versionstring import get_version_string
 logger = logging.getLogger("synapse.app.homeserver")
 
 
-def gz_wrap(r: Resource) -> Resource:
+def gz_wrap(r):
     return EncodingResourceWrapper(r, [GzipEncoderFactory()])
 
 
 class SynapseHomeServer(HomeServer):
-    DATASTORE_CLASS = DataStore  # type: ignore
+    DATASTORE_CLASS = DataStore
 
-    def _listener_http(
-        self, config: HomeServerConfig, listener_config: ListenerConfig
-    ) -> Iterable[Port]:
+    def _listener_http(self, config: HomeServerConfig, listener_config: ListenerConfig):
         port = listener_config.port
         bind_addresses = listener_config.bind_addresses
         tls = listener_config.tls
-        # Must exist since this is an HTTP listener.
-        assert listener_config.http_options is not None
         site_tag = listener_config.http_options.tag
         if site_tag is None:
             site_tag = str(port)
 
         # We always include a health resource.
-        resources: Dict[str, Resource] = {"/health": HealthResource()}
+        resources = {"/health": HealthResource()}
 
         for res in listener_config.http_options.resources:
             for name in res.names:
@@ -115,7 +111,7 @@ class SynapseHomeServer(HomeServer):
                 ("listeners", site_tag, "additional_resources", "<%s>" % (path,)),
             )
             handler = handler_cls(config, module_api)
-            if isinstance(handler, Resource):
+            if IResource.providedBy(handler):
                 resource = handler
             elif hasattr(handler, "handle_request"):
                 resource = AdditionalResource(self, handler.handle_request)
@@ -132,7 +128,7 @@ class SynapseHomeServer(HomeServer):
 
         # try to find something useful to redirect '/' to
         if WEB_CLIENT_PREFIX in resources:
-            root_resource: Resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
+            root_resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
         elif STATIC_PREFIX in resources:
             root_resource = RootOptionsRedirectResource(STATIC_PREFIX)
         else:
@@ -149,8 +145,6 @@ class SynapseHomeServer(HomeServer):
         )
 
         if tls:
-            # refresh_certificate should have been called before this.
-            assert self.tls_server_context_factory is not None
             ports = listen_ssl(
                 bind_addresses,
                 port,
@@ -171,21 +165,20 @@ class SynapseHomeServer(HomeServer):
 
         return ports
 
-    def _configure_named_resource(
-        self, name: str, compress: bool = False
-    ) -> Dict[str, Resource]:
+    def _configure_named_resource(self, name, compress=False):
         """Build a resource map for a named resource
 
         Args:
-            name: named resource: one of "client", "federation", etc
-            compress: whether to enable gzip compression for this resource
+            name (str): named resource: one of "client", "federation", etc
+            compress (bool): whether to enable gzip compression for this
+                resource
 
         Returns:
-            map from path to HTTP resource
+            dict[str, Resource]: map from path to HTTP resource
         """
-        resources: Dict[str, Resource] = {}
+        resources = {}
         if name == "client":
-            client_resource: Resource = ClientRestResource(self)
+            client_resource = ClientRestResource(self)
             if compress:
                 client_resource = gz_wrap(client_resource)
 
@@ -214,7 +207,7 @@ class SynapseHomeServer(HomeServer):
         if name == "consent":
             from synapse.rest.consent.consent_resource import ConsentResource
 
-            consent_resource: Resource = ConsentResource(self)
+            consent_resource = ConsentResource(self)
             if compress:
                 consent_resource = gz_wrap(consent_resource)
             resources.update({"/_matrix/consent": consent_resource})
@@ -284,7 +277,7 @@ class SynapseHomeServer(HomeServer):
 
         return resources
 
-    def start_listening(self) -> None:
+    def start_listening(self):
         if self.config.redis.redis_enabled:
             # If redis is enabled we connect via the replication command handler
             # in the same way as the workers (since we're effectively a client
@@ -310,9 +303,7 @@ class SynapseHomeServer(HomeServer):
                     ReplicationStreamProtocolFactory(self),
                 )
                 for s in services:
-                    self.get_reactor().addSystemEventTrigger(
-                        "before", "shutdown", s.stopListening
-                    )
+                    reactor.addSystemEventTrigger("before", "shutdown", s.stopListening)
             elif listener.type == "metrics":
                 if not self.config.metrics.enable_metrics:
                     logger.warning(
@@ -327,13 +318,14 @@ class SynapseHomeServer(HomeServer):
                 logger.warning("Unrecognized listener type: %s", listener.type)
 
 
-def setup(config_options: List[str]) -> SynapseHomeServer:
+def setup(config_options):
     """
     Args:
-        config_options_options: The options passed to Synapse. Usually `sys.argv[1:]`.
+        config_options_options: The options passed to Synapse. Usually
+            `sys.argv[1:]`.
 
     Returns:
-        A homeserver instance.
+        HomeServer
     """
     try:
         config = HomeServerConfig.load_or_generate_config(
@@ -372,7 +364,7 @@ def setup(config_options: List[str]) -> SynapseHomeServer:
     except Exception as e:
         handle_startup_exception(e)
 
-    async def start() -> None:
+    async def start():
         # Load the OIDC provider metadatas, if OIDC is enabled.
         if hs.config.oidc.oidc_enabled:
             oidc = hs.get_oidc_handler()
@@ -412,15 +404,39 @@ def format_config_error(e: ConfigError) -> Iterator[str]:
 
     yield ":\n  %s" % (e.msg,)
 
-    parent_e = e.__cause__
+    e = e.__cause__
     indent = 1
-    while parent_e:
+    while e:
         indent += 1
-        yield ":\n%s%s" % ("  " * indent, str(parent_e))
-        parent_e = parent_e.__cause__
+        yield ":\n%s%s" % ("  " * indent, str(e))
+        e = e.__cause__
 
 
-def run(hs: HomeServer) -> None:
+def run(hs: HomeServer):
+    PROFILE_SYNAPSE = False
+    if PROFILE_SYNAPSE:
+
+        def profile(func):
+            from cProfile import Profile
+            from threading import current_thread
+
+            def profiled(*args, **kargs):
+                profile = Profile()
+                profile.enable()
+                func(*args, **kargs)
+                profile.disable()
+                ident = current_thread().ident
+                profile.dump_stats(
+                    "/tmp/%s.%s.%i.pstat" % (hs.hostname, func.__name__, ident)
+                )
+
+            return profiled
+
+        from twisted.python.threadpool import ThreadPool
+
+        ThreadPool._worker = profile(ThreadPool._worker)
+        reactor.run = profile(reactor.run)
+
     _base.start_reactor(
         "synapse-homeserver",
         soft_file_limit=hs.config.server.soft_file_limit,
@@ -432,7 +448,7 @@ def run(hs: HomeServer) -> None:
     )
 
 
-def main() -> None:
+def main():
     with LoggingContext("main"):
         # check base requirements
         check_requirements()

synapse/app/phone_stats_home.py

@@ -15,12 +15,11 @@ import logging
 import math
 import resource
 import sys
-from typing import TYPE_CHECKING, List, Sized, Tuple
+from typing import TYPE_CHECKING
 
 from prometheus_client import Gauge
 
 from synapse.metrics.background_process_metrics import wrap_as_background_process
-from synapse.types import JsonDict
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -29,7 +28,7 @@ logger = logging.getLogger("synapse.app.homeserver")
 
 # Contains the list of processes we will be monitoring
 # currently either 0 or 1
-_stats_process: List[Tuple[int, "resource.struct_rusage"]] = []
+_stats_process = []
 
 # Gauges to expose monthly active user control metrics
 current_mau_gauge = Gauge("synapse_admin_mau:current", "Current MAU")
@@ -46,15 +45,9 @@ registered_reserved_users_mau_gauge = Gauge(
 
 
 @wrap_as_background_process("phone_stats_home")
-async def phone_stats_home(
-    hs: "HomeServer",
-    stats: JsonDict,
-    stats_process: List[Tuple[int, "resource.struct_rusage"]] = _stats_process,
-) -> None:
+async def phone_stats_home(hs: "HomeServer", stats, stats_process=_stats_process):
     logger.info("Gathering stats for reporting")
     now = int(hs.get_clock().time())
-    # Ensure the homeserver has started.
-    assert hs.start_time is not None
     uptime = int(now - hs.start_time)
     if uptime < 0:
         uptime = 0
@@ -153,15 +146,15 @@ async def phone_stats_home(
         logger.warning("Error reporting stats: %s", e)
 
 
-def start_phone_stats_home(hs: "HomeServer") -> None:
+def start_phone_stats_home(hs: "HomeServer"):
     """
     Start the background tasks which report phone home stats.
     """
     clock = hs.get_clock()
 
-    stats: JsonDict = {}
+    stats = {}
 
-    def performance_stats_init() -> None:
+    def performance_stats_init():
         _stats_process.clear()
         _stats_process.append(
             (int(hs.get_clock().time()), resource.getrusage(resource.RUSAGE_SELF))
@@ -177,10 +170,10 @@ def start_phone_stats_home(hs: "HomeServer") -> None:
     hs.get_datastore().reap_monthly_active_users()
 
     @wrap_as_background_process("generate_monthly_active_users")
-    async def generate_monthly_active_users() -> None:
+    async def generate_monthly_active_users():
         current_mau_count = 0
         current_mau_count_by_service = {}
-        reserved_users: Sized = ()
+        reserved_users = ()
         store = hs.get_datastore()
         if hs.config.server.limit_usage_by_mau or hs.config.server.mau_stats_only:
             current_mau_count = await store.get_monthly_active_count()

synapse/federation/federation_client.py

@@ -277,58 +277,6 @@ class FederationClient(FederationBase):
 
         return pdus
 
-    async def get_pdu_from_destination_raw(
-        self,
-        destination: str,
-        event_id: str,
-        room_version: RoomVersion,
-        outlier: bool = False,
-        timeout: Optional[int] = None,
-    ) -> Optional[EventBase]:
-        """Requests the PDU with given origin and ID from the remote home
-        server. Does not have any caching or rate limiting!
-
-        Args:
-            destination: Which homeserver to query
-            event_id: event to fetch
-            room_version: version of the room
-            outlier: Indicates whether the PDU is an `outlier`, i.e. if
-                it's from an arbitrary point in the context as opposed to part
-                of the current block of PDUs. Defaults to `False`
-            timeout: How long to try (in ms) each destination for before
-                moving to the next destination. None indicates no timeout.
-
-        Returns:
-            The requested PDU, or None if we were unable to find it.
-
-        Raises:
-            SynapseError, NotRetryingDestination, FederationDeniedError
-        """
-        transaction_data = await self.transport_layer.get_event(
-            destination, event_id, timeout=timeout
-        )
-
-        logger.debug(
-            "retrieved event id %s from %s: %r",
-            event_id,
-            destination,
-            transaction_data,
-        )
-
-        pdu_list: List[EventBase] = [
-            event_from_pdu_json(p, room_version, outlier=outlier)
-            for p in transaction_data["pdus"]
-        ]
-
-        if pdu_list and pdu_list[0]:
-            pdu = pdu_list[0]
-
-            # Check signatures are correct.
-            signed_pdu = await self._check_sigs_and_hash(room_version, pdu)
-            return signed_pdu
-
-        return None
-
     async def get_pdu(
         self,
         destinations: Iterable[str],
@@ -373,14 +321,30 @@ class FederationClient(FederationBase):
                 continue
 
             try:
-                signed_pdu = await self.get_pdu_from_destination_raw(
-                    destination=destination,
-                    event_id=event_id,
-                    room_version=room_version,
-                    outlier=outlier,
-                    timeout=timeout,
+                transaction_data = await self.transport_layer.get_event(
+                    destination, event_id, timeout=timeout
                 )
 
+                logger.debug(
+                    "retrieved event id %s from %s: %r",
+                    event_id,
+                    destination,
+                    transaction_data,
+                )
+
+                pdu_list: List[EventBase] = [
+                    event_from_pdu_json(p, room_version, outlier=outlier)
+                    for p in transaction_data["pdus"]
+                ]
+
+                if pdu_list and pdu_list[0]:
+                    pdu = pdu_list[0]
+
+                    # Check signatures are correct.
+                    signed_pdu = await self._check_sigs_and_hash(room_version, pdu)
+
+                    break
+
                 pdu_attempts[destination] = now
 
             except SynapseError as e:

synapse/handlers/admin.py

@@ -234,7 +234,7 @@ class ExfiltrationWriter(metaclass=abc.ABCMeta):
 
     @abc.abstractmethod
     def write_invite(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
+        self, room_id: str, event: EventBase, state: StateMap[dict]
     ) -> None:
         """Write an invite for the room, with associated invite state.
 
@@ -248,7 +248,7 @@ class ExfiltrationWriter(metaclass=abc.ABCMeta):
 
     @abc.abstractmethod
     def write_knock(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
+        self, room_id: str, event: EventBase, state: StateMap[dict]
     ) -> None:
         """Write a knock for the room, with associated knock state.
 

synapse/handlers/appservice.py

@@ -188,7 +188,7 @@ class ApplicationServicesHandler:
         self,
         stream_key: str,
         new_token: Union[int, RoomStreamToken],
-        users: Collection[Union[str, UserID]],
+        users: Optional[Collection[Union[str, UserID]]] = None,
     ) -> None:
         """
         This is called by the notifier in the background when an ephemeral event is handled
@@ -203,9 +203,7 @@ class ApplicationServicesHandler:
                 value for `stream_key` will cause this function to return early.
 
                 Ephemeral events will only be pushed to appservices that have opted into
-                receiving them by setting `push_ephemeral` to true in their registration
-                file. Note that while MSC2409 is experimental, this option is called
-                `de.sorunome.msc2409.push_ephemeral`.
+                them.
 
                 Appservices will only receive ephemeral events that fall within their
                 registered user and room namespaces.
@@ -216,7 +214,6 @@ class ApplicationServicesHandler:
         if not self.notify_appservices:
             return
 
-        # Ignore any unsupported streams
         if stream_key not in ("typing_key", "receipt_key", "presence_key"):
             return
 
@@ -233,25 +230,18 @@ class ApplicationServicesHandler:
         # Additional context: https://github.com/matrix-org/synapse/pull/11137
         assert isinstance(new_token, int)
 
-        # Check whether there are any appservices which have registered to receive
-        # ephemeral events.
-        #
-        # Note that whether these events are actually relevant to these appservices
-        # is decided later on.
         services = [
             service
             for service in self.store.get_app_services()
             if service.supports_ephemeral
         ]
         if not services:
-            # Bail out early if none of the target appservices have explicitly registered
-            # to receive these ephemeral events.
             return
 
         # We only start a new background process if necessary rather than
         # optimistically (to cut down on overhead).
         self._notify_interested_services_ephemeral(
-            services, stream_key, new_token, users
+            services, stream_key, new_token, users or []
         )
 
     @wrap_as_background_process("notify_interested_services_ephemeral")
@@ -262,7 +252,7 @@ class ApplicationServicesHandler:
         new_token: int,
         users: Collection[Union[str, UserID]],
     ) -> None:
-        logger.debug("Checking interested services for %s", stream_key)
+        logger.debug("Checking interested services for %s" % (stream_key))
         with Measure(self.clock, "notify_interested_services_ephemeral"):
             for service in services:
                 if stream_key == "typing_key":
@@ -355,9 +345,6 @@ class ApplicationServicesHandler:
 
         Args:
             service: The application service to check for which events it should receive.
-            new_token: A receipts event stream token. Purely used to double-check that the
-                from_token we pull from the database isn't greater than or equal to this
-                token. Prevents accidentally duplicating work.
 
         Returns:
             A list of JSON dictionaries containing data derived from the read receipts that
@@ -395,9 +382,6 @@ class ApplicationServicesHandler:
         Args:
             service: The application service that ephemeral events are being sent to.
             users: The users that should receive the presence update.
-            new_token: A presence update stream token. Purely used to double-check that the
-                from_token we pull from the database isn't greater than or equal to this
-                token. Prevents accidentally duplicating work.
 
         Returns:
             A list of json dictionaries containing data derived from the presence events

synapse/handlers/devicemessage.py

@@ -89,13 +89,6 @@ class DeviceMessageHandler:
         )
 
     async def on_direct_to_device_edu(self, origin: str, content: JsonDict) -> None:
-        """
-        Handle receiving to-device messages from remote homeservers.
-
-        Args:
-            origin: The remote homeserver.
-            content: The JSON dictionary containing the to-device messages.
-        """
         local_messages = {}
         sender_user_id = content["sender"]
         if origin != get_domain_from_id(sender_user_id):
@@ -142,16 +135,12 @@ class DeviceMessageHandler:
                 message_type, sender_user_id, by_device
             )
 
-        # Add messages to the database.
-        # Retrieve the stream id of the last-processed to-device message.
-        last_stream_id = await self.store.add_messages_from_remote_to_device_inbox(
+        stream_id = await self.store.add_messages_from_remote_to_device_inbox(
             origin, message_id, local_messages
         )
 
-        # Notify listeners that there are new to-device messages to process,
-        # handing them the latest stream id.
         self.notifier.on_new_event(
-            "to_device_key", last_stream_id, users=local_messages.keys()
+            "to_device_key", stream_id, users=local_messages.keys()
         )
 
     async def _check_for_unknown_devices(
@@ -206,14 +195,6 @@ class DeviceMessageHandler:
         message_type: str,
         messages: Dict[str, Dict[str, JsonDict]],
     ) -> None:
-        """
-        Handle a request from a user to send to-device message(s).
-
-        Args:
-            requester: The user that is sending the to-device messages.
-            message_type: The type of to-device messages that are being sent.
-            messages: A dictionary containing recipients mapped to messages intended for them.
-        """
         sender_user_id = requester.user.to_string()
 
         message_id = random_string(16)
@@ -276,16 +257,12 @@ class DeviceMessageHandler:
                 "org.matrix.opentracing_context": json_encoder.encode(context),
             }
 
-        # Add messages to the database.
-        # Retrieve the stream id of the last-processed to-device message.
-        last_stream_id = await self.store.add_messages_to_device_inbox(
+        stream_id = await self.store.add_messages_to_device_inbox(
             local_messages, remote_edu_contents
         )
 
-        # Notify listeners that there are new to-device messages to process,
-        # handing them the latest stream id.
         self.notifier.on_new_event(
-            "to_device_key", last_stream_id, users=local_messages.keys()
+            "to_device_key", stream_id, users=local_messages.keys()
         )
 
         if self.federation_sender:

synapse/handlers/e2e_keys.py

@@ -201,19 +201,95 @@ class E2eKeysHandler:
                     r[user_id] = remote_queries[user_id]
 
             # Now fetch any devices that we don't have in our cache
+            @trace
+            async def do_remote_query(destination: str) -> None:
+                """This is called when we are querying the device list of a user on
+                a remote homeserver and their device list is not in the device list
+                cache. If we share a room with this user and we're not querying for
+                specific user we will update the cache with their device list.
+                """
+
+                destination_query = remote_queries_not_in_cache[destination]
+
+                # We first consider whether we wish to update the device list cache with
+                # the users device list. We want to track a user's devices when the
+                # authenticated user shares a room with the queried user and the query
+                # has not specified a particular device.
+                # If we update the cache for the queried user we remove them from further
+                # queries. We use the more efficient batched query_client_keys for all
+                # remaining users
+                user_ids_updated = []
+                for (user_id, device_list) in destination_query.items():
+                    if user_id in user_ids_updated:
+                        continue
+
+                    if device_list:
+                        continue
+
+                    room_ids = await self.store.get_rooms_for_user(user_id)
+                    if not room_ids:
+                        continue
+
+                    # We've decided we're sharing a room with this user and should
+                    # probably be tracking their device lists. However, we haven't
+                    # done an initial sync on the device list so we do it now.
+                    try:
+                        if self._is_master:
+                            user_devices = await self.device_handler.device_list_updater.user_device_resync(
+                                user_id
+                            )
+                        else:
+                            user_devices = await self._user_device_resync_client(
+                                user_id=user_id
+                            )
+
+                        user_devices = user_devices["devices"]
+                        user_results = results.setdefault(user_id, {})
+                        for device in user_devices:
+                            user_results[device["device_id"]] = device["keys"]
+                        user_ids_updated.append(user_id)
+                    except Exception as e:
+                        failures[destination] = _exception_to_failure(e)
+
+                if len(destination_query) == len(user_ids_updated):
+                    # We've updated all the users in the query and we do not need to
+                    # make any further remote calls.
+                    return
+
+                # Remove all the users from the query which we have updated
+                for user_id in user_ids_updated:
+                    destination_query.pop(user_id)
+
+                try:
+                    remote_result = await self.federation.query_client_keys(
+                        destination, {"device_keys": destination_query}, timeout=timeout
+                    )
+
+                    for user_id, keys in remote_result["device_keys"].items():
+                        if user_id in destination_query:
+                            results[user_id] = keys
+
+                    if "master_keys" in remote_result:
+                        for user_id, key in remote_result["master_keys"].items():
+                            if user_id in destination_query:
+                                cross_signing_keys["master_keys"][user_id] = key
+
+                    if "self_signing_keys" in remote_result:
+                        for user_id, key in remote_result["self_signing_keys"].items():
+                            if user_id in destination_query:
+                                cross_signing_keys["self_signing_keys"][user_id] = key
+
+                except Exception as e:
+                    failure = _exception_to_failure(e)
+                    failures[destination] = failure
+                    set_tag("error", True)
+                    set_tag("reason", failure)
+
             await make_deferred_yieldable(
                 defer.gatherResults(
                     [
-                        run_in_background(
-                            self._query_devices_for_destination,
-                            results,
-                            cross_signing_keys,
-                            failures,
-                            destination,
-                            queries,
-                            timeout,
-                        )
-                        for destination, queries in remote_queries_not_in_cache.items()
+                        run_in_background(do_remote_query, destination)
+                        for destination in remote_queries_not_in_cache
                     ],
                     consumeErrors=True,
                 ).addErrback(unwrapFirstError)
@@ -225,121 +301,6 @@ class E2eKeysHandler:
 
             return ret
 
-    @trace
-    async def _query_devices_for_destination(
-        self,
-        results: JsonDict,
-        cross_signing_keys: JsonDict,
-        failures: Dict[str, JsonDict],
-        destination: str,
-        destination_query: Dict[str, Iterable[str]],
-        timeout: int,
-    ) -> None:
-        """This is called when we are querying the device list of a user on
-        a remote homeserver and their device list is not in the device list
-        cache. If we share a room with this user and we're not querying for
-        specific user we will update the cache with their device list.
-
-        Args:
-            results: A map from user ID to their device keys, which gets
-                updated with the newly fetched keys.
-            cross_signing_keys: Map from user ID to their cross signing keys,
-                which gets updated with the newly fetched keys.
-            failures: Map of destinations to failures that have occurred while
-                attempting to fetch keys.
-            destination: The remote server to query
-            destination_query: The query dict of devices to query the remote
-                server for.
-            timeout: The timeout for remote HTTP requests.
-        """
-
-        # We first consider whether we wish to update the device list cache with
-        # the users device list. We want to track a user's devices when the
-        # authenticated user shares a room with the queried user and the query
-        # has not specified a particular device.
-        # If we update the cache for the queried user we remove them from further
-        # queries. We use the more efficient batched query_client_keys for all
-        # remaining users
-        user_ids_updated = []
-        for (user_id, device_list) in destination_query.items():
-            if user_id in user_ids_updated:
-                continue
-
-            if device_list:
-                continue
-
-            room_ids = await self.store.get_rooms_for_user(user_id)
-            if not room_ids:
-                continue
-
-            # We've decided we're sharing a room with this user and should
-            # probably be tracking their device lists. However, we haven't
-            # done an initial sync on the device list so we do it now.
-            try:
-                if self._is_master:
-                    resync_results = await self.device_handler.device_list_updater.user_device_resync(
-                        user_id
-                    )
-                else:
-                    resync_results = await self._user_device_resync_client(
-                        user_id=user_id
-                    )
-
-                # Add the device keys to the results.
-                user_devices = resync_results["devices"]
-                user_results = results.setdefault(user_id, {})
-                for device in user_devices:
-                    user_results[device["device_id"]] = device["keys"]
-                user_ids_updated.append(user_id)
-
-                # Add any cross signing keys to the results.
-                master_key = resync_results.get("master_key")
-                self_signing_key = resync_results.get("self_signing_key")
-
-                if master_key:
-                    cross_signing_keys["master_keys"][user_id] = master_key
-
-                if self_signing_key:
-                    cross_signing_keys["self_signing_keys"][user_id] = self_signing_key
-            except Exception as e:
-                failures[destination] = _exception_to_failure(e)
-
-        if len(destination_query) == len(user_ids_updated):
-            # We've updated all the users in the query and we do not need to
-            # make any further remote calls.
-            return
-
-        # Remove all the users from the query which we have updated
-        for user_id in user_ids_updated:
-            destination_query.pop(user_id)
-
-        try:
-            remote_result = await self.federation.query_client_keys(
-                destination, {"device_keys": destination_query}, timeout=timeout
-            )
-
-            for user_id, keys in remote_result["device_keys"].items():
-                if user_id in destination_query:
-                    results[user_id] = keys
-
-            if "master_keys" in remote_result:
-                for user_id, key in remote_result["master_keys"].items():
-                    if user_id in destination_query:
-                        cross_signing_keys["master_keys"][user_id] = key
-
-            if "self_signing_keys" in remote_result:
-                for user_id, key in remote_result["self_signing_keys"].items():
-                    if user_id in destination_query:
-                        cross_signing_keys["self_signing_keys"][user_id] = key
-
-        except Exception as e:
-            failure = _exception_to_failure(e)
-            failures[destination] = failure
-            set_tag("error", True)
-            set_tag("reason", failure)
-
-        return
-
     async def get_cross_signing_keys_from_cache(
         self, query: Iterable[str], from_user_id: Optional[str]
     ) -> Dict[str, Dict[str, dict]]:

synapse/handlers/federation_event.py

@@ -981,6 +981,8 @@ class FederationEventHandler:
                 origin,
                 event,
                 context,
+                state=state,
+                backfilled=backfilled,
             )
         except AuthError as e:
             # FIXME richvdh 2021/10/07 I don't think this is reachable. Let's log it
@@ -1330,6 +1332,8 @@ class FederationEventHandler:
         origin: str,
         event: EventBase,
         context: EventContext,
+        state: Optional[Iterable[EventBase]] = None,
+        backfilled: bool = False,
     ) -> EventContext:
         """
         Checks whether an event should be rejected (for failing auth checks).
@@ -1340,6 +1344,12 @@ class FederationEventHandler:
             context:
                 The event context.
 
+            state:
+                The state events used to check the event for soft-fail. If this is
+                not provided the current state events will be used.
+
+            backfilled: True if the event was backfilled.
+
         Returns:
             The updated context object.
 

synapse/handlers/pagination.py

@@ -424,7 +424,7 @@ class PaginationHandler:
 
         if events:
             if event_filter:
-                events = await event_filter.filter(events)
+                events = event_filter.filter(events)
 
             events = await filter_events_for_client(
                 self.storage, user_id, events, is_peeking=(member_event_id is None)

synapse/handlers/room.py

@@ -12,7 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Contains functions for performing actions on rooms."""
+"""Contains functions for performing events on rooms."""
+
 import itertools
 import logging
 import math
@@ -30,8 +31,6 @@ from typing import (
     Tuple,
 )
 
-from typing_extensions import TypedDict
-
 from synapse.api.constants import (
     EventContentFields,
     EventTypes,
@@ -1159,10 +1158,8 @@ class RoomContextHandler:
         )
 
         if event_filter:
-            results["events_before"] = await event_filter.filter(
-                results["events_before"]
-            )
-            results["events_after"] = await event_filter.filter(results["events_after"])
+            results["events_before"] = event_filter.filter(results["events_before"])
+            results["events_after"] = event_filter.filter(results["events_after"])
 
         results["events_before"] = await filter_evts(results["events_before"])
         results["events_after"] = await filter_evts(results["events_after"])
@@ -1198,7 +1195,7 @@ class RoomContextHandler:
 
         state_events = list(state[last_event_id].values())
         if event_filter:
-            state_events = await event_filter.filter(state_events)
+            state_events = event_filter.filter(state_events)
 
         results["state"] = await filter_evts(state_events)
 
@@ -1278,13 +1275,6 @@ class RoomEventSource(EventSource[RoomStreamToken, EventBase]):
         return self.store.get_room_events_max_id(room_id)
 
 
-class ShutdownRoomResponse(TypedDict):
-    kicked_users: List[str]
-    failed_to_kick_users: List[str]
-    local_aliases: List[str]
-    new_room_id: Optional[str]
-
-
 class RoomShutdownHandler:
 
     DEFAULT_MESSAGE = (
@@ -1310,7 +1300,7 @@ class RoomShutdownHandler:
         new_room_name: Optional[str] = None,
         message: Optional[str] = None,
         block: bool = False,
-    ) -> ShutdownRoomResponse:
+    ) -> dict:
         """
         Shuts down a room. Moves all local users and room aliases automatically
         to a new room if `new_room_user_id` is set. Otherwise local users only
@@ -1344,13 +1334,8 @@ class RoomShutdownHandler:
                 Defaults to `Sharing illegal content on this server is not
                 permitted and rooms in violation will be blocked.`
             block:
-                If set to `True`, users will be prevented from joining the old
-                room. This option can also be used to pre-emptively block a room,
-                even if it's unknown to this homeserver. In this case, the room
-                will be blocked, and no further action will be taken. If `False`,
-                attempting to delete an unknown room is invalid.
-
-                Defaults to `False`.
+                If set to `true`, this room will be added to a blocking list,
+                preventing future attempts to join the room. Defaults to `false`.
 
         Returns: a dict containing the following keys:
             kicked_users: An array of users (`user_id`) that were kicked.
@@ -1359,9 +1344,7 @@ class RoomShutdownHandler:
             local_aliases:
                 An array of strings representing the local aliases that were
                 migrated from the old room to the new.
-            new_room_id:
-                A string representing the room ID of the new room, or None if
-                no such room was created.
+            new_room_id: A string representing the room ID of the new room.
         """
 
         if not new_room_name:
@@ -1372,27 +1355,13 @@ class RoomShutdownHandler:
         if not RoomID.is_valid(room_id):
             raise SynapseError(400, "%s is not a legal room ID" % (room_id,))
 
-        # Action the block first (even if the room doesn't exist yet)
-        if block:
-            # This will work even if the room is already blocked, but that is
-            # desirable in case the first attempt at blocking the room failed below.
-            await self.store.block_room(room_id, requester_user_id)
-
         if not await self.store.get_room(room_id):
-            if block:
-                # We allow you to block an unknown room.
-                return {
-                    "kicked_users": [],
-                    "failed_to_kick_users": [],
-                    "local_aliases": [],
-                    "new_room_id": None,
-                }
-            else:
-                # But if you don't want to preventatively block another room,
-                # this function can't do anything useful.
-                raise NotFoundError(
-                    "Cannot shut down room: unknown room id %s" % (room_id,)
-                )
+            raise NotFoundError("Unknown room id %s" % (room_id,))
+
+        # This will work even if the room is already blocked, but that is
+        # desirable in case the first attempt at blocking the room failed below.
+        if block:
+            await self.store.block_room(room_id, requester_user_id)
 
         if new_room_user_id is not None:
             if not self.hs.is_mine_id(new_room_user_id):

synapse/handlers/search.py

@@ -180,7 +180,7 @@ class SearchHandler:
                 % (set(group_keys) - {"room_id", "sender"},),
             )
 
-        search_filter = Filter(self.hs, filter_dict)
+        search_filter = Filter(filter_dict)
 
         # TODO: Search through left rooms too
         rooms = await self.store.get_rooms_for_local_user_where_membership_is(
@@ -242,7 +242,7 @@ class SearchHandler:
 
             rank_map.update({r["event"].event_id: r["rank"] for r in results})
 
-            filtered_events = await search_filter.filter([r["event"] for r in results])
+            filtered_events = search_filter.filter([r["event"] for r in results])
 
             events = await filter_events_for_client(
                 self.storage, user.to_string(), filtered_events
@@ -292,9 +292,7 @@ class SearchHandler:
 
                 rank_map.update({r["event"].event_id: r["rank"] for r in results})
 
-                filtered_events = await search_filter.filter(
-                    [r["event"] for r in results]
-                )
+                filtered_events = search_filter.filter([r["event"] for r in results])
 
                 events = await filter_events_for_client(
                     self.storage, user.to_string(), filtered_events

synapse/handlers/sync.py

@@ -510,7 +510,7 @@ class SyncHandler:
             log_kv({"limited": limited})
 
             if potential_recents:
-                recents = await sync_config.filter_collection.filter_room_timeline(
+                recents = sync_config.filter_collection.filter_room_timeline(
                     potential_recents
                 )
                 log_kv({"recents_after_sync_filtering": len(recents)})
@@ -575,8 +575,8 @@ class SyncHandler:
 
                 log_kv({"loaded_recents": len(events)})
 
-                loaded_recents = (
-                    await sync_config.filter_collection.filter_room_timeline(events)
+                loaded_recents = sync_config.filter_collection.filter_room_timeline(
+                    events
                 )
 
                 log_kv({"loaded_recents_after_sync_filtering": len(loaded_recents)})
@@ -1015,7 +1015,7 @@ class SyncHandler:
 
         return {
             (e.type, e.state_key): e
-            for e in await sync_config.filter_collection.filter_room_state(
+            for e in sync_config.filter_collection.filter_room_state(
                 list(state.values())
             )
             if e.type != EventTypes.Aliases  # until MSC2261 or alternative solution
@@ -1383,7 +1383,7 @@ class SyncHandler:
                 sync_config.user
             )
 
-        account_data_for_user = await sync_config.filter_collection.filter_account_data(
+        account_data_for_user = sync_config.filter_collection.filter_account_data(
             [
                 {"type": account_data_type, "content": content}
                 for account_data_type, content in account_data.items()
@@ -1448,7 +1448,7 @@ class SyncHandler:
             # Deduplicate the presence entries so that there's at most one per user
             presence = list({p.user_id: p for p in presence}.values())
 
-        presence = await sync_config.filter_collection.filter_presence(presence)
+        presence = sync_config.filter_collection.filter_presence(presence)
 
         sync_result_builder.presence = presence
 
@@ -2021,14 +2021,12 @@ class SyncHandler:
                 )
 
             account_data_events = (
-                await sync_config.filter_collection.filter_room_account_data(
+                sync_config.filter_collection.filter_room_account_data(
                     account_data_events
                 )
             )
 
-            ephemeral = await sync_config.filter_collection.filter_room_ephemeral(
-                ephemeral
-            )
+            ephemeral = sync_config.filter_collection.filter_room_ephemeral(ephemeral)
 
             if not (
                 always_include