Add historical to commentdoc

Complement tests: https://github.com/matrix-org/complement/pull/221

CHANGES.md

@@ -1,97 +1,3 @@
-Synapse 1.47.0rc2 (2021-11-10)
-==============================
-
-This fixes an issue with publishing the Debian packages for 1.47.0rc1.
-It is otherwise identical to 1.47.0rc1.
-
-
-Synapse 1.47.0rc1 (2021-11-09)
-==============================
-
-Deprecations and Removals
--------------------------
-
-- The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information. ([\#11206](https://github.com/matrix-org/synapse/issues/11206))
-- Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). ([\#11213](https://github.com/matrix-org/synapse/issues/11213))
-
-
-Features
---------
-
-- Advertise support for Client-Server API r0.6.1. ([\#11097](https://github.com/matrix-org/synapse/issues/11097))
-- Add search by room ID and room alias to the List Room admin API. ([\#11099](https://github.com/matrix-org/synapse/issues/11099))
-- Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room. ([\#11126](https://github.com/matrix-org/synapse/issues/11126))
-- Add a module API method to update a user's membership in a room. ([\#11147](https://github.com/matrix-org/synapse/issues/11147))
-- Add metrics for thread pool usage. ([\#11178](https://github.com/matrix-org/synapse/issues/11178))
-- Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288). ([\#11187](https://github.com/matrix-org/synapse/issues/11187))
-- Add a module API method to retrieve the current state of a room. ([\#11204](https://github.com/matrix-org/synapse/issues/11204))
-- Calculate a default value for `public_baseurl` based on `server_name`. ([\#11210](https://github.com/matrix-org/synapse/issues/11210))
-- Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443. ([\#11211](https://github.com/matrix-org/synapse/issues/11211))
-- Add admin APIs to pause, start and check the status of background updates. ([\#11263](https://github.com/matrix-org/synapse/issues/11263))
-
-
-Bugfixes
---------
-
-- Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat. ([\#10097](https://github.com/matrix-org/synapse/issues/10097))
-- Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine. ([\#10969](https://github.com/matrix-org/synapse/issues/10969), [\#11212](https://github.com/matrix-org/synapse/issues/11212))
-- Do not accept events if a third-party rule `check_event_allowed` callback raises an exception. ([\#11033](https://github.com/matrix-org/synapse/issues/11033))
-- Fix long-standing bug where verification requests could fail in certain cases if a federation whitelist was in place but did not include your own homeserver. ([\#11129](https://github.com/matrix-org/synapse/issues/11129))
-- Allow an empty list of `state_events_at_start` to be sent when using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint and the author of the historical messages is already part of the current room state at the given `?prev_event_id`. ([\#11188](https://github.com/matrix-org/synapse/issues/11188))
-- Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p. ([\#11191](https://github.com/matrix-org/synapse/issues/11191))
-- Delete `to_device` messages for hidden devices that will never be read, reducing database size. ([\#11199](https://github.com/matrix-org/synapse/issues/11199))
-- Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error. ([\#11200](https://github.com/matrix-org/synapse/issues/11200))
-- Fix a long-standing bug which could result in serialization errors and potentially duplicate transaction data when sending ephemeral events to application services. Contributed by @Fizzadar at Beeper. ([\#11207](https://github.com/matrix-org/synapse/issues/11207))
-- Fix a bug introduced in Synapse 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats. ([\#11217](https://github.com/matrix-org/synapse/issues/11217))
-- Fix long-standing bug where cross signing keys were not included in the response to `/r0/keys/query` the first time a remote user was queried. ([\#11234](https://github.com/matrix-org/synapse/issues/11234))
-- Fix a long-standing bug where all requests that read events from the database could get stuck as a result of losing the database connection. ([\#11240](https://github.com/matrix-org/synapse/issues/11240))
-- Fix a bug preventing Synapse from being rolled back to an earlier version when using workers. ([\#11255](https://github.com/matrix-org/synapse/issues/11255), [\#11276](https://github.com/matrix-org/synapse/issues/11276))
-- Fix a bug introduced in Synapse 1.37.1 which caused a remote event being processed by a worker to not get processed on restart if the worker was killed. ([\#11262](https://github.com/matrix-org/synapse/issues/11262))
-- Only allow old Element/Riot Android clients to send read receipts without a request body. All other clients must include a request body as required by the specification. Contributed by @rogersheu. ([\#11157](https://github.com/matrix-org/synapse/issues/11157))
-
-
-Updates to the Docker image
----------------------------
-
-- Avoid changing user ID when started as a non-root user, and no explicit `UID` is set. ([\#11209](https://github.com/matrix-org/synapse/issues/11209))
-
-
-Improved Documentation
-----------------------
-
-- Improve example HAProxy config in the docs to properly handle HTTP `Host` headers with port information. This is required for federation over port 443 to work correctly. ([\#11128](https://github.com/matrix-org/synapse/issues/11128))
-- Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5. ([\#11151](https://github.com/matrix-org/synapse/issues/11151))
-- Clarify lack of support for Windows. ([\#11198](https://github.com/matrix-org/synapse/issues/11198))
-- Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper. ([\#11221](https://github.com/matrix-org/synapse/issues/11221))
-- Add documentation for using LemonLDAP as an OpenID Connect Identity Provider. Contributed by @l00ptr. ([\#11257](https://github.com/matrix-org/synapse/issues/11257))
-
-
-Internal Changes
-----------------
-
-- Add type annotations for the `log_function` decorator. ([\#10943](https://github.com/matrix-org/synapse/issues/10943))
-- Add type hints to `synapse.events`. ([\#11098](https://github.com/matrix-org/synapse/issues/11098))
-- Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code. ([\#11137](https://github.com/matrix-org/synapse/issues/11137))
-- Add type hints so that `synapse.http` passes `mypy` checks. ([\#11164](https://github.com/matrix-org/synapse/issues/11164))
-- Update scripts to pass Shellcheck lints. ([\#11166](https://github.com/matrix-org/synapse/issues/11166))
-- Add knock information in admin export. Contributed by Rafael Gonçalves. ([\#11171](https://github.com/matrix-org/synapse/issues/11171))
-- Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly. ([\#11179](https://github.com/matrix-org/synapse/issues/11179))
-- Refactor `Filter` to check different fields depending on the data type. ([\#11194](https://github.com/matrix-org/synapse/issues/11194))
-- Improve type hints for the relations datastore. ([\#11205](https://github.com/matrix-org/synapse/issues/11205))
-- Replace outdated links in the pull request checklist with links to the rendered documentation. ([\#11225](https://github.com/matrix-org/synapse/issues/11225))
-- Fix a bug in unit test `test_block_room_and_not_purge`. ([\#11226](https://github.com/matrix-org/synapse/issues/11226))
-- In `ObservableDeferred`, run observers in the order they were registered. ([\#11229](https://github.com/matrix-org/synapse/issues/11229))
-- Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`. ([\#11231](https://github.com/matrix-org/synapse/issues/11231))
-- Add `twine` and `towncrier` as dev dependencies, as they're used by the release script. ([\#11233](https://github.com/matrix-org/synapse/issues/11233))
-- Allow `stream_writers.typing` config to be a list of one worker. ([\#11237](https://github.com/matrix-org/synapse/issues/11237))
-- Remove debugging statement in tests. ([\#11239](https://github.com/matrix-org/synapse/issues/11239))
-- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers. ([\#11244](https://github.com/matrix-org/synapse/issues/11244))
-- Add an additional test for the `cachedList` method decorator. ([\#11246](https://github.com/matrix-org/synapse/issues/11246))
-- Make minor correction to the type of `auth_checkers` callbacks. ([\#11253](https://github.com/matrix-org/synapse/issues/11253))
-- Clean up trivial aspects of the Debian package build tooling. ([\#11269](https://github.com/matrix-org/synapse/issues/11269), [\#11273](https://github.com/matrix-org/synapse/issues/11273))
-- Blacklist new SyTest that checks that key uploads are valid pending the validation being implemented in Synapse. ([\#11270](https://github.com/matrix-org/synapse/issues/11270))
-
-
 Synapse 1.46.0 (2021-11-02)
 ===========================
 

changelog.d/10097.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat.

changelog.d/10943.misc

@@ -0,0 +1 @@
+Add type annotations for the `log_function` decorator.

changelog.d/10969.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.

changelog.d/11033.bugfix

@@ -0,0 +1 @@
+Do not accept events if a third-party rule module API callback raises an exception.

changelog.d/11097.feature

@@ -0,0 +1 @@
+Advertise support for Client-Server API r0.6.1.

changelog.d/11098.misc

@@ -0,0 +1 @@
+Add type hints to `synapse.events`.

changelog.d/11099.feature

@@ -0,0 +1 @@
+Add search by room ID and room alias to List Room admin API.

changelog.d/11126.feature

@@ -0,0 +1 @@
+Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room.

changelog.d/11128.doc

@@ -0,0 +1 @@
+Improve example HAProxy config in the docs to properly handle host headers with port information. This is required for federation over port 443 to work correctly.

changelog.d/11129.bugfix

@@ -0,0 +1 @@
+Fix long-standing bug where verification requests could fail in certain cases if whitelist was in place but did not include your own homeserver.

changelog.d/11137.misc

@@ -0,0 +1 @@
+Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code.

changelog.d/11147.feature

@@ -0,0 +1 @@
+Add a module API method to update a user's membership in a room.

changelog.d/11151.doc

@@ -0,0 +1 @@
+Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5.

changelog.d/11164.misc

@@ -0,0 +1 @@
+Add type hints so that `synapse.http` passes `mypy` checks.

changelog.d/11166.misc

@@ -0,0 +1 @@
+Update scripts to pass Shellcheck lints.

changelog.d/11171.misc

@@ -0,0 +1 @@
+Add knock information in admin export. Contributed by Rafael Gonçalves.

changelog.d/11178.feature

@@ -0,0 +1 @@
+Add metrics for thread pool usage.

changelog.d/11179.misc

@@ -0,0 +1 @@
+Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly.

changelog.d/11187.feature

@@ -0,0 +1 @@
+Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288).

changelog.d/11191.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p.

changelog.d/11194.misc

@@ -0,0 +1 @@
+Refactor `Filter` to check different fields depending on the data type.

changelog.d/11198.doc

@@ -0,0 +1 @@
+Clarify lack of support for Windows.

changelog.d/11199.bugfix

@@ -0,0 +1 @@
+Delete `to_device` messages for hidden devices that will never be read, reducing database size.

changelog.d/11200.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error.

changelog.d/11204.feature

@@ -0,0 +1 @@
+Add a module API method to retrieve the current state of a room.

changelog.d/11205.misc

@@ -0,0 +1 @@
+Improve type hints for the relations datastore.

changelog.d/11206.removal

@@ -0,0 +1 @@
+The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information.

changelog.d/11209.docker

@@ -0,0 +1 @@
+Avoid changing userid when started as a non-root user, and no explicit `UID` is set.

changelog.d/11211.feature

@@ -0,0 +1 @@
+Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443.

changelog.d/11212.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.

changelog.d/11213.removal

@@ -0,0 +1 @@
+Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`).

changelog.d/11217.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats.

changelog.d/11221.doc

@@ -0,0 +1 @@
+Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper.

changelog.d/11225.misc

@@ -0,0 +1 @@
+Replace outdated links in the pull request checklist with links to the rendered documentation.

changelog.d/11226.misc

@@ -0,0 +1 @@
+Fix a bug in unit test `test_block_room_and_not_purge`.

changelog.d/11228.feature

@@ -1 +0,0 @@
-Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it.

changelog.d/11229.misc

@@ -0,0 +1 @@
+`ObservableDeferred`: run registered observers in order.

changelog.d/11231.misc

@@ -0,0 +1 @@
+Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`.

changelog.d/11236.feature

@@ -1 +0,0 @@
-Support filtering by relation senders & types per [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440).

changelog.d/11242.misc

@@ -1 +0,0 @@
-Split out federated PDU retrieval function into a non-cached version.

changelog.d/11247.misc

@@ -1 +0,0 @@
-Clean up code relating to to-device messages and sending ephemeral events to application services.

changelog.d/11265.bugfix

@@ -0,0 +1 @@
+Prevent [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical state events from being pushed to an application service via `/transactions`.

changelog.d/11278.misc

@@ -1 +0,0 @@
-Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`.

changelog.d/11282.misc

@@ -1 +0,0 @@
-Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.

changelog.d/11285.misc

@@ -1 +0,0 @@
-Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.

changelog.d/11286.doc

@@ -1 +0,0 @@
-Fix typo in the word `available` and fix HTTP method (should be `GET`) for the `username_available` admin API. Contributed by Stanislav Motylkov.

changelog.d/11287.misc

@@ -1 +0,0 @@
-Add missing type hints to `synapse.app`.

changelog.d/11288.bugfix

@@ -1 +0,0 @@
-Fix a long-standing bug where uploading extremely thin images (e.g. 1000x1) would fail. Contributed by @Neeeflix.

changelog.d/11292.misc

@@ -1 +0,0 @@
-Remove unused parameters on `FederationEventHandler._check_event_auth`.

changelog.d/11297.misc

@@ -1 +0,0 @@
-Add type hints to `synapse._scripts`.

changelog.d/11298.doc

@@ -1 +0,0 @@
-Add Single Sign-On, SAML and CAS pages to the documentation.

debian/build_virtualenv

@@ -40,7 +40,6 @@ dh_virtualenv \
     --upgrade-pip \
     --preinstall="lxml" \
     --preinstall="mock" \
-    --preinstall="wheel" \
     --extra-pip-arg="--no-cache-dir" \
     --extra-pip-arg="--compile" \
     --extras="all,systemd,test"

debian/changelog

@@ -1,16 +1,8 @@
-matrix-synapse-py3 (1.47.0~rc2) stable; urgency=medium
+matrix-synapse-py3 (1.47.0+nmu1) UNRELEASED; urgency=medium
 
-  [ Dan Callahan ]
   * Update scripts to pass Shellcheck lints.
-  * Remove unused Vagrant scripts from debian/ directory.
-  * Allow building Debian packages for any architecture, not just amd64.
-  * Preinstall the "wheel" package when building virtualenvs.
-  * Do not error if /etc/default/matrix-synapse is missing.
 
-  [ Synapse Packaging team ]
-  * New synapse release 1.47.0~rc2.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 10 Nov 2021 09:41:01 +0000
+ -- root <root@cae79a6e79d7>  Fri, 22 Oct 2021 22:20:31 +0000
 
 matrix-synapse-py3 (1.46.0) stable; urgency=medium
 

debian/control

@@ -19,7 +19,7 @@ Standards-Version: 3.9.8
 Homepage: https://github.com/matrix-org/synapse
 
 Package: matrix-synapse-py3
-Architecture: any
+Architecture: amd64
 Provides: matrix-synapse
 Conflicts:
  matrix-synapse (<< 0.34.0.1-0matrix2),

debian/matrix-synapse.service

@@ -5,7 +5,7 @@ Description=Synapse Matrix homeserver
 Type=notify
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=-/etc/default/matrix-synapse
+EnvironmentFile=/etc/default/matrix-synapse
 ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
 ExecReload=/bin/kill -HUP $MAINPID

debian/test/.gitignore

@@ -0,0 +1,2 @@
+.vagrant
+*.log

debian/test/provision.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+#
+# provisioning script for vagrant boxes for testing the matrix-synapse debs.
+#
+# Will install the most recent matrix-synapse-py3 deb for this platform from
+# the /debs directory.
+
+set -e
+
+apt-get update
+apt-get install -y lsb-release
+
+deb=$(find /debs -name "matrix-synapse-py3_*+$(lsb_release -cs)*.deb" | sort | tail -n1)
+
+debconf-set-selections <<EOF
+matrix-synapse matrix-synapse/report-stats boolean false
+matrix-synapse matrix-synapse/server-name string localhost:18448
+EOF
+
+dpkg -i "$deb"
+
+sed -i -e 's/port: 8448$/port: 18448/; s/port: 8008$/port: 18008' /etc/matrix-synapse/homeserver.yaml
+echo 'registration_shared_secret: secret' >> /etc/matrix-synapse/homeserver.yaml
+systemctl restart matrix-synapse

debian/test/stretch/Vagrantfile

@@ -0,0 +1,13 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+ver = `cd ../../..; dpkg-parsechangelog -S Version`.strip()
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "debian/stretch64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "../../../../debs", "/debs", type: "nfs"
+
+  config.vm.provision "shell", path: "../provision.sh"
+end

debian/test/xenial/Vagrantfile

@@ -0,0 +1,10 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "ubuntu/xenial64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "../../../../debs", "/debs"
+  config.vm.provision "shell", path: "../provision.sh"
+end

docs/SUMMARY.md

@@ -23,10 +23,10 @@
     - [Structured Logging](structured_logging.md)
     - [Templates](templates.md)
     - [User Authentication](usage/configuration/user_authentication/README.md)
-      - [Single-Sign On](usage/configuration/user_authentication/single_sign_on/README.md)
+      - [Single-Sign On]()
         - [OpenID Connect](openid.md)
-        - [SAML](usage/configuration/user_authentication/single_sign_on/saml.md)
-        - [CAS](usage/configuration/user_authentication/single_sign_on/cas.md)
+        - [SAML]()
+        - [CAS]()
         - [SSO Mapping Providers](sso_mapping_providers.md)
       - [Password Auth Providers](password_auth_providers.md)
       - [JSON Web Tokens](jwt.md)
@@ -51,7 +51,6 @@
   - [Administration](usage/administration/README.md)
     - [Admin API](usage/administration/admin_api/README.md)
       - [Account Validity](admin_api/account_validity.md)
-      - [Background Updates](usage/administration/admin_api/background_updates.md)
       - [Delete Group](admin_api/delete_group.md)
       - [Event Reports](admin_api/event_reports.md)
       - [Media](admin_api/media_admin_api.md)

docs/admin_api/rooms.md

@@ -385,7 +385,7 @@ A response body like the following is returned:
 
 # Delete Room API
 
-The Delete Room admin API allows server admins to remove rooms from the server
+The Delete Room admin API allows server admins to remove rooms from server
 and block these rooms.
 
 Shuts down a room. Moves all local users and room aliases automatically to a
@@ -396,17 +396,13 @@ The new room will be created with the user specified by the `new_room_user_id` p
 as room administrator and will contain a message explaining what happened. Users invited
 to the new room will have power level `-10` by default, and thus be unable to speak.
 
-If `block` is `true`, users will be prevented from joining the old room.
-This option can also be used to pre-emptively block a room, even if it's unknown
-to this homeserver. In this case, the room will be blocked, and no further action
-will be taken. If `block` is `false`, attempting to delete an unknown room is
-invalid and will be rejected as a bad request.
+If `block` is `True` it prevents new joins to the old room.
 
 This API will remove all trace of the old room from your database after removing
 all local users. If `purge` is `true` (the default), all traces of the old room will
 be removed from your database after removing all local users. If you do not want
 this to happen, set `purge` to `false`.
-Depending on the amount of history being purged, a call to the API may take
+Depending on the amount of history being purged a call to the API may take
 several minutes or longer.
 
 The local server will only have the power to move local user and room aliases to
@@ -468,9 +464,8 @@ The following JSON body parameters are available:
               `new_room_user_id` in the new room. Ideally this will clearly convey why the
                original room was shut down. Defaults to `Sharing illegal content on this server
                is not permitted and rooms in violation will be blocked.`
-* `block` - Optional. If set to `true`, this room will be added to a blocking list,
-            preventing future attempts to join the room. Rooms can be blocked
-            even if they're not yet known to the homeserver. Defaults to `false`.
+* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing
+            future attempts to join the room. Defaults to `false`.
 * `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
             Defaults to `true`.
 * `force_purge` - Optional, and ignored unless `purge` is `true`. If set to `true`, it
@@ -488,8 +483,7 @@ The following fields are returned in the JSON response body:
 * `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
 * `local_aliases` - An array of strings representing the local aliases that were migrated from
                     the old room to the new.
-* `new_room_id` - A string representing the room ID of the new room, or `null` if
-                  no such room was created.
+* `new_room_id` - A string representing the room ID of the new room.
 
 
 ## Undoing room deletions

docs/admin_api/user_admin_api.md

@@ -1107,7 +1107,7 @@ This endpoint will work even if registration is disabled on the server, unlike
 The API is:
 
 ```
-GET /_synapse/admin/v1/username_available?username=$localpart
+POST /_synapse/admin/v1/username_availabile?username=$localpart
 ```
 
 The request and response format is the same as the [/_matrix/client/r0/register/available](https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available) API.

docs/modules/password_auth_provider_callbacks.md

@@ -11,7 +11,7 @@ registered by using the Module API's `register_password_auth_provider_callbacks`
 _First introduced in Synapse v1.46.0_
 
 ```python
-auth_checkers: Dict[Tuple[str, Tuple[str, ...]], Callable]
+ auth_checkers: Dict[Tuple[str,Tuple], Callable]
 ```
 
 A dict mapping from tuples of a login type identifier (such as `m.login.password`) and a

docs/openid.md

@@ -22,7 +22,6 @@ such as [Github][github-idp].
 [google-idp]: https://developers.google.com/identity/protocols/oauth2/openid-connect
 [auth0]: https://auth0.com/
 [authentik]: https://goauthentik.io/
-[lemonldap]: https://lemonldap-ng.org/
 [okta]: https://www.okta.com/
 [dex-idp]: https://github.com/dexidp/dex
 [keycloak-idp]: https://www.keycloak.org/docs/latest/server_admin/#sso-protocols
@@ -244,43 +243,6 @@ oidc_providers:
         display_name_template: "{{ user.preferred_username|capitalize }}" # TO BE FILLED: If your users have names in Authentik and you want those in Synapse, this should be replaced with user.name|capitalize.
 ```
 
-### LemonLDAP
-
-[LemonLDAP::NG][lemonldap] is an open-source IdP solution.
-
-1. Create an OpenID Connect Relying Parties in LemonLDAP::NG
-2. The parameters are:
-- Client ID under the basic menu of the new Relying Parties (`Options > Basic >
-  Client ID`)
-- Client secret (`Options > Basic > Client secret`)
-- JWT Algorithm: RS256 within the security menu of the new Relying Parties
-  (`Options > Security > ID Token signature algorithm` and `Options > Security >
-  Access Token signature algorithm`)
-- Scopes: OpenID, Email and Profile
-- Allowed redirection addresses for login (`Options > Basic > Allowed
-  redirection addresses for login` ) :
-  `[synapse public baseurl]/_synapse/client/oidc/callback`
-
-Synapse config:
-```yaml
-oidc_providers:
-  - idp_id: lemonldap
-    idp_name: lemonldap
-    discover: true
-    issuer: "https://auth.example.org/" # TO BE FILLED: replace with your domain
-    client_id: "your client id" # TO BE FILLED
-    client_secret: "your client secret" # TO BE FILLED
-    scopes:
-      - "openid"
-      - "profile"
-      - "email"
-    user_mapping_provider:
-      config:
-        localpart_template: "{{ user.preferred_username }}}"
-        # TO BE FILLED: If your users have names in LemonLDAP::NG and you want those in Synapse, this should be replaced with user.name|capitalize or any valid filter.
-        display_name_template: "{{ user.preferred_username|capitalize }}"
-```
-
 ### GitHub
 
 [GitHub][github-idp] is a bit special as it is not an OpenID Connect compliant provider, but

docs/sample_config.yaml

@@ -91,8 +91,6 @@ pid_file: DATADIR/homeserver.pid
 # Otherwise, it should be the URL to reach Synapse's client HTTP listener (see
 # 'listeners' below).
 #
-# Defaults to 'https://<server_name>/'.
-#
 #public_baseurl: https://example.com/
 
 # Uncomment the following to tell other servers to send federation traffic on
@@ -1267,7 +1265,7 @@ oembed:
 # in on this server.
 #
 # (By default, no suggestion is made, so it is left up to the client.
-# This setting is ignored unless public_baseurl is also explicitly set.)
+# This setting is ignored unless public_baseurl is also set.)
 #
 #default_identity_server: https://matrix.org
 
@@ -1292,6 +1290,8 @@ oembed:
 # by the Matrix Identity Service API specification:
 # https://matrix.org/docs/spec/identity_service/latest
 #
+# If a delegate is specified, the config option public_baseurl must also be filled out.
+#
 account_threepid_delegates:
     #email: https://example.com     # Delegate email sending to example.com
     #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
@@ -1981,10 +1981,11 @@ sso:
     # phishing attacks from evil.site. To avoid this, include a slash after the
     # hostname: "https://my.client/".
     #
-    # The login fallback page (used by clients that don't natively support the
-    # required login flows) is whitelisted in addition to any URLs in this list.
+    # If public_baseurl is set, then the login fallback page (used by clients
+    # that don't natively support the required login flows) is whitelisted in
+    # addition to any URLs in this list.
     #
-    # By default, this list contains only the login fallback page.
+    # By default, this list is empty.
     #
     #client_whitelist:
     #  - https://riot.im/develop

docs/systemd-with-workers/system/matrix-synapse-worker@.service

@@ -15,7 +15,7 @@ Type=notify
 NotifyAccess=main
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=-/etc/default/matrix-synapse
+EnvironmentFile=/etc/default/matrix-synapse
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.generic_worker --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always

docs/systemd-with-workers/system/matrix-synapse.service

@@ -10,7 +10,7 @@ Type=notify
 NotifyAccess=main
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=-/etc/default/matrix-synapse
+EnvironmentFile=/etc/default/matrix-synapse
 ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
 ExecReload=/bin/kill -HUP $MAINPID

docs/usage/administration/admin_api/background_updates.md

@@ -1,84 +0,0 @@
-# Background Updates API
-
-This API allows a server administrator to manage the background updates being
-run against the database.
-
-## Status
-
-This API gets the current status of the background updates.
-
-
-The API is:
-
-```
-GET /_synapse/admin/v1/background_updates/status
-```
-
-Returning:
-
-```json
-{
-    "enabled": true,
-    "current_updates": {
-        "<db_name>": {
-            "name": "<background_update_name>",
-            "total_item_count": 50,
-            "total_duration_ms": 10000.0,
-            "average_items_per_ms": 2.2,
-        },
-    }
-}
-```
-
-`enabled` whether the background updates are enabled or disabled.
-
-`db_name` the database name (usually Synapse is configured with a single database named 'master').
-
-For each update:
-
-`name` the name of the update.
-`total_item_count` total number of "items" processed (the meaning of 'items' depends on the update in question).
-`total_duration_ms` how long the background process has been running, not including time spent sleeping.
-`average_items_per_ms` how many items are processed per millisecond based on an exponential average.
-
-
-
-## Enabled
-
-This API allow pausing background updates.
-
-Background updates should *not* be paused for significant periods of time, as
-this can affect the performance of Synapse.
-
-*Note*: This won't persist over restarts.
-
-*Note*: This won't cancel any update query that is currently running. This is
-usually fine since most queries are short lived, except for `CREATE INDEX`
-background updates which won't be cancelled once started.
-
-
-The API is:
-
-```
-POST /_synapse/admin/v1/background_updates/enabled
-```
-
-with the following body:
-
-```json
-{
-    "enabled": false
-}
-```
-
-`enabled` sets whether the background updates are enabled or disabled.
-
-The API returns the `enabled` param.
-
-```json
-{
-    "enabled": false
-}
-```
-
-There is also a `GET` version which returns the `enabled` state.

docs/usage/configuration/user_authentication/single_sign_on/README.md

@@ -1,5 +0,0 @@
-# Single Sign-On
-
-Synapse supports single sign-on through the SAML, Open ID Connect or CAS protocols. 
-LDAP and other login methods are supported through first and third-party password
-auth provider modules.

docs/usage/configuration/user_authentication/single_sign_on/cas.md

@@ -1,8 +0,0 @@
-# CAS
-
-Synapse supports authenticating users via the [Central Authentication
-Service protocol](https://en.wikipedia.org/wiki/Central_Authentication_Service)
-(CAS) natively.
-
-Please see the `cas_config` and `sso` sections of the [Synapse configuration
-file](../../../configuration/homeserver_sample_config.md) for more details.

docs/usage/configuration/user_authentication/single_sign_on/saml.md

@@ -1,8 +0,0 @@
-# SAML
-
-Synapse supports authenticating users via the [Security Assertion
-Markup Language](https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language)
-(SAML) protocol natively.
-
-Please see the `saml2_config` and `sso` sections of the [Synapse configuration
-file](../../../configuration/homeserver_sample_config.md) for more details.

mypy.ini

@@ -10,162 +10,86 @@ warn_unreachable = True
 local_partial_types = True
 no_implicit_optional = True
 
+# To find all folders that pass mypy you run:
+#
+#   find synapse/* -type d -not -name __pycache__ -exec bash -c "mypy '{}' > /dev/null"  \; -print
+
 files =
   scripts-dev/sign_json,
-  setup.py,
-  synapse/,
-  tests/
-
-# Note: Better exclusion syntax coming in mypy > 0.910
-# https://github.com/python/mypy/pull/11329
-#
-# For now, set the (?x) flag enable "verbose" regexes
-# https://docs.python.org/3/library/re.html#re.X
-exclude = (?x)
-  ^(
-   |synapse/storage/databases/__init__.py
-   |synapse/storage/databases/main/__init__.py
-   |synapse/storage/databases/main/account_data.py
-   |synapse/storage/databases/main/cache.py
-   |synapse/storage/databases/main/censor_events.py
-   |synapse/storage/databases/main/deviceinbox.py
-   |synapse/storage/databases/main/devices.py
-   |synapse/storage/databases/main/directory.py
-   |synapse/storage/databases/main/e2e_room_keys.py
-   |synapse/storage/databases/main/end_to_end_keys.py
-   |synapse/storage/databases/main/event_federation.py
-   |synapse/storage/databases/main/event_push_actions.py
-   |synapse/storage/databases/main/events_bg_updates.py
-   |synapse/storage/databases/main/events_forward_extremities.py
-   |synapse/storage/databases/main/events_worker.py
-   |synapse/storage/databases/main/filtering.py
-   |synapse/storage/databases/main/group_server.py
-   |synapse/storage/databases/main/lock.py
-   |synapse/storage/databases/main/media_repository.py
-   |synapse/storage/databases/main/metrics.py
-   |synapse/storage/databases/main/monthly_active_users.py
-   |synapse/storage/databases/main/openid.py
-   |synapse/storage/databases/main/presence.py
-   |synapse/storage/databases/main/profile.py
-   |synapse/storage/databases/main/purge_events.py
-   |synapse/storage/databases/main/push_rule.py
-   |synapse/storage/databases/main/receipts.py
-   |synapse/storage/databases/main/rejections.py
-   |synapse/storage/databases/main/room.py
-   |synapse/storage/databases/main/room_batch.py
-   |synapse/storage/databases/main/roommember.py
-   |synapse/storage/databases/main/search.py
-   |synapse/storage/databases/main/signatures.py
-   |synapse/storage/databases/main/state.py
-   |synapse/storage/databases/main/state_deltas.py
-   |synapse/storage/databases/main/stats.py
-   |synapse/storage/databases/main/tags.py
-   |synapse/storage/databases/main/transactions.py
-   |synapse/storage/databases/main/user_directory.py
-   |synapse/storage/databases/main/user_erasure_store.py
-   |synapse/storage/schema/
-
-   |tests/api/test_auth.py
-   |tests/api/test_ratelimiting.py
-   |tests/app/test_openid_listener.py
-   |tests/appservice/test_scheduler.py
-   |tests/config/test_cache.py
-   |tests/config/test_tls.py
-   |tests/crypto/test_keyring.py
-   |tests/events/test_presence_router.py
-   |tests/events/test_utils.py
-   |tests/federation/test_federation_catch_up.py
-   |tests/federation/test_federation_sender.py
-   |tests/federation/test_federation_server.py
-   |tests/federation/transport/test_knocking.py
-   |tests/federation/transport/test_server.py
-   |tests/handlers/test_cas.py
-   |tests/handlers/test_directory.py
-   |tests/handlers/test_e2e_keys.py
-   |tests/handlers/test_federation.py
-   |tests/handlers/test_oidc.py
-   |tests/handlers/test_presence.py
-   |tests/handlers/test_profile.py
-   |tests/handlers/test_saml.py
-   |tests/handlers/test_typing.py
-   |tests/http/federation/test_matrix_federation_agent.py
-   |tests/http/federation/test_srv_resolver.py
-   |tests/http/test_fedclient.py
-   |tests/http/test_proxyagent.py
-   |tests/http/test_servlet.py
-   |tests/http/test_site.py
-   |tests/logging/__init__.py
-   |tests/logging/test_terse_json.py
-   |tests/module_api/test_api.py
-   |tests/push/test_email.py
-   |tests/push/test_http.py
-   |tests/push/test_presentable_names.py
-   |tests/push/test_push_rule_evaluator.py
-   |tests/rest/admin/test_admin.py
-   |tests/rest/admin/test_device.py
-   |tests/rest/admin/test_media.py
-   |tests/rest/admin/test_server_notice.py
-   |tests/rest/admin/test_user.py
-   |tests/rest/admin/test_username_available.py
-   |tests/rest/client/test_account.py
-   |tests/rest/client/test_events.py
-   |tests/rest/client/test_filter.py
-   |tests/rest/client/test_groups.py
-   |tests/rest/client/test_register.py
-   |tests/rest/client/test_report_event.py
-   |tests/rest/client/test_rooms.py
-   |tests/rest/client/test_third_party_rules.py
-   |tests/rest/client/test_transactions.py
-   |tests/rest/client/test_typing.py
-   |tests/rest/client/utils.py
-   |tests/rest/key/v2/test_remote_key_resource.py
-   |tests/rest/media/v1/test_base.py
-   |tests/rest/media/v1/test_media_storage.py
-   |tests/rest/media/v1/test_url_preview.py
-   |tests/scripts/test_new_matrix_user.py
-   |tests/server.py
-   |tests/server_notices/test_resource_limits_server_notices.py
-   |tests/state/test_v2.py
-   |tests/storage/test_account_data.py
-   |tests/storage/test_appservice.py
-   |tests/storage/test_background_update.py
-   |tests/storage/test_base.py
-   |tests/storage/test_client_ips.py
-   |tests/storage/test_database.py
-   |tests/storage/test_event_federation.py
-   |tests/storage/test_id_generators.py
-   |tests/storage/test_roommember.py
-   |tests/test_metrics.py
-   |tests/test_phone_home.py
-   |tests/test_server.py
-   |tests/test_state.py
-   |tests/test_terms_auth.py
-   |tests/test_visibility.py
-   |tests/unittest.py
-   |tests/util/caches/test_cached_call.py
-   |tests/util/caches/test_deferred_cache.py
-   |tests/util/caches/test_descriptors.py
-   |tests/util/caches/test_response_cache.py
-   |tests/util/caches/test_ttlcache.py
-   |tests/util/test_async_helpers.py
-   |tests/util/test_batching_queue.py
-   |tests/util/test_dict_cache.py
-   |tests/util/test_expiring_cache.py
-   |tests/util/test_file_consumer.py
-   |tests/util/test_linearizer.py
-   |tests/util/test_logcontext.py
-   |tests/util/test_lrucache.py
-   |tests/util/test_rwlock.py
-   |tests/util/test_wheel_timer.py
-   |tests/utils.py
-   )$
+  synapse/__init__.py,
+  synapse/api,
+  synapse/appservice,
+  synapse/config,
+  synapse/crypto,
+  synapse/event_auth.py,
+  synapse/events,
+  synapse/federation,
+  synapse/groups,
+  synapse/handlers,
+  synapse/http,
+  synapse/logging,
+  synapse/metrics,
+  synapse/module_api,
+  synapse/notifier.py,
+  synapse/push,
+  synapse/replication,
+  synapse/rest,
+  synapse/server.py,
+  synapse/server_notices,
+  synapse/spam_checker_api,
+  synapse/state,
+  synapse/storage/__init__.py,
+  synapse/storage/_base.py,
+  synapse/storage/background_updates.py,
+  synapse/storage/databases/main/appservice.py,
+  synapse/storage/databases/main/client_ips.py,
+  synapse/storage/databases/main/events.py,
+  synapse/storage/databases/main/keys.py,
+  synapse/storage/databases/main/pusher.py,
+  synapse/storage/databases/main/registration.py,
+  synapse/storage/databases/main/relations.py,
+  synapse/storage/databases/main/session.py,
+  synapse/storage/databases/main/stream.py,
+  synapse/storage/databases/main/ui_auth.py,
+  synapse/storage/databases/state,
+  synapse/storage/database.py,
+  synapse/storage/engines,
+  synapse/storage/keys.py,
+  synapse/storage/persist_events.py,
+  synapse/storage/prepare_database.py,
+  synapse/storage/purge_events.py,
+  synapse/storage/push_rule.py,
+  synapse/storage/relations.py,
+  synapse/storage/roommember.py,
+  synapse/storage/state.py,
+  synapse/storage/types.py,
+  synapse/storage/util,
+  synapse/streams,
+  synapse/types.py,
+  synapse/util,
+  synapse/visibility.py,
+  tests/replication,
+  tests/test_event_auth.py,
+  tests/test_utils,
+  tests/handlers/test_password_providers.py,
+  tests/handlers/test_room.py,
+  tests/handlers/test_room_summary.py,
+  tests/handlers/test_send_email.py,
+  tests/handlers/test_sync.py,
+  tests/handlers/test_user_directory.py,
+  tests/rest/client/test_login.py,
+  tests/rest/client/test_auth.py,
+  tests/rest/client/test_relations.py,
+  tests/rest/media/v1/test_filepath.py,
+  tests/rest/media/v1/test_oembed.py,
+  tests/storage/test_state.py,
+  tests/storage/test_user_directory.py,
+  tests/util/test_itertools.py,
+  tests/util/test_stream_change_cache.py
 
 [mypy-synapse.api.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.app.*]
-disallow_untyped_defs = True
-
 [mypy-synapse.crypto.*]
 disallow_untyped_defs = True
 
@@ -348,9 +272,6 @@ ignore_missing_imports = True
 [mypy-opentracing]
 ignore_missing_imports = True
 
-[mypy-parameterized.*]
-ignore_missing_imports = True
-
 [mypy-phonenumbers.*]
 ignore_missing_imports = True
 

setup.py

@@ -17,7 +17,6 @@
 # limitations under the License.
 import glob
 import os
-from typing import Any, Dict
 
 from setuptools import Command, find_packages, setup
 
@@ -50,6 +49,8 @@ here = os.path.abspath(os.path.dirname(__file__))
 # [1]: http://tox.readthedocs.io/en/2.5.0/example/basic.html#integration-with-setup-py-test-command
 # [2]: https://pypi.python.org/pypi/setuptools_trial
 class TestCommand(Command):
+    user_options = []
+
     def initialize_options(self):
         pass
 
@@ -74,7 +75,7 @@ def read_file(path_segments):
 
 def exec_file(path_segments):
     """Execute a single python file to get the variables defined in it"""
-    result: Dict[str, Any] = {}
+    result = {}
     code = read_file(path_segments)
     exec(code, result)
     return result
@@ -110,7 +111,6 @@ CONDITIONAL_REQUIREMENTS["mypy"] = [
     "types-Pillow>=8.3.4",
     "types-pyOpenSSL>=20.0.7",
     "types-PyYAML>=5.4.10",
-    "types-requests>=2.26.0",
     "types-setuptools>=57.4.0",
 ]
 
@@ -132,9 +132,6 @@ CONDITIONAL_REQUIREMENTS["dev"] = (
         "GitPython==3.1.14",
         "commonmark==0.9.1",
         "pygithub==1.55",
-        # The following are executed as commands by the release script.
-        "twine",
-        "towncrier",
     ]
 )
 

synapse/__init__.py

@@ -47,7 +47,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.47.0rc2"
+__version__ = "1.46.0"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/_scripts/register_new_matrix_user.py

@@ -1,6 +1,5 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2018 New Vector
-# Copyright 2021 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,23 +19,22 @@ import hashlib
 import hmac
 import logging
 import sys
-from typing import Callable, Optional
 
 import requests as _requests
 import yaml
 
 
 def request_registration(
-    user: str,
-    password: str,
-    server_location: str,
-    shared_secret: str,
-    admin: bool = False,
-    user_type: Optional[str] = None,
+    user,
+    password,
+    server_location,
+    shared_secret,
+    admin=False,
+    user_type=None,
     requests=_requests,
-    _print: Callable[[str], None] = print,
-    exit: Callable[[int], None] = sys.exit,
-) -> None:
+    _print=print,
+    exit=sys.exit,
+):
 
     url = "%s/_synapse/admin/v1/register" % (server_location.rstrip("/"),)
 
@@ -67,13 +65,13 @@ def request_registration(
         mac.update(b"\x00")
         mac.update(user_type.encode("utf8"))
 
-    hex_mac = mac.hexdigest()
+    mac = mac.hexdigest()
 
     data = {
         "nonce": nonce,
         "username": user,
         "password": password,
-        "mac": hex_mac,
+        "mac": mac,
         "admin": admin,
         "user_type": user_type,
     }
@@ -93,17 +91,10 @@ def request_registration(
     _print("Success!")
 
 
-def register_new_user(
-    user: str,
-    password: str,
-    server_location: str,
-    shared_secret: str,
-    admin: Optional[bool],
-    user_type: Optional[str],
-) -> None:
+def register_new_user(user, password, server_location, shared_secret, admin, user_type):
     if not user:
         try:
-            default_user: Optional[str] = getpass.getuser()
+            default_user = getpass.getuser()
         except Exception:
             default_user = None
 
@@ -132,8 +123,8 @@ def register_new_user(
             sys.exit(1)
 
     if admin is None:
-        admin_inp = input("Make admin [no]: ")
-        if admin_inp in ("y", "yes", "true"):
+        admin = input("Make admin [no]: ")
+        if admin in ("y", "yes", "true"):
             admin = True
         else:
             admin = False
@@ -143,7 +134,7 @@ def register_new_user(
     )
 
 
-def main() -> None:
+def main():
 
     logging.captureWarnings(True)
 

synapse/_scripts/review_recent_signups.py

@@ -92,7 +92,7 @@ def get_recent_users(txn: LoggingTransaction, since_ms: int) -> List[UserInfo]:
     return user_infos
 
 
-def main() -> None:
+def main():
     parser = argparse.ArgumentParser()
     parser.add_argument(
         "-c",
@@ -142,8 +142,7 @@ def main() -> None:
     engine = create_engine(database_config.config)
 
     with make_conn(database_config, engine, "review_recent_signups") as db_conn:
-        # This generates a type of Cursor, not LoggingTransaction.
-        user_infos = get_recent_users(db_conn.cursor(), since_ms)  # type: ignore[arg-type]
+        user_infos = get_recent_users(db_conn.cursor(), since_ms)
 
     for user_info in user_infos:
         if exclude_users_with_email and user_info.emails:

synapse/api/filtering.py

@@ -1,7 +1,7 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
 # Copyright 2018-2019 New Vector Ltd
-# Copyright 2019-2021 The Matrix.org Foundation C.I.C.
+# Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -86,9 +86,6 @@ ROOM_EVENT_FILTER_SCHEMA = {
         # cf https://github.com/matrix-org/matrix-doc/pull/2326
         "org.matrix.labels": {"type": "array", "items": {"type": "string"}},
         "org.matrix.not_labels": {"type": "array", "items": {"type": "string"}},
-        # MSC3440, filtering by event relations.
-        "io.element.relation_senders": {"type": "array", "items": {"type": "string"}},
-        "io.element.relation_types": {"type": "array", "items": {"type": "string"}},
     },
 }
 
@@ -149,16 +146,14 @@ def matrix_user_id_validator(user_id_str: str) -> UserID:
 
 class Filtering:
     def __init__(self, hs: "HomeServer"):
-        self._hs = hs
+        super().__init__()
         self.store = hs.get_datastore()
 
-        self.DEFAULT_FILTER_COLLECTION = FilterCollection(hs, {})
-
     async def get_user_filter(
         self, user_localpart: str, filter_id: Union[int, str]
     ) -> "FilterCollection":
         result = await self.store.get_user_filter(user_localpart, filter_id)
-        return FilterCollection(self._hs, result)
+        return FilterCollection(result)
 
     def add_user_filter(
         self, user_localpart: str, user_filter: JsonDict
@@ -196,22 +191,21 @@ FilterEvent = TypeVar("FilterEvent", EventBase, UserPresenceState, JsonDict)
 
 
 class FilterCollection:
-    def __init__(self, hs: "HomeServer", filter_json: JsonDict):
+    def __init__(self, filter_json: JsonDict):
         self._filter_json = filter_json
 
         room_filter_json = self._filter_json.get("room", {})
 
         self._room_filter = Filter(
-            hs,
-            {k: v for k, v in room_filter_json.items() if k in ("rooms", "not_rooms")},
+            {k: v for k, v in room_filter_json.items() if k in ("rooms", "not_rooms")}
         )
 
-        self._room_timeline_filter = Filter(hs, room_filter_json.get("timeline", {}))
-        self._room_state_filter = Filter(hs, room_filter_json.get("state", {}))
-        self._room_ephemeral_filter = Filter(hs, room_filter_json.get("ephemeral", {}))
-        self._room_account_data = Filter(hs, room_filter_json.get("account_data", {}))
-        self._presence_filter = Filter(hs, filter_json.get("presence", {}))
-        self._account_data = Filter(hs, filter_json.get("account_data", {}))
+        self._room_timeline_filter = Filter(room_filter_json.get("timeline", {}))
+        self._room_state_filter = Filter(room_filter_json.get("state", {}))
+        self._room_ephemeral_filter = Filter(room_filter_json.get("ephemeral", {}))
+        self._room_account_data = Filter(room_filter_json.get("account_data", {}))
+        self._presence_filter = Filter(filter_json.get("presence", {}))
+        self._account_data = Filter(filter_json.get("account_data", {}))
 
         self.include_leave = filter_json.get("room", {}).get("include_leave", False)
         self.event_fields = filter_json.get("event_fields", [])
@@ -238,37 +232,25 @@ class FilterCollection:
     def include_redundant_members(self) -> bool:
         return self._room_state_filter.include_redundant_members
 
-    async def filter_presence(
+    def filter_presence(
         self, events: Iterable[UserPresenceState]
     ) -> List[UserPresenceState]:
-        return await self._presence_filter.filter(events)
+        return self._presence_filter.filter(events)
 
-    async def filter_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return await self._account_data.filter(events)
+    def filter_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._account_data.filter(events)
 
-    async def filter_room_state(self, events: Iterable[EventBase]) -> List[EventBase]:
-        return await self._room_state_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_state(self, events: Iterable[EventBase]) -> List[EventBase]:
+        return self._room_state_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_timeline(
-        self, events: Iterable[EventBase]
-    ) -> List[EventBase]:
-        return await self._room_timeline_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_timeline(self, events: Iterable[EventBase]) -> List[EventBase]:
+        return self._room_timeline_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_ephemeral(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return await self._room_ephemeral_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_ephemeral(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._room_ephemeral_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_account_data(
-        self, events: Iterable[JsonDict]
-    ) -> List[JsonDict]:
-        return await self._room_account_data.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._room_account_data.filter(self._room_filter.filter(events))
 
     def blocks_all_presence(self) -> bool:
         return (
@@ -292,9 +274,7 @@ class FilterCollection:
 
 
 class Filter:
-    def __init__(self, hs: "HomeServer", filter_json: JsonDict):
-        self._hs = hs
-        self._store = hs.get_datastore()
+    def __init__(self, filter_json: JsonDict):
         self.filter_json = filter_json
 
         self.limit = filter_json.get("limit", 10)
@@ -317,20 +297,6 @@ class Filter:
         self.labels = filter_json.get("org.matrix.labels", None)
         self.not_labels = filter_json.get("org.matrix.not_labels", [])
 
-        # Ideally these would be rejected at the endpoint if they were provided
-        # and not supported, but that would involve modifying the JSON schema
-        # based on the homeserver configuration.
-        if hs.config.experimental.msc3440_enabled:
-            self.relation_senders = self.filter_json.get(
-                "io.element.relation_senders", None
-            )
-            self.relation_types = self.filter_json.get(
-                "io.element.relation_types", None
-            )
-        else:
-            self.relation_senders = None
-            self.relation_types = None
-
     def filters_all_types(self) -> bool:
         return "*" in self.not_types
 
@@ -340,7 +306,7 @@ class Filter:
     def filters_all_rooms(self) -> bool:
         return "*" in self.not_rooms
 
-    def _check(self, event: FilterEvent) -> bool:
+    def check(self, event: FilterEvent) -> bool:
         """Checks whether the filter matches the given event.
 
         Args:
@@ -454,30 +420,8 @@ class Filter:
 
         return room_ids
 
-    async def _check_event_relations(
-        self, events: Iterable[FilterEvent]
-    ) -> List[FilterEvent]:
-        # The event IDs to check, mypy doesn't understand the ifinstance check.
-        event_ids = [event.event_id for event in events if isinstance(event, EventBase)]  # type: ignore[attr-defined]
-        event_ids_to_keep = set(
-            await self._store.events_have_relations(
-                event_ids, self.relation_senders, self.relation_types
-            )
-        )
-
-        return [
-            event
-            for event in events
-            if not isinstance(event, EventBase) or event.event_id in event_ids_to_keep
-        ]
-
-    async def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
-        result = [event for event in events if self._check(event)]
-
-        if self.relation_senders or self.relation_types:
-            return await self._check_event_relations(result)
-
-        return result
+    def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
+        return list(filter(self.check, events))
 
     def with_room_ids(self, room_ids: Iterable[str]) -> "Filter":
         """Returns a new filter with the given room IDs appended.
@@ -489,7 +433,7 @@ class Filter:
             filter: A new filter including the given rooms and the old
                     filter's rooms.
         """
-        newFilter = Filter(self._hs, self.filter_json)
+        newFilter = Filter(self.filter_json)
         newFilter.rooms += room_ids
         return newFilter
 
@@ -500,3 +444,6 @@ def _matches_wildcard(actual_value: Optional[str], filter_value: str) -> bool:
         return actual_value.startswith(type_prefix)
     else:
         return actual_value == filter_value
+
+
+DEFAULT_FILTER_COLLECTION = FilterCollection({})

synapse/api/urls.py

@@ -38,6 +38,9 @@ class ConsentURIBuilder:
     def __init__(self, hs_config: HomeServerConfig):
         if hs_config.key.form_secret is None:
             raise ConfigError("form_secret not set in config")
+        if hs_config.server.public_baseurl is None:
+            raise ConfigError("public_baseurl not set in config")
+
         self._hmac_secret = hs_config.key.form_secret.encode("utf-8")
         self._public_baseurl = hs_config.server.public_baseurl
 

synapse/app/__init__.py

@@ -13,7 +13,6 @@
 # limitations under the License.
 import logging
 import sys
-from typing import Container
 
 from synapse import python_dependencies  # noqa: E402
 
@@ -28,9 +27,7 @@ except python_dependencies.DependencyException as e:
     sys.exit(1)
 
 
-def check_bind_error(
-    e: Exception, address: str, bind_addresses: Container[str]
-) -> None:
+def check_bind_error(e, address, bind_addresses):
     """
     This method checks an exception occurred while binding on 0.0.0.0.
     If :: is specified in the bind addresses a warning is shown.
@@ -41,9 +38,9 @@ def check_bind_error(
     When binding on 0.0.0.0 after :: this can safely be ignored.
 
     Args:
-        e: Exception that was caught.
-        address: Address on which binding was attempted.
-        bind_addresses: Addresses on which the service listens.
+        e (Exception): Exception that was caught.
+        address (str): Address on which binding was attempted.
+        bind_addresses (list): Addresses on which the service listens.
     """
     if address == "0.0.0.0" and "::" in bind_addresses:
         logger.warning(

synapse/app/_base.py

@@ -22,27 +22,13 @@ import socket
 import sys
 import traceback
 import warnings
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Awaitable,
-    Callable,
-    Collection,
-    Dict,
-    Iterable,
-    List,
-    NoReturn,
-    Tuple,
-    cast,
-)
+from typing import TYPE_CHECKING, Awaitable, Callable, Iterable
 
 from cryptography.utils import CryptographyDeprecationWarning
+from typing_extensions import NoReturn
 
 import twisted
-from twisted.internet import defer, error, reactor as _reactor
-from twisted.internet.interfaces import IOpenSSLContextFactory, IReactorSSL, IReactorTCP
-from twisted.internet.protocol import ServerFactory
-from twisted.internet.tcp import Port
+from twisted.internet import defer, error, reactor
 from twisted.logger import LoggingFile, LogLevel
 from twisted.protocols.tls import TLSMemoryBIOFactory
 from twisted.python.threadpool import ThreadPool
@@ -62,7 +48,6 @@ from synapse.logging.context import PreserveLoggingContext
 from synapse.metrics import register_threadpool
 from synapse.metrics.background_process_metrics import wrap_as_background_process
 from synapse.metrics.jemalloc import setup_jemalloc_stats
-from synapse.types import ISynapseReactor
 from synapse.util.caches.lrucache import setup_expire_lru_cache_entries
 from synapse.util.daemonize import daemonize_process
 from synapse.util.gai_resolver import GAIResolver
@@ -72,44 +57,33 @@ from synapse.util.versionstring import get_version_string
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
-# Twisted injects the global reactor to make it easier to import, this confuses
-# mypy which thinks it is a module. Tell it that it a more proper type.
-reactor = cast(ISynapseReactor, _reactor)
-
-
 logger = logging.getLogger(__name__)
 
 # list of tuples of function, args list, kwargs dict
-_sighup_callbacks: List[
-    Tuple[Callable[..., None], Tuple[Any, ...], Dict[str, Any]]
-] = []
+_sighup_callbacks = []
 
 
-def register_sighup(func: Callable[..., None], *args: Any, **kwargs: Any) -> None:
+def register_sighup(func, *args, **kwargs):
     """
     Register a function to be called when a SIGHUP occurs.
 
     Args:
-        func: Function to be called when sent a SIGHUP signal.
+        func (function): Function to be called when sent a SIGHUP signal.
         *args, **kwargs: args and kwargs to be passed to the target function.
     """
     _sighup_callbacks.append((func, args, kwargs))
 
 
-def start_worker_reactor(
-    appname: str,
-    config: HomeServerConfig,
-    run_command: Callable[[], None] = reactor.run,
-) -> None:
+def start_worker_reactor(appname, config, run_command=reactor.run):
     """Run the reactor in the main process
 
     Daemonizes if necessary, and then configures some resources, before starting
     the reactor. Pulls configuration from the 'worker' settings in 'config'.
 
     Args:
-        appname: application name which will be sent to syslog
-        config: config object
-        run_command: callable that actually runs the reactor
+        appname (str): application name which will be sent to syslog
+        config (synapse.config.Config): config object
+        run_command (Callable[]): callable that actually runs the reactor
     """
 
     logger = logging.getLogger(config.worker.worker_app)
@@ -127,32 +101,32 @@ def start_worker_reactor(
 
 
 def start_reactor(
-    appname: str,
-    soft_file_limit: int,
-    gc_thresholds: Tuple[int, int, int],
-    pid_file: str,
-    daemonize: bool,
-    print_pidfile: bool,
-    logger: logging.Logger,
-    run_command: Callable[[], None] = reactor.run,
-) -> None:
+    appname,
+    soft_file_limit,
+    gc_thresholds,
+    pid_file,
+    daemonize,
+    print_pidfile,
+    logger,
+    run_command=reactor.run,
+):
     """Run the reactor in the main process
 
     Daemonizes if necessary, and then configures some resources, before starting
     the reactor
 
     Args:
-        appname: application name which will be sent to syslog
-        soft_file_limit:
+        appname (str): application name which will be sent to syslog
+        soft_file_limit (int):
         gc_thresholds:
-        pid_file: name of pid file to write to if daemonize is True
-        daemonize: true to run the reactor in a background process
-        print_pidfile: whether to print the pid file, if daemonize is True
-        logger: logger instance to pass to Daemonize
-        run_command: callable that actually runs the reactor
+        pid_file (str): name of pid file to write to if daemonize is True
+        daemonize (bool): true to run the reactor in a background process
+        print_pidfile (bool): whether to print the pid file, if daemonize is True
+        logger (logging.Logger): logger instance to pass to Daemonize
+        run_command (Callable[]): callable that actually runs the reactor
     """
 
-    def run() -> None:
+    def run():
         logger.info("Running")
         setup_jemalloc_stats()
         change_resource_limit(soft_file_limit)
@@ -211,7 +185,7 @@ def redirect_stdio_to_logs() -> None:
     print("Redirected stdout/stderr to logs")
 
 
-def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> None:
+def register_start(cb: Callable[..., Awaitable], *args, **kwargs) -> None:
     """Register a callback with the reactor, to be called once it is running
 
     This can be used to initialise parts of the system which require an asynchronous
@@ -221,7 +195,7 @@ def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> N
     will exit.
     """
 
-    async def wrapper() -> None:
+    async def wrapper():
         try:
             await cb(*args, **kwargs)
         except Exception:
@@ -250,7 +224,7 @@ def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> N
     reactor.callWhenRunning(lambda: defer.ensureDeferred(wrapper()))
 
 
-def listen_metrics(bind_addresses: Iterable[str], port: int) -> None:
+def listen_metrics(bind_addresses, port):
     """
     Start Prometheus metrics server.
     """
@@ -262,11 +236,11 @@ def listen_metrics(bind_addresses: Iterable[str], port: int) -> None:
 
 
 def listen_manhole(
-    bind_addresses: Collection[str],
+    bind_addresses: Iterable[str],
     port: int,
     manhole_settings: ManholeConfig,
     manhole_globals: dict,
-) -> None:
+):
     # twisted.conch.manhole 21.1.0 uses "int_from_bytes", which produces a confusing
     # warning. It's fixed by https://github.com/twisted/twisted/pull/1522), so
     # suppress the warning for now.
@@ -285,18 +259,12 @@ def listen_manhole(
     )
 
 
-def listen_tcp(
-    bind_addresses: Collection[str],
-    port: int,
-    factory: ServerFactory,
-    reactor: IReactorTCP = reactor,
-    backlog: int = 50,
-) -> List[Port]:
+def listen_tcp(bind_addresses, port, factory, reactor=reactor, backlog=50):
     """
     Create a TCP socket for a port and several addresses
 
     Returns:
-        list of twisted.internet.tcp.Port listening for TCP connections
+        list[twisted.internet.tcp.Port]: listening for TCP connections
     """
     r = []
     for address in bind_addresses:
@@ -305,19 +273,12 @@ def listen_tcp(
         except error.CannotListenError as e:
             check_bind_error(e, address, bind_addresses)
 
-    # IReactorTCP returns an object implementing IListeningPort from listenTCP,
-    # but we know it will be a Port instance.
-    return r  # type: ignore[return-value]
+    return r
 
 
 def listen_ssl(
-    bind_addresses: Collection[str],
-    port: int,
-    factory: ServerFactory,
-    context_factory: IOpenSSLContextFactory,
-    reactor: IReactorSSL = reactor,
-    backlog: int = 50,
-) -> List[Port]:
+    bind_addresses, port, factory, context_factory, reactor=reactor, backlog=50
+):
     """
     Create an TLS-over-TCP socket for a port and several addresses
 
@@ -333,13 +294,10 @@ def listen_ssl(
         except error.CannotListenError as e:
             check_bind_error(e, address, bind_addresses)
 
-    # IReactorSSL incorrectly declares that an int is returned from listenSSL,
-    # it actually returns an object implementing IListeningPort, but we know it
-    # will be a Port instance.
-    return r  # type: ignore[return-value]
+    return r
 
 
-def refresh_certificate(hs: "HomeServer") -> None:
+def refresh_certificate(hs: "HomeServer"):
     """
     Refresh the TLS certificates that Synapse is using by re-reading them from
     disk and updating the TLS context factories to use them.
@@ -371,7 +329,7 @@ def refresh_certificate(hs: "HomeServer") -> None:
         logger.info("Context factories updated.")
 
 
-async def start(hs: "HomeServer") -> None:
+async def start(hs: "HomeServer"):
     """
     Start a Synapse server or worker.
 
@@ -402,7 +360,7 @@ async def start(hs: "HomeServer") -> None:
     if hasattr(signal, "SIGHUP"):
 
         @wrap_as_background_process("sighup")
-        def handle_sighup(*args: Any, **kwargs: Any) -> None:
+        def handle_sighup(*args, **kwargs):
             # Tell systemd our state, if we're using it. This will silently fail if
             # we're not using systemd.
             sdnotify(b"RELOADING=1")
@@ -415,7 +373,7 @@ async def start(hs: "HomeServer") -> None:
         # We defer running the sighup handlers until next reactor tick. This
         # is so that we're in a sane state, e.g. flushing the logs may fail
         # if the sighup happens in the middle of writing a log entry.
-        def run_sighup(*args: Any, **kwargs: Any) -> None:
+        def run_sighup(*args, **kwargs):
             # `callFromThread` should be "signal safe" as well as thread
             # safe.
             reactor.callFromThread(handle_sighup, *args, **kwargs)
@@ -478,8 +436,12 @@ async def start(hs: "HomeServer") -> None:
         atexit.register(gc.freeze)
 
 
-def setup_sentry(hs: "HomeServer") -> None:
-    """Enable sentry integration, if enabled in configuration"""
+def setup_sentry(hs: "HomeServer"):
+    """Enable sentry integration, if enabled in configuration
+
+    Args:
+        hs
+    """
 
     if not hs.config.metrics.sentry_enabled:
         return
@@ -504,7 +466,7 @@ def setup_sentry(hs: "HomeServer") -> None:
         scope.set_tag("worker_name", name)
 
 
-def setup_sdnotify(hs: "HomeServer") -> None:
+def setup_sdnotify(hs: "HomeServer"):
     """Adds process state hooks to tell systemd what we are up to."""
 
     # Tell systemd our state, if we're using it. This will silently fail if
@@ -519,7 +481,7 @@ def setup_sdnotify(hs: "HomeServer") -> None:
 sdnotify_sockaddr = os.getenv("NOTIFY_SOCKET")
 
 
-def sdnotify(state: bytes) -> None:
+def sdnotify(state):
     """
     Send a notification to systemd, if the NOTIFY_SOCKET env var is set.
 
@@ -528,7 +490,7 @@ def sdnotify(state: bytes) -> None:
     package which many OSes don't include as a matter of principle.
 
     Args:
-        state: notification to send
+        state (bytes): notification to send
     """
     if not isinstance(state, bytes):
         raise TypeError("sdnotify should be called with a bytes")

synapse/app/admin_cmd.py

@@ -17,7 +17,6 @@ import logging
 import os
 import sys
 import tempfile
-from typing import List, Optional
 
 from twisted.internet import defer, task
 
@@ -26,7 +25,6 @@ from synapse.app import _base
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
-from synapse.events import EventBase
 from synapse.handlers.admin import ExfiltrationWriter
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
@@ -42,7 +40,6 @@ from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.server import HomeServer
 from synapse.storage.databases.main.room import RoomWorkerStore
-from synapse.types import StateMap
 from synapse.util.logcontext import LoggingContext
 from synapse.util.versionstring import get_version_string
 
@@ -68,11 +65,16 @@ class AdminCmdSlavedStore(
 
 
 class AdminCmdServer(HomeServer):
-    DATASTORE_CLASS = AdminCmdSlavedStore  # type: ignore
+    DATASTORE_CLASS = AdminCmdSlavedStore
 
 
-async def export_data_command(hs: HomeServer, args: argparse.Namespace) -> None:
-    """Export data for a user."""
+async def export_data_command(hs: HomeServer, args):
+    """Export data for a user.
+
+    Args:
+        hs
+        args (argparse.Namespace)
+    """
 
     user_id = args.user_id
     directory = args.output_directory
@@ -90,12 +92,12 @@ class FileExfiltrationWriter(ExfiltrationWriter):
     Note: This writes to disk on the main reactor thread.
 
     Args:
-        user_id: The user whose data is being exfiltrated.
-        directory: The directory to write the data to, if None then will write
-            to a temporary directory.
+        user_id (str): The user whose data is being exfiltrated.
+        directory (str|None): The directory to write the data to, if None then
+            will write to a temporary directory.
     """
 
-    def __init__(self, user_id: str, directory: Optional[str] = None):
+    def __init__(self, user_id, directory=None):
         self.user_id = user_id
 
         if directory:
@@ -109,7 +111,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
         if list(os.listdir(self.base_directory)):
             raise Exception("Directory must be empty")
 
-    def write_events(self, room_id: str, events: List[EventBase]) -> None:
+    def write_events(self, room_id, events):
         room_directory = os.path.join(self.base_directory, "rooms", room_id)
         os.makedirs(room_directory, exist_ok=True)
         events_file = os.path.join(room_directory, "events")
@@ -118,9 +120,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in events:
                 print(json.dumps(event.get_pdu_json()), file=f)
 
-    def write_state(
-        self, room_id: str, event_id: str, state: StateMap[EventBase]
-    ) -> None:
+    def write_state(self, room_id, event_id, state):
         room_directory = os.path.join(self.base_directory, "rooms", room_id)
         state_directory = os.path.join(room_directory, "state")
         os.makedirs(state_directory, exist_ok=True)
@@ -131,9 +131,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event.get_pdu_json()), file=f)
 
-    def write_invite(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
-    ) -> None:
+    def write_invite(self, room_id, event, state):
         self.write_events(room_id, [event])
 
         # We write the invite state somewhere else as they aren't full events
@@ -147,9 +145,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event), file=f)
 
-    def write_knock(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
-    ) -> None:
+    def write_knock(self, room_id, event, state):
         self.write_events(room_id, [event])
 
         # We write the knock state somewhere else as they aren't full events
@@ -163,11 +159,11 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event), file=f)
 
-    def finished(self) -> str:
+    def finished(self):
         return self.base_directory
 
 
-def start(config_options: List[str]) -> None:
+def start(config_options):
     parser = argparse.ArgumentParser(description="Synapse Admin Command")
     HomeServerConfig.add_arguments_to_parser(parser)
 
@@ -235,7 +231,7 @@ def start(config_options: List[str]) -> None:
     # We also make sure that `_base.start` gets run before we actually run the
     # command.
 
-    async def run() -> None:
+    async def run():
         with LoggingContext("command"):
             await _base.start(ss)
             await args.func(ss, args)

synapse/app/generic_worker.py

@@ -14,10 +14,11 @@
 # limitations under the License.
 import logging
 import sys
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, Optional
 
 from twisted.internet import address
-from twisted.web.resource import Resource
+from twisted.web.resource import IResource
+from twisted.web.server import Request
 
 import synapse
 import synapse.events
@@ -43,7 +44,7 @@ from synapse.config.server import ListenerConfig
 from synapse.federation.transport.server import TransportLayerServer
 from synapse.http.server import JsonResource, OptionsResource
 from synapse.http.servlet import RestServlet, parse_json_object_from_request
-from synapse.http.site import SynapseRequest, SynapseSite
+from synapse.http.site import SynapseSite
 from synapse.logging.context import LoggingContext
 from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
@@ -118,7 +119,6 @@ from synapse.storage.databases.main.stats import StatsStore
 from synapse.storage.databases.main.transactions import TransactionWorkerStore
 from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
 from synapse.storage.databases.main.user_directory import UserDirectoryStore
-from synapse.types import JsonDict
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.versionstring import get_version_string
 
@@ -143,9 +143,7 @@ class KeyUploadServlet(RestServlet):
         self.http_client = hs.get_simple_http_client()
         self.main_uri = hs.config.worker.worker_main_http_uri
 
-    async def on_POST(
-        self, request: SynapseRequest, device_id: Optional[str]
-    ) -> Tuple[int, JsonDict]:
+    async def on_POST(self, request: Request, device_id: Optional[str]):
         requester = await self.auth.get_user_by_req(request, allow_guest=True)
         user_id = requester.user.to_string()
         body = parse_json_object_from_request(request)
@@ -189,8 +187,9 @@ class KeyUploadServlet(RestServlet):
                 # If the header exists, add to the comma-separated list of the first
                 # instance of the header. Otherwise, generate a new header.
                 if x_forwarded_for:
-                    x_forwarded_for = [x_forwarded_for[0] + b", " + previous_host]
-                    x_forwarded_for.extend(x_forwarded_for[1:])
+                    x_forwarded_for = [
+                        x_forwarded_for[0] + b", " + previous_host
+                    ] + x_forwarded_for[1:]
                 else:
                     x_forwarded_for = [previous_host]
             headers[b"X-Forwarded-For"] = x_forwarded_for
@@ -254,16 +253,13 @@ class GenericWorkerSlavedStore(
     SessionStore,
     BaseSlavedStore,
 ):
-    # Properties that multiple storage classes define. Tell mypy what the
-    # expected type is.
-    server_name: str
-    config: HomeServerConfig
+    pass
 
 
 class GenericWorkerServer(HomeServer):
-    DATASTORE_CLASS = GenericWorkerSlavedStore  # type: ignore
+    DATASTORE_CLASS = GenericWorkerSlavedStore
 
-    def _listen_http(self, listener_config: ListenerConfig) -> None:
+    def _listen_http(self, listener_config: ListenerConfig):
         port = listener_config.port
         bind_addresses = listener_config.bind_addresses
 
@@ -271,10 +267,10 @@ class GenericWorkerServer(HomeServer):
 
         site_tag = listener_config.http_options.tag
         if site_tag is None:
-            site_tag = str(port)
+            site_tag = port
 
         # We always include a health resource.
-        resources: Dict[str, Resource] = {"/health": HealthResource()}
+        resources: Dict[str, IResource] = {"/health": HealthResource()}
 
         for res in listener_config.http_options.resources:
             for name in res.names:
@@ -390,7 +386,7 @@ class GenericWorkerServer(HomeServer):
 
         logger.info("Synapse worker now listening on port %d", port)
 
-    def start_listening(self) -> None:
+    def start_listening(self):
         for listener in self.config.worker.worker_listeners:
             if listener.type == "http":
                 self._listen_http(listener)
@@ -415,7 +411,7 @@ class GenericWorkerServer(HomeServer):
         self.get_tcp_replication().start_replication(self)
 
 
-def start(config_options: List[str]) -> None:
+def start(config_options):
     try:
         config = HomeServerConfig.load_config("Synapse worker", config_options)
     except ConfigError as e:

synapse/app/homeserver.py

@@ -16,10 +16,10 @@
 import logging
 import os
 import sys
-from typing import Dict, Iterable, Iterator, List
+from typing import Iterator
 
-from twisted.internet.tcp import Port
-from twisted.web.resource import EncodingResourceWrapper, Resource
+from twisted.internet import reactor
+from twisted.web.resource import EncodingResourceWrapper, IResource
 from twisted.web.server import GzipEncoderFactory
 from twisted.web.static import File
 
@@ -76,27 +76,23 @@ from synapse.util.versionstring import get_version_string
 logger = logging.getLogger("synapse.app.homeserver")
 
 
-def gz_wrap(r: Resource) -> Resource:
+def gz_wrap(r):
     return EncodingResourceWrapper(r, [GzipEncoderFactory()])
 
 
 class SynapseHomeServer(HomeServer):
-    DATASTORE_CLASS = DataStore  # type: ignore
+    DATASTORE_CLASS = DataStore
 
-    def _listener_http(
-        self, config: HomeServerConfig, listener_config: ListenerConfig
-    ) -> Iterable[Port]:
+    def _listener_http(self, config: HomeServerConfig, listener_config: ListenerConfig):
         port = listener_config.port
         bind_addresses = listener_config.bind_addresses
         tls = listener_config.tls
-        # Must exist since this is an HTTP listener.
-        assert listener_config.http_options is not None
         site_tag = listener_config.http_options.tag
         if site_tag is None:
             site_tag = str(port)
 
         # We always include a health resource.
-        resources: Dict[str, Resource] = {"/health": HealthResource()}
+        resources = {"/health": HealthResource()}
 
         for res in listener_config.http_options.resources:
             for name in res.names:
@@ -115,7 +111,7 @@ class SynapseHomeServer(HomeServer):
                 ("listeners", site_tag, "additional_resources", "<%s>" % (path,)),
             )
             handler = handler_cls(config, module_api)
-            if isinstance(handler, Resource):
+            if IResource.providedBy(handler):
                 resource = handler
             elif hasattr(handler, "handle_request"):
                 resource = AdditionalResource(self, handler.handle_request)
@@ -132,7 +128,7 @@ class SynapseHomeServer(HomeServer):
 
         # try to find something useful to redirect '/' to
         if WEB_CLIENT_PREFIX in resources:
-            root_resource: Resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
+            root_resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
         elif STATIC_PREFIX in resources:
             root_resource = RootOptionsRedirectResource(STATIC_PREFIX)
         else:
@@ -149,8 +145,6 @@ class SynapseHomeServer(HomeServer):
         )
 
         if tls:
-            # refresh_certificate should have been called before this.
-            assert self.tls_server_context_factory is not None
             ports = listen_ssl(
                 bind_addresses,
                 port,
@@ -171,21 +165,20 @@ class SynapseHomeServer(HomeServer):
 
         return ports
 
-    def _configure_named_resource(
-        self, name: str, compress: bool = False
-    ) -> Dict[str, Resource]:
+    def _configure_named_resource(self, name, compress=False):
         """Build a resource map for a named resource
 
         Args:
-            name: named resource: one of "client", "federation", etc
-            compress: whether to enable gzip compression for this resource
+            name (str): named resource: one of "client", "federation", etc
+            compress (bool): whether to enable gzip compression for this
+                resource
 
         Returns:
-            map from path to HTTP resource
+            dict[str, Resource]: map from path to HTTP resource
         """
-        resources: Dict[str, Resource] = {}
+        resources = {}
         if name == "client":
-            client_resource: Resource = ClientRestResource(self)
+            client_resource = ClientRestResource(self)
             if compress:
                 client_resource = gz_wrap(client_resource)
 
@@ -214,7 +207,7 @@ class SynapseHomeServer(HomeServer):
         if name == "consent":
             from synapse.rest.consent.consent_resource import ConsentResource
 
-            consent_resource: Resource = ConsentResource(self)
+            consent_resource = ConsentResource(self)
             if compress:
                 consent_resource = gz_wrap(consent_resource)
             resources.update({"/_matrix/consent": consent_resource})
@@ -284,7 +277,7 @@ class SynapseHomeServer(HomeServer):
 
         return resources
 
-    def start_listening(self) -> None:
+    def start_listening(self):
         if self.config.redis.redis_enabled:
             # If redis is enabled we connect via the replication command handler
             # in the same way as the workers (since we're effectively a client
@@ -310,9 +303,7 @@ class SynapseHomeServer(HomeServer):
                     ReplicationStreamProtocolFactory(self),
                 )
                 for s in services:
-                    self.get_reactor().addSystemEventTrigger(
-                        "before", "shutdown", s.stopListening
-                    )
+                    reactor.addSystemEventTrigger("before", "shutdown", s.stopListening)
             elif listener.type == "metrics":
                 if not self.config.metrics.enable_metrics:
                     logger.warning(
@@ -327,13 +318,14 @@ class SynapseHomeServer(HomeServer):
                 logger.warning("Unrecognized listener type: %s", listener.type)
 
 
-def setup(config_options: List[str]) -> SynapseHomeServer:
+def setup(config_options):
     """
     Args:
-        config_options_options: The options passed to Synapse. Usually `sys.argv[1:]`.
+        config_options_options: The options passed to Synapse. Usually
+            `sys.argv[1:]`.
 
     Returns:
-        A homeserver instance.
+        HomeServer
     """
     try:
         config = HomeServerConfig.load_or_generate_config(
@@ -372,7 +364,7 @@ def setup(config_options: List[str]) -> SynapseHomeServer:
     except Exception as e:
         handle_startup_exception(e)
 
-    async def start() -> None:
+    async def start():
         # Load the OIDC provider metadatas, if OIDC is enabled.
         if hs.config.oidc.oidc_enabled:
             oidc = hs.get_oidc_handler()
@@ -412,15 +404,39 @@ def format_config_error(e: ConfigError) -> Iterator[str]:
 
     yield ":\n  %s" % (e.msg,)
 
-    parent_e = e.__cause__
+    e = e.__cause__
     indent = 1
-    while parent_e:
+    while e:
         indent += 1
-        yield ":\n%s%s" % ("  " * indent, str(parent_e))
-        parent_e = parent_e.__cause__
+        yield ":\n%s%s" % ("  " * indent, str(e))
+        e = e.__cause__
 
 
-def run(hs: HomeServer) -> None:
+def run(hs: HomeServer):
+    PROFILE_SYNAPSE = False
+    if PROFILE_SYNAPSE:
+
+        def profile(func):
+            from cProfile import Profile
+            from threading import current_thread
+
+            def profiled(*args, **kargs):
+                profile = Profile()
+                profile.enable()
+                func(*args, **kargs)
+                profile.disable()
+                ident = current_thread().ident
+                profile.dump_stats(
+                    "/tmp/%s.%s.%i.pstat" % (hs.hostname, func.__name__, ident)
+                )
+
+            return profiled
+
+        from twisted.python.threadpool import ThreadPool
+
+        ThreadPool._worker = profile(ThreadPool._worker)
+        reactor.run = profile(reactor.run)
+
     _base.start_reactor(
         "synapse-homeserver",
         soft_file_limit=hs.config.server.soft_file_limit,
@@ -432,7 +448,7 @@ def run(hs: HomeServer) -> None:
     )
 
 
-def main() -> None:
+def main():
     with LoggingContext("main"):
         # check base requirements
         check_requirements()

synapse/app/phone_stats_home.py

@@ -15,12 +15,11 @@ import logging
 import math
 import resource
 import sys
-from typing import TYPE_CHECKING, List, Sized, Tuple
+from typing import TYPE_CHECKING
 
 from prometheus_client import Gauge
 
 from synapse.metrics.background_process_metrics import wrap_as_background_process
-from synapse.types import JsonDict
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -29,7 +28,7 @@ logger = logging.getLogger("synapse.app.homeserver")
 
 # Contains the list of processes we will be monitoring
 # currently either 0 or 1
-_stats_process: List[Tuple[int, "resource.struct_rusage"]] = []
+_stats_process = []
 
 # Gauges to expose monthly active user control metrics
 current_mau_gauge = Gauge("synapse_admin_mau:current", "Current MAU")
@@ -46,15 +45,9 @@ registered_reserved_users_mau_gauge = Gauge(
 
 
 @wrap_as_background_process("phone_stats_home")
-async def phone_stats_home(
-    hs: "HomeServer",
-    stats: JsonDict,
-    stats_process: List[Tuple[int, "resource.struct_rusage"]] = _stats_process,
-) -> None:
+async def phone_stats_home(hs: "HomeServer", stats, stats_process=_stats_process):
     logger.info("Gathering stats for reporting")
     now = int(hs.get_clock().time())
-    # Ensure the homeserver has started.
-    assert hs.start_time is not None
     uptime = int(now - hs.start_time)
     if uptime < 0:
         uptime = 0
@@ -153,15 +146,15 @@ async def phone_stats_home(
         logger.warning("Error reporting stats: %s", e)
 
 
-def start_phone_stats_home(hs: "HomeServer") -> None:
+def start_phone_stats_home(hs: "HomeServer"):
     """
     Start the background tasks which report phone home stats.
     """
     clock = hs.get_clock()
 
-    stats: JsonDict = {}
+    stats = {}
 
-    def performance_stats_init() -> None:
+    def performance_stats_init():
         _stats_process.clear()
         _stats_process.append(
             (int(hs.get_clock().time()), resource.getrusage(resource.RUSAGE_SELF))
@@ -177,10 +170,10 @@ def start_phone_stats_home(hs: "HomeServer") -> None:
     hs.get_datastore().reap_monthly_active_users()
 
     @wrap_as_background_process("generate_monthly_active_users")
-    async def generate_monthly_active_users() -> None:
+    async def generate_monthly_active_users():
         current_mau_count = 0
         current_mau_count_by_service = {}
-        reserved_users: Sized = ()
+        reserved_users = ()
         store = hs.get_datastore()
         if hs.config.server.limit_usage_by_mau or hs.config.server.mau_stats_only:
             current_mau_count = await store.get_monthly_active_count()

synapse/appservice/api.py

@@ -231,13 +231,32 @@ class ApplicationServiceApi(SimpleHttpClient):
                 json_body=body,
                 args={"access_token": service.hs_token},
             )
+            if logger.isEnabledFor(logging.DEBUG):
+                logger.debug(
+                    "push_bulk to %s succeeded! events=%s",
+                    uri,
+                    [event.get("event_id") for event in events],
+                )
             sent_transactions_counter.labels(service.id).inc()
             sent_events_counter.labels(service.id).inc(len(events))
             return True
         except CodeMessageException as e:
-            logger.warning("push_bulk to %s received %s", uri, e.code)
+            logger.warning(
+                "push_bulk to %s received code=%s msg=%s",
+                uri,
+                e.code,
+                e.msg,
+                exc_info=logger.isEnabledFor(logging.DEBUG),
+            )
         except Exception as ex:
-            logger.warning("push_bulk to %s threw exception %s", uri, ex)
+            logger.warning(
+                "push_bulk to %s threw exception(%s) %s args=%s",
+                uri,
+                type(ex).__name__,
+                ex,
+                ex.args,
+                exc_info=logger.isEnabledFor(logging.DEBUG),
+            )
         failed_transactions_counter.labels(service.id).inc()
         return False
 

synapse/config/account_validity.py

@@ -75,6 +75,10 @@ class AccountValidityConfig(Config):
                 self.account_validity_period * 10.0 / 100.0
             )
 
+        if self.account_validity_renew_by_email_enabled:
+            if not self.root.server.public_baseurl:
+                raise ConfigError("Can't send renewal emails without 'public_baseurl'")
+
         # Load account validity templates.
         account_validity_template_dir = account_validity_config.get("template_dir")
         if account_validity_template_dir is not None:

synapse/config/cas.py

@@ -16,7 +16,7 @@ from typing import Any, List
 
 from synapse.config.sso import SsoAttributeRequirement
 
-from ._base import Config
+from ._base import Config, ConfigError
 from ._util import validate_config
 
 
@@ -35,10 +35,14 @@ class CasConfig(Config):
         if self.cas_enabled:
             self.cas_server_url = cas_config["server_url"]
 
-            # TODO Update this to a _synapse URL.
+            # The public baseurl is required because it is used by the redirect
+            # template.
             public_baseurl = self.root.server.public_baseurl
-            self.cas_service_url = public_baseurl + "_matrix/client/r0/login/cas/ticket"
+            if not public_baseurl:
+                raise ConfigError("cas_config requires a public_baseurl to be set")
 
+            # TODO Update this to a _synapse URL.
+            self.cas_service_url = public_baseurl + "_matrix/client/r0/login/cas/ticket"
             self.cas_displayname_attribute = cas_config.get("displayname_attribute")
             required_attributes = cas_config.get("required_attributes") or {}
             self.cas_required_attributes = _parsed_required_attributes_def(

synapse/config/emailconfig.py

@@ -186,6 +186,11 @@ class EmailConfig(Config):
             if not self.email_notif_from:
                 missing.append("email.notif_from")
 
+            # public_baseurl is required to build password reset and validation links that
+            # will be emailed to users
+            if config.get("public_baseurl") is None:
+                missing.append("public_baseurl")
+
             if missing:
                 raise ConfigError(
                     MISSING_PASSWORD_RESET_CONFIG_ERROR % (", ".join(missing),)
@@ -291,6 +296,9 @@ class EmailConfig(Config):
             if not self.email_notif_from:
                 missing.append("email.notif_from")
 
+            if config.get("public_baseurl") is None:
+                missing.append("public_baseurl")
+
             if missing:
                 raise ConfigError(
                     "email.enable_notifs is True but required keys are missing: %s"

synapse/config/oidc.py

@@ -59,6 +59,8 @@ class OIDCConfig(Config):
                 )
 
         public_baseurl = self.root.server.public_baseurl
+        if public_baseurl is None:
+            raise ConfigError("oidc_config requires a public_baseurl to be set")
         self.oidc_callback_url = public_baseurl + "_synapse/client/oidc/callback"
 
     @property

synapse/config/registration.py

@@ -45,6 +45,17 @@ class RegistrationConfig(Config):
         account_threepid_delegates = config.get("account_threepid_delegates") or {}
         self.account_threepid_delegate_email = account_threepid_delegates.get("email")
         self.account_threepid_delegate_msisdn = account_threepid_delegates.get("msisdn")
+        if (
+            self.account_threepid_delegate_msisdn
+            and not self.root.server.public_baseurl
+        ):
+            raise ConfigError(
+                "The configuration option `public_baseurl` is required if "
+                "`account_threepid_delegate.msisdn` is set, such that "
+                "clients know where to submit validation tokens to. Please "
+                "configure `public_baseurl`."
+            )
+
         self.default_identity_server = config.get("default_identity_server")
         self.allow_guest_access = config.get("allow_guest_access", False)
 
@@ -229,7 +240,7 @@ class RegistrationConfig(Config):
         # in on this server.
         #
         # (By default, no suggestion is made, so it is left up to the client.
-        # This setting is ignored unless public_baseurl is also explicitly set.)
+        # This setting is ignored unless public_baseurl is also set.)
         #
         #default_identity_server: https://matrix.org
 
@@ -254,6 +265,8 @@ class RegistrationConfig(Config):
         # by the Matrix Identity Service API specification:
         # https://matrix.org/docs/spec/identity_service/latest
         #
+        # If a delegate is specified, the config option public_baseurl must also be filled out.
+        #
         account_threepid_delegates:
             #email: https://example.com     # Delegate email sending to example.com
             #msisdn: http://localhost:8090  # Delegate SMS sending to this local process

synapse/config/saml2.py

@@ -199,11 +199,14 @@ class SAML2Config(Config):
         """
         import saml2
 
+        public_baseurl = self.root.server.public_baseurl
+        if public_baseurl is None:
+            raise ConfigError("saml2_config requires a public_baseurl to be set")
+
         if self.saml2_grandfathered_mxid_source_attribute:
             optional_attributes.add(self.saml2_grandfathered_mxid_source_attribute)
         optional_attributes -= required_attributes
 
-        public_baseurl = self.root.server.public_baseurl
         metadata_url = public_baseurl + "_synapse/client/saml2/metadata.xml"
         response_url = public_baseurl + "_synapse/client/saml2/authn_response"
         return {

synapse/config/server.py

@@ -16,7 +16,6 @@ import itertools
 import logging
 import os.path
 import re
-import urllib.parse
 from textwrap import indent
 from typing import Any, Dict, Iterable, List, Optional, Set, Tuple, Union
 
@@ -265,44 +264,10 @@ class ServerConfig(Config):
         self.use_frozen_dicts = config.get("use_frozen_dicts", False)
         self.serve_server_wellknown = config.get("serve_server_wellknown", False)
 
-        # Whether we should serve a "client well-known":
-        #  (a) at .well-known/matrix/client on our client HTTP listener
-        #  (b) in the response to /login
-        #
-        # ... which together help ensure that clients use our public_baseurl instead of
-        # whatever they were told by the user.
-        #
-        # For the sake of backwards compatibility with existing installations, this is
-        # True if public_baseurl is specified explicitly, and otherwise False. (The
-        # reasoning here is that we have no way of knowing that the default
-        # public_baseurl is actually correct for existing installations - many things
-        # will not work correctly, but that's (probably?) better than sending clients
-        # to a completely broken URL.
-        self.serve_client_wellknown = False
-
-        public_baseurl = config.get("public_baseurl")
-        if public_baseurl is None:
-            public_baseurl = f"https://{self.server_name}/"
-            logger.info("Using default public_baseurl %s", public_baseurl)
-        else:
-            self.serve_client_wellknown = True
-            if public_baseurl[-1] != "/":
-                public_baseurl += "/"
-        self.public_baseurl = public_baseurl
-
-        # check that public_baseurl is valid
-        try:
-            splits = urllib.parse.urlsplit(self.public_baseurl)
-        except Exception as e:
-            raise ConfigError(f"Unable to parse URL: {e}", ("public_baseurl",))
-        if splits.scheme not in ("https", "http"):
-            raise ConfigError(
-                f"Invalid scheme '{splits.scheme}': only https and http are supported"
-            )
-        if splits.query or splits.fragment:
-            raise ConfigError(
-                "public_baseurl cannot contain query parameters or a #-fragment"
-            )
+        self.public_baseurl = config.get("public_baseurl")
+        if self.public_baseurl is not None:
+            if self.public_baseurl[-1] != "/":
+                self.public_baseurl += "/"
 
         # Whether to enable user presence.
         presence_config = config.get("presence") or {}
@@ -808,8 +773,6 @@ class ServerConfig(Config):
         # Otherwise, it should be the URL to reach Synapse's client HTTP listener (see
         # 'listeners' below).
         #
-        # Defaults to 'https://<server_name>/'.
-        #
         #public_baseurl: https://example.com/
 
         # Uncomment the following to tell other servers to send federation traffic on

synapse/config/sso.py

@@ -101,10 +101,13 @@ class SSOConfig(Config):
         # gracefully to the client). This would make it pointless to ask the user for
         # confirmation, since the URL the confirmation page would be showing wouldn't be
         # the client's.
-        login_fallback_url = (
-            self.root.server.public_baseurl + "_matrix/static/client/login"
-        )
-        self.sso_client_whitelist.append(login_fallback_url)
+        # public_baseurl is an optional setting, so we only add the fallback's URL to the
+        # list if it's provided (because we can't figure out what that URL is otherwise).
+        if self.root.server.public_baseurl:
+            login_fallback_url = (
+                self.root.server.public_baseurl + "_matrix/static/client/login"
+            )
+            self.sso_client_whitelist.append(login_fallback_url)
 
     def generate_config_section(self, **kwargs):
         return """\
@@ -125,10 +128,11 @@ class SSOConfig(Config):
             # phishing attacks from evil.site. To avoid this, include a slash after the
             # hostname: "https://my.client/".
             #
-            # The login fallback page (used by clients that don't natively support the
-            # required login flows) is whitelisted in addition to any URLs in this list.
+            # If public_baseurl is set, then the login fallback page (used by clients
+            # that don't natively support the required login flows) is whitelisted in
+            # addition to any URLs in this list.
             #
-            # By default, this list contains only the login fallback page.
+            # By default, this list is empty.
             #
             #client_whitelist:
             #  - https://riot.im/develop

synapse/config/workers.py

@@ -63,8 +63,7 @@ class WriterLocations:
 
     Attributes:
         events: The instances that write to the event and backfill streams.
-        typing: The instances that write to the typing stream. Currently
-            can only be a single instance.
+        typing: The instance that writes to the typing stream.
         to_device: The instances that write to the to_device stream. Currently
             can only be a single instance.
         account_data: The instances that write to the account data streams. Currently
@@ -76,15 +75,9 @@ class WriterLocations:
     """
 
     events = attr.ib(
-        default=["master"],
-        type=List[str],
-        converter=_instance_to_list_converter,
-    )
-    typing = attr.ib(
-        default=["master"],
-        type=List[str],
-        converter=_instance_to_list_converter,
+        default=["master"], type=List[str], converter=_instance_to_list_converter
     )
+    typing = attr.ib(default="master", type=str)
     to_device = attr.ib(
         default=["master"],
         type=List[str],
@@ -224,11 +217,6 @@ class WorkerConfig(Config):
                         % (instance, stream)
                     )
 
-        if len(self.writers.typing) != 1:
-            raise ConfigError(
-                "Must only specify one instance to handle `typing` messages."
-            )
-
         if len(self.writers.to_device) != 1:
             raise ConfigError(
                 "Must only specify one instance to handle `to_device` messages."

synapse/federation/federation_client.py

@@ -277,58 +277,6 @@ class FederationClient(FederationBase):
 
         return pdus
 
-    async def get_pdu_from_destination_raw(
-        self,
-        destination: str,
-        event_id: str,
-        room_version: RoomVersion,
-        outlier: bool = False,
-        timeout: Optional[int] = None,
-    ) -> Optional[EventBase]:
-        """Requests the PDU with given origin and ID from the remote home
-        server. Does not have any caching or rate limiting!
-
-        Args:
-            destination: Which homeserver to query
-            event_id: event to fetch
-            room_version: version of the room
-            outlier: Indicates whether the PDU is an `outlier`, i.e. if
-                it's from an arbitrary point in the context as opposed to part
-                of the current block of PDUs. Defaults to `False`
-            timeout: How long to try (in ms) each destination for before
-                moving to the next destination. None indicates no timeout.
-
-        Returns:
-            The requested PDU, or None if we were unable to find it.
-
-        Raises:
-            SynapseError, NotRetryingDestination, FederationDeniedError
-        """
-        transaction_data = await self.transport_layer.get_event(
-            destination, event_id, timeout=timeout
-        )
-
-        logger.debug(
-            "retrieved event id %s from %s: %r",
-            event_id,
-            destination,
-            transaction_data,
-        )
-
-        pdu_list: List[EventBase] = [
-            event_from_pdu_json(p, room_version, outlier=outlier)
-            for p in transaction_data["pdus"]
-        ]
-
-        if pdu_list and pdu_list[0]:
-            pdu = pdu_list[0]
-
-            # Check signatures are correct.
-            signed_pdu = await self._check_sigs_and_hash(room_version, pdu)
-            return signed_pdu
-
-        return None
-
     async def get_pdu(
         self,
         destinations: Iterable[str],
@@ -373,14 +321,30 @@ class FederationClient(FederationBase):
                 continue
 
             try:
-                signed_pdu = await self.get_pdu_from_destination_raw(
-                    destination=destination,
-                    event_id=event_id,
-                    room_version=room_version,
-                    outlier=outlier,
-                    timeout=timeout,
+                transaction_data = await self.transport_layer.get_event(
+                    destination, event_id, timeout=timeout
                 )
 
+                logger.debug(
+                    "retrieved event id %s from %s: %r",
+                    event_id,
+                    destination,
+                    transaction_data,
+                )
+
+                pdu_list: List[EventBase] = [
+                    event_from_pdu_json(p, room_version, outlier=outlier)
+                    for p in transaction_data["pdus"]
+                ]
+
+                if pdu_list and pdu_list[0]:
+                    pdu = pdu_list[0]
+
+                    # Check signatures are correct.
+                    signed_pdu = await self._check_sigs_and_hash(room_version, pdu)
+
+                    break
+
                 pdu_attempts[destination] = now
 
             except SynapseError as e:

synapse/federation/federation_server.py

@@ -213,11 +213,6 @@ class FederationServer(FederationBase):
             self._started_handling_of_staged_events = True
             self._handle_old_staged_events()
 
-            # Start a periodic check for old staged events. This is to handle
-            # the case where locks time out, e.g. if another process gets killed
-            # without dropping its locks.
-            self._clock.looping_call(self._handle_old_staged_events, 60 * 1000)
-
         # keep this as early as possible to make the calculated origin ts as
         # accurate as possible.
         request_time = self._clock.time_msec()
@@ -1237,6 +1232,10 @@ class FederationHandlerRegistry:
 
         self.query_handlers[query_type] = handler
 
+    def register_instance_for_edu(self, edu_type: str, instance_name: str) -> None:
+        """Register that the EDU handler is on a different instance than master."""
+        self._edu_type_to_instance[edu_type] = [instance_name]
+
     def register_instances_for_edu(
         self, edu_type: str, instance_names: List[str]
     ) -> None:

synapse/handlers/admin.py

@@ -234,7 +234,7 @@ class ExfiltrationWriter(metaclass=abc.ABCMeta):
 
     @abc.abstractmethod
     def write_invite(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
+        self, room_id: str, event: EventBase, state: StateMap[dict]
     ) -> None:
         """Write an invite for the room, with associated invite state.
 
@@ -248,7 +248,7 @@ class ExfiltrationWriter(metaclass=abc.ABCMeta):
 
     @abc.abstractmethod
     def write_knock(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
+        self, room_id: str, event: EventBase, state: StateMap[dict]
     ) -> None:
         """Write a knock for the room, with associated knock state.
 

synapse/handlers/appservice.py

@@ -34,7 +34,6 @@ from synapse.metrics.background_process_metrics import (
 )
 from synapse.storage.databases.main.directory import RoomAliasMapping
 from synapse.types import JsonDict, RoomAlias, RoomStreamToken, UserID
-from synapse.util.async_helpers import Linearizer
 from synapse.util.metrics import Measure
 
 if TYPE_CHECKING:
@@ -59,10 +58,6 @@ class ApplicationServicesHandler:
         self.current_max = 0
         self.is_processing = False
 
-        self._ephemeral_events_linearizer = Linearizer(
-            name="appservice_ephemeral_events"
-        )
-
     def notify_interested_services(self, max_token: RoomStreamToken) -> None:
         """Notifies (pushes) all application services interested in this event.
 
@@ -188,7 +183,7 @@ class ApplicationServicesHandler:
         self,
         stream_key: str,
         new_token: Union[int, RoomStreamToken],
-        users: Collection[Union[str, UserID]],
+        users: Optional[Collection[Union[str, UserID]]] = None,
     ) -> None:
         """
         This is called by the notifier in the background when an ephemeral event is handled
@@ -203,9 +198,7 @@ class ApplicationServicesHandler:
                 value for `stream_key` will cause this function to return early.
 
                 Ephemeral events will only be pushed to appservices that have opted into
-                receiving them by setting `push_ephemeral` to true in their registration
-                file. Note that while MSC2409 is experimental, this option is called
-                `de.sorunome.msc2409.push_ephemeral`.
+                them.
 
                 Appservices will only receive ephemeral events that fall within their
                 registered user and room namespaces.
@@ -216,7 +209,6 @@ class ApplicationServicesHandler:
         if not self.notify_appservices:
             return
 
-        # Ignore any unsupported streams
         if stream_key not in ("typing_key", "receipt_key", "presence_key"):
             return
 
@@ -233,25 +225,18 @@ class ApplicationServicesHandler:
         # Additional context: https://github.com/matrix-org/synapse/pull/11137
         assert isinstance(new_token, int)
 
-        # Check whether there are any appservices which have registered to receive
-        # ephemeral events.
-        #
-        # Note that whether these events are actually relevant to these appservices
-        # is decided later on.
         services = [
             service
             for service in self.store.get_app_services()
             if service.supports_ephemeral
         ]
         if not services:
-            # Bail out early if none of the target appservices have explicitly registered
-            # to receive these ephemeral events.
             return
 
         # We only start a new background process if necessary rather than
         # optimistically (to cut down on overhead).
         self._notify_interested_services_ephemeral(
-            services, stream_key, new_token, users
+            services, stream_key, new_token, users or []
         )
 
     @wrap_as_background_process("notify_interested_services_ephemeral")
@@ -262,7 +247,7 @@ class ApplicationServicesHandler:
         new_token: int,
         users: Collection[Union[str, UserID]],
     ) -> None:
-        logger.debug("Checking interested services for %s", stream_key)
+        logger.debug("Checking interested services for %s" % (stream_key))
         with Measure(self.clock, "notify_interested_services_ephemeral"):
             for service in services:
                 if stream_key == "typing_key":
@@ -275,37 +260,26 @@ class ApplicationServicesHandler:
                     events = await self._handle_typing(service, new_token)
                     if events:
                         self.scheduler.submit_ephemeral_events_for_as(service, events)
-                    continue
 
-                # Since we read/update the stream position for this AS/stream
-                with (
-                    await self._ephemeral_events_linearizer.queue(
-                        (service.id, stream_key)
+                elif stream_key == "receipt_key":
+                    events = await self._handle_receipts(service)
+                    if events:
+                        self.scheduler.submit_ephemeral_events_for_as(service, events)
+
+                    # Persist the latest handled stream token for this appservice
+                    await self.store.set_type_stream_id_for_appservice(
+                        service, "read_receipt", new_token
                     )
-                ):
-                    if stream_key == "receipt_key":
-                        events = await self._handle_receipts(service, new_token)
-                        if events:
-                            self.scheduler.submit_ephemeral_events_for_as(
-                                service, events
-                            )
 
-                        # Persist the latest handled stream token for this appservice
-                        await self.store.set_type_stream_id_for_appservice(
-                            service, "read_receipt", new_token
-                        )
+                elif stream_key == "presence_key":
+                    events = await self._handle_presence(service, users)
+                    if events:
+                        self.scheduler.submit_ephemeral_events_for_as(service, events)
 
-                    elif stream_key == "presence_key":
-                        events = await self._handle_presence(service, users, new_token)
-                        if events:
-                            self.scheduler.submit_ephemeral_events_for_as(
-                                service, events
-                            )
-
-                        # Persist the latest handled stream token for this appservice
-                        await self.store.set_type_stream_id_for_appservice(
-                            service, "presence", new_token
-                        )
+                    # Persist the latest handled stream token for this appservice
+                    await self.store.set_type_stream_id_for_appservice(
+                        service, "presence", new_token
+                    )
 
     async def _handle_typing(
         self, service: ApplicationService, new_token: int
@@ -342,9 +316,7 @@ class ApplicationServicesHandler:
         )
         return typing
 
-    async def _handle_receipts(
-        self, service: ApplicationService, new_token: Optional[int]
-    ) -> List[JsonDict]:
+    async def _handle_receipts(self, service: ApplicationService) -> List[JsonDict]:
         """
         Return the latest read receipts that the given application service should receive.
 
@@ -355,9 +327,6 @@ class ApplicationServicesHandler:
 
         Args:
             service: The application service to check for which events it should receive.
-            new_token: A receipts event stream token. Purely used to double-check that the
-                from_token we pull from the database isn't greater than or equal to this
-                token. Prevents accidentally duplicating work.
 
         Returns:
             A list of JSON dictionaries containing data derived from the read receipts that
@@ -366,12 +335,6 @@ class ApplicationServicesHandler:
         from_key = await self.store.get_type_stream_id_for_appservice(
             service, "read_receipt"
         )
-        if new_token is not None and new_token <= from_key:
-            logger.debug(
-                "Rejecting token lower than or equal to stored: %s" % (new_token,)
-            )
-            return []
-
         receipts_source = self.event_sources.sources.receipt
         receipts, _ = await receipts_source.get_new_events_as(
             service=service, from_key=from_key
@@ -379,10 +342,7 @@ class ApplicationServicesHandler:
         return receipts
 
     async def _handle_presence(
-        self,
-        service: ApplicationService,
-        users: Collection[Union[str, UserID]],
-        new_token: Optional[int],
+        self, service: ApplicationService, users: Collection[Union[str, UserID]]
     ) -> List[JsonDict]:
         """
         Return the latest presence updates that the given application service should receive.
@@ -395,9 +355,6 @@ class ApplicationServicesHandler:
         Args:
             service: The application service that ephemeral events are being sent to.
             users: The users that should receive the presence update.
-            new_token: A presence update stream token. Purely used to double-check that the
-                from_token we pull from the database isn't greater than or equal to this
-                token. Prevents accidentally duplicating work.
 
         Returns:
             A list of json dictionaries containing data derived from the presence events
@@ -408,12 +365,6 @@ class ApplicationServicesHandler:
         from_key = await self.store.get_type_stream_id_for_appservice(
             service, "presence"
         )
-        if new_token is not None and new_token <= from_key:
-            logger.debug(
-                "Rejecting token lower than or equal to stored: %s" % (new_token,)
-            )
-            return []
-
         for user in users:
             if isinstance(user, str):
                 user = UserID.from_string(user)

synapse/handlers/auth.py

@@ -1989,9 +1989,7 @@ class PasswordAuthProvider:
         self,
         check_3pid_auth: Optional[CHECK_3PID_AUTH_CALLBACK] = None,
         on_logged_out: Optional[ON_LOGGED_OUT_CALLBACK] = None,
-        auth_checkers: Optional[
-            Dict[Tuple[str, Tuple[str, ...]], CHECK_AUTH_CALLBACK]
-        ] = None,
+        auth_checkers: Optional[Dict[Tuple[str, Tuple], CHECK_AUTH_CALLBACK]] = None,
     ) -> None:
         # Register check_3pid_auth callback
         if check_3pid_auth is not None: