maxmustermann/synapse
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Merge branch 'release-v1.45' into matrix-org-hotfixes

Browse source
This commit is contained in:
Brendan Abolivier 2021-10-12 10:21:26 +01:00
parent ebbd37b66d e0f11ae4a5
commit 6ce0dc0620
No known key found for this signature in database
GPG key ID: 1E015C145F1916CD
260 changed files with 4119 additions and 2179 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.ci/scripts/test_synapse_port_db.sh
View file

@ -25,7 +25,7 @@ python -m synapse.app.homeserver --generate-keys -c .ci/sqlite-config.yaml
echo "--- Prepare test database"
# Make sure the SQLite3 database is using the latest schema and has no pending background update.
scripts-dev/update_database --database-config .ci/sqlite-config.yaml
scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
# Create the PostgreSQL database.
.ci/scripts/postgres_exec.py "CREATE DATABASE synapse"
@ -46,7 +46,7 @@ echo "--- Prepare empty SQLite database"
# we do this by deleting the sqlite db, and then doing the same again.
rm .ci/test_db.db
scripts-dev/update_database --database-config .ci/sqlite-config.yaml
scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
# re-create the PostgreSQL database.
.ci/scripts/postgres_exec.py \

2
.github/CODEOWNERS vendored Normal file
View file

@ -0,0 +1,2 @@
# Automatically request reviews from the synapse-core team when a pull request comes in.
* @matrix-org/synapse-core

23
.github/workflows/tests.yml vendored
View file

@ -76,22 +76,25 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.6", "3.7", "3.8", "3.9"]
python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
database: ["sqlite"]
toxenv: ["py"]
include:
# Newest Python without optional deps
- python-version: "3.9"
toxenv: "py-noextras,combine"
- python-version: "3.10"
toxenv: "py-noextras"
# Oldest Python with PostgreSQL
- python-version: "3.6"
database: "postgres"
postgres-version: "9.6"
toxenv: "py"
# Newest Python with PostgreSQL
- python-version: "3.9"
# Newest Python with newest PostgreSQL
- python-version: "3.10"
database: "postgres"
postgres-version: "13"
postgres-version: "14"
toxenv: "py"
steps:
- uses: actions/checkout@v2
@ -111,7 +114,7 @@ jobs:
if: ${{ matrix.postgres-version }}
timeout-minutes: 2
run: until pg_isready -h localhost; do sleep 1; done
- run: tox -e py,combine
- run: tox -e ${{ matrix.toxenv }}
env:
TRIAL_FLAGS: "--jobs=2"
SYNAPSE_POSTGRES: ${{ matrix.database == 'postgres' || '' }}
@ -169,7 +172,7 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
- run: pip install tox
- run: tox -e py,combine
- run: tox -e py
env:
TRIAL_FLAGS: "--jobs=2"
- name: Dump logs
@ -256,8 +259,8 @@ jobs:
- python-version: "3.6"
postgres-version: "9.6"
- python-version: "3.9"
postgres-version: "13"
- python-version: "3.10"
postgres-version: "14"
services:
postgres:

16
CHANGES.md
View file

@ -1,3 +1,19 @@
Synapse 1.44.0 (2021-10-05)
===========================
No significant changes since 1.44.0rc3.
Synapse 1.44.0rc3 (2021-10-04)
==============================
Bugfixes
--------
- Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933))
- Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
Synapse 1.44.0rc2 (2021-09-30)
==============================

9
README.rst
View file

@ -55,11 +55,8 @@ solutions. The hope is for Matrix to act as the building blocks for a new
generation of fully open and interoperable messaging and VoIP apps for the
internet.
Synapse is a reference "homeserver" implementation of Matrix from the core
development team at matrix.org, written in Python/Twisted. It is intended to
showcase the concept of Matrix and let folks see the spec in the context of a
codebase and let you run your own homeserver and generally help bootstrap the
ecosystem.
Synapse is a Matrix "homeserver" implementation developed by the matrix.org core
team, written in Python 3/Twisted.
In Matrix, every user runs one or more Matrix clients, which connect through to
a Matrix homeserver. The homeserver stores all their personal chat history and
@ -301,7 +298,7 @@ to install using pip and a virtualenv::
python3 -m venv ./env
source ./env/bin/activate
pip install -e ".[all,test]"
pip install -e ".[all,dev]"
This will run a process of downloading and installing all the needed
dependencies into a virtual env. If any dependencies fail to install,

1
changelog.d/10822.feature Normal file
View file

@ -0,0 +1 @@
Support autodiscovery of oEmbed previews.

1
changelog.d/10877.feature Normal file
View file

@ -0,0 +1 @@
Ensure `(room_id, next_batch_id)` is unique across [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms to avoid cross-talk/conflicts between batches.

1
changelog.d/10888.misc Normal file
View file

@ -0,0 +1 @@
Improve type hinting in `synapse.util`.

1
changelog.d/10892.misc Normal file
View file

@ -0,0 +1 @@
Add further type hints to `synapse.storage.util`.

1
changelog.d/10894.feature Normal file
View file

@ -0,0 +1 @@
Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites.

1
changelog.d/10895.misc Normal file
View file

@ -0,0 +1 @@
Fix type hints to be compatible with an upcoming change to Twisted.

1
changelog.d/10902.misc Normal file
View file

@ -0,0 +1 @@
Update utility code to handle C implementations of frozendict.

1
changelog.d/10903.misc Normal file
View file

@ -0,0 +1 @@
Drop old functionality which maintained database compatibility with Synapse versions before 1.31.

1
changelog.d/10910.feature Normal file
View file

@ -0,0 +1 @@
Add a spam checker callback to allow or deny room joins.

1
changelog.d/10915.misc Normal file
View file

@ -0,0 +1 @@
Clean-up configuration helper classes for the `ServerConfig` class.

1
changelog.d/10916.misc Normal file
View file

@ -0,0 +1 @@
Use direct references to config flags.

1
changelog.d/10922.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki.

1
changelog.d/10924.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory.

1
changelog.d/10926.misc Normal file
View file

@ -0,0 +1 @@
Clean up some of the federation event authentication code for clarity.

1
changelog.d/10927.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8/9 could be applied to earlier room versions in some situations.

1
changelog.d/10933.bugfix
View file

@ -1 +0,0 @@
Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error.

1
changelog.d/10934.misc Normal file
View file

@ -0,0 +1 @@
Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings.

1
changelog.d/10935.misc Normal file
View file

@ -0,0 +1 @@
Refactor user directory tests in preparation for upcoming changes.

1
changelog.d/10936.misc Normal file
View file

@ -0,0 +1 @@
Include the event id in the logcontext when handling PDUs received over federation.

1
changelog.d/10939.misc Normal file
View file

@ -0,0 +1 @@
Fix logged errors in unit tests.

1
changelog.d/10940.misc Normal file
View file

@ -0,0 +1 @@
Clean up some of the federation event authentication code for clarity.

1
changelog.d/10945.misc Normal file
View file

@ -0,0 +1 @@
Fix a broken test to ensure that consent configuration works during registration.

1
changelog.d/10947.bugfix Normal file
View file

@ -0,0 +1 @@
Fixes a long-standing bug wherin deactivated users still count towards the mau limit.

1
changelog.d/10954.feature Normal file
View file

@ -0,0 +1 @@
Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper.

1
changelog.d/10956.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.

1
changelog.d/10958.misc Normal file
View file

@ -0,0 +1 @@
Add type hints to filtering classes.

1
changelog.d/10959.misc Normal file
View file

@ -0,0 +1 @@
Use direct references to config flags.

1
changelog.d/10960.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and disabled users.

1
changelog.d/10961.misc Normal file
View file

@ -0,0 +1 @@
Add type-hint to `HomeserverTestcase.setup_test_homeserver`.

1
changelog.d/10962.bugfix Normal file
View file

@ -0,0 +1 @@
Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator.

1
changelog.d/10963.misc Normal file
View file

@ -0,0 +1 @@
Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr.

1
changelog.d/10966.misc Normal file
View file

@ -0,0 +1 @@
Make the release script more robust and transparent.

1
changelog.d/10968.bugfix
View file

@ -1 +0,0 @@
Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1.

1
changelog.d/10971.doc Normal file
View file

@ -0,0 +1 @@
Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz.

1
changelog.d/10973.doc Normal file
View file

@ -0,0 +1 @@
Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz.

1
changelog.d/10974.misc Normal file
View file

@ -0,0 +1 @@
Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` mega function into smaller handler functions.

1
changelog.d/10981.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt.

1
changelog.d/10982.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch.

1
changelog.d/10983.misc Normal file
View file

@ -0,0 +1 @@
Log stack traces when a missing opentracing span is detected.

1
changelog.d/10985.misc Normal file
View file

@ -0,0 +1 @@
Use direct references to config flags.

1
changelog.d/10986.misc Normal file
View file

@ -0,0 +1 @@
Clean up some of the federation event authentication code for clarity.

1
changelog.d/10987.misc Normal file
View file

@ -0,0 +1 @@
Clean up some of the federation event authentication code for clarity.

1
changelog.d/10988.misc Normal file
View file

@ -0,0 +1 @@
Clean up some of the federation event authentication code for clarity.

1
changelog.d/10990.doc Normal file
View file

@ -0,0 +1 @@
Add additional content to the Welcome and Overview page of the documentation.

1
changelog.d/10991.doc Normal file
View file

@ -0,0 +1 @@
Update links to MSCs in documentation. Contributed by @dklimpel.

1
changelog.d/10992.misc Normal file
View file

@ -0,0 +1 @@
Update GHA config to run tests against Python 3.10 and PostgreSQL 14.

1
changelog.d/10993.misc Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit.

1
changelog.d/10994.misc Normal file
View file

@ -0,0 +1 @@
Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests.

1
changelog.d/10995.bugfix Normal file
View file

@ -0,0 +1 @@
Correct a bugfix introduced in Synapse v1.44.0 that wouldn't catch every error of the connection breaks before a response could be written to it.

1
changelog.d/11002.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user_directory.

1
changelog.d/11003.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public.

1
changelog.d/11004.misc Normal file
View file

@ -0,0 +1 @@
Add further type hints to `synapse.state`.

1
changelog.d/11005.misc Normal file
View file

@ -0,0 +1 @@
Remove the deprecated `BaseHandler` object.

1
changelog.d/11006.misc Normal file
View file

@ -0,0 +1 @@
Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies.

1
changelog.d/11010.misc Normal file
View file

@ -0,0 +1 @@
Clean up some of the federation event authentication code for clarity.

1
changelog.d/11011.misc Normal file
View file

@ -0,0 +1 @@
Clean up some of the federation event authentication code for clarity.

1
changelog.d/11017.misc Normal file
View file

@ -0,0 +1 @@
Fix CI to run the unit tests without optional deps.

1
changelog.d/11019.misc Normal file
View file

@ -0,0 +1 @@
Ensure that cache config tests do not share state.

1
changelog.d/11021.misc Normal file
View file

@ -0,0 +1 @@
Add additional type hints to `synapse.server_notices`.

1
changelog.d/11023.misc Normal file
View file

@ -0,0 +1 @@
Add additional type hints for `synapse.push`.

1
changelog.d/11028.feature Normal file
View file

@ -0,0 +1 @@
Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper.

1
changelog.d/11034.misc Normal file
View file

@ -0,0 +1 @@
When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing.

1
changelog.d/11042.bugfix Normal file
View file

@ -0,0 +1 @@
Work around a regression, introduced in Synapse 1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored.

1
changelog.d/11043.misc Normal file
View file

@ -0,0 +1 @@
Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements.

1
changelog.d/9655.feature Normal file
View file

@ -0,0 +1 @@
Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`.

19
debian/changelog vendored
View file

@ -1,3 +1,22 @@
matrix-synapse-py3 (1.44.0~rc2+nmu1) UNRELEASED; urgency=medium
[ Nick @ Beeper ]
* Include an `update_synapse_database` script in the distribution.
-- root <root@f7b8a71098d3> Mon, 04 Oct 2021 13:29:26 +0000
matrix-synapse-py3 (1.44.0) stable; urgency=medium
* New synapse release 1.44.0.
-- Synapse Packaging team <packages@matrix.org> Tue, 05 Oct 2021 13:43:57 +0100
matrix-synapse-py3 (1.44.0~rc3) stable; urgency=medium
* New synapse release 1.44.0~rc3.
-- Synapse Packaging team <packages@matrix.org> Mon, 04 Oct 2021 14:57:22 +0100
matrix-synapse-py3 (1.44.0~rc2) stable; urgency=medium
* New synapse release 1.44.0~rc2.

1
debian/matrix-synapse-py3.links vendored
View file

@ -3,3 +3,4 @@ opt/venvs/matrix-synapse/bin/register_new_matrix_user usr/bin/register_new_matri
opt/venvs/matrix-synapse/bin/synapse_port_db usr/bin/synapse_port_db
opt/venvs/matrix-synapse/bin/synapse_review_recent_signups usr/bin/synapse_review_recent_signups
opt/venvs/matrix-synapse/bin/synctl usr/bin/synctl
opt/venvs/matrix-synapse/bin/update_synapse_database usr/bin/update_synapse_database

4
docs/MSC1711_certificates_FAQ.md
View file

@ -3,7 +3,7 @@
## Historical Note
This document was originally written to guide server admins through the upgrade
path towards Synapse 1.0. Specifically,
[MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md)
[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md)
required that all servers present valid TLS certificates on their federation
API. Admins were encouraged to achieve compliance from version 0.99.0 (released
in February 2019) ahead of version 1.0 (released June 2019) enforcing the
@ -282,7 +282,7 @@ coffin of the Perspectives project (which was already pretty dead). So, the
Spec Core Team decided that a better approach would be to mandate valid TLS
certificates for federation alongside the rest of the Web. More details can be
found in
[MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach).
[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach).
This results in a breaking change, which is disruptive, but absolutely critical
for the security model. However, the existence of Let's Encrypt as a trivial

6
docs/README.md
View file

@ -6,9 +6,9 @@ Please update any links to point to the new website instead.
## About
This directory currently holds a series of markdown files documenting how to install, use
and develop Synapse, the reference Matrix homeserver. The documentation is readable directly
from this repository, but it is recommended to instead browse through the
[website](https://matrix-org.github.io/synapse) for easier discoverability.
and develop Synapse. The documentation is readable directly from this repository, but it is
recommended to instead browse through the [website](https://matrix-org.github.io/synapse) for
easier discoverability.
## Adding to the documentation

4
docs/development/contributing_guide.md
View file

@ -50,7 +50,7 @@ setup a *virtualenv*, as follows:
cd path/where/you/have/cloned/the/repository
python3 -m venv ./env
source ./env/bin/activate
pip install -e ".[all,lint,mypy,test]"
pip install -e ".[all,dev]"
pip install tox
```
@ -63,7 +63,7 @@ TBD
# 5. Get in touch.
Join our developer community on Matrix: #synapse-dev:matrix.org !
Join our developer community on Matrix: [#synapse-dev:matrix.org](https://matrix.to/#/#synapse-dev:matrix.org)!
# 6. Pick an issue.

11
docs/development/saml.md
View file

@ -1,10 +1,9 @@
# How to test SAML as a developer without a server
https://capriza.github.io/samling/samling.html (https://github.com/capriza/samling) is a great
resource for being able to tinker with the SAML options within Synapse without needing to
deploy and configure a complicated software stack.
https://fujifish.github.io/samling/samling.html (https://github.com/fujifish/samling) is a great resource for being able to tinker with the
SAML options within Synapse without needing to deploy and configure a complicated software stack.
To make Synapse (and therefore Riot) use it:
To make Synapse (and therefore Element) use it:
1. Use the samling.html URL above or deploy your own and visit the IdP Metadata tab.
2. Copy the XML to your clipboard.
@ -26,9 +25,9 @@ To make Synapse (and therefore Riot) use it:
the dependencies are installed and ready to go.
7. Restart Synapse.
Then in Riot:
Then in Element:
1. Visit the login page with a Riot pointing at your homeserver.
1. Visit the login page and point Element towards your homeserver using the `public_baseurl` above.
2. Click the Single Sign-On button.
3. On the samling page, enter a Name Identifier and add a SAML Attribute for `uid=your_localpart`.
The response must also be signed.

50
docs/modules/spam_checker_callbacks.md
View file

@ -19,6 +19,21 @@ either a `bool` to indicate whether the event must be rejected because of spam,
to indicate the event must be rejected because of spam and to give a rejection reason to
forward to clients.
### `user_may_join_room`
```python
async def user_may_join_room(user: str, room: str, is_invited: bool) -> bool
```
Called when a user is trying to join a room. The module must return a `bool` to indicate
whether the user can join the room. The user is represented by their Matrix user ID (e.g.
`@alice:example.com`) and the room is represented by its Matrix ID (e.g.
`!room:example.com`). The module is also given a boolean to indicate whether the user
currently has a pending invite in the room.
This callback isn't called if the join is performed by a server administrator, or in the
context of a room creation.
### `user_may_invite`
```python
@ -29,6 +44,41 @@ Called when processing an invitation. The module must return a `bool` indicating
the inviter can invite the invitee to the given room. Both inviter and invitee are
represented by their Matrix user ID (e.g. `@alice:example.com`).
### `user_may_send_3pid_invite`
```python
async def user_may_send_3pid_invite(
inviter: str,
medium: str,
address: str,
room_id: str,
) -> bool
```
Called when processing an invitation using a third-party identifier (also called a 3PID,
e.g. an email address or a phone number). The module must return a `bool` indicating
whether the inviter can invite the invitee to the given room.
The inviter is represented by their Matrix user ID (e.g. `@alice:example.com`), and the
invitee is represented by its medium (e.g. "email") and its address
(e.g. `alice@example.com`). See [the Matrix specification](https://matrix.org/docs/spec/appendices#pid-types)
for more information regarding third-party identifiers.
For example, a call to this callback to send an invitation to the email address
`alice@example.com` would look like this:
```python
await user_may_send_3pid_invite(
"@bob:example.com", # The inviter's user ID
"email", # The medium of the 3PID to invite
"alice@example.com", # The address of the 3PID to invite
"!some_room:example.com", # The ID of the room to send the invite into
)
```
**Note**: If the third-party identifier is already associated with a matrix user ID,
[`user_may_invite`](#user_may_invite) will be used instead.
### `user_may_create_room`
```python

3
docs/usage/administration/admin_api/registration_tokens.md
View file

@ -1,7 +1,8 @@
# Registration Tokens
This API allows you to manage tokens which can be used to authenticate
registration requests, as proposed in [MSC3231](https://github.com/govynnus/matrix-doc/blob/token-registration/proposals/3231-token-authenticated-registration.md).
registration requests, as proposed in
[MSC3231](https://github.com/matrix-org/matrix-doc/blob/main/proposals/3231-token-authenticated-registration.md).
To use it, you will need to enable the `registration_requires_token` config
option, and authenticate by providing an `access_token` for a server admin:
see [Admin API](../../usage/administration/admin_api).

79
docs/welcome_and_overview.md
View file

@ -1,4 +1,79 @@
# Introduction
Welcome to the documentation repository for Synapse, the reference
[Matrix](https://matrix.org) homeserver implementation.
Welcome to the documentation repository for Synapse, a
[Matrix](https://matrix.org) homeserver implementation developed by the matrix.org core
team.
## Installing and using Synapse
This documentation covers topics for **installation**, **configuration** and
**maintainence** of your Synapse process:
* Learn how to [install](setup/installation.md) and
[configure](usage/configuration/index.html) your own instance, perhaps with [Single
Sign-On](usage/configuration/user_authentication/index.html).
* See how to [upgrade](upgrade.md) between Synapse versions.
* Administer your instance using the [Admin
API](usage/administration/admin_api/index.html), installing [pluggable
modules](modules/index.html), or by accessing the [manhole](manhole.md).
* Learn how to [read log lines](usage/administration/request_log.md), configure
[logging](usage/configuration/logging_sample_config.md) or set up [structured
logging](structured_logging.md).
* Scale Synapse through additional [worker processes](workers.md).
* Set up [monitoring and metrics](metrics-howto.md) to keep an eye on your
Synapse instance's performance.
## Developing on Synapse
Contributions are welcome! Synapse is primarily written in
[Python](https://python.org). As a developer, you may be interested in the
following documentation:
* Read the [Contributing Guide](development/contributing_guide.md). It is meant
to walk new contributors through the process of developing and submitting a
change to the Synapse codebase (which is [hosted on
GitHub](https://github.com/matrix-org/synapse)).
* Set up your [development
environment](development/contributing_guide.md#2-what-do-i-need), then learn
how to [lint](development/contributing_guide.md#run-the-linters) and
[test](development/contributing_guide.md#8-test-test-test) your code.
* Look at [the issue tracker](https://github.com/matrix-org/synapse/issues) for
bugs to fix or features to add. If you're new, it may be best to start with
those labeled [good first
issue](https://github.com/matrix-org/synapse/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
* Understand [how Synapse is
built](development/internal_documentation/index.html), how to [migrate
database schemas](development/database_schema.md), learn about
[federation](federate.md) and how to [set up a local
federation](federate.md#running-a-demo-federation-of-synapses) for development.
* We like to keep our `git` history clean. [Learn](development/git.md) how to
do so!
* And finally, contribute to this documentation! The source for which is
[located here](https://github.com/matrix-org/synapse/tree/develop/docs).
## Donating to Synapse development
Want to help keep Synapse going but don't know how to code? Synapse is a
[Matrix.org Foundation](https://matrix.org) project. Consider becoming a
supportor on [Liberapay](https://liberapay.com/matrixdotorg),
[Patreon](https://patreon.com/matrixdotorg) or through
[PayPal](https://paypal.me/matrixdotorg) via a one-time donation.
If you are an organisation or enterprise and would like to sponsor development,
reach out to us over email at: support (at) matrix.org
## Reporting a security vulnerability
If you've found a security issue in Synapse or any other Matrix.org Foundation
project, please report it to us in accordance with our [Security Disclosure
Policy](https://www.matrix.org/security-disclosure-policy/). Thank you!

178
mypy.ini
View file

@ -96,15 +96,48 @@ files =
[mypy-synapse.handlers.*]
disallow_untyped_defs = True
[mypy-synapse.push.*]
disallow_untyped_defs = True
[mypy-synapse.rest.*]
disallow_untyped_defs = True
[mypy-synapse.server_notices.*]
disallow_untyped_defs = True
[mypy-synapse.state.*]
disallow_untyped_defs = True
[mypy-synapse.storage.util.*]
disallow_untyped_defs = True
[mypy-synapse.streams.*]
disallow_untyped_defs = True
[mypy-synapse.util.batching_queue]
disallow_untyped_defs = True
[mypy-synapse.util.caches.cached_call]
disallow_untyped_defs = True
[mypy-synapse.util.caches.dictionary_cache]
disallow_untyped_defs = True
[mypy-synapse.util.caches.lrucache]
disallow_untyped_defs = True
[mypy-synapse.util.caches.response_cache]
disallow_untyped_defs = True
[mypy-synapse.util.caches.stream_change_cache]
disallow_untyped_defs = True
[mypy-synapse.util.caches.ttl_cache]
disallow_untyped_defs = True
[mypy-synapse.util.daemonize]
disallow_untyped_defs = True
[mypy-synapse.util.file_consumer]
disallow_untyped_defs = True
@ -141,6 +174,9 @@ disallow_untyped_defs = True
[mypy-synapse.util.msisdn]
disallow_untyped_defs = True
[mypy-synapse.util.patch_inline_callbacks]
disallow_untyped_defs = True
[mypy-synapse.util.ratelimitutils]
disallow_untyped_defs = True
@ -162,98 +198,106 @@ disallow_untyped_defs = True
[mypy-synapse.util.wheel_timer]
disallow_untyped_defs = True
[mypy-pymacaroons.*]
ignore_missing_imports = True
[mypy-synapse.util.versionstring]
disallow_untyped_defs = True
[mypy-zope]
[mypy-tests.handlers.test_user_directory]
disallow_untyped_defs = True
[mypy-tests.storage.test_user_directory]
disallow_untyped_defs = True
;; Dependencies without annotations
;; Before ignoring a module, check to see if type stubs are available.
;; The `typeshed` project maintains stubs here:
;; https://github.com/python/typeshed/tree/master/stubs
;; and for each package `foo` there's a corresponding `types-foo` package on PyPI,
;; which we can pull in as a dev dependency by adding to `setup.py`'s
;; `CONDITIONAL_REQUIREMENTS["mypy"]` list.
[mypy-authlib.*]
ignore_missing_imports = True
[mypy-bcrypt]
ignore_missing_imports = True
[mypy-constantly]
ignore_missing_imports = True
[mypy-twisted.*]
ignore_missing_imports = True
[mypy-treq.*]
ignore_missing_imports = True
[mypy-hyperlink]
ignore_missing_imports = True
[mypy-h11]
ignore_missing_imports = True
[mypy-msgpack]
ignore_missing_imports = True
[mypy-opentracing]
ignore_missing_imports = True
[mypy-OpenSSL.*]
ignore_missing_imports = True
[mypy-netaddr]
ignore_missing_imports = True
[mypy-saml2.*]
ignore_missing_imports = True
[mypy-canonicaljson]
ignore_missing_imports = True
[mypy-jaeger_client.*]
ignore_missing_imports = True
[mypy-jsonschema]
ignore_missing_imports = True
[mypy-signedjson.*]
ignore_missing_imports = True
[mypy-prometheus_client.*]
ignore_missing_imports = True
[mypy-service_identity.*]
[mypy-constantly]
ignore_missing_imports = True
[mypy-daemonize]
ignore_missing_imports = True
[mypy-sentry_sdk]
ignore_missing_imports = True
[mypy-PIL.*]
ignore_missing_imports = True
[mypy-lxml]
ignore_missing_imports = True
[mypy-jwt.*]
ignore_missing_imports = True
[mypy-authlib.*]
ignore_missing_imports = True
[mypy-rust_python_jaeger_reporter.*]
ignore_missing_imports = True
[mypy-nacl.*]
[mypy-h11]
ignore_missing_imports = True
[mypy-hiredis]
ignore_missing_imports = True
[mypy-hyperlink]
ignore_missing_imports = True
[mypy-ijson.*]
ignore_missing_imports = True
[mypy-jaeger_client.*]
ignore_missing_imports = True
[mypy-josepy.*]
ignore_missing_imports = True
[mypy-pympler.*]
[mypy-jwt.*]
ignore_missing_imports = True
[mypy-lxml]
ignore_missing_imports = True
[mypy-msgpack]
ignore_missing_imports = True
[mypy-nacl.*]
ignore_missing_imports = True
[mypy-netaddr]
ignore_missing_imports = True
[mypy-opentracing]
ignore_missing_imports = True
[mypy-phonenumbers.*]
ignore_missing_imports = True
[mypy-ijson.*]
[mypy-prometheus_client.*]
ignore_missing_imports = True
[mypy-pymacaroons.*]
ignore_missing_imports = True
[mypy-pympler.*]
ignore_missing_imports = True
[mypy-rust_python_jaeger_reporter.*]
ignore_missing_imports = True
[mypy-saml2.*]
ignore_missing_imports = True
[mypy-sentry_sdk]
ignore_missing_imports = True
[mypy-service_identity.*]
ignore_missing_imports = True
[mypy-signedjson.*]
ignore_missing_imports = True
[mypy-treq.*]
ignore_missing_imports = True
[mypy-twisted.*]
ignore_missing_imports = True
[mypy-zope]
ignore_missing_imports = True

2
scripts-dev/lint.sh
View file

@ -90,10 +90,10 @@ else
"scripts/hash_password"
"scripts/register_new_matrix_user"
"scripts/synapse_port_db"
"scripts/update_synapse_database"
"scripts-dev"
"scripts-dev/build_debian_packages"
"scripts-dev/sign_json"
"scripts-dev/update_database"
"contrib" "synctl" "setup.py" "synmark" "stubs" ".ci"
)
fi

2
scripts-dev/make_full_schema.sh
View file

@ -147,7 +147,7 @@ python -m synapse.app.homeserver --generate-keys -c "$SQLITE_CONFIG"
# Make sure the SQLite3 database is using the latest schema and has no pending background update.
echo "Running db background jobs..."
scripts-dev/update_database --database-config "$SQLITE_CONFIG"
scripts/update_synapse_database --database-config --run-background-updates "$SQLITE_CONFIG"
# Create the PostgreSQL database.
echo "Creating postgres database..."

36
scripts-dev/release.py
View file

@ -35,6 +35,19 @@ from github import Github
from packaging import version
def run_until_successful(command, *args, **kwargs):
while True:
completed_process = subprocess.run(command, *args, **kwargs)
exit_code = completed_process.returncode
if exit_code == 0:
# successful, so nothing more to do here.
return completed_process
print(f"The command {command!r} failed with exit code {exit_code}.")
print("Please try to correct the failure and then re-run.")
click.confirm("Try again?", abort=True)
@click.group()
def cli():
"""An interactive script to walk through the parts of creating a release.
@ -197,7 +210,7 @@ def prepare():
f.write(parsed_synapse_ast.dumps())
# Generate changelogs
subprocess.run("python3 -m towncrier", shell=True)
run_until_successful("python3 -m towncrier", shell=True)
# Generate debian changelogs
if parsed_new_version.pre is not None:
@ -209,11 +222,11 @@ def prepare():
else:
debian_version = new_version
subprocess.run(
run_until_successful(
f'dch -M -v {debian_version} "New synapse release {debian_version}."',
shell=True,
)
subprocess.run('dch -M -r -D stable ""', shell=True)
run_until_successful('dch -M -r -D stable ""', shell=True)
# Show the user the changes and ask if they want to edit the change log.
repo.git.add("-u")
@ -224,7 +237,7 @@ def prepare():
# Commit the changes.
repo.git.add("-u")
repo.git.commit(f"-m {new_version}")
repo.git.commit("-m", new_version)
# We give the option to bail here in case the user wants to make sure things
# are OK before pushing.
@ -239,6 +252,8 @@ def prepare():
# Otherwise, push and open the changelog in the browser.
repo.git.push("-u", repo.remote().name, repo.active_branch.name)
print("Opening the changelog in your browser...")
print("Please ask others to give it a check.")
click.launch(
f"https://github.com/matrix-org/synapse/blob/{repo.active_branch.name}/CHANGES.md"
)
@ -290,7 +305,19 @@ def tag(gh_token: Optional[str]):
# If no token was given, we bail here
if not gh_token:
print("Launching the GitHub release page in your browser.")
print("Please correct the title and create a draft.")
if current_version.is_prerelease:
print("As this is an RC, remember to mark it as a pre-release!")
print("(by the way, this step can be automated by passing --gh-token,")
print("or one of the GH_TOKEN or GITHUB_TOKEN env vars.)")
click.launch(f"https://github.com/matrix-org/synapse/releases/edit/{tag_name}")
print("Once done, you need to wait for the release assets to build.")
if click.confirm("Launch the release assets actions page?", default=True):
click.launch(
f"https://github.com/matrix-org/synapse/actions?query=branch%3A{tag_name}"
)
return
# Create a new draft release
@ -305,6 +332,7 @@ def tag(gh_token: Optional[str]):
)
# Open the release and the actions where we are building the assets.
print("Launching the release page and the actions page.")
click.launch(release.html_url)
click.launch(
f"https://github.com/matrix-org/synapse/actions?query=branch%3A{tag_name}"

6
scripts/synapse_port_db
View file

@ -215,7 +215,7 @@ class MockHomeserver:
def __init__(self, config):
self.clock = Clock(reactor)
self.config = config
self.hostname = config.server_name
self.hostname = config.server.server_name
self.version_string = "Synapse/" + get_version_string(synapse)
def get_clock(self):
@ -583,7 +583,7 @@ class Porter(object):
return
self.postgres_store = self.build_db_store(
self.hs_config.get_single_database()
self.hs_config.database.get_single_database()
)
await self.run_background_updates_on_postgres()
@ -1069,7 +1069,7 @@ class CursesProgress(Progress):
self.stdscr.addstr(0, 0, status, curses.A_BOLD)
max_len = max([len(t) for t in self.tables.keys()])
max_len = max(len(t) for t in self.tables.keys())
left_margin = 5
middle_space = 1

50
scripts-dev/update_database → scripts/update_synapse_database
View file

@ -36,16 +36,35 @@ class MockHomeserver(HomeServer):
def __init__(self, config, **kwargs):
super(MockHomeserver, self).__init__(
config.server_name, reactor=reactor, config=config, **kwargs
config.server.server_name, reactor=reactor, config=config, **kwargs
)
self.version_string = "Synapse/" + get_version_string(synapse)
if __name__ == "__main__":
def run_background_updates(hs):
store = hs.get_datastore()
async def run_background_updates():
await store.db_pool.updates.run_background_updates(sleep=False)
# Stop the reactor to exit the script once every background update is run.
reactor.stop()
def run():
# Apply all background updates on the database.
defer.ensureDeferred(
run_as_background_process("background_updates", run_background_updates)
)
reactor.callWhenRunning(run)
reactor.run()
def main():
parser = argparse.ArgumentParser(
description=(
"Updates a synapse database to the latest schema and runs background updates"
"Updates a synapse database to the latest schema and optionally runs background updates"
" on it."
)
)
@ -54,7 +73,13 @@ if __name__ == "__main__":
"--database-config",
type=argparse.FileType("r"),
required=True,
help="A database config file for either a SQLite3 database or a PostgreSQL one.",
help="Synapse configuration file, giving the details of the database to be updated",
)
parser.add_argument(
"--run-background-updates",
action="store_true",
required=False,
help="run background updates after upgrading the database schema",
)
args = parser.parse_args()
@ -82,19 +107,10 @@ if __name__ == "__main__":
# Setup instantiates the store within the homeserver object and updates the
# DB.
hs.setup()
store = hs.get_datastore()
async def run_background_updates():
await store.db_pool.updates.run_background_updates(sleep=False)
# Stop the reactor to exit the script once every background update is run.
reactor.stop()
if args.run_background_updates:
run_background_updates(hs)
def run():
# Apply all background updates on the database.
defer.ensureDeferred(
run_as_background_process("background_updates", run_background_updates)
)
reactor.callWhenRunning(run)
reactor.run()
if __name__ == "__main__":
main()

32
setup.py
View file

@ -103,17 +103,17 @@ CONDITIONAL_REQUIREMENTS["lint"] = [
"flake8",
]
CONDITIONAL_REQUIREMENTS["dev"] = CONDITIONAL_REQUIREMENTS["lint"] + [
# The following are used by the release script
"click==7.1.2",
"redbaron==0.9.2",
"GitPython==3.1.14",
"commonmark==0.9.1",
"pygithub==1.55",
CONDITIONAL_REQUIREMENTS["mypy"] = [
"mypy==0.910",
"mypy-zope==0.3.2",
"types-bleach>=4.1.0",
"types-jsonschema>=3.2.0",
"types-Pillow>=8.3.4",
"types-pyOpenSSL>=20.0.7",
"types-PyYAML>=5.4.10",
"types-setuptools>=57.4.0",
]
CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.812", "mypy-zope==0.2.13"]
# Dependencies which are exclusively required by unit test code. This is
# NOT a list of all modules that are necessary to run the unit tests.
# Tests assume that all optional dependencies are installed.
@ -121,6 +121,20 @@ CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.812", "mypy-zope==0.2.13"]
# parameterized_class decorator was introduced in parameterized 0.7.0
CONDITIONAL_REQUIREMENTS["test"] = ["parameterized>=0.7.0"]
CONDITIONAL_REQUIREMENTS["dev"] = (
CONDITIONAL_REQUIREMENTS["lint"]
+ CONDITIONAL_REQUIREMENTS["mypy"]
+ CONDITIONAL_REQUIREMENTS["test"]
+ [
# The following are used by the release script
"click==7.1.2",
"redbaron==0.9.2",
"GitPython==3.1.14",
"commonmark==0.9.1",
"pygithub==1.55",
]
)
setup(
name="matrix-synapse",
version=version,

2
synapse/__init__.py
View file

@ -47,7 +47,7 @@ try:
except ImportError:
pass
__version__ = "1.44.0rc2"
__version__ = "1.44.0"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when

117
synapse/api/filtering.py
View file

@ -15,7 +15,17 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import json
from typing import List
from typing import (
TYPE_CHECKING,
Awaitable,
Container,
Iterable,
List,
Optional,
Set,
TypeVar,
Union,
)
import jsonschema
from jsonschema import FormatChecker
@ -23,7 +33,11 @@ from jsonschema import FormatChecker
from synapse.api.constants import EventContentFields
from synapse.api.errors import SynapseError
from synapse.api.presence import UserPresenceState
from synapse.types import RoomID, UserID
from synapse.events import EventBase
from synapse.types import JsonDict, RoomID, UserID
if TYPE_CHECKING:
from synapse.server import HomeServer
FILTER_SCHEMA = {
"additionalProperties": False,
@ -120,25 +134,29 @@ USER_FILTER_SCHEMA = {
@FormatChecker.cls_checks("matrix_room_id")
def matrix_room_id_validator(room_id_str):
def matrix_room_id_validator(room_id_str: str) -> RoomID:
return RoomID.from_string(room_id_str)
@FormatChecker.cls_checks("matrix_user_id")
def matrix_user_id_validator(user_id_str):
def matrix_user_id_validator(user_id_str: str) -> UserID:
return UserID.from_string(user_id_str)
class Filtering:
def __init__(self, hs):
def __init__(self, hs: "HomeServer"):
super().__init__()
self.store = hs.get_datastore()
async def get_user_filter(self, user_localpart, filter_id):
async def get_user_filter(
self, user_localpart: str, filter_id: Union[int, str]
) -> "FilterCollection":
result = await self.store.get_user_filter(user_localpart, filter_id)
return FilterCollection(result)
def add_user_filter(self, user_localpart, user_filter):
def add_user_filter(
self, user_localpart: str, user_filter: JsonDict
) -> Awaitable[int]:
self.check_valid_filter(user_filter)
return self.store.add_user_filter(user_localpart, user_filter)
@ -146,13 +164,13 @@ class Filtering:
# replace_user_filter at some point? There's no REST API specified for
# them however
def check_valid_filter(self, user_filter_json):
def check_valid_filter(self, user_filter_json: JsonDict) -> None:
"""Check if the provided filter is valid.
This inspects all definitions contained within the filter.
Args:
user_filter_json(dict): The filter
user_filter_json: The filter
Raises:
SynapseError: If the filter is not valid.
"""
@ -167,8 +185,12 @@ class Filtering:
raise SynapseError(400, str(e))
# Filters work across events, presence EDUs, and account data.
FilterEvent = TypeVar("FilterEvent", EventBase, UserPresenceState, JsonDict)
class FilterCollection:
def __init__(self, filter_json):
def __init__(self, filter_json: JsonDict):
self._filter_json = filter_json
room_filter_json = self._filter_json.get("room", {})
@ -188,25 +210,25 @@ class FilterCollection:
self.event_fields = filter_json.get("event_fields", [])
self.event_format = filter_json.get("event_format", "client")
def __repr__(self):
def __repr__(self) -> str:
return "<FilterCollection %s>" % (json.dumps(self._filter_json),)
def get_filter_json(self):
def get_filter_json(self) -> JsonDict:
return self._filter_json
def timeline_limit(self):
def timeline_limit(self) -> int:
return self._room_timeline_filter.limit()
def presence_limit(self):
def presence_limit(self) -> int:
return self._presence_filter.limit()
def ephemeral_limit(self):
def ephemeral_limit(self) -> int:
return self._room_ephemeral_filter.limit()
def lazy_load_members(self):
def lazy_load_members(self) -> bool:
return self._room_state_filter.lazy_load_members()
def include_redundant_members(self):
def include_redundant_members(self) -> bool:
return self._room_state_filter.include_redundant_members()
def filter_presence(self, events):
@ -218,29 +240,31 @@ class FilterCollection:
def filter_room_state(self, events):
return self._room_state_filter.filter(self._room_filter.filter(events))
def filter_room_timeline(self, events):
def filter_room_timeline(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
return self._room_timeline_filter.filter(self._room_filter.filter(events))
def filter_room_ephemeral(self, events):
def filter_room_ephemeral(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
return self._room_ephemeral_filter.filter(self._room_filter.filter(events))
def filter_room_account_data(self, events):
def filter_room_account_data(
self, events: Iterable[FilterEvent]
) -> List[FilterEvent]:
return self._room_account_data.filter(self._room_filter.filter(events))
def blocks_all_presence(self):
def blocks_all_presence(self) -> bool:
return (
self._presence_filter.filters_all_types()
or self._presence_filter.filters_all_senders()
)
def blocks_all_room_ephemeral(self):
def blocks_all_room_ephemeral(self) -> bool:
return (
self._room_ephemeral_filter.filters_all_types()
or self._room_ephemeral_filter.filters_all_senders()
or self._room_ephemeral_filter.filters_all_rooms()
)
def blocks_all_room_timeline(self):
def blocks_all_room_timeline(self) -> bool:
return (
self._room_timeline_filter.filters_all_types()
or self._room_timeline_filter.filters_all_senders()
@ -249,7 +273,7 @@ class FilterCollection:
class Filter:
def __init__(self, filter_json):
def __init__(self, filter_json: JsonDict):
self.filter_json = filter_json
self.types = self.filter_json.get("types", None)
@ -266,20 +290,20 @@ class Filter:
self.labels = self.filter_json.get("org.matrix.labels", None)
self.not_labels = self.filter_json.get("org.matrix.not_labels", [])
def filters_all_types(self):
def filters_all_types(self) -> bool:
return "*" in self.not_types
def filters_all_senders(self):
def filters_all_senders(self) -> bool:
return "*" in self.not_senders
def filters_all_rooms(self):
def filters_all_rooms(self) -> bool:
return "*" in self.not_rooms
def check(self, event):
def check(self, event: FilterEvent) -> bool:
"""Checks whether the filter matches the given event.
Returns:
bool: True if the event matches
True if the event matches
"""
# We usually get the full "events" as dictionaries coming through,
# except for presence which actually gets passed around as its own
@ -305,18 +329,25 @@ class Filter:
room_id = event.get("room_id", None)
ev_type = event.get("type", None)
content = event.get("content", {})
content = event.get("content") or {}
# check if there is a string url field in the content for filtering purposes
contains_url = isinstance(content.get("url"), str)
labels = content.get(EventContentFields.LABELS, [])
return self.check_fields(room_id, sender, ev_type, labels, contains_url)
def check_fields(self, room_id, sender, event_type, labels, contains_url):
def check_fields(
self,
room_id: Optional[str],
sender: Optional[str],
event_type: Optional[str],
labels: Container[str],
contains_url: bool,
) -> bool:
"""Checks whether the filter matches the given event fields.
Returns:
bool: True if the event fields match
True if the event fields match
"""
literal_keys = {
"rooms": lambda v: room_id == v,
@ -343,14 +374,14 @@ class Filter:
return True
def filter_rooms(self, room_ids):
def filter_rooms(self, room_ids: Iterable[str]) -> Set[str]:
"""Apply the 'rooms' filter to a given list of rooms.
Args:
room_ids (list): A list of room_ids.
room_ids: A list of room_ids.
Returns:
list: A list of room_ids that match the filter
A list of room_ids that match the filter
"""
room_ids = set(room_ids)
@ -363,23 +394,23 @@ class Filter:
return room_ids
def filter(self, events):
def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
return list(filter(self.check, events))
def limit(self):
def limit(self) -> int:
return self.filter_json.get("limit", 10)
def lazy_load_members(self):
def lazy_load_members(self) -> bool:
return self.filter_json.get("lazy_load_members", False)
def include_redundant_members(self):
def include_redundant_members(self) -> bool:
return self.filter_json.get("include_redundant_members", False)
def with_room_ids(self, room_ids):
def with_room_ids(self, room_ids: Iterable[str]) -> "Filter":
"""Returns a new filter with the given room IDs appended.
Args:
room_ids (iterable[unicode]): The room_ids to add
room_ids: The room_ids to add
Returns:
filter: A new filter including the given rooms and the old
@ -390,8 +421,8 @@ class Filter:
return newFilter
def _matches_wildcard(actual_value, filter_value):
if filter_value.endswith("*"):
def _matches_wildcard(actual_value: Optional[str], filter_value: str) -> bool:
if filter_value.endswith("*") and isinstance(actual_value, str):
type_prefix = filter_value[:-1]
return actual_value.startswith(type_prefix)
else:

86
synapse/api/ratelimiting.py
View file

@ -17,6 +17,7 @@ from collections import OrderedDict
from typing import Hashable, Optional, Tuple
from synapse.api.errors import LimitExceededError
from synapse.config.ratelimiting import RateLimitConfig
from synapse.storage.databases.main import DataStore
from synapse.types import Requester
from synapse.util import Clock
@ -233,3 +234,88 @@ class Ratelimiter:
raise LimitExceededError(
retry_after_ms=int(1000 * (time_allowed - time_now_s))
)
class RequestRatelimiter:
def __init__(
self,
store: DataStore,
clock: Clock,
rc_message: RateLimitConfig,
rc_admin_redaction: Optional[RateLimitConfig],
):
self.store = store
self.clock = clock
# The rate_hz and burst_count are overridden on a per-user basis
self.request_ratelimiter = Ratelimiter(
store=self.store, clock=self.clock, rate_hz=0, burst_count=0
)
self._rc_message = rc_message
# Check whether ratelimiting room admin message redaction is enabled
# by the presence of rate limits in the config
if rc_admin_redaction:
self.admin_redaction_ratelimiter: Optional[Ratelimiter] = Ratelimiter(
store=self.store,
clock=self.clock,
rate_hz=rc_admin_redaction.per_second,
burst_count=rc_admin_redaction.burst_count,
)
else:
self.admin_redaction_ratelimiter = None
async def ratelimit(
self,
requester: Requester,
update: bool = True,
is_admin_redaction: bool = False,
) -> None:
"""Ratelimits requests.
Args:
requester
update: Whether to record that a request is being processed.
Set to False when doing multiple checks for one request (e.g.
to check up front if we would reject the request), and set to
True for the last call for a given request.
is_admin_redaction: Whether this is a room admin/moderator
redacting an event. If so then we may apply different
ratelimits depending on config.
Raises:
LimitExceededError if the request should be ratelimited
"""
user_id = requester.user.to_string()
# The AS user itself is never rate limited.
app_service = self.store.get_app_service_by_user_id(user_id)
if app_service is not None:
return # do not ratelimit app service senders
messages_per_second = self._rc_message.per_second
burst_count = self._rc_message.burst_count
# Check if there is a per user override in the DB.
override = await self.store.get_ratelimit_for_user(user_id)
if override:
# If overridden with a null Hz then ratelimiting has been entirely
# disabled for the user
if not override.messages_per_second:
return
messages_per_second = override.messages_per_second
burst_count = override.burst_count
if is_admin_redaction and self.admin_redaction_ratelimiter:
# If we have separate config for admin redactions, use a separate
# ratelimiter as to not have user_ids clash
await self.admin_redaction_ratelimiter.ratelimit(requester, update=update)
else:
# Override rate and burst count per-user
await self.request_ratelimiter.ratelimit(
requester,
rate_hz=messages_per_second,
burst_count=burst_count,
update=update,
)

10
synapse/app/_base.py
View file

@ -86,11 +86,11 @@ def start_worker_reactor(appname, config, run_command=reactor.run):
start_reactor(
appname,
soft_file_limit=config.soft_file_limit,
gc_thresholds=config.gc_thresholds,
soft_file_limit=config.server.soft_file_limit,
gc_thresholds=config.server.gc_thresholds,
pid_file=config.worker.worker_pid_file,
daemonize=config.worker.worker_daemonize,
print_pidfile=config.print_pidfile,
print_pidfile=config.server.print_pidfile,
logger=logger,
run_command=run_command,
)
@ -298,10 +298,10 @@ def refresh_certificate(hs):
Refresh the TLS certificates that Synapse is using by re-reading them from
disk and updating the TLS context factories to use them.
"""
if not hs.config.has_tls_listener():
if not hs.config.server.has_tls_listener():
return
hs.config.read_certificate_from_disk()
hs.config.tls.read_certificate_from_disk()
hs.tls_server_context_factory = context_factory.ServerContextFactory(hs.config)
if hs._listening_services:

8
synapse/app/admin_cmd.py
View file

@ -195,14 +195,14 @@ def start(config_options):
config.logging.no_redirect_stdio = True
# Explicitly disable background processes
config.update_user_directory = False
config.server.update_user_directory = False
config.worker.run_background_tasks = False
config.start_pushers = False
config.worker.start_pushers = False
config.pusher_shard_config.instances = []
config.send_federation = False
config.worker.send_federation = False
config.federation_shard_config.instances = []
synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
synapse.events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
ss = AdminCmdServer(
config.server.server_name,

2
synapse/app/generic_worker.py
View file

@ -462,7 +462,7 @@ def start(config_options):
# For other worker types we force this to off.
config.server.update_user_directory = False
synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
synapse.events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
synapse.util.caches.TRACK_MEMORY_USAGE = config.caches.track_memory_usage
if config.server.gc_seconds:

16
synapse/app/homeserver.py
View file

@ -234,7 +234,7 @@ class SynapseHomeServer(HomeServer):
)
if name in ["media", "federation", "client"]:
if self.config.media.enable_media_repo:
if self.config.server.enable_media_repo:
media_repo = self.get_media_repository_resource()
resources.update(
{MEDIA_PREFIX: media_repo, LEGACY_MEDIA_PREFIX: media_repo}
@ -248,7 +248,7 @@ class SynapseHomeServer(HomeServer):
resources[SERVER_KEY_V2_PREFIX] = KeyApiV2Resource(self)
if name == "webclient":
webclient_loc = self.config.web_client_location
webclient_loc = self.config.server.web_client_location
if webclient_loc is None:
logger.warning(
@ -343,7 +343,7 @@ def setup(config_options):
# generating config files and shouldn't try to continue.
sys.exit(0)
events.USE_FROZEN_DICTS = config.use_frozen_dicts
events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
synapse.util.caches.TRACK_MEMORY_USAGE = config.caches.track_memory_usage
if config.server.gc_seconds:
@ -439,11 +439,11 @@ def run(hs):
_base.start_reactor(
"synapse-homeserver",
soft_file_limit=hs.config.soft_file_limit,
gc_thresholds=hs.config.gc_thresholds,
pid_file=hs.config.pid_file,
daemonize=hs.config.daemonize,
print_pidfile=hs.config.print_pidfile,
soft_file_limit=hs.config.server.soft_file_limit,
gc_thresholds=hs.config.server.gc_thresholds,
pid_file=hs.config.server.pid_file,
daemonize=hs.config.server.daemonize,
print_pidfile=hs.config.server.print_pidfile,
logger=logger,
)

8
synapse/app/phone_stats_home.py
View file

@ -74,7 +74,7 @@ async def phone_stats_home(hs, stats, stats_process=_stats_process):
store = hs.get_datastore()
stats["homeserver"] = hs.config.server.server_name
stats["server_context"] = hs.config.server_context
stats["server_context"] = hs.config.server.server_context
stats["timestamp"] = now
stats["uptime_seconds"] = uptime
version = sys.version_info
@ -171,7 +171,7 @@ def start_phone_stats_home(hs):
current_mau_count_by_service = {}
reserved_users = ()
store = hs.get_datastore()
if hs.config.limit_usage_by_mau or hs.config.mau_stats_only:
if hs.config.server.limit_usage_by_mau or hs.config.server.mau_stats_only:
current_mau_count = await store.get_monthly_active_count()
current_mau_count_by_service = (
await store.get_monthly_active_count_by_service()
@ -183,9 +183,9 @@ def start_phone_stats_home(hs):
current_mau_by_service_gauge.labels(app_service).set(float(count))
registered_reserved_users_mau_gauge.set(float(len(reserved_users)))
max_mau_gauge.set(float(hs.config.max_mau_value))
max_mau_gauge.set(float(hs.config.server.max_mau_value))
if hs.config.limit_usage_by_mau or hs.config.mau_stats_only:
if hs.config.server.limit_usage_by_mau or hs.config.server.mau_stats_only:
generate_monthly_active_users()
clock.looping_call(generate_monthly_active_users, 5 * 60 * 1000)
# End of monthly active user settings

64
synapse/config/_base.py
View file

@ -118,21 +118,6 @@ class Config:
"synapse", "res/templates"
)
def __getattr__(self, item: str) -> Any:
"""
Try and fetch a configuration option that does not exist on this class.
This is so that existing configs that rely on `self.value`, where value
is actually from a different config section, continue to work.
"""
if item in ["generate_config_section", "read_config"]:
raise AttributeError(item)
if self.root is None:
raise AttributeError(item)
else:
return self.root._get_unclassed_config(self.section, item)
@staticmethod
def parse_size(value):
if isinstance(value, int):
@ -289,7 +274,9 @@ class Config:
env.filters.update(
{
"format_ts": _format_ts_filter,
"mxc_to_http": _create_mxc_to_http_filter(self.public_baseurl),
"mxc_to_http": _create_mxc_to_http_filter(
self.root.server.public_baseurl
),
}
)
@ -311,8 +298,6 @@ class RootConfig:
config_classes = []
def __init__(self):
self._configs = OrderedDict()
for config_class in self.config_classes:
if config_class.section is None:
raise ValueError("%r requires a section name" % (config_class,))
@ -321,42 +306,7 @@ class RootConfig:
conf = config_class(self)
except Exception as e:
raise Exception("Failed making %s: %r" % (config_class.section, e))
self._configs[config_class.section] = conf
def __getattr__(self, item: str) -> Any:
"""
Redirect lookups on this object either to config objects, or values on
config objects, so that `config.tls.blah` works, as well as legacy uses
of things like `config.server_name`. It will first look up the config
section name, and then values on those config classes.
"""
if item in self._configs.keys():
return self._configs[item]
return self._get_unclassed_config(None, item)
def _get_unclassed_config(self, asking_section: Optional[str], item: str):
"""
Fetch a config value from one of the instantiated config classes that
has not been fetched directly.
Args:
asking_section: If this check is coming from a Config child, which
one? This section will not be asked if it has the value.
item: The configuration value key.
Raises:
AttributeError if no config classes have the config key. The body
will contain what sections were checked.
"""
for key, val in self._configs.items():
if key == asking_section:
continue
if item in dir(val):
return getattr(val, item)
raise AttributeError(item, "not found in %s" % (list(self._configs.keys()),))
setattr(self, config_class.section, conf)
def invoke_all(self, func_name: str, *args, **kwargs) -> MutableMapping[str, Any]:
"""
@ -373,9 +323,11 @@ class RootConfig:
"""
res = OrderedDict()
for name, config in self._configs.items():
for config_class in self.config_classes:
config = getattr(self, config_class.section)
if hasattr(config, func_name):
res[name] = getattr(config, func_name)(*args, **kwargs)
res[config_class.section] = getattr(config, func_name)(*args, **kwargs)
return res

2
synapse/config/account_validity.py
View file

@ -76,7 +76,7 @@ class AccountValidityConfig(Config):
)
if self.account_validity_renew_by_email_enabled:
if not self.public_baseurl:
if not self.root.server.public_baseurl:
raise ConfigError("Can't send renewal emails without 'public_baseurl'")
# Load account validity templates.

2
synapse/config/cas.py
View file

@ -37,7 +37,7 @@ class CasConfig(Config):
# The public baseurl is required because it is used by the redirect
# template.
public_baseurl = self.public_baseurl
public_baseurl = self.root.server.public_baseurl
if not public_baseurl:
raise ConfigError("cas_config requires a public_baseurl to be set")

9
synapse/config/emailconfig.py
View file

@ -19,7 +19,6 @@ import email.utils
import logging
import os
from enum import Enum
from typing import Optional
import attr
@ -135,7 +134,7 @@ class EmailConfig(Config):
# msisdn is currently always remote while Synapse does not support any method of
# sending SMS messages
ThreepidBehaviour.REMOTE
if self.account_threepid_delegate_email
if self.root.registration.account_threepid_delegate_email
else ThreepidBehaviour.LOCAL
)
# Prior to Synapse v1.4.0, there was another option that defined whether Synapse would
@ -144,7 +143,7 @@ class EmailConfig(Config):
# identity server in the process.
self.using_identity_server_from_trusted_list = False
if (
not self.account_threepid_delegate_email
not self.root.registration.account_threepid_delegate_email
and config.get("trust_identity_server_for_password_resets", False) is True
):
# Use the first entry in self.trusted_third_party_id_servers instead
@ -156,7 +155,7 @@ class EmailConfig(Config):
# trusted_third_party_id_servers does not contain a scheme whereas
# account_threepid_delegate_email is expected to. Presume https
self.account_threepid_delegate_email: Optional[str] = (
self.root.registration.account_threepid_delegate_email = (
"https://" + first_trusted_identity_server
)
self.using_identity_server_from_trusted_list = True
@ -335,7 +334,7 @@ class EmailConfig(Config):
"client_base_url", email_config.get("riot_base_url", None)
)
if self.account_validity_renew_by_email_enabled:
if self.root.account_validity.account_validity_renew_by_email_enabled:
expiry_template_html = email_config.get(
"expiry_template_html", "notice_expiry.html"
)

6
synapse/config/key.py
View file

@ -145,11 +145,13 @@ class KeyConfig(Config):
# list of TrustedKeyServer objects
self.key_servers = list(
_parse_key_servers(key_servers, self.federation_verify_certificates)
_parse_key_servers(
key_servers, self.root.tls.federation_verify_certificates
)
)
self.macaroon_secret_key = config.get(
"macaroon_secret_key", self.registration_shared_secret
"macaroon_secret_key", self.root.registration.registration_shared_secret
)
if not self.macaroon_secret_key:

2
synapse/config/oidc.py
View file

@ -58,7 +58,7 @@ class OIDCConfig(Config):
"Multiple OIDC providers have the idp_id %r." % idp_id
)
public_baseurl = self.public_baseurl
public_baseurl = self.root.server.public_baseurl
if public_baseurl is None:
raise ConfigError("oidc_config requires a public_baseurl to be set")
self.oidc_callback_url = public_baseurl + "_synapse/client/oidc/callback"

Some files were not shown because too many files have changed in this diff Show more