Merge branch 'release-v1.47' into matrix-org-hotfixes

This commit is contained in:
Olivier Wilkinson (reivilibre) 2021-11-09 11:58:35 +00:00
commit 9d3b04e749
226 changed files with 3985 additions and 1288 deletions

View file

@ -3,7 +3,7 @@
# Test for the export-data admin command against sqlite and postgres # Test for the export-data admin command against sqlite and postgres
set -xe set -xe
cd `dirname $0`/../.. cd "$(dirname "$0")/../.."
echo "--- Install dependencies" echo "--- Install dependencies"

View file

@ -7,7 +7,7 @@
set -xe set -xe
cd `dirname $0`/../.. cd "$(dirname "$0")/../.."
echo "--- Install dependencies" echo "--- Install dependencies"

View file

@ -1,12 +1,13 @@
### Pull Request Checklist ### Pull Request Checklist
<!-- Please read CONTRIBUTING.md before submitting your pull request --> <!-- Please read https://matrix-org.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request -->
* [ ] Pull request is based on the develop branch * [ ] Pull request is based on the develop branch
* [ ] Pull request includes a [changelog file](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#changelog). The entry should: * [ ] Pull request includes a [changelog file](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`. - Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!). - End with either a period (.) or an exclamation mark (!).
- Start with a capital letter. - Start with a capital letter.
* [ ] Pull request includes a [sign off](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#sign-off) * [ ] Pull request includes a [sign off](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#sign-off)
* [ ] Code style is correct (run the [linters](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#code-style)) * [ ] [Code style](https://matrix-org.github.io/synapse/latest/code_style.html) is correct
(run the [linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

View file

@ -1,8 +1,17 @@
Synapse 1.46.0rc1 (2021-10-27) Synapse 1.46.0 (2021-11-02)
============================== ===========================
The cause of the [performance regression affecting Synapse 1.44](https://github.com/matrix-org/synapse/issues/11049) has been identified and fixed. ([\#11177](https://github.com/matrix-org/synapse/issues/11177)) The cause of the [performance regression affecting Synapse 1.44](https://github.com/matrix-org/synapse/issues/11049) has been identified and fixed. ([\#11177](https://github.com/matrix-org/synapse/issues/11177))
Bugfixes
--------
- Fix a bug introduced in v1.46.0rc1 where URL previews of some XML documents would fail. ([\#11196](https://github.com/matrix-org/synapse/issues/11196))
Synapse 1.46.0rc1 (2021-10-27)
==============================
Features Features
-------- --------

1
changelog.d/10097.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat.

1
changelog.d/10943.misc Normal file
View file

@ -0,0 +1 @@
Add type annotations for the `log_function` decorator.

1
changelog.d/10969.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.

1
changelog.d/11033.bugfix Normal file
View file

@ -0,0 +1 @@
Do not accept events if a third-party rule module API callback raises an exception.

View file

@ -0,0 +1 @@
Advertise support for Client-Server API r0.6.1.

1
changelog.d/11098.misc Normal file
View file

@ -0,0 +1 @@
Add type hints to `synapse.events`.

View file

@ -0,0 +1 @@
Add search by room ID and room alias to List Room admin API.

View file

@ -0,0 +1 @@
Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room.

1
changelog.d/11128.doc Normal file
View file

@ -0,0 +1 @@
Improve example HAProxy config in the docs to properly handle host headers with port information. This is required for federation over port 443 to work correctly.

1
changelog.d/11129.bugfix Normal file
View file

@ -0,0 +1 @@
Fix long-standing bug where verification requests could fail in certain cases if whitelist was in place but did not include your own homeserver.

1
changelog.d/11137.misc Normal file
View file

@ -0,0 +1 @@
Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code.

View file

@ -0,0 +1 @@
Add a module API method to update a user's membership in a room.

1
changelog.d/11151.doc Normal file
View file

@ -0,0 +1 @@
Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5.

1
changelog.d/11157.misc Normal file
View file

@ -0,0 +1 @@
Only allow old Element/Riot Android clients to send read receipts without a request body. All other clients must include a request body as required by the specification. Contributed by @rogersheu.

1
changelog.d/11164.misc Normal file
View file

@ -0,0 +1 @@
Add type hints so that `synapse.http` passes `mypy` checks.

1
changelog.d/11166.misc Normal file
View file

@ -0,0 +1 @@
Update scripts to pass Shellcheck lints.

1
changelog.d/11171.misc Normal file
View file

@ -0,0 +1 @@
Add knock information in admin export. Contributed by Rafael Gonçalves.

View file

@ -0,0 +1 @@
Add metrics for thread pool usage.

1
changelog.d/11179.misc Normal file
View file

@ -0,0 +1 @@
Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly.

View file

@ -0,0 +1 @@
Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288).

1
changelog.d/11188.bugfix Normal file
View file

@ -0,0 +1 @@
Allow an empty list of `state_events_at_start` to be sent when using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint and the author of the historical messages is already part of the current room state at the given `?prev_event_id`.

1
changelog.d/11191.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p.

1
changelog.d/11194.misc Normal file
View file

@ -0,0 +1 @@
Refactor `Filter` to check different fields depending on the data type.

View file

@ -1 +0,0 @@
Fix a bug introduced in v1.46.0rc1 where URL previews of some XML documents would fail.

1
changelog.d/11198.doc Normal file
View file

@ -0,0 +1 @@
Clarify lack of support for Windows.

1
changelog.d/11199.bugfix Normal file
View file

@ -0,0 +1 @@
Delete `to_device` messages for hidden devices that will never be read, reducing database size.

1
changelog.d/11200.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error.

View file

@ -0,0 +1 @@
Add a module API method to retrieve the current state of a room.

1
changelog.d/11205.misc Normal file
View file

@ -0,0 +1 @@
Improve type hints for the relations datastore.

View file

@ -0,0 +1 @@
The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information.

1
changelog.d/11207.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug which could result in serialization errors and potentially duplicate transaction data when sending ephemeral events to application services. Contributed by @Fizzadar at Beeper.

1
changelog.d/11209.docker Normal file
View file

@ -0,0 +1 @@
Avoid changing userid when started as a non-root user, and no explicit `UID` is set.

View file

@ -0,0 +1 @@
Calculate a default value for `public_baseurl` based on `server_name`.

View file

@ -0,0 +1 @@
Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443.

1
changelog.d/11212.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.

View file

@ -0,0 +1 @@
Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`).

1
changelog.d/11217.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a bug introduced in 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats.

1
changelog.d/11221.doc Normal file
View file

@ -0,0 +1 @@
Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper.

1
changelog.d/11225.misc Normal file
View file

@ -0,0 +1 @@
Replace outdated links in the pull request checklist with links to the rendered documentation.

1
changelog.d/11226.misc Normal file
View file

@ -0,0 +1 @@
Fix a bug in unit test `test_block_room_and_not_purge`.

1
changelog.d/11229.misc Normal file
View file

@ -0,0 +1 @@
`ObservableDeferred`: run registered observers in order.

1
changelog.d/11231.misc Normal file
View file

@ -0,0 +1 @@
Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`.

1
changelog.d/11233.misc Normal file
View file

@ -0,0 +1 @@
Add `twine` and `towncrier` as dev dependencies, as they're used by the release script.

1
changelog.d/11234.bugfix Normal file
View file

@ -0,0 +1 @@
Fix long-standing bug where cross signing keys were not included in the response to `/r0/keys/query` the first time a remote user was queried.

1
changelog.d/11237.misc Normal file
View file

@ -0,0 +1 @@
Allow `stream_writers.typing` config to be a list of one worker.

1
changelog.d/11239.misc Normal file
View file

@ -0,0 +1 @@
Remove debugging statement in tests.

1
changelog.d/11240.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a long-standing bug where all requests that read events from the database could get stuck as a result of losing the database connection.

1
changelog.d/11244.misc Normal file
View file

@ -0,0 +1 @@
Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers.

1
changelog.d/11246.misc Normal file
View file

@ -0,0 +1 @@
Add an additional test for the `cachedList` method decorator.

1
changelog.d/11253.misc Normal file
View file

@ -0,0 +1 @@
Make minor correction to the type of `auth_checkers` callbacks.

1
changelog.d/11255.bugfix Normal file
View file

@ -0,0 +1 @@
Fix rolling back Synapse version when using workers.

1
changelog.d/11257.doc Normal file
View file

@ -0,0 +1 @@
Add documentation for using LemonLDAP as an OpenID Connect Identity Provider. Contributed by @l00ptr.

1
changelog.d/11262.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a bug where if a remote event is being processed by a worker when it gets killed then it won't get processed on restart. Introduced in v1.37.1.

View file

@ -0,0 +1 @@
Add some background update admin APIs.

1
changelog.d/11269.misc Normal file
View file

@ -0,0 +1 @@
Clean up trivial aspects of the Debian package build tooling.

1
changelog.d/11270.misc Normal file
View file

@ -0,0 +1 @@
Blacklist new SyTest that checks that key uploads are valid pending the validation being implemented in Synapse.

1
changelog.d/11273.misc Normal file
View file

@ -0,0 +1 @@
Clean up trivial aspects of the Debian package build tooling.

1
changelog.d/11276.bugfix Normal file
View file

@ -0,0 +1 @@
Fix rolling back Synapse version when using workers.

View file

@ -84,7 +84,9 @@ AUTH="Authorization: Bearer $TOKEN"
################################################################################################### ###################################################################################################
# finally start pruning the room: # finally start pruning the room:
################################################################################################### ###################################################################################################
POSTDATA='{"delete_local_events":"true"}' # this will really delete local events, so the messages in the room really disappear unless they are restored by remote federation # this will really delete local events, so the messages in the room really
# disappear unless they are restored by remote federation. This is because
# we pass {"delete_local_events":true} to the curl invocation below.
for ROOM in "${ROOMS_ARRAY[@]}"; do for ROOM in "${ROOMS_ARRAY[@]}"; do
echo "########################################### $(date) ################# " echo "########################################### $(date) ################# "
@ -104,7 +106,7 @@ for ROOM in "${ROOMS_ARRAY[@]}"; do
SLEEP=2 SLEEP=2
set -x set -x
# call purge # call purge
OUT=$(curl --header "$AUTH" -s -d $POSTDATA POST "$API_URL/admin/purge_history/$ROOM/$EVENT_ID") OUT=$(curl --header "$AUTH" -s -d '{"delete_local_events":true}' POST "$API_URL/admin/purge_history/$ROOM/$EVENT_ID")
PURGE_ID=$(echo "$OUT" |grep purge_id|cut -d'"' -f4 ) PURGE_ID=$(echo "$OUT" |grep purge_id|cut -d'"' -f4 )
if [ "$PURGE_ID" == "" ]; then if [ "$PURGE_ID" == "" ]; then
# probably the history purge is already in progress for $ROOM # probably the history purge is already in progress for $ROOM

View file

@ -15,7 +15,7 @@ export DH_VIRTUALENV_INSTALL_ROOT=/opt/venvs
# python won't look in the right directory. At least this way, the error will # python won't look in the right directory. At least this way, the error will
# be a *bit* more obvious. # be a *bit* more obvious.
# #
SNAKE=`readlink -e /usr/bin/python3` SNAKE=$(readlink -e /usr/bin/python3)
# try to set the CFLAGS so any compiled C extensions are compiled with the most # try to set the CFLAGS so any compiled C extensions are compiled with the most
# generic as possible x64 instructions, so that compiling it on a new Intel chip # generic as possible x64 instructions, so that compiling it on a new Intel chip
@ -24,7 +24,7 @@ SNAKE=`readlink -e /usr/bin/python3`
# TODO: add similar things for non-amd64, or figure out a more generic way to # TODO: add similar things for non-amd64, or figure out a more generic way to
# do this. # do this.
case `dpkg-architecture -q DEB_HOST_ARCH` in case $(dpkg-architecture -q DEB_HOST_ARCH) in
amd64) amd64)
export CFLAGS=-march=x86-64 export CFLAGS=-march=x86-64
;; ;;
@ -40,6 +40,7 @@ dh_virtualenv \
--upgrade-pip \ --upgrade-pip \
--preinstall="lxml" \ --preinstall="lxml" \
--preinstall="mock" \ --preinstall="mock" \
--preinstall="wheel" \
--extra-pip-arg="--no-cache-dir" \ --extra-pip-arg="--no-cache-dir" \
--extra-pip-arg="--compile" \ --extra-pip-arg="--compile" \
--extras="all,systemd,test" --extras="all,systemd,test"
@ -56,8 +57,8 @@ case "$DEB_BUILD_OPTIONS" in
*) *)
# Copy tests to a temporary directory so that we can put them on the # Copy tests to a temporary directory so that we can put them on the
# PYTHONPATH without putting the uninstalled synapse on the pythonpath. # PYTHONPATH without putting the uninstalled synapse on the pythonpath.
tmpdir=`mktemp -d` tmpdir=$(mktemp -d)
trap "rm -r $tmpdir" EXIT trap 'rm -r $tmpdir' EXIT
cp -r tests "$tmpdir" cp -r tests "$tmpdir"
@ -98,7 +99,7 @@ esac
--output-file="${PACKAGE_BUILD_DIR}/etc/matrix-synapse/log.yaml" --output-file="${PACKAGE_BUILD_DIR}/etc/matrix-synapse/log.yaml"
# add a dependency on the right version of python to substvars. # add a dependency on the right version of python to substvars.
PYPKG=`basename $SNAKE` PYPKG=$(basename "$SNAKE")
echo "synapse:pydepends=$PYPKG" >> debian/matrix-synapse-py3.substvars echo "synapse:pydepends=$PYPKG" >> debian/matrix-synapse-py3.substvars

18
debian/changelog vendored
View file

@ -1,8 +1,22 @@
matrix-synapse-py3 (1.46.0~rc1ubuntu1) UNRELEASED; urgency=medium matrix-synapse-py3 (1.47.0+nmu1) UNRELEASED; urgency=medium
* Update scripts to pass Shellcheck lints.
* Remove unused Vagrant scripts from debian/ directory.
* Allow building Debian packages for any architecture, not just amd64.
* Preinstall the "wheel" package when building virtualenvs.
* Do not error if /etc/default/matrix-synapse is missing.
-- Dan Callahan <danc@element.io> Fri, 22 Oct 2021 22:20:31 +0000
matrix-synapse-py3 (1.46.0) stable; urgency=medium
[ Richard van der Hoff ]
* Compress debs with xz, to fix incompatibility of impish debs with reprepro. * Compress debs with xz, to fix incompatibility of impish debs with reprepro.
-- Richard van der Hoff <richard@matrix.org> Wed, 27 Oct 2021 15:32:51 +0100 [ Synapse Packaging team ]
* New synapse release 1.46.0.
-- Synapse Packaging team <packages@matrix.org> Tue, 02 Nov 2021 13:22:53 +0000
matrix-synapse-py3 (1.46.0~rc1) stable; urgency=medium matrix-synapse-py3 (1.46.0~rc1) stable; urgency=medium

2
debian/control vendored
View file

@ -19,7 +19,7 @@ Standards-Version: 3.9.8
Homepage: https://github.com/matrix-org/synapse Homepage: https://github.com/matrix-org/synapse
Package: matrix-synapse-py3 Package: matrix-synapse-py3
Architecture: amd64 Architecture: any
Provides: matrix-synapse Provides: matrix-synapse
Conflicts: Conflicts:
matrix-synapse (<< 0.34.0.1-0matrix2), matrix-synapse (<< 0.34.0.1-0matrix2),

View file

@ -2,6 +2,7 @@
set -e set -e
# shellcheck disable=SC1091
. /usr/share/debconf/confmodule . /usr/share/debconf/confmodule
# try to update the debconf db according to whatever is in the config files # try to update the debconf db according to whatever is in the config files

View file

@ -1,5 +1,6 @@
#!/bin/sh -e #!/bin/sh -e
# shellcheck disable=SC1091
. /usr/share/debconf/confmodule . /usr/share/debconf/confmodule
CONFIGFILE_SERVERNAME="/etc/matrix-synapse/conf.d/server_name.yaml" CONFIGFILE_SERVERNAME="/etc/matrix-synapse/conf.d/server_name.yaml"

View file

@ -5,7 +5,7 @@ Description=Synapse Matrix homeserver
Type=notify Type=notify
User=matrix-synapse User=matrix-synapse
WorkingDirectory=/var/lib/matrix-synapse WorkingDirectory=/var/lib/matrix-synapse
EnvironmentFile=/etc/default/matrix-synapse EnvironmentFile=-/etc/default/matrix-synapse
ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
ExecReload=/bin/kill -HUP $MAINPID ExecReload=/bin/kill -HUP $MAINPID

View file

@ -1,2 +0,0 @@
.vagrant
*.log

View file

@ -1,23 +0,0 @@
#!/bin/bash
#
# provisioning script for vagrant boxes for testing the matrix-synapse debs.
#
# Will install the most recent matrix-synapse-py3 deb for this platform from
# the /debs directory.
set -e
apt-get update
apt-get install -y lsb-release
deb=`ls /debs/matrix-synapse-py3_*+$(lsb_release -cs)*.deb | sort | tail -n1`
debconf-set-selections <<EOF
matrix-synapse matrix-synapse/report-stats boolean false
matrix-synapse matrix-synapse/server-name string localhost:18448
EOF
dpkg -i "$deb"
sed -i -e '/port: 8...$/{s/8448/18448/; s/8008/18008/}' -e '$aregistration_shared_secret: secret' /etc/matrix-synapse/homeserver.yaml
systemctl restart matrix-synapse

View file

@ -1,13 +0,0 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
ver = `cd ../../..; dpkg-parsechangelog -S Version`.strip()
Vagrant.configure("2") do |config|
config.vm.box = "debian/stretch64"
config.vm.synced_folder ".", "/vagrant", disabled: true
config.vm.synced_folder "../../../../debs", "/debs", type: "nfs"
config.vm.provision "shell", path: "../provision.sh"
end

View file

@ -1,10 +0,0 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.synced_folder ".", "/vagrant", disabled: true
config.vm.synced_folder "../../../../debs", "/debs"
config.vm.provision "shell", path: "../provision.sh"
end

View file

@ -6,14 +6,14 @@ DIR="$( cd "$( dirname "$0" )" && pwd )"
PID_FILE="$DIR/servers.pid" PID_FILE="$DIR/servers.pid"
if [ -f $PID_FILE ]; then if [ -f "$PID_FILE" ]; then
echo "servers.pid exists!" echo "servers.pid exists!"
exit 1 exit 1
fi fi
for port in 8080 8081 8082; do for port in 8080 8081 8082; do
rm -rf $DIR/$port rm -rf "${DIR:?}/$port"
rm -rf $DIR/media_store.$port rm -rf "$DIR/media_store.$port"
done done
rm -rf $DIR/etc rm -rf "${DIR:?}/etc"

View file

@ -4,21 +4,22 @@ DIR="$( cd "$( dirname "$0" )" && pwd )"
CWD=$(pwd) CWD=$(pwd)
cd "$DIR/.." cd "$DIR/.." || exit
mkdir -p demo/etc mkdir -p demo/etc
export PYTHONPATH=$(readlink -f $(pwd)) PYTHONPATH=$(readlink -f "$(pwd)")
export PYTHONPATH
echo $PYTHONPATH echo "$PYTHONPATH"
for port in 8080 8081 8082; do for port in 8080 8081 8082; do
echo "Starting server on port $port... " echo "Starting server on port $port... "
https_port=$((port + 400)) https_port=$((port + 400))
mkdir -p demo/$port mkdir -p demo/$port
pushd demo/$port pushd demo/$port || exit
#rm $DIR/etc/$port.config #rm $DIR/etc/$port.config
python3 -m synapse.app.homeserver \ python3 -m synapse.app.homeserver \
@ -27,75 +28,78 @@ for port in 8080 8081 8082; do
--config-path "$DIR/etc/$port.config" \ --config-path "$DIR/etc/$port.config" \
--report-stats no --report-stats no
if ! grep -F "Customisation made by demo/start.sh" -q $DIR/etc/$port.config; then if ! grep -F "Customisation made by demo/start.sh" -q "$DIR/etc/$port.config"; then
printf '\n\n# Customisation made by demo/start.sh\n' >> $DIR/etc/$port.config
echo "public_baseurl: http://localhost:$port/" >> $DIR/etc/$port.config
echo 'enable_registration: true' >> $DIR/etc/$port.config
# Warning, this heredoc depends on the interaction of tabs and spaces. Please don't
# accidentaly bork me with your fancy settings.
listeners=$(cat <<-PORTLISTENERS
# Configure server to listen on both $https_port and $port
# This overides some of the default settings above
listeners:
- port: $https_port
type: http
tls: true
resources:
- names: [client, federation]
- port: $port
tls: false
bind_addresses: ['::1', '127.0.0.1']
type: http
x_forwarded: true
resources:
- names: [client, federation]
compress: false
PORTLISTENERS
)
echo "${listeners}" >> $DIR/etc/$port.config
# Disable tls for the servers
printf '\n\n# Disable tls on the servers.' >> $DIR/etc/$port.config
echo '# DO NOT USE IN PRODUCTION' >> $DIR/etc/$port.config
echo 'use_insecure_ssl_client_just_for_testing_do_not_use: true' >> $DIR/etc/$port.config
echo 'federation_verify_certificates: false' >> $DIR/etc/$port.config
# Set tls paths
echo "tls_certificate_path: \"$DIR/etc/localhost:$https_port.tls.crt\"" >> $DIR/etc/$port.config
echo "tls_private_key_path: \"$DIR/etc/localhost:$https_port.tls.key\"" >> $DIR/etc/$port.config
# Generate tls keys # Generate tls keys
openssl req -x509 -newkey rsa:4096 -keyout $DIR/etc/localhost\:$https_port.tls.key -out $DIR/etc/localhost\:$https_port.tls.crt -days 365 -nodes -subj "/O=matrix" openssl req -x509 -newkey rsa:4096 -keyout "$DIR/etc/localhost:$https_port.tls.key" -out "$DIR/etc/localhost:$https_port.tls.crt" -days 365 -nodes -subj "/O=matrix"
# Ignore keys from the trusted keys server # Regenerate configuration
echo '# Ignore keys from the trusted keys server' >> $DIR/etc/$port.config {
echo 'trusted_key_servers:' >> $DIR/etc/$port.config printf '\n\n# Customisation made by demo/start.sh\n'
echo ' - server_name: "matrix.org"' >> $DIR/etc/$port.config echo "public_baseurl: http://localhost:$port/"
echo ' accept_keys_insecurely: true' >> $DIR/etc/$port.config echo 'enable_registration: true'
# Reduce the blacklist # Warning, this heredoc depends on the interaction of tabs and spaces.
blacklist=$(cat <<-BLACK # Please don't accidentaly bork me with your fancy settings.
# Set the blacklist so that it doesn't include 127.0.0.1, ::1 listeners=$(cat <<-PORTLISTENERS
federation_ip_range_blacklist: # Configure server to listen on both $https_port and $port
- '10.0.0.0/8' # This overides some of the default settings above
- '172.16.0.0/12' listeners:
- '192.168.0.0/16' - port: $https_port
- '100.64.0.0/10' type: http
- '169.254.0.0/16' tls: true
- 'fe80::/64' resources:
- 'fc00::/7' - names: [client, federation]
BLACK
) - port: $port
echo "${blacklist}" >> $DIR/etc/$port.config tls: false
bind_addresses: ['::1', '127.0.0.1']
type: http
x_forwarded: true
resources:
- names: [client, federation]
compress: false
PORTLISTENERS
)
echo "${listeners}"
# Disable tls for the servers
printf '\n\n# Disable tls on the servers.'
echo '# DO NOT USE IN PRODUCTION'
echo 'use_insecure_ssl_client_just_for_testing_do_not_use: true'
echo 'federation_verify_certificates: false'
# Set tls paths
echo "tls_certificate_path: \"$DIR/etc/localhost:$https_port.tls.crt\""
echo "tls_private_key_path: \"$DIR/etc/localhost:$https_port.tls.key\""
# Ignore keys from the trusted keys server
echo '# Ignore keys from the trusted keys server'
echo 'trusted_key_servers:'
echo ' - server_name: "matrix.org"'
echo ' accept_keys_insecurely: true'
# Reduce the blacklist
blacklist=$(cat <<-BLACK
# Set the blacklist so that it doesn't include 127.0.0.1, ::1
federation_ip_range_blacklist:
- '10.0.0.0/8'
- '172.16.0.0/12'
- '192.168.0.0/16'
- '100.64.0.0/10'
- '169.254.0.0/16'
- 'fe80::/64'
- 'fc00::/7'
BLACK
)
echo "${blacklist}"
} >> "$DIR/etc/$port.config"
fi fi
# Check script parameters # Check script parameters
if [ $# -eq 1 ]; then if [ $# -eq 1 ]; then
if [ $1 = "--no-rate-limit" ]; then if [ "$1" = "--no-rate-limit" ]; then
# Disable any rate limiting # Disable any rate limiting
ratelimiting=$(cat <<-RC ratelimiting=$(cat <<-RC
@ -137,22 +141,22 @@ for port in 8080 8081 8082; do
burst_count: 1000 burst_count: 1000
RC RC
) )
echo "${ratelimiting}" >> $DIR/etc/$port.config echo "${ratelimiting}" >> "$DIR/etc/$port.config"
fi fi
fi fi
if ! grep -F "full_twisted_stacktraces" -q $DIR/etc/$port.config; then if ! grep -F "full_twisted_stacktraces" -q "$DIR/etc/$port.config"; then
echo "full_twisted_stacktraces: true" >> $DIR/etc/$port.config echo "full_twisted_stacktraces: true" >> "$DIR/etc/$port.config"
fi fi
if ! grep -F "report_stats" -q $DIR/etc/$port.config ; then if ! grep -F "report_stats" -q "$DIR/etc/$port.config" ; then
echo "report_stats: false" >> $DIR/etc/$port.config echo "report_stats: false" >> "$DIR/etc/$port.config"
fi fi
python3 -m synapse.app.homeserver \ python3 -m synapse.app.homeserver \
--config-path "$DIR/etc/$port.config" \ --config-path "$DIR/etc/$port.config" \
-D \ -D \
popd popd || exit
done done
cd "$CWD" cd "$CWD" || exit

View file

@ -8,7 +8,7 @@ for pid_file in $FILES; do
pid=$(cat "$pid_file") pid=$(cat "$pid_file")
if [[ $pid ]]; then if [[ $pid ]]; then
echo "Killing $pid_file with $pid" echo "Killing $pid_file with $pid"
kill $pid kill "$pid"
fi fi
done done

View file

@ -65,7 +65,8 @@ The following environment variables are supported in `generate` mode:
* `SYNAPSE_DATA_DIR`: where the generated config will put persistent data * `SYNAPSE_DATA_DIR`: where the generated config will put persistent data
such as the database and media store. Defaults to `/data`. such as the database and media store. Defaults to `/data`.
* `UID`, `GID`: the user id and group id to use for creating the data * `UID`, `GID`: the user id and group id to use for creating the data
directories. Defaults to `991`, `991`. directories. If unset, and no user is set via `docker run --user`, defaults
to `991`, `991`.
## Running synapse ## Running synapse
@ -97,7 +98,9 @@ The following environment variables are supported in `run` mode:
`<SYNAPSE_CONFIG_DIR>/homeserver.yaml`. `<SYNAPSE_CONFIG_DIR>/homeserver.yaml`.
* `SYNAPSE_WORKER`: module to execute, used when running synapse with workers. * `SYNAPSE_WORKER`: module to execute, used when running synapse with workers.
Defaults to `synapse.app.homeserver`, which is suitable for non-worker mode. Defaults to `synapse.app.homeserver`, which is suitable for non-worker mode.
* `UID`, `GID`: the user and group id to run Synapse as. Defaults to `991`, `991`. * `UID`, `GID`: the user and group id to run Synapse as. If unset, and no user
is set via `docker run --user`, defaults to `991`, `991`. Note that this user
must have permission to read the config files, and write to the data directories.
* `TZ`: the [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) the container will run with. Defaults to `UTC`. * `TZ`: the [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) the container will run with. Defaults to `UTC`.
For more complex setups (e.g. for workers) you can also pass your args directly to synapse using `run` mode. For example like this: For more complex setups (e.g. for workers) you can also pass your args directly to synapse using `run` mode. For example like this:
@ -186,7 +189,7 @@ point to another Dockerfile.
## Disabling the healthcheck ## Disabling the healthcheck
If you are using a non-standard port or tls inside docker you can disable the healthcheck If you are using a non-standard port or tls inside docker you can disable the healthcheck
whilst running the above `docker run` commands. whilst running the above `docker run` commands.
``` ```
--no-healthcheck --no-healthcheck
@ -212,7 +215,7 @@ If you wish to point the healthcheck at a different port with docker command, ad
## Setting the healthcheck in docker-compose file ## Setting the healthcheck in docker-compose file
You can add the following to set a custom healthcheck in a docker compose file. You can add the following to set a custom healthcheck in a docker compose file.
You will need docker-compose version >2.1 for this to work. You will need docker-compose version >2.1 for this to work.
``` ```
healthcheck: healthcheck:
@ -226,5 +229,5 @@ healthcheck:
## Using jemalloc ## Using jemalloc
Jemalloc is embedded in the image and will be used instead of the default allocator. Jemalloc is embedded in the image and will be used instead of the default allocator.
You can read about jemalloc by reading the Synapse You can read about jemalloc by reading the Synapse
[README](https://github.com/matrix-org/synapse/blob/HEAD/README.rst#help-synapse-is-slow-and-eats-all-my-ram-cpu). [README](https://github.com/matrix-org/synapse/blob/HEAD/README.rst#help-synapse-is-slow-and-eats-all-my-ram-cpu).

View file

@ -5,7 +5,7 @@
set -ex set -ex
# Get the codename from distro env # Get the codename from distro env
DIST=`cut -d ':' -f2 <<< $distro` DIST=$(cut -d ':' -f2 <<< "${distro:?}")
# we get a read-only copy of the source: make a writeable copy # we get a read-only copy of the source: make a writeable copy
cp -aT /synapse/source /synapse/build cp -aT /synapse/source /synapse/build
@ -17,7 +17,7 @@ cd /synapse/build
# Section to determine which "component" it should go into (see # Section to determine which "component" it should go into (see
# https://manpages.debian.org/stretch/reprepro/reprepro.1.en.html#GUESSING) # https://manpages.debian.org/stretch/reprepro/reprepro.1.en.html#GUESSING)
DEB_VERSION=`dpkg-parsechangelog -SVersion` DEB_VERSION=$(dpkg-parsechangelog -SVersion)
case $DEB_VERSION in case $DEB_VERSION in
*~rc*|*~a*|*~b*|*~c*) *~rc*|*~a*|*~b*|*~c*)
sed -ie '/^Section:/c\Section: prerelease' debian/control sed -ie '/^Section:/c\Section: prerelease' debian/control

View file

@ -120,6 +120,7 @@ def generate_config_from_template(config_dir, config_path, environ, ownership):
] ]
if ownership is not None: if ownership is not None:
log(f"Setting ownership on /data to {ownership}")
subprocess.check_output(["chown", "-R", ownership, "/data"]) subprocess.check_output(["chown", "-R", ownership, "/data"])
args = ["gosu", ownership] + args args = ["gosu", ownership] + args
@ -144,12 +145,18 @@ def run_generate_config(environ, ownership):
config_path = environ.get("SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml") config_path = environ.get("SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml")
data_dir = environ.get("SYNAPSE_DATA_DIR", "/data") data_dir = environ.get("SYNAPSE_DATA_DIR", "/data")
if ownership is not None:
# make sure that synapse has perms to write to the data dir.
log(f"Setting ownership on {data_dir} to {ownership}")
subprocess.check_output(["chown", ownership, data_dir])
# create a suitable log config from our template # create a suitable log config from our template
log_config_file = "%s/%s.log.config" % (config_dir, server_name) log_config_file = "%s/%s.log.config" % (config_dir, server_name)
if not os.path.exists(log_config_file): if not os.path.exists(log_config_file):
log("Creating log config %s" % (log_config_file,)) log("Creating log config %s" % (log_config_file,))
convert("/conf/log.config", log_config_file, environ) convert("/conf/log.config", log_config_file, environ)
# generate the main config file, and a signing key.
args = [ args = [
"python", "python",
"-m", "-m",
@ -168,29 +175,23 @@ def run_generate_config(environ, ownership):
"--open-private-ports", "--open-private-ports",
] ]
# log("running %s" % (args, )) # log("running %s" % (args, ))
os.execv("/usr/local/bin/python", args)
if ownership is not None:
# make sure that synapse has perms to write to the data dir.
subprocess.check_output(["chown", ownership, data_dir])
args = ["gosu", ownership] + args
os.execv("/usr/sbin/gosu", args)
else:
os.execv("/usr/local/bin/python", args)
def main(args, environ): def main(args, environ):
mode = args[1] if len(args) > 1 else "run" mode = args[1] if len(args) > 1 else "run"
desired_uid = int(environ.get("UID", "991"))
desired_gid = int(environ.get("GID", "991"))
synapse_worker = environ.get("SYNAPSE_WORKER", "synapse.app.homeserver")
if (desired_uid == os.getuid()) and (desired_gid == os.getgid()):
ownership = None
else:
ownership = "{}:{}".format(desired_uid, desired_gid)
if ownership is None: # if we were given an explicit user to switch to, do so
log("Will not perform chmod/gosu as UserID already matches request") ownership = None
if "UID" in environ:
desired_uid = int(environ["UID"])
desired_gid = int(environ.get("GID", "991"))
ownership = f"{desired_uid}:{desired_gid}"
elif os.getuid() == 0:
# otherwise, if we are running as root, use user 991
ownership = "991:991"
synapse_worker = environ.get("SYNAPSE_WORKER", "synapse.app.homeserver")
# In generate mode, generate a configuration and missing keys, then exit # In generate mode, generate a configuration and missing keys, then exit
if mode == "generate": if mode == "generate":

View file

@ -15,12 +15,12 @@ in `homeserver.yaml`, to the list of authorized domains. If you have not set
1. Agree to the terms of service and submit. 1. Agree to the terms of service and submit.
1. Copy your site key and secret key and add them to your `homeserver.yaml` 1. Copy your site key and secret key and add them to your `homeserver.yaml`
configuration file configuration file
``` ```yaml
recaptcha_public_key: YOUR_SITE_KEY recaptcha_public_key: YOUR_SITE_KEY
recaptcha_private_key: YOUR_SECRET_KEY recaptcha_private_key: YOUR_SECRET_KEY
``` ```
1. Enable the CAPTCHA for new registrations 1. Enable the CAPTCHA for new registrations
``` ```yaml
enable_registration_captcha: true enable_registration_captcha: true
``` ```
1. Go to the settings page for the CAPTCHA you just created 1. Go to the settings page for the CAPTCHA you just created

View file

@ -51,6 +51,7 @@
- [Administration](usage/administration/README.md) - [Administration](usage/administration/README.md)
- [Admin API](usage/administration/admin_api/README.md) - [Admin API](usage/administration/admin_api/README.md)
- [Account Validity](admin_api/account_validity.md) - [Account Validity](admin_api/account_validity.md)
- [Background Updates](usage/administration/admin_api/background_updates.md)
- [Delete Group](admin_api/delete_group.md) - [Delete Group](admin_api/delete_group.md)
- [Event Reports](admin_api/event_reports.md) - [Event Reports](admin_api/event_reports.md)
- [Media](admin_api/media_admin_api.md) - [Media](admin_api/media_admin_api.md)

View file

@ -99,7 +99,7 @@ server admin: see [Admin API](../usage/administration/admin_api).
It returns a JSON body like the following: It returns a JSON body like the following:
```jsonc ```json
{ {
"event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY", "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
"event_json": { "event_json": {
@ -132,7 +132,7 @@ It returns a JSON body like the following:
}, },
"type": "m.room.message", "type": "m.room.message",
"unsigned": { "unsigned": {
"age_ts": 1592291711430, "age_ts": 1592291711430
} }
}, },
"id": <report_id>, "id": <report_id>,

View file

@ -27,7 +27,7 @@ Room state data (such as joins, leaves, topic) is always preserved.
To delete local message events as well, set `delete_local_events` in the body: To delete local message events as well, set `delete_local_events` in the body:
``` ```json
{ {
"delete_local_events": true "delete_local_events": true
} }

View file

@ -28,7 +28,7 @@ server admin: see [Admin API](../usage/administration/admin_api).
Response: Response:
``` ```json
{ {
"room_id": "!636q39766251:server.com" "room_id": "!636q39766251:server.com"
} }

View file

@ -38,9 +38,14 @@ The following query parameters are available:
- `history_visibility` - Rooms are ordered alphabetically by visibility of history of the room. - `history_visibility` - Rooms are ordered alphabetically by visibility of history of the room.
- `state_events` - Rooms are ordered by number of state events. Largest to smallest. - `state_events` - Rooms are ordered by number of state events. Largest to smallest.
* `dir` - Direction of room order. Either `f` for forwards or `b` for backwards. Setting * `dir` - Direction of room order. Either `f` for forwards or `b` for backwards. Setting
this value to `b` will reverse the above sort order. Defaults to `f`. this value to `b` will reverse the above sort order. Defaults to `f`.
* `search_term` - Filter rooms by their room name. Search term can be contained in any * `search_term` - Filter rooms by their room name, canonical alias and room id.
part of the room name. Defaults to no filtering. Specifically, rooms are selected if the search term is contained in
- the room's name,
- the local part of the room's canonical alias, or
- the complete (local and server part) room's id (case sensitive).
Defaults to no filtering.
**Response** **Response**
@ -87,7 +92,7 @@ GET /_synapse/admin/v1/rooms
A response body like the following is returned: A response body like the following is returned:
```jsonc ```json
{ {
"rooms": [ "rooms": [
{ {
@ -170,7 +175,7 @@ GET /_synapse/admin/v1/rooms?order_by=size
A response body like the following is returned: A response body like the following is returned:
```jsonc ```json
{ {
"rooms": [ "rooms": [
{ {
@ -208,7 +213,7 @@ A response body like the following is returned:
} }
], ],
"offset": 0, "offset": 0,
"total_rooms": 150 "total_rooms": 150,
"next_token": 100 "next_token": 100
} }
``` ```
@ -224,7 +229,7 @@ GET /_synapse/admin/v1/rooms?order_by=size&from=100
A response body like the following is returned: A response body like the following is returned:
```jsonc ```json
{ {
"rooms": [ "rooms": [
{ {
@ -380,7 +385,7 @@ A response body like the following is returned:
# Delete Room API # Delete Room API
The Delete Room admin API allows server admins to remove rooms from server The Delete Room admin API allows server admins to remove rooms from the server
and block these rooms. and block these rooms.
Shuts down a room. Moves all local users and room aliases automatically to a Shuts down a room. Moves all local users and room aliases automatically to a
@ -520,16 +525,6 @@ With all that being said, if you still want to try and recover the room:
4. If `new_room_user_id` was given, a 'Content Violation' will have been 4. If `new_room_user_id` was given, a 'Content Violation' will have been
created. Consider whether you want to delete that roomm. created. Consider whether you want to delete that roomm.
## Deprecated endpoint
The previous deprecated API will be removed in a future release, it was:
```
POST /_synapse/admin/v1/rooms/<room_id>/delete
```
It behaves the same way than the current endpoint except the path and the method.
# Make Room Admin API # Make Room Admin API
Grants another user the highest power available to a local user who is in the room. Grants another user the highest power available to a local user who is in the room.

View file

@ -10,7 +10,9 @@ The necessary tools are detailed below.
First install them with: First install them with:
pip install -e ".[lint,mypy]" ```sh
pip install -e ".[lint,mypy]"
```
- **black** - **black**
@ -21,7 +23,9 @@ First install them with:
Have `black` auto-format your code (it shouldn't change any Have `black` auto-format your code (it shouldn't change any
functionality) with: functionality) with:
black . --exclude="\.tox|build|env" ```sh
black . --exclude="\.tox|build|env"
```
- **flake8** - **flake8**
@ -30,7 +34,9 @@ First install them with:
Check all application and test code with: Check all application and test code with:
flake8 synapse tests ```sh
flake8 synapse tests
```
- **isort** - **isort**
@ -39,7 +45,9 @@ First install them with:
Auto-fix imports with: Auto-fix imports with:
isort -rc synapse tests ```sh
isort -rc synapse tests
```
`-rc` means to recursively search the given directories. `-rc` means to recursively search the given directories.
@ -66,15 +74,19 @@ save as it takes a while and is very resource intensive.
Example: Example:
from synapse.types import UserID ```python
... from synapse.types import UserID
user_id = UserID(local, server) ...
user_id = UserID(local, server)
```
is preferred over: is preferred over:
from synapse import types ```python
... from synapse import types
user_id = types.UserID(local, server) ...
user_id = types.UserID(local, server)
```
(or any other variant). (or any other variant).
@ -134,28 +146,30 @@ Some guidelines follow:
Example: Example:
## Frobnication ## ```yaml
## Frobnication ##
# The frobnicator will ensure that all requests are fully frobnicated. # The frobnicator will ensure that all requests are fully frobnicated.
# To enable it, uncomment the following. # To enable it, uncomment the following.
# #
#frobnicator_enabled: true #frobnicator_enabled: true
# By default, the frobnicator will frobnicate with the default frobber. # By default, the frobnicator will frobnicate with the default frobber.
# The following will make it use an alternative frobber. # The following will make it use an alternative frobber.
# #
#frobincator_frobber: special_frobber #frobincator_frobber: special_frobber
# Settings for the frobber # Settings for the frobber
# #
frobber: frobber:
# frobbing speed. Defaults to 1. # frobbing speed. Defaults to 1.
# #
#speed: 10 #speed: 10
# frobbing distance. Defaults to 1000. # frobbing distance. Defaults to 1000.
# #
#distance: 100 #distance: 100
```
Note that the sample configuration is generated from the synapse code Note that the sample configuration is generated from the synapse code
and is maintained by a script, `scripts-dev/generate_sample_config`. and is maintained by a script, `scripts-dev/generate_sample_config`.

View file

@ -99,7 +99,7 @@ construct URIs where users can give their consent.
see if an unauthenticated user is viewing the page. This is typically see if an unauthenticated user is viewing the page. This is typically
wrapped around the form that would be used to actually agree to the document: wrapped around the form that would be used to actually agree to the document:
``` ```html
{% if not public_version %} {% if not public_version %}
<!-- The variables used here are only provided when the 'u' param is given to the homeserver --> <!-- The variables used here are only provided when the 'u' param is given to the homeserver -->
<form method="post" action="consent"> <form method="post" action="consent">

View file

@ -1,4 +1,8 @@
# Delegation # Delegation of incoming federation traffic
In the following documentation, we use the term `server_name` to refer to that setting
in your homeserver configuration file. It appears at the ends of user ids, and tells
other homeservers where they can find your server.
By default, other homeservers will expect to be able to reach yours via By default, other homeservers will expect to be able to reach yours via
your `server_name`, on port 8448. For example, if you set your `server_name` your `server_name`, on port 8448. For example, if you set your `server_name`
@ -12,13 +16,21 @@ to a different server and/or port (e.g. `synapse.example.com:443`).
## .well-known delegation ## .well-known delegation
To use this method, you need to be able to alter the To use this method, you need to be able to configure the server at
`server_name` 's https server to serve the `/.well-known/matrix/server` `https://<server_name>` to serve a file at
URL. Having an active server (with a valid TLS certificate) serving your `https://<server_name>/.well-known/matrix/server`. There are two ways to do this, shown below.
`server_name` domain is out of the scope of this documentation.
The URL `https://<server_name>/.well-known/matrix/server` should Note that the `.well-known` file is hosted on the default port for `https` (port 443).
return a JSON structure containing the key `m.server` like so:
### External server
For maximum flexibility, you need to configure an external server such as nginx, Apache
or HAProxy to serve the `https://<server_name>/.well-known/matrix/server` file. Setting
up such a server is out of the scope of this documentation, but note that it is often
possible to configure your [reverse proxy](reverse_proxy.md) for this.
The URL `https://<server_name>/.well-known/matrix/server` should be configured
return a JSON structure containing the key `m.server` like this:
```json ```json
{ {
@ -26,8 +38,9 @@ return a JSON structure containing the key `m.server` like so:
} }
``` ```
In our example, this would mean that URL `https://example.com/.well-known/matrix/server` In our example (where we want federation traffic to be routed to
should return: `https://synapse.example.com`, on port 443), this would mean that
`https://example.com/.well-known/matrix/server` should return:
```json ```json
{ {
@ -38,16 +51,29 @@ should return:
Note, specifying a port is optional. If no port is specified, then it defaults Note, specifying a port is optional. If no port is specified, then it defaults
to 8448. to 8448.
With .well-known delegation, federating servers will check for a valid TLS ### Serving a `.well-known/matrix/server` file with Synapse
certificate for the delegated hostname (in our example: `synapse.example.com`).
If you are able to set up your domain so that `https://<server_name>` is routed to
Synapse (i.e., the only change needed is to direct federation traffic to port 443
instead of port 8448), then it is possible to configure Synapse to serve a suitable
`.well-known/matrix/server` file. To do so, add the following to your `homeserver.yaml`
file:
```yaml
serve_server_wellknown: true
```
**Note**: this *only* works if `https://<server_name>` is routed to Synapse, so is
generally not suitable if Synapse is hosted at a subdomain such as
`https://synapse.example.com`.
## SRV DNS record delegation ## SRV DNS record delegation
It is also possible to do delegation using a SRV DNS record. However, that is It is also possible to do delegation using a SRV DNS record. However, that is generally
considered an advanced topic since it's a bit complex to set up, and `.well-known` not recommended, as it can be difficult to configure the TLS certificates correctly in
delegation is already enough in most cases. this case, and it offers little advantage over `.well-known` delegation.
However, if you really need it, you can find some documentation on how such a However, if you really need it, you can find some documentation on what such a
record should look like and how Synapse will use it in [the Matrix record should look like and how Synapse will use it in [the Matrix
specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names). specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names).
@ -68,27 +94,9 @@ wouldn't need any delegation set up.
domain `server_name` points to, you will need to let other servers know how to domain `server_name` points to, you will need to let other servers know how to
find it using delegation. find it using delegation.
### Do you still recommend against using a reverse proxy on the federation port? ### Should I use a reverse proxy for federation traffic?
We no longer actively recommend against using a reverse proxy. Many admins will Generally, using a reverse proxy for both the federation and client traffic is a good
find it easier to direct federation traffic to a reverse proxy and manage their idea, since it saves handling TLS traffic in Synapse. See
own TLS certificates, and this is a supported configuration. [the reverse proxy documentation](reverse_proxy.md) for information on setting up a
See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a
reverse proxy. reverse proxy.
### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy?
This is no longer necessary. If you are using a reverse proxy for all of your
TLS traffic, then you can set `no_tls: True` in the Synapse config.
In that case, the only reason Synapse needs the certificate is to populate a legacy
`tls_fingerprints` field in the federation API. This is ignored by Synapse 0.99.0
and later, and the only time pre-0.99 Synapses will check it is when attempting to
fetch the server keys - and generally this is delegated via `matrix.org`, which
is running a modern version of Synapse.
### Do I need the same certificate for the client and federation port?
No. There is nothing stopping you from using different certificates,
particularly if you are using a reverse proxy.

View file

@ -8,23 +8,23 @@ easy to run CAS implementation built on top of Django.
1. Create a new virtualenv: `python3 -m venv <your virtualenv>` 1. Create a new virtualenv: `python3 -m venv <your virtualenv>`
2. Activate your virtualenv: `source /path/to/your/virtualenv/bin/activate` 2. Activate your virtualenv: `source /path/to/your/virtualenv/bin/activate`
3. Install Django and django-mama-cas: 3. Install Django and django-mama-cas:
``` ```sh
python -m pip install "django<3" "django-mama-cas==2.4.0" python -m pip install "django<3" "django-mama-cas==2.4.0"
``` ```
4. Create a Django project in the current directory: 4. Create a Django project in the current directory:
``` ```sh
django-admin startproject cas_test . django-admin startproject cas_test .
``` ```
5. Follow the [install directions](https://django-mama-cas.readthedocs.io/en/latest/installation.html#configuring) for django-mama-cas 5. Follow the [install directions](https://django-mama-cas.readthedocs.io/en/latest/installation.html#configuring) for django-mama-cas
6. Setup the SQLite database: `python manage.py migrate` 6. Setup the SQLite database: `python manage.py migrate`
7. Create a user: 7. Create a user:
``` ```sh
python manage.py createsuperuser python manage.py createsuperuser
``` ```
1. Use whatever you want as the username and password. 1. Use whatever you want as the username and password.
2. Leave the other fields blank. 2. Leave the other fields blank.
8. Use the built-in Django test server to serve the CAS endpoints on port 8000: 8. Use the built-in Django test server to serve the CAS endpoints on port 8000:
``` ```sh
python manage.py runserver python manage.py runserver
``` ```

View file

@ -15,6 +15,11 @@ license - in our case, this is almost always Apache Software License v2 (see
# 2. What do I need? # 2. What do I need?
If you are running Windows, the Windows Subsystem for Linux (WSL) is strongly
recommended for development. More information about WSL can be found at
<https://docs.microsoft.com/en-us/windows/wsl/install>. Running Synapse natively
on Windows is not officially supported.
The code of Synapse is written in Python 3. To do pretty much anything, you'll need [a recent version of Python 3](https://wiki.python.org/moin/BeginnersGuide/Download). The code of Synapse is written in Python 3. To do pretty much anything, you'll need [a recent version of Python 3](https://wiki.python.org/moin/BeginnersGuide/Download).
The source code of Synapse is hosted on GitHub. You will also need [a recent version of git](https://github.com/git-guides/install-git). The source code of Synapse is hosted on GitHub. You will also need [a recent version of git](https://github.com/git-guides/install-git).
@ -41,8 +46,6 @@ can find many good git tutorials on the web.
# 4. Install the dependencies # 4. Install the dependencies
## Under Unix (macOS, Linux, BSD, ...)
Once you have installed Python 3 and added the source, please open a terminal and Once you have installed Python 3 and added the source, please open a terminal and
setup a *virtualenv*, as follows: setup a *virtualenv*, as follows:
@ -56,10 +59,6 @@ pip install tox
This will install the developer dependencies for the project. This will install the developer dependencies for the project.
## Under Windows
TBD
# 5. Get in touch. # 5. Get in touch.

View file

@ -89,7 +89,9 @@ To do so, use `scripts-dev/make_full_schema.sh`. This will produce new
Ensure postgres is installed, then run: Ensure postgres is installed, then run:
./scripts-dev/make_full_schema.sh -p postgres_username -o output_dir/ ```sh
./scripts-dev/make_full_schema.sh -p postgres_username -o output_dir/
```
NB at the time of writing, this script predates the split into separate `state`/`main` NB at the time of writing, this script predates the split into separate `state`/`main`
databases so will require updates to handle that correctly. databases so will require updates to handle that correctly.

View file

@ -15,7 +15,7 @@ To make Synapse (and therefore Element) use it:
sp_config: sp_config:
allow_unknown_attributes: true # Works around a bug with AVA Hashes: https://github.com/IdentityPython/pysaml2/issues/388 allow_unknown_attributes: true # Works around a bug with AVA Hashes: https://github.com/IdentityPython/pysaml2/issues/388
metadata: metadata:
local: ["samling.xml"] local: ["samling.xml"]
``` ```
5. Ensure that your `homeserver.yaml` has a setting for `public_baseurl`: 5. Ensure that your `homeserver.yaml` has a setting for `public_baseurl`:
```yaml ```yaml

View file

@ -69,9 +69,9 @@ A default policy can be defined as such, in the `retention` section of
the configuration file: the configuration file:
```yaml ```yaml
default_policy: default_policy:
min_lifetime: 1d min_lifetime: 1d
max_lifetime: 1y max_lifetime: 1y
``` ```
Here, `min_lifetime` and `max_lifetime` have the same meaning and level Here, `min_lifetime` and `max_lifetime` have the same meaning and level
@ -95,14 +95,14 @@ depending on an event's room's policy. This can be done by setting the
file. An example of such configuration could be: file. An example of such configuration could be:
```yaml ```yaml
purge_jobs: purge_jobs:
- longest_max_lifetime: 3d - longest_max_lifetime: 3d
interval: 12h interval: 12h
- shortest_max_lifetime: 3d - shortest_max_lifetime: 3d
longest_max_lifetime: 1w longest_max_lifetime: 1w
interval: 1d interval: 1d
- shortest_max_lifetime: 1w - shortest_max_lifetime: 1w
interval: 2d interval: 2d
``` ```
In this example, we define three jobs: In this example, we define three jobs:
@ -141,8 +141,8 @@ purging old events in a room. These limits can be defined as such in the
`retention` section of the configuration file: `retention` section of the configuration file:
```yaml ```yaml
allowed_lifetime_min: 1d allowed_lifetime_min: 1d
allowed_lifetime_max: 1y allowed_lifetime_max: 1y
``` ```
The limits are considered when running purge jobs. If necessary, the The limits are considered when running purge jobs. If necessary, the

View file

@ -10,8 +10,8 @@ registered by using the Module API's `register_password_auth_provider_callbacks`
_First introduced in Synapse v1.46.0_ _First introduced in Synapse v1.46.0_
``` ```python
auth_checkers: Dict[Tuple[str,Tuple], Callable] auth_checkers: Dict[Tuple[str, Tuple[str, ...]], Callable]
``` ```
A dict mapping from tuples of a login type identifier (such as `m.login.password`) and a A dict mapping from tuples of a login type identifier (such as `m.login.password`) and a

View file

@ -123,42 +123,6 @@ callback returns `True`, Synapse falls through to the next one. The value of the
callback that does not return `True` will be used. If this happens, Synapse will not call callback that does not return `True` will be used. If this happens, Synapse will not call
any of the subsequent implementations of this callback. any of the subsequent implementations of this callback.
### `user_may_create_room_with_invites`
_First introduced in Synapse v1.44.0_
```python
async def user_may_create_room_with_invites(
user: str,
invites: List[str],
threepid_invites: List[Dict[str, str]],
) -> bool
```
Called when processing a room creation request (right after `user_may_create_room`).
The module is given the Matrix user ID of the user trying to create a room, as well as a
list of Matrix users to invite and a list of third-party identifiers (3PID, e.g. email
addresses) to invite.
An invited Matrix user to invite is represented by their Matrix user IDs, and an invited
3PIDs is represented by a dict that includes the 3PID medium (e.g. "email") through its
`medium` key and its address (e.g. "alice@example.com") through its `address` key.
See [the Matrix specification](https://matrix.org/docs/spec/appendices#pid-types) for more
information regarding third-party identifiers.
If no invite and/or 3PID invite were specified in the room creation request, the
corresponding list(s) will be empty.
**Note**: This callback is not called when a room is cloned (e.g. during a room upgrade)
since no invites are sent when cloning a room. To cover this case, modules also need to
implement `user_may_create_room`.
If multiple modules implement this callback, they will be considered in order. If a
callback returns `True`, Synapse falls through to the next one. The value of the first
callback that does not return `True` will be used. If this happens, Synapse will not call
any of the subsequent implementations of this callback.
### `user_may_create_room_alias` ### `user_may_create_room_alias`
_First introduced in Synapse v1.37.0_ _First introduced in Synapse v1.37.0_

View file

@ -43,6 +43,14 @@ event with new data by returning the new event's data as a dictionary. In order
that, it is recommended the module calls `event.get_dict()` to get the current event as a that, it is recommended the module calls `event.get_dict()` to get the current event as a
dictionary, and modify the returned dictionary accordingly. dictionary, and modify the returned dictionary accordingly.
If `check_event_allowed` raises an exception, the module is assumed to have failed.
The event will not be accepted but is not treated as explicitly rejected, either.
An HTTP request causing the module check will likely result in a 500 Internal
Server Error.
When the boolean returned by the module is `False`, the event is rejected.
(Module developers should not use exceptions for rejection.)
Note that replacing the event only works for events sent by local users, not for events Note that replacing the event only works for events sent by local users, not for events
received over federation. received over federation.
@ -119,6 +127,27 @@ callback returns `True`, Synapse falls through to the next one. The value of the
callback that does not return `True` will be used. If this happens, Synapse will not call callback that does not return `True` will be used. If this happens, Synapse will not call
any of the subsequent implementations of this callback. any of the subsequent implementations of this callback.
### `on_new_event`
_First introduced in Synapse v1.47.0_
```python
async def on_new_event(
event: "synapse.events.EventBase",
state_events: "synapse.types.StateMap",
) -> None:
```
Called after sending an event into a room. The module is passed the event, as well
as the state of the room _after_ the event. This means that if the event is a state event,
it will be included in this state.
Note that this callback is called when the event has already been processed and stored
into the room, which means this callback cannot be used to deny persisting the event. To
deny an incoming event, see [`check_event_for_spam`](spam_checker_callbacks.md#check_event_for_spam) instead.
If multiple modules implement this callback, Synapse runs them all in order.
## Example ## Example
The example below is a module that implements the third-party rules callback The example below is a module that implements the third-party rules callback

View file

@ -21,6 +21,8 @@ such as [Github][github-idp].
[google-idp]: https://developers.google.com/identity/protocols/oauth2/openid-connect [google-idp]: https://developers.google.com/identity/protocols/oauth2/openid-connect
[auth0]: https://auth0.com/ [auth0]: https://auth0.com/
[authentik]: https://goauthentik.io/
[lemonldap]: https://lemonldap-ng.org/
[okta]: https://www.okta.com/ [okta]: https://www.okta.com/
[dex-idp]: https://github.com/dexidp/dex [dex-idp]: https://github.com/dexidp/dex
[keycloak-idp]: https://www.keycloak.org/docs/latest/server_admin/#sso-protocols [keycloak-idp]: https://www.keycloak.org/docs/latest/server_admin/#sso-protocols
@ -209,6 +211,76 @@ oidc_providers:
display_name_template: "{{ user.name }}" display_name_template: "{{ user.name }}"
``` ```
### Authentik
[Authentik][authentik] is an open-source IdP solution.
1. Create a provider in Authentik, with type OAuth2/OpenID.
2. The parameters are:
- Client Type: Confidential
- JWT Algorithm: RS256
- Scopes: OpenID, Email and Profile
- RSA Key: Select any available key
- Redirect URIs: `[synapse public baseurl]/_synapse/client/oidc/callback`
3. Create an application for synapse in Authentik and link it to the provider.
4. Note the slug of your application, Client ID and Client Secret.
Synapse config:
```yaml
oidc_providers:
- idp_id: authentik
idp_name: authentik
discover: true
issuer: "https://your.authentik.example.org/application/o/your-app-slug/" # TO BE FILLED: domain and slug
client_id: "your client id" # TO BE FILLED
client_secret: "your client secret" # TO BE FILLED
scopes:
- "openid"
- "profile"
- "email"
user_mapping_provider:
config:
localpart_template: "{{ user.preferred_username }}}"
display_name_template: "{{ user.preferred_username|capitalize }}" # TO BE FILLED: If your users have names in Authentik and you want those in Synapse, this should be replaced with user.name|capitalize.
```
### LemonLDAP
[LemonLDAP::NG][lemonldap] is an open-source IdP solution.
1. Create an OpenID Connect Relying Parties in LemonLDAP::NG
2. The parameters are:
- Client ID under the basic menu of the new Relying Parties (`Options > Basic >
Client ID`)
- Client secret (`Options > Basic > Client secret`)
- JWT Algorithm: RS256 within the security menu of the new Relying Parties
(`Options > Security > ID Token signature algorithm` and `Options > Security >
Access Token signature algorithm`)
- Scopes: OpenID, Email and Profile
- Allowed redirection addresses for login (`Options > Basic > Allowed
redirection addresses for login` ) :
`[synapse public baseurl]/_synapse/client/oidc/callback`
Synapse config:
```yaml
oidc_providers:
- idp_id: lemonldap
idp_name: lemonldap
discover: true
issuer: "https://auth.example.org/" # TO BE FILLED: replace with your domain
client_id: "your client id" # TO BE FILLED
client_secret: "your client secret" # TO BE FILLED
scopes:
- "openid"
- "profile"
- "email"
user_mapping_provider:
config:
localpart_template: "{{ user.preferred_username }}}"
# TO BE FILLED: If your users have names in LemonLDAP::NG and you want those in Synapse, this should be replaced with user.name|capitalize or any valid filter.
display_name_template: "{{ user.preferred_username|capitalize }}"
```
### GitHub ### GitHub
[GitHub][github-idp] is a bit special as it is not an OpenID Connect compliant provider, but [GitHub][github-idp] is a bit special as it is not an OpenID Connect compliant provider, but

View file

@ -29,16 +29,20 @@ connect to a postgres database.
Assuming your PostgreSQL database user is called `postgres`, first authenticate as the database user with: Assuming your PostgreSQL database user is called `postgres`, first authenticate as the database user with:
su - postgres ```sh
# Or, if your system uses sudo to get administrative rights su - postgres
sudo -u postgres bash # Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
```
Then, create a postgres user and a database with: Then, create a postgres user and a database with:
# this will prompt for a password for the new user ```sh
createuser --pwprompt synapse_user # this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
```
The above will create a user called `synapse_user`, and a database called The above will create a user called `synapse_user`, and a database called
`synapse`. `synapse`.
@ -145,20 +149,26 @@ Firstly, shut down the currently running synapse server and copy its
database file (typically `homeserver.db`) to another location. Once the database file (typically `homeserver.db`) to another location. Once the
copy is complete, restart synapse. For instance: copy is complete, restart synapse. For instance:
./synctl stop ```sh
cp homeserver.db homeserver.db.snapshot ./synctl stop
./synctl start cp homeserver.db homeserver.db.snapshot
./synctl start
```
Copy the old config file into a new config file: Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml ```sh
cp homeserver.yaml homeserver-postgres.yaml
```
Edit the database section as described in the section *Synapse config* Edit the database section as described in the section *Synapse config*
above and with the SQLite snapshot located at `homeserver.db.snapshot` above and with the SQLite snapshot located at `homeserver.db.snapshot`
simply run: simply run:
synapse_port_db --sqlite-database homeserver.db.snapshot \ ```sh
--postgres-config homeserver-postgres.yaml synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml
```
The flag `--curses` displays a coloured curses progress UI. The flag `--curses` displays a coloured curses progress UI.
@ -170,16 +180,20 @@ To complete the conversion shut down the synapse server and run the port
script one last time, e.g. if the SQLite database is at `homeserver.db` script one last time, e.g. if the SQLite database is at `homeserver.db`
run: run:
synapse_port_db --sqlite-database homeserver.db \ ```sh
--postgres-config homeserver-postgres.yaml synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml
```
Once that has completed, change the synapse config to point at the Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file `homeserver-postgres.yaml`: PostgreSQL database configuration file `homeserver-postgres.yaml`:
./synctl stop ```sh
mv homeserver.yaml homeserver-old-sqlite.yaml ./synctl stop
mv homeserver-postgres.yaml homeserver.yaml mv homeserver.yaml homeserver-old-sqlite.yaml
./synctl start mv homeserver-postgres.yaml homeserver.yaml
./synctl start
```
Synapse should now be running against PostgreSQL. Synapse should now be running against PostgreSQL.

View file

@ -52,7 +52,7 @@ to proxied traffic.)
### nginx ### nginx
``` ```nginx
server { server {
listen 443 ssl http2; listen 443 ssl http2;
listen [::]:443 ssl http2; listen [::]:443 ssl http2;
@ -141,7 +141,7 @@ matrix.example.com {
### Apache ### Apache
``` ```apache
<VirtualHost *:443> <VirtualHost *:443>
SSLEngine on SSLEngine on
ServerName matrix.example.com ServerName matrix.example.com
@ -170,7 +170,7 @@ matrix.example.com {
**NOTE 2**: It appears that Synapse is currently incompatible with the ModSecurity module for Apache (`mod_security2`). If you need it enabled for other services on your web server, you can disable it for Synapse's two VirtualHosts by including the following lines before each of the two `</VirtualHost>` above: **NOTE 2**: It appears that Synapse is currently incompatible with the ModSecurity module for Apache (`mod_security2`). If you need it enabled for other services on your web server, you can disable it for Synapse's two VirtualHosts by including the following lines before each of the two `</VirtualHost>` above:
``` ```apache
<IfModule security2_module> <IfModule security2_module>
SecRuleEngine off SecRuleEngine off
</IfModule> </IfModule>
@ -188,7 +188,7 @@ frontend https
http-request set-header X-Forwarded-For %[src] http-request set-header X-Forwarded-For %[src]
# Matrix client traffic # Matrix client traffic
acl matrix-host hdr(host) -i matrix.example.com acl matrix-host hdr(host) -i matrix.example.com matrix.example.com:443
acl matrix-path path_beg /_matrix acl matrix-path path_beg /_matrix
acl matrix-path path_beg /_synapse/client acl matrix-path path_beg /_synapse/client

View file

@ -91,8 +91,28 @@ pid_file: DATADIR/homeserver.pid
# Otherwise, it should be the URL to reach Synapse's client HTTP listener (see # Otherwise, it should be the URL to reach Synapse's client HTTP listener (see
# 'listeners' below). # 'listeners' below).
# #
# Defaults to 'https://<server_name>/'.
#
#public_baseurl: https://example.com/ #public_baseurl: https://example.com/
# Uncomment the following to tell other servers to send federation traffic on
# port 443.
#
# By default, other servers will try to reach our server on port 8448, which can
# be inconvenient in some environments.
#
# Provided 'https://<server_name>/' on port 443 is routed to Synapse, this
# option configures Synapse to serve a file at
# 'https://<server_name>/.well-known/matrix/server'. This will tell other
# servers to send traffic to port 443 instead.
#
# See https://matrix-org.github.io/synapse/latest/delegate.html for more
# information.
#
# Defaults to 'false'.
#
#serve_server_wellknown: true
# Set the soft limit on the number of file descriptors synapse can use # Set the soft limit on the number of file descriptors synapse can use
# Zero is used to indicate synapse should set the soft limit to the # Zero is used to indicate synapse should set the soft limit to the
# hard limit. # hard limit.
@ -1247,7 +1267,7 @@ oembed:
# in on this server. # in on this server.
# #
# (By default, no suggestion is made, so it is left up to the client. # (By default, no suggestion is made, so it is left up to the client.
# This setting is ignored unless public_baseurl is also set.) # This setting is ignored unless public_baseurl is also explicitly set.)
# #
#default_identity_server: https://matrix.org #default_identity_server: https://matrix.org
@ -1272,8 +1292,6 @@ oembed:
# by the Matrix Identity Service API specification: # by the Matrix Identity Service API specification:
# https://matrix.org/docs/spec/identity_service/latest # https://matrix.org/docs/spec/identity_service/latest
# #
# If a delegate is specified, the config option public_baseurl must also be filled out.
#
account_threepid_delegates: account_threepid_delegates:
#email: https://example.com # Delegate email sending to example.com #email: https://example.com # Delegate email sending to example.com
#msisdn: http://localhost:8090 # Delegate SMS sending to this local process #msisdn: http://localhost:8090 # Delegate SMS sending to this local process
@ -1963,11 +1981,10 @@ sso:
# phishing attacks from evil.site. To avoid this, include a slash after the # phishing attacks from evil.site. To avoid this, include a slash after the
# hostname: "https://my.client/". # hostname: "https://my.client/".
# #
# If public_baseurl is set, then the login fallback page (used by clients # The login fallback page (used by clients that don't natively support the
# that don't natively support the required login flows) is whitelisted in # required login flows) is whitelisted in addition to any URLs in this list.
# addition to any URLs in this list.
# #
# By default, this list is empty. # By default, this list contains only the login fallback page.
# #
#client_whitelist: #client_whitelist:
# - https://riot.im/develop # - https://riot.im/develop

Some files were not shown because too many files have changed in this diff Show more