maxmustermann/synapse
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes

Browse source
This commit is contained in:
Erik Johnston 2019-06-28 10:04:54 +01:00
parent ad2ba70959 be3b901ccd
commit 576b62a6a3
92 changed files with 7884 additions and 618 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.circleci/config.yml
View file

@ -4,7 +4,7 @@ jobs:
machine: true
steps:
- checkout
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 --build-arg PYTHON_VERSION=3.6 .
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 .
- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
- run: docker push matrixdotorg/synapse:${CIRCLE_TAG}
- run: docker push matrixdotorg/synapse:${CIRCLE_TAG}-py3
@ -12,7 +12,7 @@ jobs:
machine: true
steps:
- checkout
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 --build-arg PYTHON_VERSION=3.6 .
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 .
- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
- run: docker push matrixdotorg/synapse:latest
- run: docker push matrixdotorg/synapse:latest-py3

22
.dockerignore
View file

@ -1,9 +1,13 @@
Dockerfile
.travis.yml
.gitignore
demo/etc
tox.ini
.git/*
.tox/*
debian/matrix-synapse/
debian/matrix-synapse-*/
# ignore everything by default
*
# things to include
!docker
!scripts
!synapse
!MANIFEST.in
!README.rst
!setup.py
!synctl
**/__pycache__

6
CHANGES.md
View file

@ -52,7 +52,11 @@ Features
- Add a script to generate new signing-key files. ([\#5361](https://github.com/matrix-org/synapse/issues/5361))
- Update upgrade and installation guides ahead of 1.0. ([\#5371](https://github.com/matrix-org/synapse/issues/5371))
- Replace the `perspectives` configuration section with `trusted_key_servers`, and make validating the signatures on responses optional (since TLS will do this job for us). ([\#5374](https://github.com/matrix-org/synapse/issues/5374))
- Add ability to perform password reset via email without trusting the identity server. ([\#5377](https://github.com/matrix-org/synapse/issues/5377))
- Add ability to perform password reset via email without trusting the identity server. **As a result of this PR, password resets will now be disabled on the default configuration.**
Password reset emails are now sent from the homeserver by default, instead of the identity server. To enable this functionality, ensure `email` and `public_baseurl` config options are filled out.
If you would like to re-enable password resets being sent from the identity server (warning: this is dangerous! See [#5345](https://github.com/matrix-org/synapse/pull/5345)), set `email.trust_identity_server_for_password_resets` to true. ([\#5377](https://github.com/matrix-org/synapse/issues/5377))
- Set default room version to v4. ([\#5379](https://github.com/matrix-org/synapse/issues/5379))

31
INSTALL.md
View file

@ -1,3 +1,4 @@
- [Choosing your server name](#choosing-your-server-name)
- [Installing Synapse](#installing-synapse)
- [Installing from source](#installing-from-source)
- [Platform-Specific Instructions](#platform-specific-instructions)
@ -10,6 +11,22 @@
- [Setting up a TURN server](#setting-up-a-turn-server)
- [URL previews](#url-previews)
# Choosing your server name
It is important to choose the name for your server before you install Synapse,
because it cannot be changed later.
The server name determines the "domain" part of user-ids for users on your
server: these will all be of the format `@user:my.domain.name`. It also
determines how other matrix servers will reach yours for federation.
For a test configuration, set this to the hostname of your server. For a more
production-ready setup, you will probably want to specify your domain
(`example.com`) rather than a matrix-specific hostname here (in the same way
that your email address is probably `user@example.com` rather than
`user@email.example.com`) - but doing so may require more advanced setup: see
[Setting up Federation](docs/federate.md).
# Installing Synapse
## Installing from source
@ -64,16 +81,7 @@ python -m synapse.app.homeserver \
--report-stats=[yes|no]
```
... substituting an appropriate value for `--server-name`. The server name
determines the "domain" part of user-ids for users on your server: these will
all be of the format `@user:my.domain.name`. It also determines how other
matrix servers will reach yours for Federation. For a test configuration,
set this to the hostname of your server. For a more production-ready setup, you
will probably want to specify your domain (`example.com`) rather than a
matrix-specific hostname here (in the same way that your email address is
probably `user@example.com` rather than `user@email.example.com`) - but
doing so may require more advanced setup: see [Setting up Federation](docs/federate.md).
Beware that the server name cannot be changed later.
... substituting an appropriate value for `--server-name`.
This command will generate you a config file that you can then customise, but it will
also generate a set of keys for you. These keys will allow your Home Server to
@ -86,9 +94,6 @@ different. See the
[spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys)
for more information on key management.)
You will need to give Synapse a TLS certficate before it will start - see [TLS
certificates](#tls-certificates).
To actually run your new homeserver, pick a working directory for Synapse to
run (e.g. `~/synapse`), and::

1
changelog.d/5051.bugfix Normal file
View file

@ -0,0 +1 @@
Prevent >1 room upgrades happening simultaneously on the same room.

1
changelog.d/5092.feature Normal file
View file

@ -0,0 +1 @@
Added possibilty to disable local password authentication. Contributed by Daniel Hoffend.

1
changelog.d/5313.misc Normal file
View file

@ -0,0 +1 @@
Update example haproxy config to a more compatible setup.

1
changelog.d/5499.misc Normal file
View file

@ -0,0 +1 @@
Some cleanups and sanity-checking in the CPU and database metrics.

1
changelog.d/5523.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a regression where homeservers on private IP addresses were incorrectly blacklisted.

1
changelog.d/5524.feature Normal file
View file

@ -0,0 +1 @@
Add --data-dir and --open-private-ports options.

1
changelog.d/5534.feature Normal file
View file

@ -0,0 +1 @@
Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the "restrict_public_rooms_to_local_users" configuration setting. If "restrict_public_rooms_to_local_users" is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests.

1
changelog.d/5537.misc Normal file
View file

@ -0,0 +1 @@
Add information about how to install and run `black` on the codebase to code_style.rst.

1
changelog.d/5543.misc Normal file
View file

@ -0,0 +1 @@
Make the config clearer in that email.template_dir is relative to the Synapse's root directory, not the `synapse/` folder within it.

1
changelog.d/5545.misc Normal file
View file

@ -0,0 +1 @@
Update v1.0.0 release changelog to include more information about changes to password resets.

1
changelog.d/5546.feature Normal file
View file

@ -0,0 +1 @@
Update docker image to use Python 3.7.

1
changelog.d/5547.feature Normal file
View file

@ -0,0 +1 @@
Increase default log level for docker image to INFO. It can still be changed by editing the generated log.config file.

1
changelog.d/5548.misc Normal file
View file

@ -0,0 +1 @@
Remove non-functioning check_event_hash.py dev script.

1
changelog.d/5550.feature Normal file
View file

@ -0,0 +1 @@
The minimum TLS version used for outgoing federation requests can now be set with `federation_client_minimum_tls_version`.

1
changelog.d/5550.misc Normal file
View file

@ -0,0 +1 @@
Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapse's allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour.

1
changelog.d/5555.bugfix Normal file
View file

@ -0,0 +1 @@
Fixed m.login.jwt using unregistred user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill.

1
changelog.d/5558.misc Normal file
View file

@ -0,0 +1 @@
Improve install docs on choosing server_name.

1
changelog.d/5559.feature Normal file
View file

@ -0,0 +1 @@
Optimise devices changed query to not pull unnecessary rows from the database, reducing database load.

1
changelog.d/5561.feature Normal file
View file

@ -0,0 +1 @@
Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.

1
changelog.d/5562.feature Normal file
View file

@ -0,0 +1 @@
Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.

1
changelog.d/5563.bugfix Normal file
View file

@ -0,0 +1 @@
Docker: Use a sensible location for data files when generating a config file.

1
changelog.d/5564.misc Normal file
View file

@ -0,0 +1 @@
Reduce the amount of stuff we send in the docker context.

1
changelog.d/5565.feature Normal file
View file

@ -0,0 +1 @@
Docker: Send synapse logs to the docker logging system, by default.

1
changelog.d/5566.feature Normal file
View file

@ -0,0 +1 @@
Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.

1
changelog.d/5567.feature Normal file
View file

@ -0,0 +1 @@
Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.

1
changelog.d/5568.feature Normal file
View file

@ -0,0 +1 @@
Docker image: open the non-TLS port by default.

1
changelog.d/5570.misc Normal file
View file

@ -0,0 +1 @@
Point the reverse links in the Purge History contrib scripts at the intended location.

4
contrib/docker/README.md
View file

@ -1,5 +1,9 @@
# Synapse Docker
FIXME: this is out-of-date as of
https://github.com/matrix-org/synapse/issues/5518. Contributions to bring it up
to date would be welcome.
### Automated configuration
It is recommended that you use Docker Compose to run your containers, including

6815
contrib/grafana/synapse.json
View file

File diff suppressed because one or more lines are too long

4
contrib/purge_api/README.md
View file

@ -3,7 +3,7 @@ Purge history API examples
# `purge_history.sh`
A bash file, that uses the [purge history API](/docs/admin_api/README.rst) to
A bash file, that uses the [purge history API](/docs/admin_api/purge_history_api.rst) to
purge all messages in a list of rooms up to a certain event. You can select a
timeframe or a number of messages that you want to keep in the room.
@ -12,5 +12,5 @@ the script.
# `purge_remote_media.sh`
A bash file, that uses the [purge history API](/docs/admin_api/README.rst) to
A bash file, that uses the [purge history API](/docs/admin_api/purge_history_api.rst) to
purge all old cached remote media.

2
debian/build_virtualenv vendored
View file

@ -43,7 +43,7 @@ dh_virtualenv \
--preinstall="mock" \
--extra-pip-arg="--no-cache-dir" \
--extra-pip-arg="--compile" \
--extras="all"
--extras="all,systemd"
PACKAGE_BUILD_DIR="debian/matrix-synapse-py3"
VIRTUALENV_DIR="${PACKAGE_BUILD_DIR}${DH_VIRTUALENV_INSTALL_ROOT}/matrix-synapse"

7
debian/changelog vendored
View file

@ -1,3 +1,10 @@
matrix-synapse-py3 (1.0.0+nmu1) UNRELEASED; urgency=medium
[ Silke Hofstra ]
* Include systemd-python to allow logging to the systemd journal.
-- Silke Hofstra <silke@slxh.eu> Wed, 29 May 2019 09:45:29 +0200
matrix-synapse-py3 (1.0.0) stable; urgency=medium
* New synapse release 1.0.0.

2
docker/Dockerfile
View file

@ -11,7 +11,7 @@
# docker build -f docker/Dockerfile --build-arg PYTHON_VERSION=3.6 .
#
ARG PYTHON_VERSION=2
ARG PYTHON_VERSION=3.7
###
### Stage 0: builder

227
docker/README.md
View file

@ -6,39 +6,11 @@ postgres database.
The image also does *not* provide a TURN server.
## Run
### Using docker-compose (easier)
This image is designed to run either with an automatically generated
configuration file or with a custom configuration that requires manual editing.
An easy way to make use of this image is via docker-compose. See the
[contrib/docker](https://github.com/matrix-org/synapse/tree/master/contrib/docker) section of the synapse project for
examples.
### Without Compose (harder)
If you do not wish to use Compose, you may still run this image using plain
Docker commands. Note that the following is just a guideline and you may need
to add parameters to the docker run command to account for the network situation
with your postgres database.
```
docker run \
-d \
--name synapse \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_SERVER_NAME=my.matrix.host \
-e SYNAPSE_REPORT_STATS=yes \
-p 8448:8448 \
matrixdotorg/synapse:latest
```
## Volumes
The image expects a single volume, located at ``/data``, that will hold:
By default, the image expects a single volume, located at ``/data``, that will hold:
* configuration files;
* temporary files during uploads;
* uploaded media and thumbnails;
* the SQLite database if you do not configure postgres;
@ -53,129 +25,106 @@ In order to setup an application service, simply create an ``appservices``
directory in the data volume and write the application service Yaml
configuration file there. Multiple application services are supported.
## TLS certificates
## Generating a configuration file
Synapse requires a valid TLS certificate. You can do one of the following:
The first step is to genearte a valid config file. To do this, you can run the
image with the `generate` commandline option.
* Provide your own certificate and key (as
`${DATA_PATH}/${SYNAPSE_SERVER_NAME}.tls.crt` and
`${DATA_PATH}/${SYNAPSE_SERVER_NAME}.tls.key`, or elsewhere by providing an
entire config as `${SYNAPSE_CONFIG_PATH}`). In this case, you should forward
traffic to port 8448 in the container, for example with `-p 443:8448`.
* Use a reverse proxy to terminate incoming TLS, and forward the plain http
traffic to port 8008 in the container. In this case you should set `-e
SYNAPSE_NO_TLS=1`.
* Use the ACME (Let's Encrypt) support built into Synapse. This requires
`${SYNAPSE_SERVER_NAME}` port 80 to be forwarded to port 8009 in the
container, for example with `-p 80:8009`. To enable it in the docker
container, set `-e SYNAPSE_ACME=1`.
If you don't do any of these, Synapse will fail to start with an error similar to:
synapse.config._base.ConfigError: Error accessing file '/data/<server_name>.tls.crt' (config for tls_certificate): No such file or directory
## Environment
Unless you specify a custom path for the configuration file, a very generic
file will be generated, based on the following environment settings.
These are a good starting point for setting up your own deployment.
Global settings:
* ``UID``, the user id Synapse will run as [default 991]
* ``GID``, the group id Synapse will run as [default 991]
* ``SYNAPSE_CONFIG_PATH``, path to a custom config file
If ``SYNAPSE_CONFIG_PATH`` is set, you should generate a configuration file
then customize it manually: see [Generating a config
file](#generating-a-config-file).
Otherwise, a dynamic configuration file will be used.
### Environment variables used to build a dynamic configuration file
The following environment variables are used to build the configuration file
when ``SYNAPSE_CONFIG_PATH`` is not set.
* ``SYNAPSE_SERVER_NAME`` (mandatory), the server public hostname.
* ``SYNAPSE_REPORT_STATS``, (mandatory, ``yes`` or ``no``), enable anonymous
statistics reporting back to the Matrix project which helps us to get funding.
* `SYNAPSE_NO_TLS`, (accepts `true`, `false`, `on`, `off`, `1`, `0`, `yes`, `no`]): disable
TLS in Synapse (use this if you run your own TLS-capable reverse proxy). Defaults
to `false` (ie, TLS is enabled by default).
* ``SYNAPSE_ENABLE_REGISTRATION``, set this variable to enable registration on
the Synapse instance.
* ``SYNAPSE_ALLOW_GUEST``, set this variable to allow guest joining this server.
* ``SYNAPSE_EVENT_CACHE_SIZE``, the event cache size [default `10K`].
* ``SYNAPSE_RECAPTCHA_PUBLIC_KEY``, set this variable to the recaptcha public
key in order to enable recaptcha upon registration.
* ``SYNAPSE_RECAPTCHA_PRIVATE_KEY``, set this variable to the recaptcha private
key in order to enable recaptcha upon registration.
* ``SYNAPSE_TURN_URIS``, set this variable to the coma-separated list of TURN
uris to enable TURN for this homeserver.
* ``SYNAPSE_TURN_SECRET``, set this to the TURN shared secret if required.
* ``SYNAPSE_MAX_UPLOAD_SIZE``, set this variable to change the max upload size
[default `10M`].
* ``SYNAPSE_ACME``: set this to enable the ACME certificate renewal support.
Shared secrets, that will be initialized to random values if not set:
* ``SYNAPSE_REGISTRATION_SHARED_SECRET``, secret for registrering users if
registration is disable.
* ``SYNAPSE_MACAROON_SECRET_KEY`` secret for signing access tokens
to the server.
Database specific values (will use SQLite if not set):
* `POSTGRES_DB` - The database name for the synapse postgres
database. [default: `synapse`]
* `POSTGRES_HOST` - The host of the postgres database if you wish to use
postgresql instead of sqlite3. [default: `db` which is useful when using a
container on the same docker network in a compose file where the postgres
service is called `db`]
* `POSTGRES_PASSWORD` - The password for the synapse postgres database. **If
this is set then postgres will be used instead of sqlite3.** [default: none]
**NOTE**: You are highly encouraged to use postgresql! Please use the compose
file to make it easier to deploy.
* `POSTGRES_USER` - The user for the synapse postgres database. [default:
`synapse`]
Mail server specific values (will not send emails if not set):
* ``SYNAPSE_SMTP_HOST``, hostname to the mail server.
* ``SYNAPSE_SMTP_PORT``, TCP port for accessing the mail server [default
``25``].
* ``SYNAPSE_SMTP_USER``, username for authenticating against the mail server if
any.
* ``SYNAPSE_SMTP_PASSWORD``, password for authenticating against the mail
server if any.
### Generating a config file
It is possible to generate a basic configuration file for use with
`SYNAPSE_CONFIG_PATH` using the `generate` commandline option. You will need to
specify values for `SYNAPSE_CONFIG_PATH`, `SYNAPSE_SERVER_NAME` and
`SYNAPSE_REPORT_STATS`, and mount a docker volume to store the data on. For
example:
You will need to specify values for the `SYNAPSE_SERVER_NAME` and
`SYNAPSE_REPORT_STATS` environment variable, and mount a docker volume to store
the configuration on. For example:
```
docker run -it --rm \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_CONFIG_PATH=/data/homeserver.yaml \
-e SYNAPSE_SERVER_NAME=my.matrix.host \
-e SYNAPSE_REPORT_STATS=yes \
matrixdotorg/synapse:latest generate
```
This will generate a `homeserver.yaml` in (typically)
`/var/lib/docker/volumes/synapse-data/_data`, which you can then customise and
use with:
For information on picking a suitable server name, see
https://github.com/matrix-org/synapse/blob/master/INSTALL.md.
The above command will generate a `homeserver.yaml` in (typically)
`/var/lib/docker/volumes/synapse-data/_data`. You should check this file, and
customise it to your needs.
The following environment variables are supported in `generate` mode:
* `SYNAPSE_SERVER_NAME` (mandatory): the server public hostname.
* `SYNAPSE_REPORT_STATS` (mandatory, `yes` or `no`): whether to enable
anonymous statistics reporting.
* `SYNAPSE_CONFIG_DIR`: where additional config files (such as the log config
and event signing key) will be stored. Defaults to `/data`.
* `SYNAPSE_CONFIG_PATH`: path to the file to be generated. Defaults to
`<SYNAPSE_CONFIG_DIR>/homeserver.yaml`.
* `SYNAPSE_DATA_DIR`: where the generated config will put persistent data
such as the datatase and media store. Defaults to `/data`.
* `UID`, `GID`: the user id and group id to use for creating the data
directories. Defaults to `991`, `991`.
## Running synapse
Once you have a valid configuration file, you can start synapse as follows:
```
docker run -d --name synapse \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_CONFIG_PATH=/data/homeserver.yaml \
-p 8008:8008 \
matrixdotorg/synapse:latest
```
You can then check that it has started correctly with:
```
docker logs synapse
```
If all is well, you should now be able to connect to http://localhost:8008 and
see a confirmation message.
The following environment variables are supported in run mode:
* `SYNAPSE_CONFIG_DIR`: where additional config files are stored. Defaults to
`/data`.
* `SYNAPSE_CONFIG_PATH`: path to the config file. Defaults to
`<SYNAPSE_CONFIG_DIR>/homeserver.yaml`.
* `UID`, `GID`: the user and group id to run Synapse as. Defaults to `991`, `991`.
## TLS support
The default configuration exposes a single HTTP port: http://localhost:8008. It
is suitable for local testing, but for any practical use, you will either need
to use a reverse proxy, or configure Synapse to expose an HTTPS port.
For documentation on using a reverse proxy, see
https://github.com/matrix-org/synapse/blob/master/docs/reverse_proxy.rst.
For more information on enabling TLS support in synapse itself, see
https://github.com/matrix-org/synapse/blob/master/INSTALL.md#tls-certificates. Of
course, you will need to expose the TLS port from the container with a `-p`
argument to `docker run`.
## Legacy dynamic configuration file support
For backwards-compatibility only, the docker image supports creating a dynamic
configuration file based on environment variables. This is now deprecated, but
is enabled when the `SYNAPSE_SERVER_NAME` variable is set (and `generate` is
not given).
To migrate from a dynamic configuration file to a static one, run the docker
container once with the environment variables set, and `migrate_config`
commandline option. For example:
```
docker run -it --rm \
--mount type=volume,src=synapse-data,dst=/data \
-e SYNAPSE_SERVER_NAME=my.matrix.host \
-e SYNAPSE_REPORT_STATS=yes \
matrixdotorg/synapse:latest migrate_config
```
This will generate the same configuration file as the legacy mode used, but
will store it in `/data/homeserver.yaml` instead of a temporary location. You
can then use it as shown above at [Running synapse](#running-synapse).

2
docker/conf/homeserver.yaml
View file

@ -21,7 +21,7 @@ server_name: "{{ SYNAPSE_SERVER_NAME }}"
pid_file: /homeserver.pid
web_client: False
soft_file_limit: 0
log_config: "/compiled/log.config"
log_config: "{{ SYNAPSE_LOG_CONFIG }}"
## Ports ##

7
docker/conf/log.config
View file

@ -16,14 +16,11 @@ handlers:
filters: [context]
loggers:
synapse:
level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
synapse.storage.SQL:
# beware: increasing this to DEBUG will make synapse log sensitive
# information such as access tokens.
level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
level: INFO
root:
level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
level: {{ SYNAPSE_LOG_LEVEL or "INFO" }}
handlers: [console]

268
docker/start.py
View file

@ -1,109 +1,243 @@
#!/usr/local/bin/python
import jinja2
import os
import sys
import subprocess
import glob
import codecs
import glob
import os
import subprocess
import sys
import jinja2
# Utility functions
convert = lambda src, dst, environ: open(dst, "w").write(
jinja2.Template(open(src).read()).render(**environ)
)
def log(txt):
print(txt, file=sys.stderr)
def check_arguments(environ, args):
for argument in args:
if argument not in environ:
print("Environment variable %s is mandatory, exiting." % argument)
sys.exit(2)
def error(txt):
log(txt)
sys.exit(2)
def generate_secrets(environ, secrets):
def convert(src, dst, environ):
"""Generate a file from a template
Args:
src (str): path to input file
dst (str): path to file to write
environ (dict): environment dictionary, for replacement mappings.
"""
with open(src) as infile:
template = infile.read()
rendered = jinja2.Template(template).render(**environ)
with open(dst, "w") as outfile:
outfile.write(rendered)
def generate_config_from_template(config_dir, config_path, environ, ownership):
"""Generate a homeserver.yaml from environment variables
Args:
config_dir (str): where to put generated config files
config_path (str): where to put the main config file
environ (dict): environment dictionary
ownership (str): "<user>:<group>" string which will be used to set
ownership of the generated configs
"""
for v in ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"):
if v not in environ:
error(
"Environment variable '%s' is mandatory when generating a config file."
% (v,)
)
# populate some params from data files (if they exist, else create new ones)
environ = environ.copy()
secrets = {
"registration": "SYNAPSE_REGISTRATION_SHARED_SECRET",
"macaroon": "SYNAPSE_MACAROON_SECRET_KEY",
}
for name, secret in secrets.items():
if secret not in environ:
filename = "/data/%s.%s.key" % (environ["SYNAPSE_SERVER_NAME"], name)
# if the file already exists, load in the existing value; otherwise,
# generate a new secret and write it to a file
if os.path.exists(filename):
log("Reading %s from %s" % (secret, filename))
with open(filename) as handle:
value = handle.read()
else:
print("Generating a random secret for {}".format(name))
log("Generating a random secret for {}".format(secret))
value = codecs.encode(os.urandom(32), "hex").decode()
with open(filename, "w") as handle:
handle.write(value)
environ[secret] = value
environ["SYNAPSE_APPSERVICES"] = glob.glob("/data/appservices/*.yaml")
if not os.path.exists(config_dir):
os.mkdir(config_dir)
# Prepare the configuration
mode = sys.argv[1] if len(sys.argv) > 1 else None
environ = os.environ.copy()
ownership = "{}:{}".format(environ.get("UID", 991), environ.get("GID", 991))
args = ["python", "-m", "synapse.app.homeserver"]
# Convert SYNAPSE_NO_TLS to boolean if exists
if "SYNAPSE_NO_TLS" in environ:
tlsanswerstring = str.lower(environ["SYNAPSE_NO_TLS"])
if tlsanswerstring in ("true", "on", "1", "yes"):
environ["SYNAPSE_NO_TLS"] = True
else:
if tlsanswerstring in ("false", "off", "0", "no"):
environ["SYNAPSE_NO_TLS"] = False
else:
error(
'Environment variable "SYNAPSE_NO_TLS" found but value "'
+ tlsanswerstring
+ '" unrecognized; exiting.'
)
# In generate mode, generate a configuration, missing keys, then exit
if mode == "generate":
check_arguments(
environ, ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS", "SYNAPSE_CONFIG_PATH")
if "SYNAPSE_LOG_CONFIG" not in environ:
environ["SYNAPSE_LOG_CONFIG"] = config_dir + "/log.config"
log("Generating synapse config file " + config_path)
convert("/conf/homeserver.yaml", config_path, environ)
log_config_file = environ["SYNAPSE_LOG_CONFIG"]
log("Generating log config file " + log_config_file)
convert("/conf/log.config", log_config_file, environ)
subprocess.check_output(["chown", "-R", ownership, "/data"])
# Hopefully we already have a signing key, but generate one if not.
subprocess.check_output(
[
"su-exec",
ownership,
"python",
"-m",
"synapse.app.homeserver",
"--config-path",
config_path,
# tell synapse to put generated keys in /data rather than /compiled
"--keys-directory",
config_dir,
"--generate-keys",
]
)
args += [
def run_generate_config(environ, ownership):
"""Run synapse with a --generate-config param to generate a template config file
Args:
environ (dict): env var dict
ownership (str): "userid:groupid" arg for chmod
Never returns.
"""
for v in ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"):
if v not in environ:
error("Environment variable '%s' is mandatory in `generate` mode." % (v,))
server_name = environ["SYNAPSE_SERVER_NAME"]
config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
config_path = environ.get("SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml")
data_dir = environ.get("SYNAPSE_DATA_DIR", "/data")
# create a suitable log config from our template
log_config_file = "%s/%s.log.config" % (config_dir, server_name)
if not os.path.exists(log_config_file):
log("Creating log config %s" % (log_config_file,))
convert("/conf/log.config", log_config_file, environ)
# make sure that synapse has perms to write to the data dir.
subprocess.check_output(["chown", ownership, data_dir])
args = [
"python",
"-m",
"synapse.app.homeserver",
"--server-name",
environ["SYNAPSE_SERVER_NAME"],
server_name,
"--report-stats",
environ["SYNAPSE_REPORT_STATS"],
"--config-path",
environ["SYNAPSE_CONFIG_PATH"],
config_path,
"--config-directory",
config_dir,
"--data-directory",
data_dir,
"--generate-config",
"--open-private-ports",
]
# log("running %s" % (args, ))
os.execv("/usr/local/bin/python", args)
# In normal mode, generate missing keys if any, then run synapse
else:
if "SYNAPSE_CONFIG_PATH" in environ:
config_path = environ["SYNAPSE_CONFIG_PATH"]
else:
check_arguments(environ, ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"))
generate_secrets(
environ,
{
"registration": "SYNAPSE_REGISTRATION_SHARED_SECRET",
"macaroon": "SYNAPSE_MACAROON_SECRET_KEY",
},
def main(args, environ):
mode = args[1] if len(args) > 1 else None
ownership = "{}:{}".format(environ.get("UID", 991), environ.get("GID", 991))
# In generate mode, generate a configuration and missing keys, then exit
if mode == "generate":
return run_generate_config(environ, ownership)
if mode == "migrate_config":
# generate a config based on environment vars.
config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
config_path = environ.get(
"SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml"
)
environ["SYNAPSE_APPSERVICES"] = glob.glob("/data/appservices/*.yaml")
if not os.path.exists("/compiled"):
os.mkdir("/compiled")
return generate_config_from_template(
config_dir, config_path, environ, ownership
)
if mode is not None:
error("Unknown execution mode '%s'" % (mode,))
if "SYNAPSE_SERVER_NAME" in environ:
# backwards-compatibility generate-a-config-on-the-fly mode
if "SYNAPSE_CONFIG_PATH" in environ:
error(
"SYNAPSE_SERVER_NAME and SYNAPSE_CONFIG_PATH are mutually exclusive "
"except in `generate` or `migrate_config` mode."
)
config_path = "/compiled/homeserver.yaml"
log(
"Generating config file '%s' on-the-fly from environment variables.\n"
"Note that this mode is deprecated. You can migrate to a static config\n"
"file by running with 'migrate_config'. See the README for more details."
% (config_path,)
)
# Convert SYNAPSE_NO_TLS to boolean if exists
if "SYNAPSE_NO_TLS" in environ:
tlsanswerstring = str.lower(environ["SYNAPSE_NO_TLS"])
if tlsanswerstring in ("true", "on", "1", "yes"):
environ["SYNAPSE_NO_TLS"] = True
else:
if tlsanswerstring in ("false", "off", "0", "no"):
environ["SYNAPSE_NO_TLS"] = False
else:
print(
'Environment variable "SYNAPSE_NO_TLS" found but value "'
+ tlsanswerstring
+ '" unrecognized; exiting.'
)
sys.exit(2)
generate_config_from_template("/compiled", config_path, environ, ownership)
else:
config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
config_path = environ.get(
"SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml"
)
if not os.path.exists(config_path):
error(
"Config file '%s' does not exist. You should either create a new "
"config file by running with the `generate` argument (and then edit "
"the resulting file before restarting) or specify the path to an "
"existing config file with the SYNAPSE_CONFIG_PATH variable."
% (config_path,)
)
convert("/conf/homeserver.yaml", config_path, environ)
convert("/conf/log.config", "/compiled/log.config", environ)
subprocess.check_output(["chown", "-R", ownership, "/data"])
log("Starting synapse with config file " + config_path)
args += [
args = [
"su-exec",
ownership,
"python",
"-m",
"synapse.app.homeserver",
"--config-path",
config_path,
# tell synapse to put any generated keys in /data rather than /compiled
"--keys-directory",
"/data",
]
os.execv("/sbin/su-exec", args)
# Generate missing keys and start synapse
subprocess.check_output(args + ["--generate-keys"])
os.execv("/sbin/su-exec", ["su-exec", ownership] + args)
if __name__ == "__main__":
main(sys.argv, os.environ)

2
docs/MSC1711_certificates_FAQ.md
View file

@ -14,7 +14,7 @@ upgraded, however it may be of use to those with old installs returning to the
project.
If you are setting up a server from scratch you almost certainly should look at
the [installation guide](INSTALL.md) instead.
the [installation guide](../INSTALL.md) instead.
## Introduction
The goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0.0. It

95
docs/code_style.rst
View file

@ -1,43 +1,60 @@
- Everything should comply with PEP8. Code should pass
``pep8 --max-line-length=100`` without any warnings.
# Code Style
- **Indenting**:
The Synapse codebase uses a number of code formatting tools in order to
quickly and automatically check for formatting (and sometimes logical) errors
in code.
- NEVER tabs. 4 spaces to indent.
The necessary tools are detailed below.
- follow PEP8; either hanging indent or multiline-visual indent depending
on the size and shape of the arguments and what makes more sense to the
author. In other words, both this::
## Formatting tools
print("I am a fish %s" % "moo")
The Synapse codebase uses [black](https://pypi.org/project/black/) as an
opinionated code formatter, ensuring all comitted code is properly
formatted.
and this::
First install ``black`` with::
print("I am a fish %s" %
"moo")
pip install --upgrade black
and this::
Have ``black`` auto-format your code (it shouldn't change any
functionality) with::
print(
"I am a fish %s" %
"moo",
)
black . --exclude="\.tox|build|env"
...are valid, although given each one takes up 2x more vertical space than
the previous, it's up to the author's discretion as to which layout makes
most sense for their function invocation. (e.g. if they want to add
comments per-argument, or put expressions in the arguments, or group
related arguments together, or want to deliberately extend or preserve
vertical/horizontal space)
- **flake8**
- **Line length**:
``flake8`` is a code checking tool. We require code to pass ``flake8`` before being merged into the codebase.
Max line length is 79 chars (with flexibility to overflow by a "few chars" if
the overflowing content is not semantically significant and avoids an
explosion of vertical whitespace).
Install ``flake8`` with::
Use parentheses instead of ``\`` for line continuation where ever possible
(which is pretty much everywhere).
pip install --upgrade flake8
Check all application and test code with::
flake8 synapse tests
- **isort**
``isort`` ensures imports are nicely formatted, and can suggest and
auto-fix issues such as double-importing.
Install ``isort`` with::
pip install --upgrade isort
Auto-fix imports with::
isort -rc synapse tests
``-rc`` means to recursively search the given directories.
It's worth noting that modern IDEs and text editors can run these tools
automatically on save. It may be worth looking into whether this
functionality is supported in your editor for a more convenient development
workflow. It is not, however, recommended to run ``flake8`` on save as it
takes a while and is very resource intensive.
## General rules
- **Naming**:
@ -46,26 +63,6 @@
- Use double quotes ``"foo"`` rather than single quotes ``'foo'``.
- **Blank lines**:
- There should be max a single new line between:
- statements
- functions in a class
- There should be two new lines between:
- definitions in a module (e.g., between different classes)
- **Whitespace**:
There should be spaces where spaces should be and not where there shouldn't
be:
- a single space after a comma
- a single space before and after for '=' when used as assignment
- no spaces before and after for '=' for default values and keyword arguments.
- **Comments**: should follow the `google code style
<http://google.github.io/styleguide/pyguide.html?showone=Comments#Comments>`_.
This is so that we can generate documentation with `sphinx
@ -76,7 +73,7 @@
- **Imports**:
- Prefer to import classes and functions than packages or modules.
- Prefer to import classes and functions rather than packages or modules.
Example::

6
docs/reverse_proxy.rst
View file

@ -89,8 +89,10 @@ Let's assume that we expect clients to connect to our server at
bind :::443 v4v6 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1
# Matrix client traffic
acl matrix hdr(host) -i matrix.example.com
use_backend matrix if matrix
acl matrix-host hdr(host) -i matrix.example.com
acl matrix-path path_beg /_matrix
use_backend matrix if matrix-host matrix-path
frontend matrix-federation
bind :::8448 v4v6 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1

39
docs/sample_config.yaml
View file

@ -54,11 +54,15 @@ pid_file: DATADIR/homeserver.pid
#
#require_auth_for_profile_requests: true
# If set to 'true', requires authentication to access the server's
# public rooms directory through the client API, and forbids any other
# homeserver to fetch it via federation. Defaults to 'false'.
# If set to 'false', requires authentication to access the server's public rooms
# directory through the client API. Defaults to 'true'.
#
#restrict_public_rooms_to_local_users: true
#allow_public_rooms_without_auth: false
# If set to 'false', forbids any other homeserver to fetch the server's public
# rooms directory via federation. Defaults to 'true'.
#
#allow_public_rooms_over_federation: false
# The default room version for newly created rooms.
#
@ -209,7 +213,7 @@ listeners:
- names: [client, federation]
compress: false
# example additonal_resources:
# example additional_resources:
#
#additional_resources:
# "/_matrix/my/custom/endpoint":
@ -313,6 +317,15 @@ listeners:
#
#federation_verify_certificates: false
# The minimum TLS version that will be used for outbound federation requests.
#
# Defaults to `1`. Configurable to `1`, `1.1`, `1.2`, or `1.3`. Note
# that setting this value higher than `1.2` will prevent federation to most
# of the public Matrix network: only configure it to `1.3` if you have an
# entirely private federation setup and you can ensure TLS 1.3 support.
#
#federation_client_minimum_tls_version: 1.2
# Skip federation certificate verification on the following whitelist
# of domains.
#
@ -1042,6 +1055,12 @@ password_config:
#
#enabled: false
# Uncomment to disable authentication against the local password
# database. This is ignored if `enabled` is false, and is only useful
# if you have other password_providers.
#
#localdb_enabled: false
# Uncomment and change to a secret random string for extra security.
# DO NOT CHANGE THIS AFTER INITIAL SETUP!
#
@ -1066,11 +1085,13 @@ password_config:
# app_name: Matrix
#
# # Enable email notifications by default
# #
# notif_for_new_users: True
#
# # Defining a custom URL for Riot is only needed if email notifications
# # should contain links to a self-hosted installation of Riot; when set
# # the "app_name" setting is ignored
# #
# riot_base_url: "http://localhost/riot"
#
# # Enable sending password reset emails via the configured, trusted
@ -1083,16 +1104,22 @@ password_config:
# #
# # If this option is set to false and SMTP options have not been
# # configured, resetting user passwords via email will be disabled
# #
# #trust_identity_server_for_password_resets: false
#
# # Configure the time that a validation email or text message code
# # will expire after sending
# #
# # This is currently used for password resets
# #
# #validation_token_lifetime: 1h
#
# # Template directory. All template files should be stored within this
# # directory
# # directory. If not set, default templates from within the Synapse
# # package will be used
# #
# # For the list of default templates, please see
# # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
# #
# #template_dir: res/templates
#

54
scripts-dev/check_event_hash.py
View file

@ -1,54 +0,0 @@
import argparse
import hashlib
import json
import logging
import sys
from unpaddedbase64 import encode_base64
from synapse.crypto.event_signing import (
check_event_content_hash,
compute_event_reference_hash,
)
class dictobj(dict):
def __init__(self, *args, **kargs):
dict.__init__(self, *args, **kargs)
self.__dict__ = self
def get_dict(self):
return dict(self)
def get_full_dict(self):
return dict(self)
def get_pdu_json(self):
return dict(self)
def main():
parser = argparse.ArgumentParser()
parser.add_argument(
"input_json", nargs="?", type=argparse.FileType("r"), default=sys.stdin
)
args = parser.parse_args()
logging.basicConfig()
event_json = dictobj(json.load(args.input_json))
algorithms = {"sha256": hashlib.sha256}
for alg_name in event_json.hashes:
if check_event_content_hash(event_json, algorithms[alg_name]):
print("PASS content hash %s" % (alg_name,))
else:
print("FAIL content hash %s" % (alg_name,))
for algorithm in algorithms.values():
name, h_bytes = compute_event_reference_hash(event_json, algorithm)
print("Reference hash %s: %s" % (name, encode_base64(h_bytes)))
if __name__ == "__main__":
main()

2
scripts/generate_config
View file

@ -1,4 +1,4 @@
#!/usr/bin/env python
#!/usr/bin/env python3
import argparse
import shutil

135
synapse/config/_base.py
View file

@ -136,11 +136,6 @@ class Config(object):
with open(file_path) as file_stream:
return file_stream.read()
@staticmethod
def read_config_file(file_path):
with open(file_path) as file_stream:
return yaml.safe_load(file_stream)
def invoke_all(self, name, *args, **kargs):
results = []
for cls in type(self).mro():
@ -155,12 +150,12 @@ class Config(object):
server_name,
generate_secrets=False,
report_stats=None,
open_private_ports=False,
):
"""Build a default configuration file
This is used both when the user explicitly asks us to generate a config file
(eg with --generate_config), and before loading the config at runtime (to give
a base which the config files override)
This is used when the user explicitly asks us to generate a config file
(eg with --generate_config).
Args:
config_dir_path (str): The path where the config files are kept. Used to
@ -179,23 +174,25 @@ class Config(object):
report_stats (bool|None): Initial setting for the report_stats setting.
If None, report_stats will be left unset.
open_private_ports (bool): True to leave private ports (such as the non-TLS
HTTP listener) open to the internet.
Returns:
str: the yaml config file
"""
default_config = "\n\n".join(
return "\n\n".join(
dedent(conf)
for conf in self.invoke_all(
"default_config",
"generate_config_section",
config_dir_path=config_dir_path,
data_dir_path=data_dir_path,
server_name=server_name,
generate_secrets=generate_secrets,
report_stats=report_stats,
open_private_ports=open_private_ports,
)
)
return default_config
@classmethod
def load_config(cls, description, argv):
"""Parse the commandline and config files
@ -240,9 +237,7 @@ class Config(object):
config_dir_path = os.path.abspath(config_dir_path)
data_dir_path = os.getcwd()
config_dict = obj.read_config_files(
config_files, config_dir_path=config_dir_path, data_dir_path=data_dir_path
)
config_dict = read_config_files(config_files)
obj.parse_config_dict(
config_dict, config_dir_path=config_dir_path, data_dir_path=data_dir_path
)
@ -300,6 +295,23 @@ class Config(object):
" config file."
),
)
generate_group.add_argument(
"--data-directory",
metavar="DIRECTORY",
help=(
"Specify where data such as the media store and database file should be"
" stored. Defaults to the current working directory."
),
)
generate_group.add_argument(
"--open-private-ports",
action="store_true",
help=(
"Leave private ports (such as the non-TLS HTTP listener) open to the"
" internet. Do not use this unless you know what you are doing."
),
)
config_args, remaining_args = config_parser.parse_known_args(argv)
config_files = find_config_files(search_paths=config_args.config_path)
@ -333,6 +345,12 @@ class Config(object):
if not cls.path_exists(config_path):
print("Generating config file %s" % (config_path,))
if config_args.data_directory:
data_dir_path = config_args.data_directory
else:
data_dir_path = os.getcwd()
data_dir_path = os.path.abspath(data_dir_path)
server_name = config_args.server_name
if not server_name:
raise ConfigError(
@ -346,6 +364,7 @@ class Config(object):
server_name=server_name,
report_stats=(config_args.report_stats == "yes"),
generate_secrets=True,
open_private_ports=config_args.open_private_ports,
)
if not cls.path_exists(config_dir_path):
@ -354,8 +373,8 @@ class Config(object):
config_file.write("# vim:ft=yaml\n\n")
config_file.write(config_str)
config = yaml.safe_load(config_str)
obj.invoke_all("generate_files", config)
config_dict = yaml.safe_load(config_str)
obj.generate_missing_files(config_dict, config_dir_path)
print(
(
@ -385,12 +404,9 @@ class Config(object):
obj.invoke_all("add_arguments", parser)
args = parser.parse_args(remaining_args)
config_dict = obj.read_config_files(
config_files, config_dir_path=config_dir_path, data_dir_path=data_dir_path
)
config_dict = read_config_files(config_files)
if generate_missing_configs:
obj.generate_missing_files(config_dict)
obj.generate_missing_files(config_dict, config_dir_path)
return None
obj.parse_config_dict(
@ -400,53 +416,6 @@ class Config(object):
return obj
def read_config_files(self, config_files, config_dir_path, data_dir_path):
"""Read the config files into a dict
Args:
config_files (iterable[str]): A list of the config files to read
config_dir_path (str): The path where the config files are kept. Used to
create filenames for things like the log config and the signing key.
data_dir_path (str): The path where the data files are kept. Used to create
filenames for things like the database and media store.
Returns: dict
"""
# first we read the config files into a dict
specified_config = {}
for config_file in config_files:
yaml_config = self.read_config_file(config_file)
specified_config.update(yaml_config)
# not all of the options have sensible defaults in code, so we now need to
# generate a default config file suitable for the specified server name...
if "server_name" not in specified_config:
raise ConfigError(MISSING_SERVER_NAME)
server_name = specified_config["server_name"]
config_string = self.generate_config(
config_dir_path=config_dir_path,
data_dir_path=data_dir_path,
server_name=server_name,
generate_secrets=False,
)
# ... and read it into a base config dict ...
config = yaml.safe_load(config_string)
# ... and finally, overlay it with the actual configuration.
config.pop("log_config")
config.update(specified_config)
if "report_stats" not in config:
raise ConfigError(
MISSING_REPORT_STATS_CONFIG_INSTRUCTIONS
+ "\n"
+ MISSING_REPORT_STATS_SPIEL
)
return config
def parse_config_dict(self, config_dict, config_dir_path, data_dir_path):
"""Read the information from the config dict into this Config object.
@ -466,8 +435,32 @@ class Config(object):
data_dir_path=data_dir_path,
)
def generate_missing_files(self, config_dict):
self.invoke_all("generate_files", config_dict)
def generate_missing_files(self, config_dict, config_dir_path):
self.invoke_all("generate_files", config_dict, config_dir_path)
def read_config_files(config_files):
"""Read the config files into a dict
Args:
config_files (iterable[str]): A list of the config files to read
Returns: dict
"""
specified_config = {}
for config_file in config_files:
with open(config_file) as file_stream:
yaml_config = yaml.safe_load(file_stream)
specified_config.update(yaml_config)
if "server_name" not in specified_config:
raise ConfigError(MISSING_SERVER_NAME)
if "report_stats" not in specified_config:
raise ConfigError(
MISSING_REPORT_STATS_CONFIG_INSTRUCTIONS + "\n" + MISSING_REPORT_STATS_SPIEL
)
return specified_config
def find_config_files(search_paths):

2
synapse/config/api.py
View file

@ -30,7 +30,7 @@ class ApiConfig(Config):
],
)
def default_config(cls, **kwargs):
def generate_config_section(cls, **kwargs):
return """\
## API Configuration ##

2
synapse/config/appservice.py
View file

@ -34,7 +34,7 @@ class AppServiceConfig(Config):
self.notify_appservices = config.get("notify_appservices", True)
self.track_appservice_user_ips = config.get("track_appservice_user_ips", False)
def default_config(cls, **kwargs):
def generate_config_section(cls, **kwargs):
return """\
# A list of application service config files to use
#

2
synapse/config/captcha.py
View file

@ -28,7 +28,7 @@ class CaptchaConfig(Config):
"https://www.recaptcha.net/recaptcha/api/siteverify",
)
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return """\
## Captcha ##
# See docs/CAPTCHA_SETUP for full details of configuring this.

2
synapse/config/cas.py
View file

@ -35,7 +35,7 @@ class CasConfig(Config):
self.cas_service_url = None
self.cas_required_attributes = {}
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """
# Enable CAS for registration and login.
#

2
synapse/config/consent_config.py
View file

@ -111,5 +111,5 @@ class ConsentConfig(Config):
"policy_name", "Privacy Policy"
)
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return DEFAULT_CONFIG

2
synapse/config/database.py
View file

@ -38,7 +38,7 @@ class DatabaseConfig(Config):
self.set_databasepath(config.get("database_path"))
def default_config(self, data_dir_path, **kwargs):
def generate_config_section(self, data_dir_path, **kwargs):
database_path = os.path.join(data_dir_path, "homeserver.db")
return (
"""\

12
synapse/config/emailconfig.py
View file

@ -214,7 +214,7 @@ class EmailConfig(Config):
if not os.path.isfile(p):
raise ConfigError("Unable to find email template file %s" % (p,))
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """
# Enable sending emails for password resets, notification events or
# account expiry notices
@ -233,11 +233,13 @@ class EmailConfig(Config):
# app_name: Matrix
#
# # Enable email notifications by default
# #
# notif_for_new_users: True
#
# # Defining a custom URL for Riot is only needed if email notifications
# # should contain links to a self-hosted installation of Riot; when set
# # the "app_name" setting is ignored
# #
# riot_base_url: "http://localhost/riot"
#
# # Enable sending password reset emails via the configured, trusted
@ -250,16 +252,22 @@ class EmailConfig(Config):
# #
# # If this option is set to false and SMTP options have not been
# # configured, resetting user passwords via email will be disabled
# #
# #trust_identity_server_for_password_resets: false
#
# # Configure the time that a validation email or text message code
# # will expire after sending
# #
# # This is currently used for password resets
# #
# #validation_token_lifetime: 1h
#
# # Template directory. All template files should be stored within this
# # directory
# # directory. If not set, default templates from within the Synapse
# # package will be used
# #
# # For the list of default templates, please see
# # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
# #
# #template_dir: res/templates
#

2
synapse/config/groups.py
View file

@ -21,7 +21,7 @@ class GroupsConfig(Config):
self.enable_group_creation = config.get("enable_group_creation", False)
self.group_creation_prefix = config.get("group_creation_prefix", "")
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return """\
# Uncomment to allow non-server-admin users to create groups on this server
#

2
synapse/config/jwt_config.py
View file

@ -41,7 +41,7 @@ class JWTConfig(Config):
self.jwt_secret = None
self.jwt_algorithm = None
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return """\
# The JWT needs to contain a globally unique "sub" (subject) claim.
#

24
synapse/config/key.py
View file

@ -65,13 +65,18 @@ class TrustedKeyServer(object):
class KeyConfig(Config):
def read_config(self, config, **kwargs):
def read_config(self, config, config_dir_path, **kwargs):
# the signing key can be specified inline or in a separate file
if "signing_key" in config:
self.signing_key = read_signing_keys([config["signing_key"]])
else:
self.signing_key_path = config["signing_key_path"]
self.signing_key = self.read_signing_key(self.signing_key_path)
signing_key_path = config.get("signing_key_path")
if signing_key_path is None:
signing_key_path = os.path.join(
config_dir_path, config["server_name"] + ".signing.key"
)
self.signing_key = self.read_signing_key(signing_key_path)
self.old_signing_keys = self.read_old_signing_keys(
config.get("old_signing_keys", {})
@ -117,7 +122,7 @@ class KeyConfig(Config):
# falsification of values
self.form_secret = config.get("form_secret", None)
def default_config(
def generate_config_section(
self, config_dir_path, server_name, generate_secrets=False, **kwargs
):
base_key_name = os.path.join(config_dir_path, server_name)
@ -237,8 +242,15 @@ class KeyConfig(Config):
)
return keys
def generate_files(self, config):
signing_key_path = config["signing_key_path"]
def generate_files(self, config, config_dir_path):
if "signing_key" in config:
return
signing_key_path = config.get("signing_key_path")
if signing_key_path is None:
signing_key_path = os.path.join(
config_dir_path, config["server_name"] + ".signing.key"
)
if not self.path_exists(signing_key_path):
print("Generating signing key file %s" % (signing_key_path,))

4
synapse/config/logger.py
View file

@ -80,7 +80,7 @@ class LoggingConfig(Config):
self.log_config = self.abspath(config.get("log_config"))
self.log_file = self.abspath(config.get("log_file"))
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
log_config = os.path.join(config_dir_path, server_name + ".log.config")
return (
"""\
@ -133,7 +133,7 @@ class LoggingConfig(Config):
help="Do not redirect stdout/stderr to the log",
)
def generate_files(self, config):
def generate_files(self, config, config_dir_path):
log_config = config.get("log_config")
if log_config and not os.path.exists(log_config):
log_file = self.abspath("homeserver.log")

2
synapse/config/metrics.py
View file

@ -40,7 +40,7 @@ class MetricsConfig(Config):
"sentry.dsn field is required when sentry integration is enabled"
)
def default_config(self, report_stats=None, **kwargs):
def generate_config_section(self, report_stats=None, **kwargs):
res = """\
## Metrics ###

9
synapse/config/password.py
View file

@ -26,15 +26,22 @@ class PasswordConfig(Config):
password_config = {}
self.password_enabled = password_config.get("enabled", True)
self.password_localdb_enabled = password_config.get("localdb_enabled", True)
self.password_pepper = password_config.get("pepper", "")
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """\
password_config:
# Uncomment to disable password login
#
#enabled: false
# Uncomment to disable authentication against the local password
# database. This is ignored if `enabled` is false, and is only useful
# if you have other password_providers.
#
#localdb_enabled: false
# Uncomment and change to a secret random string for extra security.
# DO NOT CHANGE THIS AFTER INITIAL SETUP!
#

2
synapse/config/password_auth_providers.py
View file

@ -46,7 +46,7 @@ class PasswordAuthProviderConfig(Config):
self.password_providers.append((provider_class, provider_config))
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return """\
#password_providers:
# - module: "ldap_auth_provider.LdapAuthProvider"

2
synapse/config/push.py
View file

@ -42,7 +42,7 @@ class PushConfig(Config):
)
self.push_include_content = not redact_content
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """
# Clients requesting push notifications can either have the body of
# the message sent in the notification poke along with other details

2
synapse/config/ratelimiting.py
View file

@ -80,7 +80,7 @@ class RatelimitConfig(Config):
"federation_rr_transactions_per_room_per_second", 50
)
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return """\
## Ratelimiting ##

2
synapse/config/registration.py
View file

@ -85,7 +85,7 @@ class RegistrationConfig(Config):
"disable_msisdn_registration", False
)
def default_config(self, generate_secrets=False, **kwargs):
def generate_config_section(self, generate_secrets=False, **kwargs):
if generate_secrets:
registration_shared_secret = 'registration_shared_secret: "%s"' % (
random_string_with_symbols(50),

8
synapse/config/repository.py
View file

@ -91,7 +91,9 @@ class ContentRepositoryConfig(Config):
self.max_image_pixels = self.parse_size(config.get("max_image_pixels", "32M"))
self.max_spider_size = self.parse_size(config.get("max_spider_size", "10M"))
self.media_store_path = self.ensure_directory(config["media_store_path"])
self.media_store_path = self.ensure_directory(
config.get("media_store_path", "media_store")
)
backup_media_store_path = config.get("backup_media_store_path")
@ -148,7 +150,7 @@ class ContentRepositoryConfig(Config):
(provider_class, parsed_config, wrapper_config)
)
self.uploads_path = self.ensure_directory(config["uploads_path"])
self.uploads_path = self.ensure_directory(config.get("uploads_path", "uploads"))
self.dynamic_thumbnails = config.get("dynamic_thumbnails", False)
self.thumbnail_requirements = parse_thumbnail_requirements(
config.get("thumbnail_sizes", DEFAULT_THUMBNAIL_SIZES)
@ -188,7 +190,7 @@ class ContentRepositoryConfig(Config):
self.url_preview_url_blacklist = config.get("url_preview_url_blacklist", ())
def default_config(self, data_dir_path, **kwargs):
def generate_config_section(self, data_dir_path, **kwargs):
media_store = os.path.join(data_dir_path, "media_store")
uploads_path = os.path.join(data_dir_path, "uploads")

2
synapse/config/room_directory.py
View file

@ -46,7 +46,7 @@ class RoomDirectoryConfig(Config):
_RoomDirectoryRule("room_list_publication_rules", {"action": "allow"})
]
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """
# Uncomment to disable searching the public room list. When disabled
# blocks searching local and remote room lists for local and remote

2
synapse/config/saml2_config.py
View file

@ -61,7 +61,7 @@ class SAML2Config(Config):
},
}
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """\
# Enable SAML2 for registration and login. Uses pysaml2.
#

61
synapse/config/server.py
View file

@ -82,12 +82,32 @@ class ServerConfig(Config):
"require_auth_for_profile_requests", False
)
# If set to 'True', requires authentication to access the server's
# public rooms directory through the client API, and forbids any other
# homeserver to fetch it via federation.
self.restrict_public_rooms_to_local_users = config.get(
"restrict_public_rooms_to_local_users", False
)
if "restrict_public_rooms_to_local_users" in config and (
"allow_public_rooms_without_auth" in config
or "allow_public_rooms_over_federation" in config
):
raise ConfigError(
"Can't use 'restrict_public_rooms_to_local_users' if"
" 'allow_public_rooms_without_auth' and/or"
" 'allow_public_rooms_over_federation' is set."
)
# Check if the legacy "restrict_public_rooms_to_local_users" flag is set. This
# flag is now obsolete but we need to check it for backward-compatibility.
if config.get("restrict_public_rooms_to_local_users", False):
self.allow_public_rooms_without_auth = False
self.allow_public_rooms_over_federation = False
else:
# If set to 'False', requires authentication to access the server's public
# rooms directory through the client API. Defaults to 'True'.
self.allow_public_rooms_without_auth = config.get(
"allow_public_rooms_without_auth", True
)
# If set to 'False', forbids any other homeserver to fetch the server's public
# rooms directory via federation. Defaults to 'True'.
self.allow_public_rooms_over_federation = config.get(
"allow_public_rooms_over_federation", True
)
default_room_version = config.get("default_room_version", DEFAULT_ROOM_VERSION)
@ -307,7 +327,9 @@ class ServerConfig(Config):
def has_tls_listener(self):
return any(l["tls"] for l in self.listeners)
def default_config(self, server_name, data_dir_path, **kwargs):
def generate_config_section(
self, server_name, data_dir_path, open_private_ports, **kwargs
):
_, bind_port = parse_and_validate_server_name(server_name)
if bind_port is not None:
unsecure_port = bind_port - 400
@ -320,6 +342,13 @@ class ServerConfig(Config):
# Bring DEFAULT_ROOM_VERSION into the local-scope for use in the
# default config string
default_room_version = DEFAULT_ROOM_VERSION
unsecure_http_binding = "port: %i\n tls: false" % (unsecure_port,)
if not open_private_ports:
unsecure_http_binding += (
"\n bind_addresses: ['::1', '127.0.0.1']"
)
return (
"""\
## Server ##
@ -366,11 +395,15 @@ class ServerConfig(Config):
#
#require_auth_for_profile_requests: true
# If set to 'true', requires authentication to access the server's
# public rooms directory through the client API, and forbids any other
# homeserver to fetch it via federation. Defaults to 'false'.
# If set to 'false', requires authentication to access the server's public rooms
# directory through the client API. Defaults to 'true'.
#
#restrict_public_rooms_to_local_users: true
#allow_public_rooms_without_auth: false
# If set to 'false', forbids any other homeserver to fetch the server's public
# rooms directory via federation. Defaults to 'true'.
#
#allow_public_rooms_over_federation: false
# The default room version for newly created rooms.
#
@ -511,9 +544,7 @@ class ServerConfig(Config):
# If you plan to use a reverse proxy, please see
# https://github.com/matrix-org/synapse/blob/master/docs/reverse_proxy.rst.
#
- port: %(unsecure_port)s
tls: false
bind_addresses: ['::1', '127.0.0.1']
- %(unsecure_http_binding)s
type: http
x_forwarded: true
@ -521,7 +552,7 @@ class ServerConfig(Config):
- names: [client, federation]
compress: false
# example additonal_resources:
# example additional_resources:
#
#additional_resources:
# "/_matrix/my/custom/endpoint":

2
synapse/config/server_notices_config.py
View file

@ -78,5 +78,5 @@ class ServerNoticesConfig(Config):
# todo: i18n
self.server_notices_room_name = c.get("room_name", "Server Notices")
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return DEFAULT_CONFIG

2
synapse/config/spam_checker.py
View file

@ -26,7 +26,7 @@ class SpamCheckerConfig(Config):
if provider is not None:
self.spam_checker = load_module(provider)
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return """\
#spam_checker:
# module: "my_custom_project.SuperSpamChecker"

2
synapse/config/stats.py
View file

@ -42,7 +42,7 @@ class StatsConfig(Config):
/ 1000
)
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """
# Local statistics collection. Used in populating the room directory.
#

2
synapse/config/third_party_event_rules.py
View file

@ -26,7 +26,7 @@ class ThirdPartyRulesConfig(Config):
if provider is not None:
self.third_party_event_rules = load_module(provider)
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return """\
# Server admins can define a Python module that implements extra rules for
# allowing or denying incoming events. In order to work, this module needs to

36
synapse/config/tls.py
View file

@ -23,7 +23,7 @@ import six
from unpaddedbase64 import encode_base64
from OpenSSL import crypto
from OpenSSL import SSL, crypto
from twisted.internet._sslverify import Certificate, trustRootFromCertificates
from synapse.config._base import Config, ConfigError
@ -81,6 +81,27 @@ class TlsConfig(Config):
"federation_verify_certificates", True
)
# Minimum TLS version to use for outbound federation traffic
self.federation_client_minimum_tls_version = str(
config.get("federation_client_minimum_tls_version", 1)
)
if self.federation_client_minimum_tls_version not in ["1", "1.1", "1.2", "1.3"]:
raise ConfigError(
"federation_client_minimum_tls_version must be one of: 1, 1.1, 1.2, 1.3"
)
# Prevent people shooting themselves in the foot here by setting it to
# the biggest number blindly
if self.federation_client_minimum_tls_version == "1.3":
if getattr(SSL, "OP_NO_TLSv1_3", None) is None:
raise ConfigError(
(
"federation_client_minimum_tls_version cannot be 1.3, "
"your OpenSSL does not support it"
)
)
# Whitelist of domains to not verify certificates for
fed_whitelist_entries = config.get(
"federation_certificate_verification_whitelist", []
@ -217,7 +238,9 @@ class TlsConfig(Config):
if sha256_fingerprint not in sha256_fingerprints:
self.tls_fingerprints.append({"sha256": sha256_fingerprint})
def default_config(self, config_dir_path, server_name, data_dir_path, **kwargs):
def generate_config_section(
self, config_dir_path, server_name, data_dir_path, **kwargs
):
base_key_name = os.path.join(config_dir_path, server_name)
tls_certificate_path = base_key_name + ".tls.crt"
@ -259,6 +282,15 @@ class TlsConfig(Config):
#
#federation_verify_certificates: false
# The minimum TLS version that will be used for outbound federation requests.
#
# Defaults to `1`. Configurable to `1`, `1.1`, `1.2`, or `1.3`. Note
# that setting this value higher than `1.2` will prevent federation to most
# of the public Matrix network: only configure it to `1.3` if you have an
# entirely private federation setup and you can ensure TLS 1.3 support.
#
#federation_client_minimum_tls_version: 1.2
# Skip federation certificate verification on the following whitelist
# of domains.
#

2
synapse/config/user_directory.py
View file

@ -33,7 +33,7 @@ class UserDirectoryConfig(Config):
"search_all_users", False
)
def default_config(self, config_dir_path, server_name, **kwargs):
def generate_config_section(self, config_dir_path, server_name, **kwargs):
return """
# User Directory configuration
#

2
synapse/config/voip.py
View file

@ -26,7 +26,7 @@ class VoipConfig(Config):
)
self.turn_allow_guests = config.get("turn_allow_guests", True)
def default_config(self, **kwargs):
def generate_config_section(self, **kwargs):
return """\
## TURN ##

39
synapse/crypto/context_factory.py
View file

@ -24,12 +24,25 @@ from OpenSSL import SSL, crypto
from twisted.internet._sslverify import _defaultCurveName
from twisted.internet.abstract import isIPAddress, isIPv6Address
from twisted.internet.interfaces import IOpenSSLClientConnectionCreator
from twisted.internet.ssl import CertificateOptions, ContextFactory, platformTrust
from twisted.internet.ssl import (
CertificateOptions,
ContextFactory,
TLSVersion,
platformTrust,
)
from twisted.python.failure import Failure
logger = logging.getLogger(__name__)
_TLS_VERSION_MAP = {
"1": TLSVersion.TLSv1_0,
"1.1": TLSVersion.TLSv1_1,
"1.2": TLSVersion.TLSv1_2,
"1.3": TLSVersion.TLSv1_3,
}
class ServerContextFactory(ContextFactory):
"""Factory for PyOpenSSL SSL contexts that are used to handle incoming
connections."""
@ -43,16 +56,18 @@ class ServerContextFactory(ContextFactory):
try:
_ecCurve = crypto.get_elliptic_curve(_defaultCurveName)
context.set_tmp_ecdh(_ecCurve)
except Exception:
logger.exception("Failed to enable elliptic curve for TLS")
context.set_options(SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
context.set_options(
SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3 | SSL.OP_NO_TLSv1 | SSL.OP_NO_TLSv1_1
)
context.use_certificate_chain_file(config.tls_certificate_file)
context.use_privatekey(config.tls_private_key)
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
context.set_cipher_list(
"ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1"
"ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1:!AESCCM"
)
def getContext(self):
@ -79,10 +94,22 @@ class ClientTLSOptionsFactory(object):
# Use CA root certs provided by OpenSSL
trust_root = platformTrust()
self._verify_ssl_context = CertificateOptions(trustRoot=trust_root).getContext()
# "insecurelyLowerMinimumTo" is the argument that will go lower than
# Twisted's default, which is why it is marked as "insecure" (since
# Twisted's defaults are reasonably secure). But, since Twisted is
# moving to TLS 1.2 by default, we want to respect the config option if
# it is set to 1.0 (which the alternate option, raiseMinimumTo, will not
# let us do).
minTLS = _TLS_VERSION_MAP[config.federation_client_minimum_tls_version]
self._verify_ssl = CertificateOptions(
trustRoot=trust_root, insecurelyLowerMinimumTo=minTLS
)
self._verify_ssl_context = self._verify_ssl.getContext()
self._verify_ssl_context.set_info_callback(self._context_info_cb)
self._no_verify_ssl_context = CertificateOptions().getContext()
self._no_verify_ssl = CertificateOptions(insecurelyLowerMinimumTo=minTLS)
self._no_verify_ssl_context = self._no_verify_ssl.getContext()
self._no_verify_ssl_context.set_info_callback(self._context_info_cb)
def get_options(self, host):

8
synapse/federation/transport/server.py
View file

@ -721,15 +721,15 @@ class PublicRoomList(BaseFederationServlet):
PATH = "/publicRooms"
def __init__(self, handler, authenticator, ratelimiter, server_name, deny_access):
def __init__(self, handler, authenticator, ratelimiter, server_name, allow_access):
super(PublicRoomList, self).__init__(
handler, authenticator, ratelimiter, server_name
)
self.deny_access = deny_access
self.allow_access = allow_access
@defer.inlineCallbacks
def on_GET(self, origin, content, query):
if self.deny_access:
if not self.allow_access:
raise FederationDeniedError(origin)
limit = parse_integer_from_args(query, "limit", 0)
@ -1436,7 +1436,7 @@ def register_servlets(hs, resource, authenticator, ratelimiter, servlet_groups=N
authenticator=authenticator,
ratelimiter=ratelimiter,
server_name=hs.hostname,
deny_access=hs.config.restrict_public_rooms_to_local_users,
allow_access=hs.config.allow_public_rooms_over_federation,
).register(resource)
if "group_server" in servlet_groups:

2
synapse/handlers/auth.py
View file

@ -743,7 +743,7 @@ class AuthHandler(BaseHandler):
result = (result, None)
defer.returnValue(result)
if login_type == LoginType.PASSWORD:
if login_type == LoginType.PASSWORD and self.hs.config.password_localdb_enabled:
known_login_type = True
canonical_user_id = yield self._check_local_password(

14
synapse/handlers/device.py
View file

@ -101,9 +101,13 @@ class DeviceWorkerHandler(BaseHandler):
room_ids = yield self.store.get_rooms_for_user(user_id)
# First we check if any devices have changed
changed = yield self.store.get_user_whose_devices_changed(
from_token.device_list_key
# First we check if any devices have changed for users that we share
# rooms with.
users_who_share_room = yield self.store.get_users_who_share_room_with_user(
user_id
)
changed = yield self.store.get_users_whose_devices_changed(
from_token.device_list_key, users_who_share_room
)
# Then work out if any users have since joined
@ -188,10 +192,6 @@ class DeviceWorkerHandler(BaseHandler):
break
if possibly_changed or possibly_left:
users_who_share_room = yield self.store.get_users_who_share_room_with_user(
user_id
)
# Take the intersection of the users whose devices may have changed
# and those that actually still share a room with the user
possibly_joined = possibly_changed & users_who_share_room

142
synapse/handlers/room.py
View file

@ -32,6 +32,7 @@ from synapse.storage.state import StateFilter
from synapse.types import RoomAlias, RoomID, RoomStreamToken, StreamToken, UserID
from synapse.util import stringutils
from synapse.util.async_helpers import Linearizer
from synapse.util.caches.response_cache import ResponseCache
from synapse.visibility import filter_events_for_client
from ._base import BaseHandler
@ -40,6 +41,8 @@ logger = logging.getLogger(__name__)
id_server_scheme = "https://"
FIVE_MINUTES_IN_MS = 5 * 60 * 1000
class RoomCreationHandler(BaseHandler):
@ -75,6 +78,12 @@ class RoomCreationHandler(BaseHandler):
# linearizer to stop two upgrades happening at once
self._upgrade_linearizer = Linearizer("room_upgrade_linearizer")
# If a user tries to update the same room multiple times in quick
# succession, only process the first attempt and return its result to
# subsequent requests
self._upgrade_response_cache = ResponseCache(
hs, "room_upgrade", timeout_ms=FIVE_MINUTES_IN_MS
)
self._server_notices_mxid = hs.config.server_notices_mxid
self.third_party_event_rules = hs.get_third_party_event_rules()
@ -95,67 +104,96 @@ class RoomCreationHandler(BaseHandler):
user_id = requester.user.to_string()
with (yield self._upgrade_linearizer.queue(old_room_id)):
# start by allocating a new room id
r = yield self.store.get_room(old_room_id)
if r is None:
raise NotFoundError("Unknown room id %s" % (old_room_id,))
new_room_id = yield self._generate_room_id(
creator_id=user_id, is_public=r["is_public"]
)
logger.info("Creating new room %s to replace %s", new_room_id, old_room_id)
# we create and auth the tombstone event before properly creating the new
# room, to check our user has perms in the old room.
tombstone_event, tombstone_context = (
yield self.event_creation_handler.create_event(
requester,
{
"type": EventTypes.Tombstone,
"state_key": "",
"room_id": old_room_id,
"sender": user_id,
"content": {
"body": "This room has been replaced",
"replacement_room": new_room_id,
},
},
token_id=requester.access_token_id,
# Check if this room is already being upgraded by another person
for key in self._upgrade_response_cache.pending_result_cache:
if key[0] == old_room_id and key[1] != user_id:
# Two different people are trying to upgrade the same room.
# Send the second an error.
#
# Note that this of course only gets caught if both users are
# on the same homeserver.
raise SynapseError(
400, "An upgrade for this room is currently in progress"
)
)
old_room_version = yield self.store.get_room_version(old_room_id)
yield self.auth.check_from_context(
old_room_version, tombstone_event, tombstone_context
)
yield self.clone_existing_room(
# Upgrade the room
#
# If this user has sent multiple upgrade requests for the same room
# and one of them is not complete yet, cache the response and
# return it to all subsequent requests
ret = yield self._upgrade_response_cache.wrap(
(old_room_id, user_id),
self._upgrade_room,
requester,
old_room_id,
new_version, # args for _upgrade_room
)
defer.returnValue(ret)
@defer.inlineCallbacks
def _upgrade_room(self, requester, old_room_id, new_version):
user_id = requester.user.to_string()
# start by allocating a new room id
r = yield self.store.get_room(old_room_id)
if r is None:
raise NotFoundError("Unknown room id %s" % (old_room_id,))
new_room_id = yield self._generate_room_id(
creator_id=user_id, is_public=r["is_public"]
)
logger.info("Creating new room %s to replace %s", new_room_id, old_room_id)
# we create and auth the tombstone event before properly creating the new
# room, to check our user has perms in the old room.
tombstone_event, tombstone_context = (
yield self.event_creation_handler.create_event(
requester,
old_room_id=old_room_id,
new_room_id=new_room_id,
new_room_version=new_version,
tombstone_event_id=tombstone_event.event_id,
{
"type": EventTypes.Tombstone,
"state_key": "",
"room_id": old_room_id,
"sender": user_id,
"content": {
"body": "This room has been replaced",
"replacement_room": new_room_id,
},
},
token_id=requester.access_token_id,
)
)
old_room_version = yield self.store.get_room_version(old_room_id)
yield self.auth.check_from_context(
old_room_version, tombstone_event, tombstone_context
)
# now send the tombstone
yield self.event_creation_handler.send_nonmember_event(
requester, tombstone_event, tombstone_context
)
yield self.clone_existing_room(
requester,
old_room_id=old_room_id,
new_room_id=new_room_id,
new_room_version=new_version,
tombstone_event_id=tombstone_event.event_id,
)
old_room_state = yield tombstone_context.get_current_state_ids(self.store)
# now send the tombstone
yield self.event_creation_handler.send_nonmember_event(
requester, tombstone_event, tombstone_context
)
# update any aliases
yield self._move_aliases_to_new_room(
requester, old_room_id, new_room_id, old_room_state
)
old_room_state = yield tombstone_context.get_current_state_ids(self.store)
# and finally, shut down the PLs in the old room, and update them in the new
# room.
yield self._update_upgraded_room_pls(
requester, old_room_id, new_room_id, old_room_state
)
# update any aliases
yield self._move_aliases_to_new_room(
requester, old_room_id, new_room_id, old_room_state
)
defer.returnValue(new_room_id)
# and finally, shut down the PLs in the old room, and update them in the new
# room.
yield self._update_upgraded_room_pls(
requester, old_room_id, new_room_id, old_room_state
)
defer.returnValue(new_room_id)
@defer.inlineCallbacks
def _update_upgraded_room_pls(

3
synapse/handlers/set_password.py
View file

@ -33,6 +33,9 @@ class SetPasswordHandler(BaseHandler):
@defer.inlineCallbacks
def set_password(self, user_id, newpassword, requester=None):
if not self.hs.config.password_localdb_enabled:
raise SynapseError(403, "Password change disabled", errcode=Codes.FORBIDDEN)
password_hash = yield self._auth_handler.hash(newpassword)
except_device_id = requester.device_id if requester else None

82
synapse/handlers/sync.py
View file

@ -1061,40 +1061,74 @@ class SyncHandler(object):
newly_left_rooms,
newly_left_users,
):
"""Generate the DeviceLists section of sync
Args:
sync_result_builder (SyncResultBuilder)
newly_joined_rooms (set[str]): Set of rooms user has joined since
previous sync
newly_joined_or_invited_users (set[str]): Set of users that have
joined or been invited to a room since previous sync.
newly_left_rooms (set[str]): Set of rooms user has left since
previous sync
newly_left_users (set[str]): Set of users that have left a room
we're in since previous sync
Returns:
Deferred[DeviceLists]
"""
user_id = sync_result_builder.sync_config.user.to_string()
since_token = sync_result_builder.since_token
# We're going to mutate these fields, so lets copy them rather than
# assume they won't get used later.
newly_joined_or_invited_users = set(newly_joined_or_invited_users)
newly_left_users = set(newly_left_users)
if since_token and since_token.device_list_key:
changed = yield self.store.get_user_whose_devices_changed(
since_token.device_list_key
)
# TODO: Be more clever than this, i.e. remove users who we already
# share a room with?
for room_id in newly_joined_rooms:
joined_users = yield self.state.get_current_users_in_room(room_id)
newly_joined_or_invited_users.update(joined_users)
for room_id in newly_left_rooms:
left_users = yield self.state.get_current_users_in_room(room_id)
newly_left_users.update(left_users)
# TODO: Check that these users are actually new, i.e. either they
# weren't in the previous sync *or* they left and rejoined.
changed.update(newly_joined_or_invited_users)
if not changed and not newly_left_users:
defer.returnValue(DeviceLists(changed=[], left=newly_left_users))
# We want to figure out what user IDs the client should refetch
# device keys for, and which users we aren't going to track changes
# for anymore.
#
# For the first step we check:
# a. if any users we share a room with have updated their devices,
# and
# b. we also check if we've joined any new rooms, or if a user has
# joined a room we're in.
#
# For the second step we just find any users we no longer share a
# room with by looking at all users that have left a room plus users
# that were in a room we've left.
users_who_share_room = yield self.store.get_users_who_share_room_with_user(
user_id
)
# Step 1a, check for changes in devices of users we share a room with
users_that_have_changed = yield self.store.get_users_whose_devices_changed(
since_token.device_list_key, users_who_share_room
)
# Step 1b, check for newly joined rooms
for room_id in newly_joined_rooms:
joined_users = yield self.state.get_current_users_in_room(room_id)
newly_joined_or_invited_users.update(joined_users)
# TODO: Check that these users are actually new, i.e. either they
# weren't in the previous sync *or* they left and rejoined.
users_that_have_changed.update(newly_joined_or_invited_users)
# Now find users that we no longer track
for room_id in newly_left_rooms:
left_users = yield self.state.get_current_users_in_room(room_id)
newly_left_users.update(left_users)
# Remove any users that we still share a room with.
newly_left_users -= users_who_share_room
defer.returnValue(
DeviceLists(
changed=users_who_share_room & changed,
left=set(newly_left_users) - users_who_share_room,
)
DeviceLists(changed=users_that_have_changed, left=newly_left_users)
)
else:
defer.returnValue(DeviceLists(changed=[], left=[]))

1
synapse/python_dependencies.py
View file

@ -95,6 +95,7 @@ CONDITIONAL_REQUIREMENTS = {
"url_preview": ["lxml>=3.5.0"],
"test": ["mock>=2.0", "parameterized"],
"sentry": ["sentry-sdk>=0.7.2"],
"jwt": ["pyjwt>=1.6.4"],
}
ALL_OPTIONAL_REQUIREMENTS = set()

2
synapse/rest/client/v1/login.py
View file

@ -336,7 +336,7 @@ class LoginRestServlet(RestServlet):
}
else:
user_id, access_token = (
yield self.handlers.registration_handler.register(localpart=user)
yield self.registration_handler.register(localpart=user)
)
device_id = login_submission.get("device_id")

2
synapse/rest/client/v1/room.py
View file

@ -311,7 +311,7 @@ class PublicRoomListRestServlet(TransactionRestServlet):
# Option to allow servers to require auth when accessing
# /publicRooms via CS API. This is especially helpful in private
# federations.
if self.hs.config.restrict_public_rooms_to_local_users:
if not self.hs.config.allow_public_rooms_without_auth:
raise
# We allow people to not be authed if they're just looking at our

35
synapse/storage/_base.py
View file

@ -175,22 +175,22 @@ class PerformanceCounters(object):
self.current_counters = {}
self.previous_counters = {}
def update(self, key, start_time, end_time=None):
if end_time is None:
end_time = time.time()
duration = end_time - start_time
def update(self, key, duration_secs):
count, cum_time = self.current_counters.get(key, (0, 0))
count += 1
cum_time += duration
cum_time += duration_secs
self.current_counters[key] = (count, cum_time)
return end_time
def interval(self, interval_duration, limit=3):
def interval(self, interval_duration_secs, limit=3):
counters = []
for name, (count, cum_time) in iteritems(self.current_counters):
prev_count, prev_time = self.previous_counters.get(name, (0, 0))
counters.append(
((cum_time - prev_time) / interval_duration, count - prev_count, name)
(
(cum_time - prev_time) / interval_duration_secs,
count - prev_count,
name,
)
)
self.previous_counters = dict(self.current_counters)
@ -221,7 +221,6 @@ class SQLBaseStore(object):
# is running in mainline, and we have some nice monitoring frontends
# to watch it
self._txn_perf_counters = PerformanceCounters()
self._get_event_counters = PerformanceCounters()
self._get_event_cache = Cache(
"*getEvent*", keylen=3, max_entries=hs.config.event_cache_size
@ -369,21 +368,13 @@ class SQLBaseStore(object):
time_then = self._previous_loop_ts
self._previous_loop_ts = time_now
ratio = (curr - prev) / (time_now - time_then)
duration = time_now - time_then
ratio = (curr - prev) / duration
top_three_counters = self._txn_perf_counters.interval(
time_now - time_then, limit=3
)
top_3_event_counters = self._get_event_counters.interval(
time_now - time_then, limit=3
)
top_three_counters = self._txn_perf_counters.interval(duration, limit=3)
perf_logger.info(
"Total database time: %.3f%% {%s} {%s}",
ratio * 100,
top_three_counters,
top_3_event_counters,
"Total database time: %.3f%% {%s}", ratio * 100, top_three_counters
)
self._clock.looping_call(loop, 10000)
@ -465,7 +456,7 @@ class SQLBaseStore(object):
transaction_logger.debug("[TXN END] {%s} %f sec", name, duration)
self._current_txn_total_time += duration
self._txn_perf_counters.update(desc, start, end)
self._txn_perf_counters.update(desc, duration)
sql_txn_timer.labels(desc).observe(duration)
@defer.inlineCallbacks

50
synapse/storage/devices.py
View file

@ -24,6 +24,7 @@ from synapse.api.errors import StoreError
from synapse.metrics.background_process_metrics import run_as_background_process
from synapse.storage._base import Cache, SQLBaseStore, db_to_json
from synapse.storage.background_updates import BackgroundUpdateStore
from synapse.util import batch_iter
from synapse.util.caches.descriptors import cached, cachedInlineCallbacks, cachedList
logger = logging.getLogger(__name__)
@ -391,22 +392,47 @@ class DeviceWorkerStore(SQLBaseStore):
return now_stream_id, []
@defer.inlineCallbacks
def get_user_whose_devices_changed(self, from_key):
"""Get set of users whose devices have changed since `from_key`.
def get_users_whose_devices_changed(self, from_key, user_ids):
"""Get set of users whose devices have changed since `from_key` that
are in the given list of user_ids.
Args:
from_key (str): The device lists stream token
user_ids (Iterable[str])
Returns:
Deferred[set[str]]: The set of user_ids whose devices have changed
since `from_key`
"""
from_key = int(from_key)
changed = self._device_list_stream_cache.get_all_entities_changed(from_key)
if changed is not None:
defer.returnValue(set(changed))
sql = """
SELECT DISTINCT user_id FROM device_lists_stream WHERE stream_id > ?
"""
rows = yield self._execute(
"get_user_whose_devices_changed", None, sql, from_key
# Get set of users who *may* have changed. Users not in the returned
# list have definitely not changed.
to_check = list(
self._device_list_stream_cache.get_entities_changed(user_ids, from_key)
)
if not to_check:
return defer.succeed(set())
def _get_users_whose_devices_changed_txn(txn):
changes = set()
sql = """
SELECT DISTINCT user_id FROM device_lists_stream
WHERE stream_id > ?
AND user_id IN (%s)
"""
for chunk in batch_iter(to_check, 100):
txn.execute(sql % (",".join("?" for _ in chunk),), (from_key,) + chunk)
changes.update(user_id for user_id, in txn)
return changes
return self.runInteraction(
"get_users_whose_devices_changed", _get_users_whose_devices_changed_txn
)
defer.returnValue(set(row[0] for row in rows))
def get_all_device_list_changes_for_remotes(self, from_key, to_key):
"""Return a list of `(stream_id, user_id, destination)` which is the

2
synapse/util/caches/response_cache.py
View file

@ -137,7 +137,7 @@ class ResponseCache(object):
*args: positional parameters to pass to the callback, if it is used
**kwargs: named paramters to pass to the callback, if it is used
**kwargs: named parameters to pass to the callback, if it is used
Returns:
twisted.internet.defer.Deferred: yieldable result

46
synapse/util/logcontext.py
View file

@ -336,10 +336,9 @@ class LoggingContext(object):
logger.warning("Called stop on logcontext %s without calling start", self)
return
usage_end = get_thread_resource_usage()
self._resource_usage.ru_utime += usage_end.ru_utime - self.usage_start.ru_utime
self._resource_usage.ru_stime += usage_end.ru_stime - self.usage_start.ru_stime
utime_delta, stime_delta = self._get_cputime()
self._resource_usage.ru_utime += utime_delta
self._resource_usage.ru_stime += stime_delta
self.usage_start = None
@ -357,13 +356,44 @@ class LoggingContext(object):
# can include resource usage so far.
is_main_thread = threading.current_thread() is self.main_thread
if self.alive and self.usage_start and is_main_thread:
current = get_thread_resource_usage()
res.ru_utime += current.ru_utime - self.usage_start.ru_utime
res.ru_stime += current.ru_stime - self.usage_start.ru_stime
utime_delta, stime_delta = self._get_cputime()
res.ru_utime += utime_delta
res.ru_stime += stime_delta
return res
def _get_cputime(self):
"""Get the cpu usage time so far
Returns: Tuple[float, float]: seconds in user mode, seconds in system mode
"""
current = get_thread_resource_usage()
utime_delta = current.ru_utime - self.usage_start.ru_utime
stime_delta = current.ru_stime - self.usage_start.ru_stime
# sanity check
if utime_delta < 0:
logger.error(
"utime went backwards! %f < %f",
current.ru_utime,
self.usage_start.ru_utime,
)
utime_delta = 0
if stime_delta < 0:
logger.error(
"stime went backwards! %f < %f",
current.ru_stime,
self.usage_start.ru_stime,
)
stime_delta = 0
return utime_delta, stime_delta
def add_database_transaction(self, duration_sec):
if duration_sec < 0:
raise ValueError("DB txn time can only be non-negative")
self._resource_usage.db_txn_count += 1
self._resource_usage.db_txn_duration_sec += duration_sec
@ -374,6 +404,8 @@ class LoggingContext(object):
sched_sec (float): number of seconds it took us to get a
connection
"""
if sched_sec < 0:
raise ValueError("DB scheduling time can only be non-negative")
self._resource_usage.db_sched_duration_sec += sched_sec
def record_event_fetch(self, event_count):

115
tests/config/test_tls.py
View file

@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright 2019 New Vector Ltd
# Copyright 2019 Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -15,7 +16,10 @@
import os
from synapse.config.tls import TlsConfig
from OpenSSL import SSL
from synapse.config.tls import ConfigError, TlsConfig
from synapse.crypto.context_factory import ClientTLSOptionsFactory
from tests.unittest import TestCase
@ -78,3 +82,112 @@ s4niecZKPBizL6aucT59CsunNmmb5Glq8rlAcU+1ZTZZzGYqVYhF6axB9Qg=
"or use Synapse's ACME support to provision one."
),
)
def test_tls_client_minimum_default(self):
"""
The default client TLS version is 1.0.
"""
config = {}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
self.assertEqual(t.federation_client_minimum_tls_version, "1")
def test_tls_client_minimum_set(self):
"""
The default client TLS version can be set to 1.0, 1.1, and 1.2.
"""
config = {"federation_client_minimum_tls_version": 1}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
self.assertEqual(t.federation_client_minimum_tls_version, "1")
config = {"federation_client_minimum_tls_version": 1.1}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
self.assertEqual(t.federation_client_minimum_tls_version, "1.1")
config = {"federation_client_minimum_tls_version": 1.2}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
self.assertEqual(t.federation_client_minimum_tls_version, "1.2")
# Also test a string version
config = {"federation_client_minimum_tls_version": "1"}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
self.assertEqual(t.federation_client_minimum_tls_version, "1")
config = {"federation_client_minimum_tls_version": "1.2"}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
self.assertEqual(t.federation_client_minimum_tls_version, "1.2")
def test_tls_client_minimum_1_point_3_missing(self):
"""
If TLS 1.3 support is missing and it's configured, it will raise a
ConfigError.
"""
# thanks i hate it
if hasattr(SSL, "OP_NO_TLSv1_3"):
OP_NO_TLSv1_3 = SSL.OP_NO_TLSv1_3
delattr(SSL, "OP_NO_TLSv1_3")
self.addCleanup(setattr, SSL, "SSL.OP_NO_TLSv1_3", OP_NO_TLSv1_3)
assert not hasattr(SSL, "OP_NO_TLSv1_3")
config = {"federation_client_minimum_tls_version": 1.3}
t = TestConfig()
with self.assertRaises(ConfigError) as e:
t.read_config(config, config_dir_path="", data_dir_path="")
self.assertEqual(
e.exception.args[0],
(
"federation_client_minimum_tls_version cannot be 1.3, "
"your OpenSSL does not support it"
),
)
def test_tls_client_minimum_1_point_3_exists(self):
"""
If TLS 1.3 support exists and it's configured, it will be settable.
"""
# thanks i hate it, still
if not hasattr(SSL, "OP_NO_TLSv1_3"):
SSL.OP_NO_TLSv1_3 = 0x00
self.addCleanup(lambda: delattr(SSL, "OP_NO_TLSv1_3"))
assert hasattr(SSL, "OP_NO_TLSv1_3")
config = {"federation_client_minimum_tls_version": 1.3}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
self.assertEqual(t.federation_client_minimum_tls_version, "1.3")
def test_tls_client_minimum_set_passed_through_1_2(self):
"""
The configured TLS version is correctly configured by the ContextFactory.
"""
config = {"federation_client_minimum_tls_version": 1.2}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
cf = ClientTLSOptionsFactory(t)
# The context has had NO_TLSv1_1 and NO_TLSv1_0 set, but not NO_TLSv1_2
self.assertNotEqual(cf._verify_ssl._options & SSL.OP_NO_TLSv1, 0)
self.assertNotEqual(cf._verify_ssl._options & SSL.OP_NO_TLSv1_1, 0)
self.assertEqual(cf._verify_ssl._options & SSL.OP_NO_TLSv1_2, 0)
def test_tls_client_minimum_set_passed_through_1_0(self):
"""
The configured TLS version is correctly configured by the ContextFactory.
"""
config = {"federation_client_minimum_tls_version": 1}
t = TestConfig()
t.read_config(config, config_dir_path="", data_dir_path="")
cf = ClientTLSOptionsFactory(t)
# The context has not had any of the NO_TLS set.
self.assertEqual(cf._verify_ssl._options & SSL.OP_NO_TLSv1, 0)
self.assertEqual(cf._verify_ssl._options & SSL.OP_NO_TLSv1_1, 0)
self.assertEqual(cf._verify_ssl._options & SSL.OP_NO_TLSv1_2, 0)

2
tests/rest/client/v1/test_rooms.py
View file

@ -920,7 +920,7 @@ class PublicRoomsRestrictedTestCase(unittest.HomeserverTestCase):
self.url = b"/_matrix/client/r0/publicRooms"
config = self.default_config()
config["restrict_public_rooms_to_local_users"] = True
config["allow_public_rooms_without_auth"] = False
self.hs = self.setup_test_homeserver(config=config)
return self.hs