Commit graph

174 commits

Author SHA1 Message Date
David Robertson
e9ce4d089b
Use and recommend poetry 1.1.14, up from 1.1.12 (#13285) 2022-07-15 16:18:47 +01:00
Richard van der Hoff
512486bbeb
Docker: copy postgres from base image (#13279)
When building the docker images for complement testing, copy a preinstalled
complement over from a base image, rather than apt installing it. This avoids
network traffic and is much faster.
2022-07-15 11:13:40 +01:00
reivilibre
65e675504f
Add the ability to set the log level using the SYNAPSE_TEST_LOG_LEVEL environment when using complement.sh. (#13152) 2022-07-05 09:46:20 +00:00
Till
fa10468eb4
[Complement] Allow device_name lookup over federation (#13167) 2022-07-04 12:34:50 +00:00
reivilibre
c04e25789e
Enable Complement testing in the 'Twisted Trunk' CI runs. (#13079) 2022-07-01 15:42:49 +00:00
reivilibre
9667bad55d
Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments. (#13127)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-30 11:58:12 +00:00
reivilibre
3c5549e74a
Refactor the Dockerfile-workers configuration script to use Jinja2 templates in Synapse workers' Supervisord blocks. (#13054)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-27 11:43:20 +01:00
reivilibre
90cadcd403
Add a Subject Alternative Name to the certificate generated for Complement tests. (#13071) 2022-06-16 12:43:21 +01:00
Richard van der Hoff
1e0044e8f9
Complement: use SQLite by default (#13075)
If no database is configured explicitly, use sqlite.

This means that you don't have to pass any variables into the image.
2022-06-16 12:12:26 +01:00
Erik Johnston
c95b04bb0e
Change default sync_response_cache_duration (#13042) 2022-06-15 16:55:20 +00:00
reivilibre
212be2edc1
Use updated update_user_directory_from_worker options in Complement tests. (#13069) 2022-06-15 15:54:32 +01:00
reivilibre
538044ac01
Collapse Docker build commands in Complement CI runs to make the logs easier to read. (#13058) 2022-06-15 14:42:27 +00:00
reivilibre
67f51c84f8
Merge the Complement testing Docker images into a single, multi-purpose image. (#12881)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-08 09:57:05 +00:00
Nick Mills-Barrett
1f9013ce60
Add the batch_send endpoint to generic workers (#12868) 2022-05-25 09:51:07 +00:00
Erik Johnston
f5b1c09909
Pin poetry.core in Docker images (#12853) 2022-05-24 11:35:08 +01:00
reivilibre
67aae05ece
Support registering Application Services when running with workers under Complement. (#12826)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-05-23 14:11:06 +01:00
reivilibre
438925c422
Fix Complement TestCanRegisterAdmin with workers, by adding Complement's shared registration secret. (#12819) 2022-05-23 12:20:30 +01:00
reivilibre
4fef76ca34
Remove Caddy from the Synapse workers image used in Complement. (#12818) 2022-05-23 10:29:24 +01:00
Till
e8ae472d3b
Update configs used by Complement to allow more invites (#12731) 2022-05-13 16:45:47 +01:00
Richard van der Hoff
34e84fee68
Tweaks to workers-under-complement (#12637)
* Bump the HS startup timeout
* Log prefixes for more processes
* Bump the overall timeout
2022-05-09 22:41:06 +01:00
reivilibre
c2d50e9f6c
Add the notify_appservices_from_worker configuration option (superseding notify_appservices) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452) 2022-05-06 11:43:53 +01:00
reivilibre
d743b25c8f
Use supervisord to supervise Postgres and Caddy in the Complement image. (#12480)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-27 14:39:41 +01:00
Jason Robinson
706456de1f
Mark Dockerfile as requiring BuildKit (#12541)
Co-authored-by: David Robertson <davidr@element.io>
2022-04-26 15:31:52 +01:00
David Robertson
8bac3e0435
disallow-untyped-defs in docker and stubs directories (#12528) 2022-04-25 12:32:35 +00:00
Richard van der Hoff
f5668f0b4a
Await un-partial-stating after a partial-state join (#12399)
When we join a room via the faster-joins mechanism, we end up with "partial
state" at some points on the event DAG. Many parts of the codebase need to
wait for the full state to load. So, we implement a mechanism to keep track of
which events have partial state, and wait for them to be fully-populated.
2022-04-21 07:42:03 +01:00
David Robertson
e5a76ec00b
Dump setuptools; correct pyproject version number (#12478) 2022-04-20 17:33:20 +01:00
Richard van der Hoff
aaaff98202
Dockerfile-workers: reduce the amount we install (#12464)
This is an attempt to reduce the rebuild time. In short, we reduce the amount
of stuff that the dockerfile installs, so as to give a faster startup.
2022-04-14 15:36:49 +01:00
Richard van der Hoff
960b4fb409
complement-synapse-workers: factor out separate entry point script (#12467)
... with a bit more verbosity.
2022-04-14 14:56:10 +01:00
Richard van der Hoff
8af8a9bce5
Dockerfile-workers: give the master its own log config (#12466)
When we run a worker-mode synapse under docker, everything gets logged to stdout. Currently, output from the workers is tacked with a worker name, for example:

```
2022-04-13 15:27:56,810 - worker:frontend_proxy1 - synapse.util.caches.lrucache - 154 - INFO - LruCache._expire_old_entries-0 - Dropped 0 items from caches
```

- note `worker:frontend_proxy1`. No such tag is applied to log lines from the master, which makes somewhat confusing reading.

To fix this, we generate a dedicated log config file for the master in the same way that we do for the workers, and use that.
2022-04-13 20:50:08 +01:00
David Robertson
5f8173dd80
Workaround pip bug installing latest treq and not-latest twisted from hashes (#12439)
The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0.

When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true

> ```
> #15 9.204 Collecting Twisted[tls]>=18.7.0
> #15 9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
> #15 9.205     Twisted[tls]>=18.7.0 from 38622ff95b/Twisted-22.4.0-py3-none-any.whl (sha256)=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724))
> #15 ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1
> ```

The underlying pip issue is https://github.com/pypa/pip/issues/9644. A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so.

(At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311, but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).
2022-04-12 10:16:01 +01:00
Richard van der Hoff
214f3b7d21
Enable certificate checking during complement tests (#12435) 2022-04-11 16:35:41 +01:00
Richard van der Hoff
3cdf5a1386
Fix up healthcheck generation for workers docker image (#12405)
This wasn't quite generating the right thing.
2022-04-11 13:38:58 +00:00
Richard van der Hoff
5f72ea1bde
Move complement setup stuff into the Synapse repo (#12404)
Fixes matrix-org/complement#330 (or it will, once we remove the old files).

It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
2022-04-11 11:39:28 +01:00
Jorge Florian
78e4d96a4d
Add missing type definitions for scripts in docker folder (#12280)
Signed-off-by: Jorge Florian <jafn28@gmail.com>
2022-04-08 10:10:58 +00:00
David Robertson
3a7e97c7ad
Poetry: use locked environment in Docker images (#12385) 2022-04-07 11:43:31 +00:00
Richard van der Hoff
34a8370d7b
README-testing.md: fix minor error 2022-03-31 13:08:03 +01:00
David Robertson
15cdcf8f30
Remove dockerfile-pgtests (#12336) 2022-03-31 11:30:07 +01:00
David Robertson
4aeb00ca20
Move synctl into synapse._scripts and expose as an entrypoint (#12140) 2022-03-04 11:58:49 +00:00
David Robertson
f3f0ab10fe
Move scripts directory inside synapse, exposing as setuptools entry_points (#12118)
* Two scripts are basically entry_points already
* Move and rename scripts/* to synapse/_scripts/*.py
* Delete sync_room_to_group.pl
* Expose entry points in setup.py
* Update linter script and config
* Fixup scripts & docs mentioning scripts that moved

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2022-03-02 13:00:16 +00:00
Andrew Morgan
f26e390a40
Use Python 3.9 in Synapse dockerfiles by default (#12112) 2022-03-01 13:55:18 +00:00
Richard van der Hoff
5598556b77
Docker: remove VOLUME directive (#11997)
The driver for this is to stop Complement complaining about it, but as far as I can tell it was pointless and needed to go away anyway.

I'm a bit unclear about what exactly VOLUME does, but I think what it means is that, if you don't override it with an explicit -v argument, then docker run will create a temporary volume, and copy things into it. The temporary volume is then deleted when the container finishes.

That only sounds useful if your image has something to copy into it (otherwise you may as well just use the default root filesystem), and our image notably doesn't copy anything into /data.

So... this wasn't doing anything, except annoying Complement?
2022-02-15 13:59:15 +00:00
Shay
9006ee36d1
Drop support for and remove references to EOL Python 3.6 (#11683)
* remove reference in comments to python3.6

* upgrade tox python env in script

* bump python version in example for completeness

* upgrade python version requirement in setup doc

* upgrade necessary python version in __init__.py

* upgrade python version in setup.py

* newsfragment

* drops refs to bionic and replace with focal

* bump refs to postgres 9.6 to 10

* fix hanging ci

* try installing tzdata first

* revert change made in b979f336

* ignore new random mypy error while debugging other error

* fix lint error for temporary workaround

* revert change to install list

* try passing env var

* export debian frontend var?

* move line and add comment

* bump pillow dependency

* bump lxml depenency

* install libjpeg-dev for pillow

* bump automat version to one compatible with py3.8

* add libwebp for pillow

* bump twisted trunk python version

* change suffix of newsfragment

* remove redundant python 3.7 checks

* lint
2022-01-21 14:23:26 -08:00
Richard van der Hoff
d41c4654db
Use buildkit's cache feature to speed up docker builds (#11691)
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
2022-01-12 10:37:57 +00:00
Callum Macdonald
7013e06e2f
Improve Docker docs for use with Postgres (#11640) 2022-01-05 10:50:28 +00:00
Shay
13c974ed35
Drop Bionic from Debian builds (#11633)
* update Trove classifiers to remove py36

* stop building bionic

* update dh-virtualenv

* newsfragment

* fix newsfragment

* update version refs

* another try at correct tag

* Update changelog
2022-01-03 11:17:16 -08:00
Michael Kaye
e2c300e7e4
Create healthcheck script for synapse-workers container (#11429)
The intent is to iterate through all the worker ports and only
report healthy when all are healthy, starting with the main process.
2021-11-26 14:05:20 +00:00
Shay
7cebaf9644
Remove code invalidated by deprecated config flag 'trust_identity_servers_for_password_resets' (#11395)
* remove background update code related to deprecated config flag

* changelog entry

* update changelog

* Delete 11394.removal

Duplicate, wrong number

* add no-op background update and change newfragment so it will be consolidated with associated work

* remove unused code

* Remove code associated with deprecated flag from legacy docker dynamic config file

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-23 06:46:40 -08:00
Aaron R
d993c3bb1e
Add support for /_matrix/media/v3 APIs (#11371)
* Add support for `/_matrix/media/v3` APIs

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Update `workers.md` to use v3 client and media APIs

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-11-17 15:30:24 +00:00
Richard van der Hoff
0b99d4c8d2
Docker: avoid changing userid unnecessarily (#11209)
* Docker image: avoid changing user during `generate`

The intention was always that the config files get written as the initial user
(normally root) - only the data directory needs to be writable by Synapse. This
got changed in https://github.com/matrix-org/synapse/pull/5970, but that seems
to have been a mistake.

* Avoid changing user if no explicit UID is given

* changelog
2021-11-01 13:55:30 +00:00
Dan Callahan
d7141e0b8b
Fix Shellcheck SC2006: Use $(...) notation
Use $(...) notation instead of legacy backticked `...`.

https://github.com/koalaman/shellcheck/wiki/SC2006

Signed-off-by: Dan Callahan <danc@element.io>
2021-10-22 23:08:55 +01:00