0
0
Fork 1
mirror of https://mau.dev/maunium/synapse.git synced 2024-12-14 17:34:00 +01:00
Commit graph

8017 commits

Author SHA1 Message Date
Erik Johnston
df366966b4
Speed up pruning of user_ips table (#16667)
Silly query planner
2023-11-29 11:54:42 +00:00
Patrick Cloke
77882b6a7d
Document which versions of Synapse have compatible schema versions. (#16661) 2023-11-28 11:01:24 -05:00
Mathieu Velten
b0ed14d815
Ignore encryption_enabled_by_default_for_room_type for notices room (#16677) 2023-11-28 13:15:26 +00:00
Patrick Cloke
d199b84006
Remove old full schema dumps. (#16697)
These are not useful and make it difficult to search for
table definitions, etc.
2023-11-28 07:28:07 -05:00
David Robertson
8751f0ef32
Fix poetry version typo in contributors' guide (#16695) 2023-11-27 15:16:20 +00:00
David Robertson
0619c2bbd2
Move media retention tests out of rest tests (#16684)
* Move media retention tests out of rest tests

AFAICS this doesn't make any HTTP requests and so it ought not to belong
in `tests.rest`.

* Changelog
2023-11-27 01:29:46 +00:00
David Robertson
c3627d0f99
Correctly read to-device stream pos on SQLite (#16682) 2023-11-24 13:42:38 +00:00
David Robertson
32a59a6495
Keep track of user_ips and monthly_active_users when delegating auth (#16672)
* Describe `insert_client_ip`
* Pull out client_ips and MAU tracking to BaseAuth
* Define HAS_AUTHLIB once in tests

sick of copypasting

* Track ips and token usage when delegating auth
* Test that we track MAU and user_ips
* Don't track `__oidc_admin`
2023-11-23 12:35:37 +00:00
Charles Wright
1a5f9bb651
Enable refreshable tokens on the admin registration endpoint (#16642)
Signed-off-by: Charles Wright <cvwright@futo.org>
2023-11-22 15:01:09 +00:00
V02460
f2430b16d1
Bump pyo3 (0.20), pythonize (0.20), pyo3-log (0.9) (#16673)
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
2023-11-22 14:55:43 +00:00
Mathieu Velten
c432d8f18f
Admin API for server notice: consistently bypass rate limits (#16670)
* Admin API for server notice: disable rate limit for all calls

* Add changelog

* Update changelog.d/16670.bugfix
2023-11-22 13:47:29 +00:00
Jason Little
460743da16
Filter out auth chain queries that don't exist (#16552) 2023-11-22 10:59:16 +00:00
David Robertson
bb86eb9814
1.97.0rc1 2023-11-21 12:38:46 +00:00
Erik Johnston
6088303efb
Speed up how quickly we launch new tasks (#16660)
Now that we're reducing concurrency (#16656), this is more important.
2023-11-17 16:36:02 +00:00
Patrick Cloke
d9dcfe2a35
Bump requests-toolbelt from 0.10.1 to 1.0.0. (#16659) 2023-11-17 10:23:07 -05:00
Erik Johnston
9c02ef21e0
Speed up purge room by adding index (#16657)
What it says on the tin
2023-11-17 14:15:44 +00:00
Erik Johnston
6fec2d035f
Also discard 'caches' and 'backfill' stream POSITIONS (#16655)
Follow on from #16640
2023-11-17 14:14:29 +00:00
Michael Weimann
518e4de758
Update admin user API return types in docs. (#16654) 2023-11-17 13:38:25 +00:00
Erik Johnston
700c8a0de5
Reduce task concurrency (#16656) 2023-11-17 13:14:26 +00:00
Erik Johnston
4d6b800385
Revert "Fix test not detecting tables with missing primary keys and missing replica identities, then add more replica identities. (#16647)" (#16652)
This reverts commit 830988ae72.
2023-11-16 16:57:26 +00:00
Erik Johnston
ef5329a9f9
Revert "Add a Postgres REPLICA IDENTITY to tables that do not have an implicit one. This should allow use of Postgres logical replication. (#16456)" (#16651)
This reverts commit 69afe3f7a0.
2023-11-16 16:48:48 +00:00
Erik Johnston
3e8531d3ba
Speed up deleting device messages (#16643)
Keeping track of a lower bound of stream ID where we've deleted everything below makes the queries much faster. Otherwise, every time we scan for rows to delete we'd re-scan across all the rows that have previously deleted (until the next table VACUUM).
2023-11-16 15:19:35 +00:00
Erik Johnston
1b238e8837
Speed up persisting large number of outliers (#16649)
Recalculating the roots tuple every iteration could be very expensive, so instead let's do a topological sort.
2023-11-16 14:25:35 +00:00
Erik Johnston
fef08cbee8
Fix sending out of order POSITION over replication (#16639)
If a worker reconnects to Redis we send out the current positions of all our streams. However, if we're also trying to send out a backlog of RDATA at the same time then we can end up sending a `POSITION` with the current token *before* we've sent all the RDATA before the current token.

This doesn't cause actual bugs as the receiving servers see the POSITION, fetch the relevant rows from the DB, and then ignore the old RDATA as they come in. However, this is inefficient so it'd be better if we didn't  send out-of-order positions
2023-11-16 13:05:09 +00:00
Erik Johnston
898655fd12
More efficiently handle no-op POSITION (#16640)
We may receive `POSITION` commands where we already know that worker has
advanced past that position, so there is no point in handling it.
2023-11-16 12:32:17 +00:00
reivilibre
830988ae72
Fix test not detecting tables with missing primary keys and missing replica identities, then add more replica identities. (#16647)
* Fix the CI query that did not detect all cases of missing primary keys

* Add more missing REPLICA IDENTITY entries

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-11-16 12:26:27 +00:00
David Robertson
43d1aa75e8
Add an Admin API to temporarily grant the ability to update an existing cross-signing key without UIA (#16634) 2023-11-15 17:28:10 +00:00
Sumner Evans
999bd77d3a
Asynchronous Uploads (#15503)
Support asynchronous uploads as defined in MSC2246.
2023-11-15 09:19:24 -05:00
Patrick Cloke
80922dc46e
Add links to pre-1.0 changelog issue/PR references. (#16638) 2023-11-15 13:31:24 +00:00
Patrick Cloke
f2f2c7c1f0
Use full GitHub links instead of bare issue numbers. (#16637) 2023-11-15 08:02:11 -05:00
Will Hunt
4dd18bdc2e
Improve documentation for /_synapse/admin/v1/rooms/<room_id>/timestamp_to_event (#16631) 2023-11-14 11:43:44 -05:00
Nick Mills-Barrett
0e36a57b60
Remove whole table locks on push rule add/delete (#16051)
The statements are already executed within a transaction thus a table
level lock is unnecessary.
2023-11-13 16:57:44 +00:00
reivilibre
69afe3f7a0
Add a Postgres REPLICA IDENTITY to tables that do not have an implicit one. This should allow use of Postgres logical replication. (#16456)
* Add Postgres replica identities to tables that don't have an implicit one

Fixes #16224

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Move the delta to version 83 as we missed the boat for 82

* Add a test that all tables have a REPLICA IDENTITY

* Extend the test to include when indices are deleted

* isort

* black

* Fully qualify `oid` as it is a 'hidden attribute' in Postgres 11

* Update tests/storage/test_database.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Add missed tables

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-11-13 16:03:22 +00:00
David Robertson
fb2554b11f
Fix outbound_federation_restricted_to docs & note when added (#16628) 2023-11-13 14:26:49 +00:00
Patrick Cloke
2c6a7dfcbf
Use attempt_to_set_autocommit everywhere. (#16615)
To avoid asserting the type of the database connection.
2023-11-09 16:19:42 -05:00
reivilibre
dc7f068d9c
Fix a long-standing bug where Synapse would not unbind third-party identifiers for Application Service users when deactivated and would not emit a compliant response. (#16617)
* Don't skip unbinding 3PIDs and returning success status when deactivating AS user

Fixes #16608

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-11-09 20:18:25 +00:00
Patrick Cloke
bc4372ad81
Use dbname instead of database for Postgres config. (#16618) 2023-11-09 14:40:45 -05:00
Patrick Cloke
9f514dd0fb
Use _invalidate_cache_and_stream_bulk in more places. (#16616)
This takes advantage of the new bulk method in more places to
invalidate caches for many keys at once (and then to stream that
over replication).
2023-11-09 14:40:30 -05:00
Patrick Cloke
ab3f1b3b53
Convert simple_select_one_txn and simple_select_one to return tuples. (#16612) 2023-11-09 11:13:31 -05:00
Patrick Cloke
ff716b483b
Return attrs for more media repo APIs. (#16611) 2023-11-09 11:00:30 -05:00
David Robertson
91587d4cf9
Bulk-invalidate e2e cached queries after claiming keys (#16613)
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-11-09 15:57:09 +00:00
Patrick Cloke
455ef04187
Avoid updating the same rows multiple times with simple_update_many_txn. (#16609)
simple_update_many_txn had a bug in it which would cause each
update to be applied twice.
2023-11-07 14:02:09 -05:00
Patrick Cloke
9738b1c497
Avoid executing no-op queries. (#16583)
If simple_{insert,upsert,update}_many_txn is called without any data
to modify then return instead of executing the query.

This matches the behavior of simple_{select,delete}_many_txn.
2023-11-07 14:00:25 -05:00
Patrick Cloke
ec9ff389f4
More tests for the simple_* methods. (#16596)
Expand tests for the simple_* database methods, additionally
test against both PostgreSQL and SQLite variants.
2023-11-07 09:34:23 -05:00
Patrick Cloke
7e5d3b06fa
Collect information for PushRuleEvaluator in parallel. (#16590)
Fetch information needed for push rule evaluation in parallel.
Ideally this would use query pipelining, but this is not
available in psycopg2.

Due to the database thread pool this may result in little
to no parallelization.
2023-11-06 15:41:57 -05:00
Patrick Cloke
1dd3074629
Bump setuptools_rust to match pinned version. (#16605) 2023-11-06 09:13:53 -05:00
Patrick Cloke
cc4fe68adf
Support reactor timing metric on more reactors. (#16532)
Previously only Twisted's EPollReactor was compatible with the
reactor timing metric, notably not working when asyncio was used.

After this change, the following configurations support the reactor
timing metric:

* poll, epoll, or select reactors
* asyncio reactor with a poll, epoll, select, /dev/poll, or kqueue event loop.
2023-11-06 08:31:22 -05:00
Patrick Cloke
92828a7f95
Simplify event persistence code (#16584)
The event persistence code used to handle multiple rooms
at a time, but was simplified to only ever be called with a
single room at a time (different rooms are now handled in
parallel). The code is still generic to multiple rooms causing
a lot of work that is unnecessary (e.g. unnecessary loops, and
partitioning data by room).

This strips out the ability to handle multiple rooms at once, greatly
simplifying the code.
2023-11-03 07:30:31 -04:00
Patrick Cloke
0afbef30cf
Use simple_select_many_txn in event persistance code. (#16585)
Just to standardize on the normal helpers, it might also have
a slight perf improvement on PostgreSQL which will now use
`ANY (?)` instead of `IN (?, ?, ...)`.
2023-11-02 09:41:00 -04:00
dependabot[bot]
c812f43bd7
Bump twisted from 23.8.0 to 23.10.0 (#16588) 2023-11-01 10:23:13 +00:00
Patrick Cloke
ed1b879576
Do not call getfullargspec on every call. (#16589)
getfullargspec is relatively expensive and the results will
not change between calls, so precalculate it outside the
wrapper.
2023-10-31 20:16:17 +00:00
Patrick Cloke
cfb6d38c47
Remove remaining usage of cursor_to_dict. (#16564) 2023-10-31 13:13:28 -04:00
Erik Johnston
4bb2b4aa9a 1.96.0rc1 2023-10-31 14:09:13 +00:00
Erik Johnston
60c5e8d79b Revert "1.96.0rc1"
This reverts commit 4724a6ded1.
2023-10-31 14:08:45 +00:00
Erik Johnston
4724a6ded1 1.96.0rc1 2023-10-31 13:47:08 +00:00
David Robertson
de981ae567
Claim local one-time-keys in bulk (#16565)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-10-30 21:25:21 +00:00
Niranjan Kurhade
91aa52c911
Clients link fixed in README (#16569) 2023-10-30 16:05:34 +00:00
Erik Johnston
408c13801a
Add fast path for replication events stream fetch (#16580)
We can bail early if the from token is greater than or equal to the
current token.
2023-10-30 14:47:57 +00:00
David Robertson
fdce83ee60
Claim fallback keys in bulk (#16570) 2023-10-30 14:34:37 +00:00
dependabot[bot]
a3f6200d65
Bump setuptools-rust from 1.7.0 to 1.8.0 (#16574)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Robertson <davidr@element.io>
2023-10-30 13:40:17 +00:00
David Robertson
4e1a19d375
Run actions/setup-go after checking out complement (#16567) 2023-10-30 13:07:08 +00:00
Erik Johnston
8c63e93286
Fix HTTP repl response to use minimum token (#16578) 2023-10-30 12:27:14 +00:00
David Robertson
747416e94c
Portdb: don't copy a table that gets rebuilt (#16563) 2023-10-27 20:14:02 +01:00
kegsay
11a8ae0632
complement: enable dirty runs (#16520)
* complement: enable dirty runs

* Add changelog

* Set a low connpool limit when running in Complement

Dirty runs can cause many containers to be running concurrently,
which seems to easily exhaust resources on the host. The increased
speedup from dirty runs also seems to use more db connections on
workers, which are misconfigured currently to have
`SUM(workers * cp_max) > max_connections`, causing
```
FATAL:  sorry, too many clients already
```
which results in tests failing.

* Try p=2 concurrency to restrict slowness of servers which causes partial state join tests to flake

* Debug logging

* Only run flakey tests

* Only adjust connection pool limits in worker mode

* Move cp vars to somewhere where they get executed in CI

* Move cp values back to where they actually work

* Debug logging

* Try p=1 to see if this makes worker mode happier

* Remove debug logging
2023-10-27 18:29:20 +01:00
Patrick Cloke
2bf9341406
Ensure local invited & knocking users leave before purge. (#16559)
This is mostly useful for federated rooms where some users
would get stuck in the invite or knock state when the room
was purged from their homeserver.
2023-10-27 12:50:50 -04:00
Erik Johnston
5413cefe32
Reduce amount of caches POSITIONS we send (#16561)
Follow on from / actually correctly does #16557
2023-10-27 16:07:11 +01:00
Erik Johnston
89dbbd68e1
Reduce spurious replication catchup (#16555) 2023-10-27 13:27:20 +00:00
Erik Johnston
928e964857
Fix cross-worker ratelimiting (#16558)
c.f. #16481
2023-10-27 12:52:40 +01:00
Erik Johnston
0680d76659
Reduce replication traffic due to reflected cache stream POSITION (#16557) 2023-10-27 12:51:08 +01:00
Erik Johnston
c02406ac71
Add new module API for adding custom fields to events unsigned section (#16549) 2023-10-27 09:04:08 +00:00
Patrick Cloke
679c691f6f
Remove more usages of cursor_to_dict. (#16551)
Mostly to improve type safety.
2023-10-26 15:12:28 -04:00
Patrick Cloke
85e5f2dc25
Add a new module API to update user presence state. (#16544)
This adds a module API which allows a module to update a user's
presence state/status message. This is useful for controlling presence
from an external system.

To fully control presence from the module the presence.enabled config
parameter gains a new state of "untracked" which disables internal tracking
of presence changes via user actions, etc. Only updates from the module will
be persisted and sent down sync properly).
2023-10-26 15:11:24 -04:00
Patrick Cloke
9407d5ba78
Convert simple_select_list and simple_select_list_txn to return lists of tuples (#16505)
This should use fewer allocations and improves type hints.
2023-10-26 13:01:36 -04:00
David Robertson
c14a7de6af
Pin the recommended poetry version in contributors' guide (#16550) 2023-10-25 16:31:15 +01:00
Erik Johnston
ba47fea528
Allow multiple workers to write to receipts stream. (#16432)
Fixes #16417
2023-10-25 16:16:19 +01:00
Patrick Cloke
e182dbb5b9
Fix tests on Twisted trunk. (#16528)
Twisted trunk makes a change to the `TLSMemoryBIOFactory` where
the underlying protocol is changed from `TLSMemoryBIOProtocol` to
`BufferingTLSTransport` to improve performance of TLS code (see
https://github.com/twisted/twisted/issues/11989).

In order to properly hook this code up in tests we need to pass the test
reactor's clock into `TLSMemoryBIOFactory` to avoid the global (trial)
reactor being used by default.

Twisted does something similar internally for tests:
157cd8e659/src/twisted/web/test/test_agent.py (L871-L874)
2023-10-25 07:39:45 -04:00
Richard Brežák
95076f77c1
Fix http/s proxy authentication with long username/passwords (#16504) 2023-10-24 13:45:21 +00:00
David Robertson
6ec98810e3
Rework alias and public room list rules docs (#16541) 2023-10-24 13:26:41 +01:00
Jason Little
ffbe9b7666
Remove duplicate call to wake a remote destination when using federation sending worker (#16515) 2023-10-24 08:09:59 -04:00
Michael Sasser
3df70aa800
Replace all Prometheus datasource UIDs of the Grafana Dashboard with the variable ${DS_PROMETHEUS} and remove __inputs (#16471) 2023-10-23 19:50:50 +01:00
Patrick Cloke
3ab861ab9e
Fix type hint errors from Twisted trunk (#16526) 2023-10-23 14:28:05 -04:00
Erik Johnston
8f35f8148e
Fix bug where a new writer advances their token too quickly (#16473)
* Fix bug where a new writer advances their token too quickly

When starting a new writer (for e.g. persisting events), the
`MultiWriterIdGenerator` doesn't have a minimum token for it as there
are no rows matching that new writer in the DB.

This results in the the first stream ID it acquired being announced as
persisted *before* it actually finishes persisting, if another writer
gets and persists a subsequent stream ID. This is due to the logic of
setting the minimum persisted position to the minimum known position of
across all writers, and the new writer starts off not being considered.

* Fix sending out POSITIONs when our token advances without update

Broke in #14820

* For replication HTTP requests, only wait for minimal position
2023-10-23 16:57:30 +01:00
Erik Johnston
3bc23cc45c
Fix bug that could cause a /sync to tightloop with sqlite after restart (#16540)
This could happen if the last rows in the account data stream were inserted into `account_data`. After a restart the max account ID would be calculated without looking at the `account_data` table, and so have an old ID.
2023-10-23 13:39:25 +00:00
Marcel
3bcb6a059f
Mention how to redirect the Jaeger traces to a specific Jaeger instance (#16531) 2023-10-23 11:55:36 +00:00
Denis Kasak
3a0aa6fe76
Force TLS certificate verification in registration script. (#16530)
If using the script remotely, there's no particularly convincing reason
to disable certificate verification, as this makes the connection
interceptible.

If on the other hand, the script is used locally (the most common use
case), you can simply target the HTTP listener and avoid TLS altogether.
This is what the script already attempts to do if passed a homeserver
configuration YAML file.
2023-10-23 07:38:51 -04:00
Patrick Cloke
12ca87f5ea
Remove the last reference to event_txn_id. (#16521)
This table was no longer used, except for a background process
which purged old entries in it.
2023-10-23 07:37:45 -04:00
David Robertson
478a6c65eb
Bump matrix-synapse-ldap3 from 0.2.2 to 0.3.0 (#16539) 2023-10-23 12:28:29 +01:00
Patrick Cloke
d2eab22de7
Clarify presence router docs. (#16529) 2023-10-20 11:40:26 -04:00
Erik Johnston
e9069c9f91
Mark sync as limited if there is a gap in the timeline (#16485)
This splits thinsg into two queries, but most of the time we won't have
new event backwards extremities so this shouldn't actually add an extra
RTT for the majority of cases.

Note this removes the check for events with no prev events, but that was
part of MSC2716 work that has since been removed.
2023-10-19 15:04:18 +01:00
Patrick Cloke
49c9745b45
Avoid sending massive replication updates when purging a room. (#16510) 2023-10-18 12:26:01 -04:00
Mathieu Velten
bcff01b406
Improve performance of delete device messages query (#16492) 2023-10-18 16:42:01 +01:00
Patrick Cloke
8841db4d27
Run trial/integration tests if .ci is modified. (#16512) 2023-10-18 07:19:53 -04:00
Patrick Cloke
68d9559fef
Test against Python 3.12 release (#16511) 2023-10-17 14:41:10 -04:00
Patrick Cloke
6e6d611f55 1.95.0rc1 2023-10-17 11:54:45 -04:00
Patrick Cloke
6ad1f9eac2
Convert DeviceLastConnectionInfo to attrs. (#16507)
To improve type safety & memory usage.
2023-10-17 12:47:42 +00:00
Patrick Cloke
77dfc1f939
Fix a bug where servers could be marked as up when they were failing (#16506)
After this change a server will only be reported as back online
if they were previously having requests fail.
2023-10-17 07:32:40 -04:00
reivilibre
7291c68eea
Update the release script to remind releaser to check for special release notes. (#16461)
* Add reminder to check special release notes board in release script

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Update release.py

* Bah, black

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-10-16 15:22:50 +01:00
Patrick Cloke
e3e0ae4ab1
Convert state delta processing from a dict to attrs. (#16469)
For improved type checking & memory usage.
2023-10-16 07:35:22 -04:00
Mathieu Velten
eee6474bce
Remove useless async job to delete device messages on sync (#16491) 2023-10-16 11:06:27 +01:00
Richard van der Hoff
109882230c
Clean up logging on event persister endpoints (#16488) 2023-10-14 17:57:27 +01:00