Commit graph

21360 commits

Author SHA1 Message Date
Eric Eastwood
957e3d74fc
Keep track when we try and fail to process a pulled event (#13589)
We can follow-up this PR with:

 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622
 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623
 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700

Fix https://github.com/matrix-org/synapse/issues/13621

Part of https://github.com/matrix-org/synapse/issues/13356

Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)
2022-09-14 13:57:50 -05:00
Patrick Cloke
666ae87729
Update event push action and receipt tables to support threads. (#13753)
Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`,
and `event_push_summary` tables. This will notifications to be segmented by the thread
in a future pull request. The `thread_id` column stores the root event ID or the special
value `"main"`.

The `thread_id` column for `event_push_actions` and `event_push_summary` is
backfilled with `"main"` for all existing rows. New entries into `event_push_actions`
and `event_push_actions_staging` will get the proper thread ID.

`receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar,
except `NULL` is a special value meaning the receipt is "unthreaded".

See MSC3771 and MSC3773 for where this data will be useful.
2022-09-14 17:11:16 +00:00
Patrick Cloke
f2d12ccabe
Use partial indices on SQLIte. (#13802)
Partial indices have been supported since SQLite 3.8, but Synapse
now requires >= 3.27, so we can enable support for them.

This requires rebuilding previous indices which were partial on
PostgreSQL, but not on SQLite.
2022-09-14 12:01:42 -04:00
reivilibre
6302753012
Deduplicate is_server_notices_room. (#13780) 2022-09-14 15:53:18 +00:00
reivilibre
cf65433de2
Fix a memory leak when running the unit tests. (#13798) 2022-09-14 15:29:05 +00:00
Quentin Gliech
eaed4e6113
Remove unused method in synapse.api.auth.Auth. (#13795)
Clean-up from b19060a29b (#13094)
and 73af10f419 (#13093) which removed
all callers.
2022-09-14 10:33:54 -04:00
David Robertson
51a77e990b
Remove incorrect migration file from state logical DB (#13788)
* Remove incorrect migration file from `state` logical DB

The table `ex_outlier_stream` is part of the `main` logical DB; it
should not have been created in the `state` logical DB. We remove this
migration now as a tidy-up.

Note: we cannot `DROP TABLE IF EXISTS ex_outlier_stream` in a new
migration, because some (most) instances of Synapse host both of these
logical DBs on the same DB cluster.

* Changelog
2022-09-14 14:16:12 +01:00
Sean Quah
c73774467e
Fix bug in device list caching when remote users leave rooms (#13749)
When a remote user leaves the last room shared with the homeserver, we
have to mark their device list as unsubscribed, otherwise we would hold
on to a stale device list in our cache. Crucially, the device list would
remain cached even after the remote user rejoined the room, which could
lead to E2EE failures until the next change to the remote user's device
list.

Fixes #13651.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-14 10:42:57 +01:00
reivilibre
21687ec189
Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of /get_missing_events federation requests. (#13789)
* Don't accept a trailing slash on the end of /get_missing_events

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-09-14 09:28:12 +01:00
Mathieu Velten
12dacecabd
Make sequence cache_invalidation_stream_seq begin at 2 (#13766)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-09-13 16:14:28 +02:00
Erik Johnston
9772e362aa Merge branch 'master' into develop 2022-09-13 12:11:53 +01:00
David Robertson
b60d47ab2c
Updates to the schema dump script (#13770) 2022-09-13 10:53:11 +01:00
David Robertson
540afb0bfc
Simplify CI tests DAG (#13784)
* Simplify CI tests DAG

* Changelog
2022-09-13 10:17:23 +01:00
Richard van der Hoff
41df25bbbd
installation.md: require libpq on M1 macs (#13480) 2022-09-13 09:01:21 +00:00
Erik Johnston
80bb098d87 Fixup changelog 2022-09-13 09:55:10 +01:00
Erik Johnston
4b678b20a2 1.67.0 2022-09-13 09:20:28 +01:00
Nick Mills-Barrett
cdbb641232
Add receipts event stream ordering (#13703) 2022-09-13 08:16:37 +01:00
Mathieu Velten
fa2f3d8d0c
Fix GHA skippable syntax (#13778)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-09-12 17:31:23 +00:00
Brendan Abolivier
7571337445
Fix typo in ratelimiting documentation (#13727) 2022-09-12 14:11:18 +01:00
Erik Johnston
dd7484b562
Fix CI on non-PR builds (#13769)
Mark cargo-test as skippable since it only runs on Rust code change.
2022-09-12 13:26:33 +01:00
Nick Mills-Barrett
da41a7cd61
Remove check current state membership up to date (#13745)
* Remove checks for membership column in current_state_events
* Add schema script to force through the
  `current_state_events_membership` background job

Contributed by Nick @ Beeper (@fizzadar).
2022-09-12 12:58:33 +01:00
Erik Johnston
ebfeac7c5d
Check if Rust lib needs rebuilding. (#13759)
This protects against the common mistake of failing to remember to rebuild Rust code after making changes.
2022-09-12 10:03:42 +00:00
Nick Mills-Barrett
4c4889cac0
Concurrently collect room unread counts for push badges (#13765)
Most of the time this function is heavily cached, but when that isn't
the case fetching the counts room by room slows down push delivery on
users with many (thousands) of rooms.

Signed off by Nick @ Beeper.
2022-09-09 19:00:21 +01:00
Eric Eastwood
a911ffb42c
Tag trace with instance name (#13761)
We tag the Synapse instance name so that it's an easy jumping off point into the logs. Can also be used to filter for an instance that is under load.

As suggested by @clokep and @reivilibre in,

 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964719258
 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964733578
2022-09-09 11:31:37 -05:00
Eric Eastwood
f694bb71b7
Strip number suffix from instance name to consolidate services that traces are spread over (#13729)
The problem with many services is that it makes it hard to find which service has the trace you want, see https://github.com/jaegertracing/jaeger-ui/issues/985

Previously, we split traces out into services based on their instance name like `matrix.org client_reader-1`, etc but there are many worker instances of the same `client_reader` so there is a lot to click through.

With this PR, all of the traces are just collected under the worker type like `client_reader`, `event_persister` 😇

Note: A Synapse worker instance name is an opaque string with the number convention only being our own thing for the `matrix.org` deployment. But seems pretty sensible to group things this way.
2022-09-09 11:30:06 -05:00
Patrick Cloke
3d9f82efcb
Use an upsert for receipts_graph. (#13752)
Instead of a delete, then insert.

This was previously done for `receipts_linearized` in
2dc430d36e (#7607).
2022-09-09 07:08:41 -04:00
Erik Johnston
c85c5ace52
Add rust to CI (#13763) 2022-09-09 11:29:04 +01:00
David Robertson
f2d2481e56
Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
Sean Quah
69fa29700e
Re-type hint some collections in /sync code as read-only (#13754)
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08 20:13:39 +01:00
reivilibre
5261d2e2e8
Remove unused Prometheus recording rules from synapse-v2.rules and add comments describing where the rest are used. (#13756) 2022-09-08 17:50:15 +00:00
Dirk Klimpel
f799eac7ea
Add timestamp to user's consent (#13741)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-09-08 15:41:48 +00:00
Sean Quah
906cead9ca
Update docstrings to explain the impact of partial state (#13750)
Update the docstrings for `get_users_in_room` and
`get_current_hosts_in_room` to explain the impact of partial state.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08 15:55:29 +01:00
Sean Quah
89e8b98b65
Avoid raising errors due to malformed IDs in get_current_hosts_in_room (#13748)
Handle malformed user IDs with no colons in `get_current_hosts_in_room`.
It's not currently possible for a malformed user ID to join a room, so
this error would never be hit.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08 15:55:03 +01:00
Sean Quah
8ef0c8ff14
Fix error in is_mine_id when encountering a malformed ID (#13746)
Previously, `is_mine_id` would raise an exception when passed an ID with
no colons. Return `False` instead.

Fixes #13040.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08 15:54:36 +01:00
reivilibre
cf11919ddd
Fix cache metrics not being updated when not using the legacy exposition module. (#13717) 2022-09-08 15:30:48 +01:00
reivilibre
526f84bc2e
Fix Prometheus recording rules to not use legacy metric names. (#13718) 2022-09-08 15:01:42 +01:00
Erik Johnston
1cc729c177
Fix latest deps (#13743) 2022-09-08 13:58:31 +01:00
David Robertson
9d11842562
Also cite upgrade notes 2022-09-08 12:04:29 +01:00
David Robertson
a7c71686ca
Add minimum version bump for sqlite to the release notes (#13742)
* Notify that SQLite min version will be bumped
* Mention in upgrade notes

Co-authored-by: reivilibre <oliverw@matrix.org>
2022-09-08 12:00:03 +01:00
reivilibre
b7e4bfd005
Fix a bug where Synapse fails to start if a signing key file contains an empty line. (#13738) 2022-09-08 11:18:03 +01:00
Eric Eastwood
d4d3249ded
Instrument get_metadata_for_events for tracing (#13730)
When backfilling, `_get_state_ids_after_missing_prev_event` calls [`get_metadata_for_events`](26bc26586b/synapse/handlers/federation_event.py (L1133)). For `#matrix:matrix.org`, it's called with 77k `state_events` which means 77 calls to the database and takes 28 seconds.
2022-09-07 11:41:52 -05:00
Erik Johnston
8d7fcf9b76
Fix latest deps CI (#13734) 2022-09-07 14:07:06 +00:00
Erik Johnston
dc0e896b68
Add some rust caching to CI (#13735) 2022-09-07 13:56:59 +00:00
David Robertson
c46fecd1f2
Correct out-of-date doc for event_cache_size (#13726) 2022-09-07 14:46:11 +01:00
David Robertson
77f3986451
Define SQLite compat policy (#13728) 2022-09-07 12:07:42 +00:00
David Robertson
b58386e37e
A second batch of Pydantic models for rest/client/account.py (#13687) 2022-09-07 12:16:10 +01:00
reivilibre
d3d9ca156e
Cancel the processing of key query requests when they time out. (#13680) 2022-09-07 12:03:32 +01:00
reivilibre
c2fe48a6ff
Rename the EventFormatVersions enum values so that they line up with room version numbers. (#13706) 2022-09-07 11:08:20 +01:00
Connor Davis
bb5b47b62a
Add Admin API to Fetch Messages Within a Particular Window (#13672)
This adds two new admin APIs that allow us to fetch messages from a room within a particular time.
2022-09-07 10:54:44 +01:00
reivilibre
26bc26586b
Remove the unspecced room_id field in the /hierarchy response. (#13506)
This is a re-do of 57d334a13d (#13365),
which was backed out in 12abd72497 (#13501).

The `room_id` field represented the parent space for each room
and was made redundant by changes in the API shape where the
`children_state` is now nested underneath each `room`.

The room ID of each child is in the `state_key` field and is still
available.
2022-09-06 15:28:44 -04:00