0
0
Fork 1
mirror of https://mau.dev/maunium/synapse.git synced 2024-12-15 02:33:51 +01:00
Commit graph

3668 commits

Author SHA1 Message Date
Eric Eastwood
3fee32ed6b
Order heroes by stream_ordering (as spec'ed) (#17435)
The spec specifically mentions `stream_ordering` but that's a Synapse specific concept. In any case, the essence of the spec is basically the first 5 members of the room which `stream_ordering` accomplishes.

Split off from https://github.com/element-hq/synapse/pull/17419#discussion_r1671342794

## Spec compliance

> This should be the first 5 members of the room, **ordered by stream ordering**, which are joined or invited. The list must never include the client’s own user ID. When no joined or invited members are available, this should consist of the banned and left users.
>
> *-- https://spec.matrix.org/v1.10/client-server-api/#_matrixclientv3sync_roomsummary*

Related to https://github.com/matrix-org/matrix-spec/issues/1334
2024-07-17 13:10:15 -05:00
dependabot[bot]
79924aebef
Bump mypy from 1.9.0 to 1.10.1 (#17445) 2024-07-16 12:08:06 +01:00
Shay
429ecb7564
Handle remote download responses with UNKNOWN_LENGTH more gracefully (#17439)
Prior to this PR, remote downloads which did not provide a
`content-length` were decremented from the remote download ratelimiter
at the max allowable size, leading to excessive ratelimiting - see
https://github.com/element-hq/synapse/issues/17394.

This PR adds a linearizer to limit concurrent remote downloads to 6 per
IP address, and decrements remote downloads without a `content-length`
from the ratelimiter *after* the download is complete and the response
length is known.

Also adds logic to ensure that responses with a known length respect the
`max_download_size`.
2024-07-16 11:13:55 +01:00
Shay
899d33f2ba
Remove unnecessary call to resume producing in fake channel (#17449)
This fell out of the authenticated media work - this bit of code masked
a bug but does not break anything when removed, so probably should be
removed.
2024-07-16 09:52:39 +01:00
Erik Johnston
df11af14db
Fix bug where sync could get stuck when using workers (#17438)
This is because we serialized the token wrong if the instance map
contained entries from before the minimum token.
2024-07-15 16:13:04 +01:00
Eric Eastwood
ab62aa09da
Add room subscriptions to Sliding Sync /sync (#17432)
Add room subscriptions to Sliding Sync `/sync`

Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync

Currently, you can only subscribe to rooms you have had *any* membership
in before.

In the future, we will allow `world_readable` rooms to be subscribed to
without joining.
2024-07-15 10:37:10 +01:00
Eric Eastwood
fb66e938b2
Add is_dm room field to Sliding Sync /sync (#17429)
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
2024-07-11 18:19:26 -05:00
Eric Eastwood
5a97bbd895
Add heroes and room summary fields to Sliding Sync /sync (#17419)
Additional room summary fields: `joined_count`, `invited_count`

Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
2024-07-11 14:05:38 -05:00
Erik Johnston
606da398fc
Fix filtering room types on remote rooms (#17434)
We can only fetch room types for rooms the server is in, so we need to
only filter rooms that we're joined to.

Also includes a perf fix to bulk fetch room types.
2024-07-11 16:00:44 +01:00
Erik Johnston
4ca13ce0dd
Handle to-device extensions to Sliding Sync (#17416)
Implements MSC3885

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-07-10 11:58:42 +01:00
Eric Eastwood
1cf3ff6b40
Add rooms name and avatar to Sliding Sync /sync (#17418)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-07-09 12:26:45 -05:00
Erik Johnston
8cdd2d214e
Fix bug in sliding sync when using old DB. (#17398)
We don't necessarily have `instance_name` for old events (before we
support multiple event persisters). We treat those as if the
`instance_name` was "master".

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-07-08 20:30:23 +01:00
Eric Eastwood
3fef535ff2
Add rooms.bump_stamp to Sliding Sync /sync for easier client-side sorting (#17395)
`bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves.

Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies).

In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types.

---

In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for.

The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622
2024-07-08 13:17:08 -05:00
Shay
cf69f8d59b
Support MSC3916 by adding a federation /thumbnail endpoint and authenticated _matrix/client/v1/media/thumbnail endpoint (#17388)
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)
added the endpoints `_matrix/federation/v1/media/thumbnail` and the
authenticated `_matrix/client/v1/media/thumbnail`.

This PR implements those endpoints, along with stabilizing
`_matrix/client/v1/media/config` and
`_matrix/client/v1/media/preview_url`.

Complement tests are at
https://github.com/matrix-org/complement/pull/728
2024-07-08 10:11:20 +01:00
dependabot[bot]
20de685a4b
Bump ruff from 0.3.7 to 0.5.0 (#17381) 2024-07-05 12:35:57 +00:00
Erik Johnston
57538eb4d9
Finish up work to allow per-user feature flags (#17392)
Follows on from @H-Shay's great work at
https://github.com/matrix-org/synapse/pull/15344 and MSC4026.

Also enables its use for MSC3881, mainly as an easy but concrete example
of how to use it.
2024-07-05 13:02:35 +01:00
Travis Ralston
2ec257d608
Upon deactivation, forget all of the user's rooms (#17400)
This can help ensure that the rooms are eventually purged if the other
local users also forget them. Synapse already clears some of the room
information as part of the `_background_remove_left_rooms` background
task, but this doesn't catch `events`, `event_json`, etc.
2024-07-05 11:56:52 +01:00
Eric Eastwood
22aeb78b77
Add rooms.required_state to Sliding Sync /sync (#17342)
Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.
2024-07-04 12:25:36 -05:00
Eric Eastwood
7be03d854b
Add room_types/not_room_types filtering to Sliding Sync /sync (#17337)
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
2024-07-02 12:46:27 -05:00
Eric Eastwood
fa91655805
Return some room data in Sliding Sync /sync (#17320)
- Timeline events
 - Stripped `invite_state`

Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-07-02 11:07:05 -05:00
Erik Johnston
99c107920d Merge remote-tracking branch 'origin/release-v1.110' into develop 2024-07-02 15:12:36 +01:00
Erik Johnston
1609855ff8 Limit size of presence EDUs (#17371)
Otherwise they are unbounded.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-07-02 14:07:11 +01:00
Shay
8f890447b0
Support MSC3916 by adding _matrix/client/v1/media/download endpoint (#17365) 2024-07-02 14:07:04 +01:00
Erik Johnston
b905ae27ca
Fix regression when bounding future tokens (#17391)
Fix bug added in #17386, where we accidentally used `room_key` for the
receipts stream. See first commit.

Reviewable commit-by-commit
2024-07-02 14:06:36 +01:00
Erik Johnston
1ce59d7ba0 Fix sync waiting for an invalid token from the "future" (#17386)
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.

Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).

We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
2024-07-02 12:40:03 +01:00
Erik Johnston
b3b793786c
Fix sync waiting for an invalid token from the "future" (#17386)
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.

Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).

We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
2024-07-02 12:39:49 +01:00
Erik Johnston
2f5a77efae
Limit size of presence EDUs (#17371)
Otherwise they are unbounded.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-07-01 12:48:36 +01:00
Shay
a023538822
Re-introduce federation /download endpoint (#17350) 2024-06-25 14:35:37 +00:00
Erik Johnston
554a92601a
Reintroduce "Reduce device lists replication traffic."" (#17361)
Reintroduces https://github.com/element-hq/synapse/pull/17333


Turns out the reason for revert was down two master instances running
2024-06-25 10:34:34 +01:00
Erik Johnston
a98cb87bee
Revert "Reduce device lists replication traffic." (#17360)
Reverts element-hq/synapse#17333

It looks like master was still sending out replication RDATA with the
old format... somehow
2024-06-25 09:57:34 +01:00
Eric Eastwood
6e8af83193
Add is_invite filtering to Sliding Sync /sync (#17335)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-24 19:07:56 -05:00
Erik Johnston
930a64b6c1
Reintroduce #17291. (#17338)
This is #17291 (which got reverted), with some added fixups, and change
so that tests actually pick up the error.

The problem was that we were not calculating any new chain IDs due to a
missing `not` in a condition.
2024-06-24 14:40:28 +00:00
Shay
7a11c0ac4f
Add support for MSC3823 - Account Suspension Part 2 (#17255) 2024-06-24 14:21:22 +01:00
Erik Johnston
cf711ac03c
Reduce device lists replication traffic. (#17333)
Reduce the replication traffic of device lists, by not sending every
destination that needs to be sent the device list update over
replication. Instead a "hosts to send to have been calculated"
notification over replication, and then federation senders read the
destinations from the DB.

For non federation senders this should heavily reduce the impact of a
user in many large rooms changing a device.
2024-06-24 14:15:13 +01:00
Erik Johnston
4243c1f074
Revert "Handle large chain calc better (#17291)" (#17334)
This reverts commit bdf82efea5  (#17291)

This seems to have stopped persisting auth chains for new events, and so
is causing state res to fall back to the slow methods
2024-06-19 17:39:33 +01:00
Erik Johnston
3239b7459c
Register sliding sync under a different path (#17331)
As the API is slightly incompatible.
2024-06-19 16:18:45 +00:00
Alexander Fechler
9104a9f0d0
Filter added to Admin-API GET /rooms (#17276) 2024-06-19 11:45:48 +01:00
Erik Johnston
bdf82efea5
Handle large chain calc better (#17291)
We calculate the auth chain links outside of the main persist event
transaction to ensure that we do not block other event sending during
the calculation.
2024-06-19 10:33:53 +01:00
Andrew Morgan
afaf2d9388
Require the 'from' parameter for /notifications be an integer (#17283)
Co-authored-by: Erik Johnston <erikj@element.io>
2024-06-19 10:05:39 +01:00
Andrew Morgan
199223062a
Revert "Support MSC3916 by adding a federation /download endpoint" (#17325) 2024-06-18 16:54:19 +01:00
Eric Eastwood
a5485437cf
Add is_encrypted filtering to Sliding Sync /sync (#17281)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-17 12:06:18 -05:00
Eric Eastwood
e5b8a3e37f
Add stream_ordering sort to Sliding Sync /sync (#17293)
Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
2024-06-17 11:27:14 -05:00
Quentin Gliech
e88332b5f4
Merge branch 'release-v1.109' into develop 2024-06-17 15:51:16 +02:00
Quentin Gliech
f983a77ab0
Set our own stream position from the current sequence value on startup (#17309) 2024-06-17 11:50:00 +00:00
Erik Johnston
a3cb244755
Automatically apply SQL for inconsistent sequence (#17305)
Rather than forcing the server operator to apply the SQL manually.

This should be safe, as there should be only one writer for these
sequences.
2024-06-14 16:40:29 +01:00
Richard van der Hoff
3aae60f17b
Enable cross-signing key upload without UIA (#17284)
Per MSC3967, which is now stable, we should not require UIA when
uploading cross-signing keys for the first time.

Fixes: #17227
2024-06-14 11:14:56 +01:00
Richard van der Hoff
2c36a679ae
Include user membership on events (#17282)
MSC4115 has now completed FCP, so we can enable it by default and switch
to the stable identifier.
2024-06-13 21:45:54 +00:00
Eric Eastwood
c12ee0d5ba
Add is_dm filtering to Sliding Sync /sync (#17277)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-13 13:56:58 -05:00
Eric Eastwood
8aaff851b1
Fix newly_left rooms not appearing if we returned early (Sliding Sync) (#17301)
Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`. 

Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync)

The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`.

Found while working on https://github.com/element-hq/synapse/pull/17293
2024-06-13 11:36:57 -05:00
Eric Eastwood
8c58eb7f17
Add event.internal_metadata.instance_name (#17300)
Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`.

`instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`.

This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293

Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see 09609cb0db (diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410)
2024-06-13 11:32:50 -05:00