Erik Johnston
44ac2aa3b6
SSS: Implement PREVIOUSLY room tracking ( #17535 )
...
Implement tracking of rooms that have had updates that have not been
sent down to clients.
Simplified Sliding Sync (SSS)
2024-08-08 10:44:17 +01:00
Eric Eastwood
11db575218
Sliding Sync: Use stream_ordering based timeline pagination for incremental sync ( #17510 )
...
Use `stream_ordering` based `timeline` pagination for incremental
`/sync` in Sliding Sync. Previously, we were always using a
`topological_ordering` but we should only be using that for historical
scenarios (initial `/sync`, newly joined, or haven't sent the room down
the connection before).
This is slightly different than what the [spec
suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing )
> Events are ordered in this API according to the arrival time of the
event on the homeserver. This can conflict with other APIs which order
events based on their partial ordering in the event graph. This can
result in duplicate events being received (once per distinct API
called). Clients SHOULD de-duplicate events based on the event ID when
this happens.
But we've had a [discussion below in this
PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569 )
and this matches what Sync v2 already does and seems like it makes
sense. Created a spec issue
https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this.
Related issues:
- https://github.com/matrix-org/matrix-spec/issues/1917
- https://github.com/matrix-org/matrix-spec/issues/852
- https://github.com/matrix-org/matrix-spec-proposals/pull/4033
2024-08-07 11:27:50 -05:00
dependabot[bot]
30e9f6e469
Bump bytes from 1.6.1 to 1.7.1 ( #17526 )
2024-08-07 10:37:54 +01:00
dependabot[bot]
eb62d12063
Bump regex from 1.10.5 to 1.10.6 ( #17527 )
2024-08-07 10:37:13 +01:00
Erik Johnston
ceb3686dcd
Fixup sliding sync comment ( #17531 )
...
c.f.
https://github.com/element-hq/synapse/pull/17529#discussion_r1705780925
2024-08-07 10:32:36 +01:00
Eric Eastwood
1dfa59b238
Sliding Sync: Add more tracing ( #17514 )
...
Spawning from looking at a couple traces and wanting a little more info.
Follow-up to github.com/element-hq/synapse/pull/17501
The changes in this PR allow you to find slow Sliding Sync traces ignoring the
`wait_for_events` time. In Jaeger, you can now filter for the `current_sync_for_user`
operation with `RESULT.result=true` indicating that it actually returned non-empty results.
If you want to find traces for your own user, you can use
`RESULT.result=true ARG.sync_config.user="@madlittlemods:matrix.org"`
2024-08-06 11:43:43 -05:00
Andrew Morgan
bef6568537
Merge branch 'release-v1.113' into develop
2024-08-06 14:19:12 +01:00
Andrew Morgan
244a255065
Clarify auto_accept_invites.worker_to_run_on config docs ( #17515 )
2024-08-06 13:26:51 +01:00
Andrew Morgan
932cb0a928
1.113.0rc1
2024-08-06 12:24:47 +01:00
dependabot[bot]
2dad718265
Bump phonenumbers from 8.13.39 to 8.13.42 ( #17521 )
2024-08-06 11:47:19 +01:00
dependabot[bot]
5d8446298c
Bump towncrier from 23.11.0 to 24.7.1 ( #17523 )
2024-08-06 11:47:06 +01:00
dependabot[bot]
d845e939a9
Bump black from 24.4.2 to 24.8.0 ( #17522 )
2024-08-06 11:46:48 +01:00
dependabot[bot]
23727869c7
Bump serde_json from 1.0.121 to 1.0.122 ( #17525 )
2024-08-06 11:45:44 +01:00
Erik Johnston
c270355349
SS: Reset connection if token is unrecognized ( #17529 )
...
This triggers the client to start a new sliding sync connection. If we
don't do this and the client asks for the full range of rooms, we end up
sending down all rooms and their state from scratch (which can be very
slow)
This causes things like
https://github.com/element-hq/element-x-ios/issues/3115 after we restart
the server
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-06 10:39:11 +01:00
Eric Eastwood
e3db7b2d81
Sliding Sync: Easier to understand timeline assertions in tests ( #17511 )
...
Added `_assertTimelineEqual(...)` because I got fed up trying to
understand the crazy diffs from the standard
`self.assertEqual(...)`/`self.assertListEqual(...)`
Before:
```
[FAIL]
Traceback (most recent call last):
File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 103, in test_rooms_limited_initial_sync
self.assertListEqual(
File "/usr/lib/python3.12/unittest/case.py", line 1091, in assertListEqual
self.assertSequenceEqual(list1, list2, msg, seq_type=list)
File "/usr/lib/python3.12/unittest/case.py", line 1073, in assertSequenceEqual
self.fail(msg)
twisted.trial.unittest.FailTest: Lists differ: ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0w[95 chars]isM'] != ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4[95 chars]nnU']
First differing element 0:
'$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA'
'$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E'
- ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA',
- '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E',
? ^
+ ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E',
? ^
- '$q4PRxQ_pBZkQI1keYuZPTtExQ23DqpUI3-Lxwfj_isM']
+ '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA',
+ '$j3Xj-t2F1wH9kUHsI8X5yqS7hkdSyN2owaArfvk8nnU']
```
After:
```
[FAIL]
Traceback (most recent call last):
File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 178, in test_rooms_limited_initial_sync
self._assertTimelineEqual(
File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 110, in _assertTimelineEqual
self._assertListEqual(
File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 79, in _assertListEqual
self.fail(f"{diff_message}\n{message}")
twisted.trial.unittest.FailTest: Items must
Expected items to be in actual ('?' = missing expected items):
[
(10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4
(11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5
? (12, master) $bgOcc3D-2QSkbk4aBxKVyOOQJGs7ZuncRJwG3cEANZg (m.room.member, @user1:test) join
]
Actual ('+' = found expected items):
[
+ (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5
+ (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4
(9, master) $FmCNyc11YeFwiJ4an7_q6H0LCCjQOKd6UCr5VKeXXUw (m.room.message, None) activity3
]
```
2024-08-05 13:20:15 -05:00
Eric Eastwood
2b620e0a15
Sliding Sync: Add typing notification extension (MSC3961) ( #17505 )
...
[MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961 ): Sliding Sync Extension: Typing Notifications
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575 ):
Sliding Sync
2024-07-31 13:20:23 -05:00
Eric Eastwood
39731bb205
Sliding Sync: Split and move tests ( #17504 )
...
Split and move Sliding Sync tests so we have some more sane test file
sizes
2024-07-31 12:20:46 -05:00
Eric Eastwood
1d6186265a
Sliding Sync: Fix limited response description (make accurate) ( #17507 )
2024-07-31 11:47:26 -05:00
Eric Eastwood
46de0ee16b
Sliding Sync: Update filters to be robust against remote invite rooms ( #17450 )
...
Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to
be robust when dealing with remote invite rooms in Sliding Sync.
Part of
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575 ):
Sliding Sync
Follow-up to https://github.com/element-hq/synapse/pull/17434
We now take into account current state, fallback to stripped state
for invite/knock rooms, then historical state. If we can't determine
the info needed to filter a room (either from state or stripped state),
it is filtered out.
2024-07-30 13:20:29 -05:00
Eric Eastwood
b221f0b84b
Sliding Sync: Add receipts extension (MSC3960) ( #17489 )
...
[MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960 ): Receipts extension
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575 ):
Sliding Sync
2024-07-30 12:49:55 -05:00
Olivier 'reivilibre
b2c55bd049
Merge branch 'master' into develop
2024-07-30 18:09:05 +01:00
Olivier 'reivilibre
ed583d9c81
Merge branch 'release-v1.112'
2024-07-30 18:07:35 +01:00
dependabot[bot]
f76dc9923c
Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726 ( #17497 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-30 17:33:43 +01:00
dependabot[bot]
7e997fb8b1
Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722 ( #17496 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-30 17:33:18 +01:00
dependabot[bot]
dbc2290cbe
Bump bcrypt from 4.1.3 to 4.2.0 ( #17495 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-30 17:32:49 +01:00
dependabot[bot]
2f6b86e79a
Bump serde_json from 1.0.120 to 1.0.121 ( #17493 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-30 17:32:16 +01:00
Olivier 'reivilibre
37f9876ccf
1.112.0
2024-07-30 17:24:09 +01:00
reivilibre
8b449a8ce6
Upgrade locked dependency on Twisted to 24.7.0rc1. ( #17502 )
...
I also update the tests and HTTP Proxy code to fix it for this new
Twisted release.
Pulls in fix for
https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
2024-07-30 17:14:14 +01:00
Olivier 'reivilibre
53db8a914e
Merge branch 'master' into develop
2024-07-30 17:10:46 +01:00
Olivier 'reivilibre
e4868f8a1e
Add bold emphasis to some parts of the changelog
2024-07-30 16:23:58 +01:00
Olivier 'reivilibre
dcad81082c
1.111.1
2024-07-30 16:16:35 +01:00
reivilibre
c56b070e6f
Upgrade locked dependency on Twisted to 24.7.0rc1. ( #17502 )
...
I also update the tests and HTTP Proxy code to fix it for this new
Twisted release.
Pulls in fix for
https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
2024-07-30 15:23:23 +01:00
dependabot[bot]
be726724a8
Bump ruff from 0.5.4 to 0.5.5 ( #17494 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-30 11:44:54 +01:00
Erik Johnston
62ae56a4ac
Add some more opentracing to sliding sync ( #17501 )
...
This will make it easier to see what it is doing in jaeger.
2024-07-30 10:54:11 +01:00
Richard van der Hoff
808dab0699
Fix failures property in /keys/query ( #17499 )
...
Fixes: https://github.com/element-hq/synapse/issues/17498
Fixes: https://github.com/element-hq/element-web/issues/27867
2024-07-30 09:51:24 +01:00
Erik Johnston
34306be5aa
Only send rooms with updates down sliding sync ( #17479 )
...
Rather than always including all rooms in range.
Also adds a pre-filter to rooms that checks the stream change cache to
see if anything might have happened.
Based on #17447
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-07-30 09:30:44 +01:00
Erik Johnston
be4a16ff44
Sliding Sync: Track whether we have sent rooms down to clients ( #17447 )
...
The basic idea is that we introduce a new token for a sliding sync
connection, which stores the mapping of room to room "status" (i.e. have
we sent the room down?). This token allows us to handle duplicate
requests properly. In future it can be used to store more
"per-connection" information safely.
In future this should be migrated into the DB, so its important that we
try to reduce the number of syncs where we need to update the
per-connection information. In this PoC this only happens when we: a)
send down a set of room for the first time, or b) we have previously
sent down a room and there are updates but we are not sending the room
down the sync (due to not falling in a list range)
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-07-29 22:45:48 +01:00
Eric Eastwood
568051c0f0
Refactor Sliding Sync tests to better utilize the SlidingSyncBase.do_sync(...) (pt. 2) ( #17482 )
...
`SlidingSyncBase.do_sync()` for tests was first introduced in
https://github.com/element-hq/synapse/pull/17452
Part 1: https://github.com/element-hq/synapse/pull/17481
2024-07-25 11:01:47 -05:00
Eric Eastwood
ebbabfe782
Refactor Sliding Sync tests to better utilize the SlidingSyncBase (pt. 1) ( #17481 )
...
`SlidingSyncBase` for tests was first introduced in
https://github.com/element-hq/synapse/pull/17452
Part 2: https://github.com/element-hq/synapse/pull/17482
2024-07-25 10:43:35 -05:00
YLong Shi
69ac4b6a6e
Update config_documentation - Change example of msisdn in allowed_local_3pids ( #17476 )
...
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-07-25 11:07:44 +00:00
Eric Eastwood
729026e604
Sliding Sync: Add Account Data extension (MSC3959) ( #17477 )
...
Extensions based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575 ):
Sliding Sync
2024-07-24 17:10:38 -05:00
Erik Johnston
bdf37ad4c4
Sliding Sync: ensure bump stamp ignores backfilled events ( #17478 )
...
Backfill events have a negative stream ordering, and so its not useful
to use to compare with other (positive) stream orderings.
Plus, the Rust SDK currently assumes `bump_stamp` is positive.
2024-07-24 15:21:56 +01:00
Erik Johnston
8bbc98e66d
Use a new token format for sliding sync ( #17452 )
...
This is in preparation for adding per-connection state.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-07-24 11:47:25 +01:00
Maciej Laskowski
4b9f4c2abf
Update debian template - new link to the delegation docs ( #17475 )
...
Update debian template - new link to the delegation docs
2024-07-24 10:32:56 +01:00
Devon Hudson
e8ee784c75
Address changelog review comments
2024-07-23 09:14:45 -06:00
Devon Hudson
48c1307911
1.112.0rc1
2024-07-23 09:01:43 -06:00
Erik Johnston
d225b6b3eb
Speed up SS room sorting ( #17468 )
...
We do this by bulk fetching the latest stream ordering.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-07-23 14:03:14 +01:00
reivilibre
1daae43f3a
Reduce volume of 'Waiting for current token' logs, which were introduced in v1.109.0. ( #17428 )
...
Introduced in: #17215
This caused us a minor bit of grief as the volume of logs produced was
much higher than normal
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
2024-07-23 11:51:34 +01:00
Michael Hollister
a9ee832e48
Fixed presence results not returning offline users on initial sync ( #17231 )
...
This is to address an issue in which `m.presence` results on initial
sync are not returning entries of users who are currently offline.
The original behaviour was from
https://github.com/element-hq/synapse/issues/1535
This change is useful for applications that use the
presence system for tracking user profile information/updates (e.g.
https://github.com/element-hq/synapse/pull/16992 or for profile status
messages).
This is gated behind a new configuration option to avoid performance
impact for applications that don't need this, as a pragmatic solution
for now.
2024-07-23 09:59:24 +00:00
dependabot[bot]
13a99fba1b
Bump hiredis from 2.3.2 to 3.0.0 ( #17464 )
...
Bumps [hiredis](https://github.com/redis/hiredis-py ) from 2.3.2 to
3.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/redis/hiredis-py/releases ">hiredis's
releases</a>.</em></p>
<blockquote>
<h2>3.0.0</h2>
<h1>Changes</h1>
<h2>Breaking Changes</h2>
<ul>
<li>Return Redis sets as Python lists (<a
href="https://redirect.github.com/redis/hiredis-py/issues/189 ">#189</a>)</li>
</ul>
<h2>🐛 Bug Fixes</h2>
<ul>
<li>Return Redis sets as Python lists (<a
href="https://redirect.github.com/redis/hiredis-py/issues/189 ">#189</a>)</li>
</ul>
<h2>Contributors</h2>
<p>We'd like to thank all the contributors who worked on this
release!</p>
<p><a href="https://github.com/gerzse "><code>@gerzse</code></a></p>
<h2>2.4.0</h2>
<h1>Changes</h1>
<h2>🧰 Maintenance</h2>
<ul>
<li>Fix small typo (<a
href="https://redirect.github.com/redis/hiredis-py/issues/192 ">#192</a>)</li>
<li>Quote version for Python setup action in CI (<a
href="https://redirect.github.com/redis/hiredis-py/issues/191 ">#191</a>)</li>
<li>Fix building the wheel for windows (<a
href="https://redirect.github.com/redis/hiredis-py/issues/190 ">#190</a>)</li>
<li>pack: Replace sdsalloc.h with alloc.h (<a
href="https://redirect.github.com/redis/hiredis-py/issues/159 ">#159</a>)</li>
<li>Bump black from 22.3.0 to 24.3.0 (<a
href="https://redirect.github.com/redis/hiredis-py/issues/185 ">#185</a>)</li>
<li>Removing python 3.7 trove (<a
href="https://redirect.github.com/redis/hiredis-py/issues/181 ">#181</a>)</li>
<li>Badge for latest released on Pypi (<a
href="https://redirect.github.com/redis/hiredis-py/issues/182 ">#182</a>)</li>
<li>Sync license in metadata with LICENSE file (<a
href="https://redirect.github.com/redis/hiredis-py/issues/183 ">#183</a>)</li>
</ul>
<h2>Contributors</h2>
<p>We'd like to thank all the contributors who worked on this
release!</p>
<p><a href="https://github.com/Apteryks "><code>@Apteryks</code></a>, <a
href="https://github.com/ArtemIsmagilov "><code>@ArtemIsmagilov</code></a>,
<a href="https://github.com/chayim "><code>@chayim</code></a>, <a
href="https://github.com/dependabot "><code>@dependabot</code></a>, <a
href="https://github.com/dependabot "><code>@dependabot</code></a>[bot],
<a href="https://github.com/gerzse "><code>@gerzse</code></a> and <a
href="https://github.com/shadchin "><code>@shadchin</code></a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="c1eefbdb76https://redirect.github.com/redis/hiredis-py/issues/189 ">#189</a>)</li>
<li><a
href="a94bb44717https://redirect.github.com/redis/hiredis-py/issues/193 ">#193</a>)</li>
<li><a
href="7792dd2338https://redirect.github.com/redis/hiredis-py/issues/192 ">#192</a>)</li>
<li><a
href="01fa2fd6f1https://redirect.github.com/redis/hiredis-py/issues/191 ">#191</a>)</li>
<li><a
href="4c970a3365https://redirect.github.com/redis/hiredis-py/issues/190 ">#190</a>)</li>
<li><a
href="f4dd0814c1https://redirect.github.com/redis/hiredis-py/issues/159 ">#159</a>)</li>
<li><a
href="e70af5b94fhttps://redirect.github.com/redis/hiredis-py/issues/185 ">#185</a>)</li>
<li><a
href="cc239705fbhttps://redirect.github.com/redis/hiredis-py/issues/181 ">#181</a>)</li>
<li><a
href="64e3394548https://redirect.github.com/redis/hiredis-py/issues/182 ">#182</a>)</li>
<li><a
href="ba18089d74https://redirect.github.com/redis/hiredis-py/issues/183 ">#183</a>)</li>
<li>See full diff in <a
href="https://github.com/redis/hiredis-py/compare/v2.3.2...v3.0.0 ">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-22 23:58:19 +00:00
