0
0
Fork 1
mirror of https://mau.dev/maunium/synapse.git synced 2024-12-16 01:43:51 +01:00
Commit graph

5632 commits

Author SHA1 Message Date
Olivier Wilkinson (reivilibre)
6a04767439 Merge branch 'release-v1.50' into develop 2022-01-07 14:22:13 +00:00
Patrick Cloke
6bf81a7a61
Bundle aggregations outside of the serialization method. (#11612)
This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.

Overall this should cause no change in behavior, but is prep work
for other improvements.
2022-01-07 09:10:46 -05:00
Patrick Cloke
6c68e874b1
Remove the /send_relation endpoint. (#11682)
This was removed from MSC2674 before that was approved
and is not used by any known clients.
2022-01-06 19:00:34 +00:00
Patrick Cloke
201c48c8de
Remove a Python 2-ism and improve type hints. (#11699)
On Python 2, indexing a byte-string gives back a byte-string,
while on Python 3 it gives back the ASCII equivalent as an int.
2022-01-06 18:08:48 +00:00
Olivier Wilkinson (reivilibre)
e87540abb1 Re-run Towncrier to add in the changelog entry for the Mjolnir workaround 2022-01-06 17:54:21 +00:00
Shay
70ce9aea71
Strip unauthorized fields from unsigned object in events received over federation (#11530)
* add some tests to verify we are stripping unauthorized fields out of unsigned

* add function to strip unauthorized fields from the unsigned object of event

* newsfragment

* update newsfragment number

* add check to on_send_membership_event

* refactor tests

* fix lint error

* slightly refactor tests and add some comments

* slight refactor

* refactor tests

* fix import error

* slight refactor

* remove unsigned filtration code from synapse/handlers/federation_event.py

* lint

* move unsigned filtering code to event base

* refactor tests

* update newsfragment

* requested changes

* remove unused retun values
2022-01-06 09:09:30 -08:00
reivilibre
eec34b1f2a
Work around Mjolnir compatibility issue by adding an import for glob_to_regex in synapse.util, where it moved from. (#11696) 2022-01-06 16:36:26 +00:00
lukasdenk
2ef1fea8d2
Make room creations denied by user_may_create_room cause an M_FORBIDDEN error to be returned, not M_UNKNOWN (#11672)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-06 13:16:42 +00:00
David Robertson
c9eb678b73
Remove debug logging for #4422 (#11693)
as per
https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
2022-01-06 12:44:36 +00:00
Travis Ralston
feb3e006d7
Fix space hierarchy endpoint to match MSC2946 (#11667)
Fixes minor discrepancies between the /hierarchy endpoint described
in MSC2946 and the implementation.

Note that the changes impact the stable and unstable /hierarchy and
unstable /spaces endpoints for both client and federation APIs.
2022-01-05 20:33:43 +00:00
Dirk Klimpel
3b51c763ba
Fix get federation status of destination if no error occured (#11593) 2022-01-05 14:46:50 -05:00
Shay
d8f94eeec2
Run pyupgrade --py37-plus --keep-percent-format on Synapse (#11685)
* newsfragment

* fix newsfragment number

* update changelog

* remove extra space
2022-01-05 09:53:05 -08:00
Patrick Cloke
83acdb23fe Re-run towncrier. 2022-01-05 12:09:15 -05:00
Patrick Cloke
877b45e812
Include io.element.thread capability for MSC3440. (#11690) 2022-01-05 12:08:03 -05:00
David Robertson
88a78c6577
Cache empty responses from /user/devices (#11587)
If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
2022-01-05 13:33:28 +00:00
Olivier Wilkinson (reivilibre)
9be5aacc2d 1.50.0rc1 2022-01-05 12:39:48 +00:00
Richard van der Hoff
0fb3dd0830
Refactor the way we set outlier (#11634)
* `_auth_and_persist_outliers`: mark persisted events as outliers

Mark any events that get persisted via `_auth_and_persist_outliers` as, well,
outliers.

Currently this will be a no-op as everything will already be flagged as an
outlier, but I'm going to change that.

* `process_remote_join`: stop flagging as outlier

The events are now flagged as outliers later on, by `_auth_and_persist_outliers`.

* `send_join`: remove `outlier=True`

The events created here are returned in the result of `send_join` to
`FederationHandler.do_invite_join`. From there they are passed into
`FederationEventHandler.process_remote_join`, which passes them to
`_auth_and_persist_outliers`... which sets the `outlier` flag.

* `get_event_auth`: remove `outlier=True`

stop flagging the events returned by `get_event_auth` as outliers. This method
is only called by `_get_remote_auth_chain_for_event`, which passes the results
into `_auth_and_persist_outliers`, which will flag them as outliers.

* `_get_remote_auth_chain_for_event`: remove `outlier=True`

we pass all the events into `_auth_and_persist_outliers`, which will now flag
the events as outliers.

* `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter

This param is now never set to True, so we can remove it.

* `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param

This is no longer set anywhere, so we can remove it.

* `get_pdu`: remove unused `outlier` parameter

... and chase it down into `get_pdu_from_destination_raw`.

* `event_from_pdu_json`: remove redundant `outlier` param

This is never set to `True`, so can be removed.

* changelog

* update docstring
2022-01-05 12:26:11 +00:00
Philipp Matthias Schäfer
eedb4527f1
Fix link from generated configuration file to documentation (#11678)
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: reivilibre <oliverw@matrix.org>
2022-01-05 12:16:52 +00:00
Fr3shTea
0201c6371c
Fix SimpleHttpClient not sending Accept header in get_json (#11677)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-05 11:59:29 +00:00
Dirk Klimpel
7a1cefc6e3
Add admin API to get users' account data (#11664)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-05 11:49:06 +00:00
reivilibre
84bfe47b01
Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common #11505 (#11687)
Co-authored-by: Sean Quah <seanq@element.io>
2022-01-05 11:41:49 +00:00
reivilibre
84d790a32e
Clarify SSO mapping provider documentation by writing def or async def before the names of methods, as appropriate. (#11681) 2022-01-05 11:25:32 +00:00
Callum Macdonald
7013e06e2f
Improve Docker docs for use with Postgres (#11640) 2022-01-05 10:50:28 +00:00
Donny Johnson
0715e77b06
Correct Synapse install command for FreeBSD. (#11267)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-05 10:38:51 +00:00
Richard van der Hoff
79f6d3550a
update ngnix reverse-proxy example (#11680)
this should not be a case-insensitive match.
2022-01-04 22:31:45 +00:00
Richard van der Hoff
b38bdae3a2
Fix AssertionErrors after purging events (#11642)
* Fix AssertionErrors after purging events

If you purged a bunch of events from your database, and then restarted synapse
without receiving more events, then you would get a bunch of AssertionErrors on
restart.

This fixes the situation by rewinding the stream processors.

* `check-newsfragment`: ignore deleted newsfiles
2022-01-04 16:36:33 +00:00
Richard van der Hoff
878aa55293
FederationClient.backfill: stop flagging events as outliers (#11632)
Events returned by `backfill` should not be flagged as outliers.

Fixes:

```
AssertionError: null
  File "synapse/handlers/federation.py", line 313, in try_backfill
    dom, room_id, limit=100, extremities=extremities
  File "synapse/handlers/federation_event.py", line 517, in backfill
    await self._process_pulled_events(dest, events, backfilled=True)
  File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events
    await self._process_pulled_event(origin, ev, backfilled=backfilled)
  File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event
    assert not event.internal_metadata.is_outlier()
```

See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992

Fixes #8894.
2022-01-04 16:31:32 +00:00
Richard van der Hoff
2359ee3864
Remove redundant get_current_events_token (#11643)
* Push `get_room_{min,max_stream_ordering}` into StreamStore

Both implementations of this are identical, so we may as well push it down and
get rid of the abstract base class nonsense.

* Remove redundant `StreamStore` class

This is empty now

* Remove redundant `get_current_events_token`

This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid
of it.

* newsfile
2022-01-04 16:10:27 +00:00
Richard van der Hoff
bd9821f7f1
Better error messages from get_create_event_for_room (#11638)
"Unknown room" can mean a multitude of things here. To help with debugging, add
some more words to the exception text.
2022-01-04 16:10:05 +00:00
Patrick Cloke
8422a7f7f6
Include the topic event in the prejoin state, per MSC3173. (#11666)
Invites and knocks will now include the topic in the stripped state
send to clients before joining the room.
2022-01-04 11:08:08 -05:00
Shay
13c974ed35
Drop Bionic from Debian builds (#11633)
* update Trove classifiers to remove py36

* stop building bionic

* update dh-virtualenv

* newsfragment

* fix newsfragment

* update version refs

* another try at correct tag

* Update changelog
2022-01-03 11:17:16 -08:00
Patrick Cloke
cbd82d0b2d
Convert all namedtuples to attrs. (#11665)
To improve type hints throughout the code.
2021-12-30 18:47:12 +00:00
Dirk Klimpel
07a3b5daba
Add type hints to synapse/storage/databases/main/events_bg_updates.py (#11654) 2021-12-30 07:22:31 -05:00
reivilibre
2c7f5e74e5
Fix a type annotation in test_account_data.py and remove it from the Mypy exclusion list. (#11657)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-29 15:12:30 +00:00
Dirk Klimpel
f82d38ed2e
Improve type hints in storage classes. (#11652)
By using cast and making ignores more specific.
2021-12-29 13:04:28 +00:00
Patrick Cloke
f58b300d27
Do not attempt to bundled aggregations for /members and /state. (#11623)
Both of those APIs return state events, which will not have bundled
aggregations added anyway.
2021-12-29 08:02:03 -05:00
Dirk Klimpel
15bb1c8511
Add type hints to synapse/storage/databases/main/stats.py (#11653) 2021-12-29 08:01:13 -05:00
Shay
fcfe67578f
Update to the current version of Black and run it on Synapse codebase (#11596)
* update black version

* run updated version of black on code

* newsfragment

* enumerate python versions
2021-12-23 20:22:15 -08:00
Patrick Cloke
66d7aa783a
Fix mypy error with opentracing.tags. (#11622) 2021-12-23 11:47:24 +00:00
AndrewFerr
c500bf37d6
Add details for how to set up TURN behind NAT (#11553)
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-12-23 10:42:50 +00:00
Shay
87da37374a
Drop EOL python 3.6 from CI (#11595)
* remove python 3.6 and postgres 9.6 from github workflow

* remove python 3.6 env from tox

* newsfragment

* correct postgres version

* add py310 to tox env list
2021-12-21 11:37:04 -08:00
Richard van der Hoff
144b9ede89 Merge branch 'master' into develop 2021-12-21 18:13:05 +00:00
reivilibre
e6897e7383
Refactor tests.util.setup_test_homeserver and tests.server.setup_test_homeserver. (#11503) 2021-12-21 16:12:05 +00:00
Patrick Cloke
b6102230a7
Add type hints to event_push_actions. (#11594) 2021-12-21 13:25:34 +00:00
Richard van der Hoff
2215954147
Various opentracing enhancements (#11619)
* Wrap `auth.get_user_by_req` in an opentracing span

give `get_user_by_req` its own opentracing span, since it can result in a
non-trivial number of sub-spans which it is useful to group together.

This requires a bit of reorganisation because it also sets some tags (and may
force tracing) on the servlet span.

* Emit opentracing span for encoding json responses

This can be a significant time sink.

* Rename all sync spans with a prefix

* Write an opentracing span for encoding sync response

* opentracing span to group generate_room_entries

* opentracing spans within sync.encode_response

* changelog

* Use the `trace` decorator instead of context managers
2021-12-21 11:10:36 +00:00
Patrick Cloke
dd47788752
Do not bundle aggregations for APIs which shouldn't include them. (#11592)
And make bundling aggregations opt-in, instead of opt-out to avoid
having APIs to include extraneous data (and being much heavier than
necessary).
2021-12-20 14:14:38 -05:00
Richard van der Hoff
c3e38b88f2
Improve opentracing support for ResponseCache (#11607)
This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
2021-12-20 18:12:08 +00:00
Richard van der Hoff
60fa4935b5
Improve opentracing for incoming HTTP requests (#11618)
* remove `start_active_span_from_request`

Instead, pull out a separate function, `span_context_from_request`, to extract
the parent span, which we can then pass into `start_active_span` as
normal. This seems to be clearer all round.

* Remove redundant tags from `incoming-federation-request`

These are all wrapped up inside a parent span generated in AsyncResource, so
there's no point duplicating all the tags that are set there.

* Leave request spans open until the request completes

It may take some time for the response to be encoded into JSON, and that JSON
to be streamed back to the client, and really we want that inside the top-level
span, so let's hand responsibility for closure to the SynapseRequest.

* opentracing logs for HTTP request events

* changelog
2021-12-20 17:45:03 +00:00
Richard van der Hoff
8e4083e2f6 Merge remote-tracking branch 'origin/release-v1.49' into develop 2021-12-20 16:34:01 +00:00
Richard van der Hoff
76aa5537ad
Disable aggregation bundling on /sync responses (#11583)
* Disable aggregation bundling on `/sync` responses

A partial revert of #11478. This turns out to have had a significant CPU impact
on initial-sync handling. For now, let's disable it, until we find a more
efficient way of achieving this.

* Fix tests.

Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2021-12-20 16:33:35 +00:00
Patrick Cloke
3e0cfd447e
Return JSON errors for unknown resources under /matrix/client. (#11602)
Instead of returning 404 errors with HTML bodies when an unknown
prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
2021-12-20 16:00:13 +00:00
V02460
7a7ca8f226
Use mock from standard library (#11588)
Instead of the backported version.
2021-12-20 10:34:46 -05:00
Shay
8ad39438fa
Add opentracing types (#11603) 2021-12-20 12:18:09 +00:00
Dirk Klimpel
8428ef66c7
Add type hints to synapse/tests/rest/admin (#11590) 2021-12-16 14:59:56 -05:00
Dirk Klimpel
1847d027e6
Add type hints to synapse/storage/databases/main/transactions.py (#11589) 2021-12-16 14:59:35 -05:00
Eric Eastwood
43f5cc7adc
Add MSC2716 and MSC3030 to /versions -> unstable_features (#11582)
As suggested in https://github.com/matrix-org/matrix-react-sdk/pull/7372#discussion_r769523369
2021-12-16 11:25:37 -06:00
Sean Quah
c7fe32edb4
Add type hints to synapse/storage/databases/main/room.py (#11575) 2021-12-15 18:00:48 +00:00
Patrick Cloke
f901f8b70e
Require Collections as the parameters for simple_* methods. (#11580)
Instead of Iterable since the generators are not allowed due
to the potential for their re-use.
2021-12-15 17:00:50 +00:00
Patrick Cloke
323151b787
Convert EventStreamResult to attrs. (#11574) 2021-12-15 11:10:02 -05:00
reivilibre
17886d2603
Add experimental support for MSC3202: allowing application services to masquerade as specific devices. (#11538) 2021-12-15 10:40:52 +00:00
Sean Quah
ecfcd9bbbe
Add type hints to synapse/storage/databases/main/e2e_room_keys.py (#11549) 2021-12-14 17:46:47 +00:00
Sean Quah
0147b3de20
Add missing type hints to synapse.logging.context (#11556) 2021-12-14 17:35:28 +00:00
Patrick Cloke
2519beaad2
Add missing type hints to synapse.appservice (#11360) 2021-12-14 17:02:46 +00:00
Sean Quah
a4dce5b53d
Remove redundant COALESCE()s around COUNT()s in database queries (#11570)
`COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a
`COUNT(NULL)` is also 0.
2021-12-14 12:34:30 +00:00
Patrick Cloke
33abbc3278
Add missing type hints to synapse.http. (#11571) 2021-12-14 07:00:47 -05:00
Richard van der Hoff
ff6fd52160
checks for generators in database functions (#11564)
A couple of safety-checks to hopefully stop people doing what I just did, and create a storage
function which only works the first time it is called (and not when it is re-run due to a database
concurrency error or similar).
2021-12-13 19:01:27 +00:00
Patrick Cloke
eb39da6782
Move HTML parsing to a separate file for URL previews. (#11566)
* Splits the logic for parsing HTML from the resource handling code.
* Fix a circular import in the oEmbed code (which uses the HTML parsing code).
* Renames some of the HTML parsing methods to:
  * Make it clear which methods are "internal" to the module.
  * Clarify what the methods do.
2021-12-13 17:55:07 +00:00
Sean Quah
5305a5e881
Type hint the constructors of the data store classes (#11555) 2021-12-13 17:05:00 +00:00
Sean Quah
1abfb15f07
Add type hints to synapse/storage/databases/main/end_to_end_keys.py (#11551) 2021-12-13 16:28:26 +00:00
Sean Quah
6da8591f2e
Add type hints to synapse/storage/databases/main/account_data.py (#11546) 2021-12-13 16:28:10 +00:00
reivilibre
e5cdb9e233
Make get_device return None if the device doesn't exist rather than raising an exception. (#11565)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-12-13 15:39:43 +00:00
Eric Eastwood
aa8708ebed
Allow events to be created with no prev_events (MSC2716) (#11243)
The event still needs to have `auth_events` defined to be valid.

Split out from https://github.com/matrix-org/synapse/pull/11114
2021-12-10 23:08:51 -06:00
Eric Eastwood
8391bd6ab5
Test to ensure we share the same state_group across the whole historical batch (MSC2716) (#11487)
Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

We did some work on making sure the `state_groups` were shared in
https://github.com/matrix-org/synapse/pull/10975
2021-12-10 20:59:20 -06:00
David Robertson
fd2dadb815
Adjust _get_rooms_changed comments (#11550)
C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
2021-12-10 19:19:48 +00:00
Richard van der Hoff
f0562183e7
skip some dict munging in event persistence (#11560)
Create a new dict helper method `simple_insert_many_values_txn`, which takes
raw row values, rather than {key=>value} dicts. This saves us a bunch of dict
munging, and makes it easier to use generators rather than creating
intermediate lists and dicts.
2021-12-10 15:02:33 +00:00
Richard van der Hoff
86e7a6d16e
Stop populating state_events.prev_state (#11558)
this field is never read, so we may as well stop populating it.
2021-12-10 14:13:23 +00:00
Patrick Cloke
9562f0c2f1
Ensure emails are canonicalized before fetching associated user. (#11547)
This should fix pushers with an email in non-canonical form is used as
the pushkey.
2021-12-10 07:17:28 -05:00
Patrick Cloke
3b8872299a
Do not allow cross-room relations, per MSC2674. (#11516) 2021-12-09 13:16:01 -05:00
Patrick Cloke
0cc3bf97b4
Additional type hints for the config module, part 2. (#11480) 2021-12-09 16:15:46 +00:00
Dirk Klimpel
941ebe49ff
Use HTTPStatus constants in place of literals in synapse.http (#11543) 2021-12-09 11:58:25 +00:00
Hubert Chathi
b47d10dc46
Support unprefixed versions of fallback key property names. (#11541) 2021-12-09 11:41:27 +00:00
Dirk Klimpel
b3bcacf3c1
Add missing errcode to parse_string and parse_boolean (#11542) 2021-12-09 11:23:34 +00:00
Robert Long
afa0a5e4fc
Allow guests to send state events (#11378) 2021-12-09 11:02:05 +00:00
Patrick Cloke
d93362d87f
Add a constant for receipt types (m.read). (#11531)
And expand some type hints in the receipts storage module.
2021-12-08 12:26:29 -05:00
Dirk Klimpel
7ecaa3b976
Clean up synapse.rest.admin (#11535) 2021-12-08 16:59:40 +00:00
reivilibre
83a74d9350
Document the usage of refresh tokens. (#11427)
Co-authored-by: David Robertson <davidr@element.io>
2021-12-08 15:31:17 +00:00
reivilibre
365e9482fe
Use HTTPStatus constants in place of literals in tests.rest.client.test_auth. (#11520) 2021-12-08 14:54:47 +00:00
Richard van der Hoff
ff7cc17b57
Improve log messages for stream ids (#11536)
Somehow I'd managed to get my database in a pickle with stream ids. These
changes were useful to debug.
2021-12-08 14:15:14 +00:00
Hubert Chathi
8541809cb9
Send and handle cross-signing messages using the stable prefix. (#10520) 2021-12-08 10:01:38 +00:00
Andrew Morgan
d6fb96e056
Fix case in wait_for_background_updates where self.store does not exist (#11331)
Pull the DataStore from the HomeServer instance, which
always exists.
2021-12-07 11:51:53 -05:00
Sean Quah
966b5d0fa0 1.49.0rc1 2021-12-07 13:56:09 +00:00
Sean Quah
088d748f2c
Revert "Move glob_to_regex and re_word_boundary to matrix-python-common (#11505) (#11527)
This reverts commit a77c369897.
2021-12-07 13:51:11 +00:00
David Robertson
14d593f72d
Refactors in _generate_sync_entry_for_rooms (#11515)
* Move sync_token up to the top
* Pull out _get_ignored_users
* Try to signpost the body of `_generate_sync_entry_for_rooms`
* Pull out _calculate_user_changes

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-07 12:42:05 +00:00
David Robertson
2a3ec6facf
Correctly register shutdown handler for presence workers (#11518)
Fixes #11517
2021-12-07 12:34:38 +00:00
Sean Quah
eccc49d755
Fix ModuleApi.looping_background_call for non-async functions (#11524)
After #10847, `looping_background_call` would print an error in the logs
every time a non-async function was called. Since the error would be
caught and ignored immediately, there were no other side effects.
2021-12-07 11:41:31 +00:00
Richard van der Hoff
b1ecd19c5d
Fix 'delete room' admin api to work on incomplete rooms (#11523)
If, for some reason, we don't have the create event, we should still be able to
purge a room.
2021-12-07 11:37:54 +00:00
David Robertson
9c55dedc8c
Correctly ignore invites from ignored users (#11511) 2021-12-07 11:24:31 +00:00
reivilibre
2d42e586a8
Fix the test breakage introduced by #11435 as a result of concurrent PRs (#11522) 2021-12-07 10:49:39 +00:00
reivilibre
2f053f3f82
Stabilise support for MSC2918 refresh tokens as they have now been merged into the Matrix specification. (#11435) 2021-12-06 19:11:43 +00:00
Quentin Gliech
a15a893df8
Save the OIDC session ID (sid) with the device on login (#11482)
As a step towards allowing back-channel logout for OIDC.
2021-12-06 12:43:06 -05:00
Dirk Klimpel
8b4b153c9e
Add admin API to get some information about federation status (#11407) 2021-12-06 16:59:50 +00:00
Patrick Cloke
494ebd7347
Include bundled aggregations in /sync and related fixes (#11478)
Due to updates to MSC2675 this includes a few fixes:

* Include bundled aggregations for /sync.
* Do not include bundled aggregations for /initialSync and /events.
* Do not bundle aggregations for state events.
* Clarifies comments and variable names.
2021-12-06 15:51:15 +00:00
Sean Quah
a77c369897
Move glob_to_regex and re_word_boundary to matrix-python-common (#11505) 2021-12-06 11:36:08 +00:00
Eric Eastwood
4eb77965cd
Update backward extremity docs to make it clear that it does not indicate whether we have fetched an events' prev_events (#11469)
Spawning from https://github.com/matrix-org/synapse/pull/9445#discussion_r758958181

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-12-03 18:25:04 -06:00
reivilibre
637df95de6
Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445) 2021-12-03 16:42:44 +00:00
Dirk Klimpel
e5f426cd54
Add type hints to synapse/tests/rest/admin (#11501) 2021-12-03 13:57:13 +00:00
Olivier Wilkinson (reivilibre)
8cd68b8102 Revert accidental commits to develop. 2021-12-03 12:31:28 +00:00
Olivier Wilkinson (reivilibre)
6cae125e20 Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2021-12-03 12:25:37 +00:00
Richard van der Hoff
5640992d17
Disambiguate queries on state_key (#11497)
We're going to add a `state_key` column to the `events` table, so we need to
add some disambiguation to queries which use it.
2021-12-02 22:42:58 +00:00
David Robertson
d26808dd85
Comments on the /sync tentacles (#11494)
This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-02 20:58:32 +00:00
reivilibre
f91624a595
Clean up tests.storage.test_appservice (#11492) 2021-12-02 18:43:33 +00:00
reivilibre
16d39a5490
Clean up tests.storage.test_main to remove use of legacy code. (#11493) 2021-12-02 18:13:43 +00:00
reivilibre
8a4c296987
Clean up tests.test_visibility to remove legacy code. (#11495) 2021-12-02 18:13:30 +00:00
Shay
49e1356ee3
Minor cleanup on recently ported doc pages (#11466)
* move wiki pages to synapse/docs and add a few titles where necessary

* update SUMMARY.md with added pages

* add changelog

* move incorrectly located newsfragment

* update changelog number

* snake case added files and update summary.md accordingly

* update issue/pr links

* update relative links to docs

* update changelog to indicate that we moved wiki pages to the docs and state reasoning

* requested changes to admin_faq.md

* requested changes to database_maintenance_tools.md

* requested changes to understanding_synapse_through_graphana_graphs.md

* add changelog

* fix leftover merge errata

* fix unwanted changes from merge

* use two spaces between entries

* outdent code blocks
2021-12-02 09:46:20 -08:00
Patrick Cloke
d2279f471b
Add most of the missing type hints to synapse.federation. (#11483)
This skips a few methods which are difficult to type.
2021-12-02 16:18:10 +00:00
Sean Quah
b50e39df57
Avoid waiting for zombie processes in synctl stop (#11490) 2021-12-02 16:07:06 +00:00
Sean Quah
858d80bf0f
Fix media repository failing when media store path contains symlinks (#11446) 2021-12-02 16:05:24 +00:00
reivilibre
435f044807
Add type annotations to tests.storage.test_appservice. (#11488) 2021-12-02 15:30:05 +00:00
Richard van der Hoff
f61462e1be
scripts-dev/sign_json: support for signing events (#11486) 2021-12-02 15:18:40 +00:00
Eric Eastwood
a6f1a3abec
Add MSC3030 experimental client and federation API endpoints to get the closest event to a given timestamp (#9445)
MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030

Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about.
```
GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}
```

Federation API endpoint:
```
GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}
```

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-12-02 01:02:20 -06:00
Shay
84dc50e160
Port wiki pages to documentation website (#11402)
* move wiki pages to synapse/docs and add a few titles where necessary

* update SUMMARY.md with added pages

* add changelog

* move incorrectly located newsfragment

* update changelog number

* snake case added files and update summary.md accordingly

* update issue/pr links

* update relative links to docs

* update changelog to indicate that we moved wiki pages to the docs and state reasoning

* revert unintentional change to CHANGES.md

* add link

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Update CHANGES.md

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-12-01 10:12:19 -08:00
Patrick Cloke
ed635d3285
Add a license header and comment. (#11479) 2021-12-01 12:51:14 -05:00
Patrick Cloke
7b62791e00
Clean-up get_version_string (#11468) 2021-12-01 12:43:32 -05:00
Brendan Abolivier
153194c771
Link background update controller docs to summary (#11475) 2021-12-01 14:13:01 +00:00
Patrick Cloke
f44d729d4c
Additional type hints for config module. (#11465)
This adds some misc. type hints to helper methods used
in the `synapse.config` module.
2021-12-01 07:28:23 -05:00
Patrick Cloke
a265fbd397
Register the login redirect endpoint for v3. (#11451)
As specified for Matrix v1.1.
2021-12-01 07:25:58 -05:00
Shay
f1795463bf
Add a note about huge pages to our Postgres doc (#11467)
* Add note to postgres doc about hugepages

* Newsfragment
2021-11-30 19:05:20 -08:00
Shay
70cbb1a5e3
Don't start Synapse master process if worker_app is set (#11416)
* Add check to catch syanpse master process starting when workers are configured

* add test to verify that starting master process with worker config raises error

* newsfragment

* specify config.worker.worker_app in check

* update test

* report specific config option that triggered the error

Co-authored-by: reivilibre <oliverw@matrix.org>

* clarify error message

Co-authored-by: reivilibre <oliverw@matrix.org>

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-30 10:12:18 -08:00
Maximilian Bosch
42bf020463
Expose worker & homeserver as entrypoints in setup.py (#11449)
Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-30 16:52:45 +00:00
Patrick Cloke
379f2650cf
Bundle relations of relations into the /relations result. (#11284)
Per updates to MSC2675 which now states that bundled
aggregations should be included from the `/relations`
endpoint.
2021-11-30 11:33:33 -05:00
Sean Quah
7ff22d6da4
Fix LruCache corruption bug with a size_callback that can return 0 (#11454)
When all entries in an `LruCache` have a size of 0 according to the
provided `size_callback`, and `drop_from_cache` is called on a cache
node, the node would be unlinked from the LRU linked list but remain in
the cache dictionary. An assertion would be later be tripped due to the
inconsistency.

Avoid unintentionally calling `__len__` and use a strict `is None`
check instead when unwrapping the weak reference.
2021-11-30 16:28:02 +00:00
Sean Quah
5a0b652d36
Eliminate a few Anys in LruCache type hints (#11453) 2021-11-30 15:39:07 +00:00
Dirk Klimpel
432a174bc1
Remove unnecessary json.dumps from tests.rest.admin (#11461)
The tests helpers automatically convert dictionaries to
JSON payloads, no need to do it manually for each
test.
2021-11-30 14:51:04 +00:00
Brendan Abolivier
b14f8a1baf Merge branch 'master' into develop 2021-11-30 14:28:30 +00:00
Patrick Cloke
28f5252c1f
Add missing copyright header. (#11460) 2021-11-30 13:23:53 +00:00
Richard van der Hoff
f13a8d1c69
synctl stop: wait for processes to exit (#11459)
If you're trying to shut down Synapse, it's rather handy if it *actually* shuts
down before you move on.
2021-11-30 11:51:12 +00:00
Marcus
a9481223d1
Improved push typing (#11409)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-11-30 11:49:20 +00:00
Dirk Klimpel
35b1900f00
Convert status codes to HTTPStatus in tests.rest.admin (#11455) 2021-11-30 09:53:54 +00:00
Dirk Klimpel
e8ae94a223
Convert status codes to HTTPStatus in synapse.rest.admin (#11452) 2021-11-29 22:19:45 +00:00
Eric Eastwood
fb58611d21
Refactor backfilled into specific behavior function arguments (_persist_events_and_state_updates) (#11417)
Part of https://github.com/matrix-org/synapse/issues/11300

Call stack:

 - `_persist_events_and_state_updates` (added `use_negative_stream_ordering`)
    - `_persist_events_txn`
       - `_update_room_depths_txn` (added `update_room_forward_stream_ordering`)
       - `_update_metadata_tables_txn`
          - `_store_room_members_txn` (added `inhibit_local_membership_updates`)

Using keyword-only arguments (`*`) to reduce the mistakes from `backfilled` being left as a positional argument somewhere and being interpreted wrong by our new arguments.
2021-11-29 16:01:54 -06:00
Patrick Cloke
a4521ce0a8
Support the stable /hierarchy endpoint from MSC2946 (#11329)
This also makes additional updates where the implementation
had drifted from the approved MSC.

Unstable endpoints will be removed at a later data.
2021-11-29 14:32:20 -05:00
Erik Johnston
d08ef6f155
Make background updates controllable via a plugin (#11306)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-11-29 17:57:06 +01:00
Patrick Cloke
9d1971a5c4
Return the stable event field from /send_join per MSC3083. (#11413)
This does not remove the unstable field and still parses both.
Handling of the unstable field will need to be removed in the
future.
2021-11-29 15:43:20 +00:00
Sean Quah
7564b8e118
Update the media repository documentation (#11415) 2021-11-29 15:37:56 +00:00
reivilibre
a82b90ab32
Add type annotations to some of the configuration surrounding refresh tokens. (#11428) 2021-11-29 13:34:14 +00:00
Richard van der Hoff
9cd13c5f63
Fix perspectives requests for multiple keys for the same server (#11440)
If we tried to request multiple keys for the same server, we would end up
dropping some of those requests.
2021-11-29 13:15:36 +00:00
Tulir Asokan
7b4e228e41
Fix using MSC2716 batch sending with event persistence workers (#11220)
Signed-off-by: Tulir Asokan <tulir@beeper.com>
2021-11-29 13:13:23 +00:00
David Robertson
dc0a3cd596
disallow-untyped-defs for the module_api (#11029) 2021-11-29 11:28:12 +00:00
David Robertson
aa457b625e
Fix changelog filename in #11441 2021-11-29 11:15:28 +00:00
David Robertson
776ad3e5e9
Add a test case for the SendJoinParser (#11441)
This would have caught the bug #11438 introduced in #11217 and fixed in #11439.
2021-11-29 11:11:46 +00:00
Daniel Molkentin
e5c5e213ea
Bump ijson dependency to 3.1 (#11438) (#11439)
Since e81fa92648, Synapse depends on
the use_float flag which has been introduced in ijson 3.1 and
is not available in 3.0. This is known to cause runtime errors
with send_join.

Signed-off-by: Daniel Molkentin <danimo@infra.run>

Co-authored-by: Daniel Molkentin <danimo@infra.run>
2021-11-26 20:51:22 +00:00
reivilibre
1b6691dce4
Update MSC2918 refresh token support to confirm with the latest revision: accept the refresh_tokens parameter in the request body rather than in the URL parameters. (#11430) 2021-11-26 19:06:16 +00:00
Sean Quah
ffd858aa68
Add type hints to synapse/storage/databases/main/events_worker.py (#11411)
Also refactor the stream ID trackers/generators a bit and try to
document them better.
2021-11-26 18:41:31 +00:00
reivilibre
1d8b80b334
Support expiry of refresh tokens and expiry of the overall session when refresh tokens are in use. (#11425) 2021-11-26 14:27:14 +00:00
Michael Kaye
e2c300e7e4
Create healthcheck script for synapse-workers container (#11429)
The intent is to iterate through all the worker ports and only
report healthy when all are healthy, starting with the main process.
2021-11-26 14:05:20 +00:00
Sean Quah
c675a18071
Track ongoing event fetches correctly (again) (#11376)
The previous fix for the ongoing event fetches counter
(8eec25a1d9) was both insufficient and
incorrect.

When the database is unreachable, `_do_fetch` never gets run and so
`_event_fetch_ongoing` is never decremented.

The previous fix also moved the `_event_fetch_ongoing` decrement outside
of the `_event_fetch_lock` which allowed race conditions to corrupt the
counter.
2021-11-26 13:47:24 +00:00
Sean Quah
7862f821de
Annotate string constants in synapse.api.constants with Final (#11356)
This change makes mypy complain if the constants are ever reassigned,
and, more usefully, makes mypy type them as `Literal`s instead of `str`s,
allowing code of the following form to pass mypy:
```py
def do_something(membership: Literal["join", "leave"], ...): ...

do_something(Membership.JOIN, ...)
```
2021-11-25 16:14:23 +00:00
Brendan Abolivier
946c102ac9 1.48.0rc1 2021-11-25 15:57:04 +00:00
Brendan Abolivier
0d88c4f903
Improve performance of remove_{hidden,deleted}_devices_from_device_inbox (#11421)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-25 15:14:54 +00:00
Brendan Abolivier
7f9841bdec
Lower minumum batch size to 1 for background updates (#11422)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-24 19:21:44 +00:00
reivilibre
f25c75d376
Rename unstable access_token_lifetime configuration option to refreshable_access_token_lifetime to make it clear it only concerns refreshable access tokens. (#11388) 2021-11-23 17:01:34 +00:00
Patrick Cloke
55669bd3de
Add missing type hints to config base classes (#11377) 2021-11-23 15:21:19 +00:00
Shay
7cebaf9644
Remove code invalidated by deprecated config flag 'trust_identity_servers_for_password_resets' (#11395)
* remove background update code related to deprecated config flag

* changelog entry

* update changelog

* Delete 11394.removal

Duplicate, wrong number

* add no-op background update and change newfragment so it will be consolidated with associated work

* remove unused code

* Remove code associated with deprecated flag from legacy docker dynamic config file

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-23 06:46:40 -08:00
Patrick Cloke
6a5dd485bd
Refactor the code to inject bundled relations during serialization. (#11408) 2021-11-23 06:43:56 -05:00
Kostas
1035663833
Add config for customizing the claim used for JWT logins. (#11361)
Allows specifying a different claim (from the default "sub") to use
when calculating the localpart of the Matrix ID used during the
JWT login.
2021-11-22 13:01:03 -05:00
Patrick Cloke
3d893b8cf2
Store arbitrary relations from events. (#11391)
Instead of only known relation types. This also reworks the background
update for thread relations to crawl events and search for any relation
type, not just threaded relations.
2021-11-22 12:01:47 -05:00
Dirk Klimpel
ea20937084
Add an admin API to run background jobs. (#11352)
Instead of having admins poke into the database directly.

Can currently run jobs to populate stats and to populate
the user directory.
2021-11-19 19:39:46 +00:00
Patrick Cloke
7ae559944a
Fix checking whether a room can be published on creation. (#11392)
If `room_list_publication_rules` was configured with a rule with a
non-wildcard alias and a room was created with an alias then an
internal server error would have been thrown.

This fixes the error and properly applies the publication rules
during room creation.
2021-11-19 15:19:32 +00:00
Patrick Cloke
4d6d38ac2f
Remove dead code from acme support. (#11393) 2021-11-19 07:07:22 -05:00
Patrick Cloke
5505da2109
Remove msc2716 from the list of tests for complement. (#11389)
As the tests are currently failing and not run in CI.
2021-11-19 07:06:16 -05:00
Hubert Chathi
eca7cffb73
Keep fallback key marked as used if it's re-uploaded (#11382) 2021-11-19 11:40:12 +00:00
Richard van der Hoff
e2e9bea1ce
Publish a develop docker image (#11380)
I'd find it helpful to have a docker image corresponding to current develop,
without having to build my own.
2021-11-19 10:56:59 +00:00
Richard van der Hoff
a6f7f84570
Fix verification of objects signed with old local keys (#11379)
Fixes a bug introduced in #11129: objects signed by the local server, but with
keys other than the current one, could not be successfully verified.

We need to check the key id in the signature, and track down the right key.
2021-11-19 10:55:09 +00:00
Eric Eastwood
7ffddd819c
Prevent historical state from being pushed to an application service via /transactions (MSC2716) (#11265)
Mark historical state from the MSC2716 `/batch_send` endpoint as `historical` which makes it `backfilled` and have a negative `stream_ordering` so it doesn't get queried by `/transactions`.

Fix https://github.com/matrix-org/synapse/issues/11241

Complement tests: https://github.com/matrix-org/complement/pull/221
2021-11-18 14:16:08 -06:00
Shay
92b75388f5
Remove legacy code related to deprecated trust_identity_server_for_password_resets config flag (#11333)
* remove code legacy code related to deprecated config flag "trust_identity_server_for_password_resets" from synapse/config/emailconfig.py

* remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/config/registration.py

* remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/handlers/identity.py

* add tests to ensure config error is thrown and synapse refuses to start when depreciated config flag is found

* add changelog

* slightly change behavior to only check for deprecated flag if set to 'true'

* Update changelog.d/11333.misc

Co-authored-by: reivilibre <oliverw@matrix.org>

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-18 10:56:32 -08:00
Dirk Klimpel
81b18fe5c0
Add dedicated admin API for blocking a room (#11324) 2021-11-18 17:43:49 +00:00
reivilibre
5f81c0ce9c
Add/Unerase annotations to Module API (#11341) 2021-11-18 16:55:33 +00:00
reivilibre
433ee159cb
Rename get_refresh_token_for_user_id to create_refresh_token_for_user_id (#11370) 2021-11-18 14:45:38 +00:00
reivilibre
539e441399
Use auto_attribs for RefreshTokenLookupResult (#11386) 2021-11-18 14:40:26 +00:00
Patrick Cloke
4bd54b263e
Do not allow MSC3440 threads to fork threads (#11161)
Adds validation to the Client-Server API to ensure that
the potential thread head does not relate to another event
already. This results in not allowing a thread to "fork" into
other threads.

If the target event is unknown for some reason (maybe it isn't
visible to your homeserver), but is the target of other events
it is assumed that the thread can be created from it. Otherwise,
it is rejected as an unknown event.
2021-11-18 13:43:09 +00:00
Nicolai Søborg
e2dabec996
Docs: Quote wildcard federation_certificate_verification_whitelist (#11381)
Otherwise I get this beautiful stacktrace:

```
python3 -m synapse.app.homeserver --config-path /etc/matrix/homeserver.yaml
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/synapse/synapse/app/homeserver.py", line 455, in <module>
    main()
  File "/root/synapse/synapse/app/homeserver.py", line 445, in main
    hs = setup(sys.argv[1:])
  File "/root/synapse/synapse/app/homeserver.py", line 345, in setup
    config = HomeServerConfig.load_or_generate_config(
  File "/root/synapse/synapse/config/_base.py", line 671, in load_or_generate_config
    config_dict = read_config_files(config_files)
  File "/root/synapse/synapse/config/_base.py", line 717, in read_config_files
    yaml_config = yaml.safe_load(file_stream)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 379, in parse_block_sequence_first_entry
    return self.parse_block_sequence_entry()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 384, in parse_block_sequence_entry
    if not self.check_token(BlockEntryToken, BlockEndToken):
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 227, in fetch_more_tokens
    return self.fetch_alias()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 610, in fetch_alias
    self.tokens.append(self.scan_anchor(AliasToken))
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 922, in scan_anchor
    raise ScannerError("while scanning an %s" % name, start_mark,
yaml.scanner.ScannerError: while scanning an alias
  in "/etc/matrix/homeserver.yaml", line 614, column 5
expected alphabetic or numeric character, but found '.'
  in "/etc/matrix/homeserver.yaml", line 614, column 6
```

Signed-off-by: Nicolai Søborg <git@xn--sb-lka.org>
2021-11-18 12:24:40 +00:00
Sean Quah
84fac0f814
Add type annotations to synapse.metrics (#10847) 2021-11-17 19:07:02 +00:00
Aaron R
d993c3bb1e
Add support for /_matrix/media/v3 APIs (#11371)
* Add support for `/_matrix/media/v3` APIs

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Update `workers.md` to use v3 client and media APIs

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-11-17 15:30:24 +00:00
reivilibre
0d86f6334a
Rename get_access_token_for_user_id method to create_access_token_for_user_id (#11369) 2021-11-17 14:10:57 +00:00
Patrick Cloke
60ecb6b4d4
Fix running complement.sh script. (#11368)
By reverting changes from #11166 in this script. Specifically commit
13f084eb58.
2021-11-17 09:04:50 -05:00
Patrick Cloke
319dcb955e
Fix incorrect return value in tests. (#11359) 2021-11-16 16:36:46 +00:00
Sean Quah
88375beeaa
Avoid sharing room hierarchy responses between users (#11355)
Different users may be allowed to see different rooms within a space,
so sharing responses between users is inadvisable.
2021-11-16 15:40:47 +00:00
Aaron R
dfa536490e
Add support for /_matrix/client/v3 APIs (#11318)
This is one of the changes required to support Matrix 1.1

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-11-16 14:47:58 +01:00
Patrick Cloke
7468723697
Add most missing type hints to synapse.util (#11328) 2021-11-16 08:47:36 -05:00
reivilibre
3a1462f7e0
Properly register all callback hooks for legacy password authentication providers (#11340)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-11-16 12:53:31 +00:00
Patrick Cloke
24b61f379a
Add ability to un-shadow-ban via the admin API. (#11347) 2021-11-16 12:43:53 +00:00
David Robertson
0dda1a7968
Misc typing fixes for tests, part 2 of N (#11330) 2021-11-16 10:41:35 +00:00
Ashwin Nair
e72135b9d3
change 'Home Server' to one word 'homeserver' (#11320)
Signed-off-by: Ashwin S. Nair <58840757+Ashwin-exe@users.noreply.github.com>
2021-11-16 10:21:01 +00:00
David Robertson
e605e4b8f2
Database storage profile passes mypy (#11342)
It already seems to pass mypy. I wonder what changed, given that it was
on the exclusion list. So this commit consists of me ensuring
`--disallow-untyped-defs` passes and a minor fixup to a function that
returned either `True` or `None`.
2021-11-15 12:59:33 +00:00
David Robertson
5562ce6a53
Get directory db file to pass mypy (#11339) 2021-11-15 12:59:05 +00:00
Tulir Asokan
6f862c5c28
Add support for the stable version of MSC2778 (#11335)
* Add support for the stable version of MSC2778

Signed-off-by: Tulir Asokan <tulir@maunium.net>

* Expect m.login.application_service in login and password provider tests

Signed-off-by: Tulir Asokan <tulir@maunium.net>
2021-11-15 10:31:22 +00:00
Shay
605921bc6b
Remove unused tables room_stats_historical and user_stats_historical (#11280)
* remove unused tables room_stats_historical and user_stats_historical

* update changelog number

* Bump schema compat version comment

* make linter happy

* Update comment to give more info

Co-authored-by: reivilibre <oliverw@matrix.org>

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-12 16:47:56 -08:00
David Robertson
fe58672546
Annotations for state_deltas.py (#11316)
I was sad that I couldn't do better for
`_curr_state_delta_stream_cache`. At least it's explicitly called out in
a comment with #TODO.
2021-11-12 20:24:12 +00:00
Patrick Cloke
3fad4e3fe5
Rollback #11322 due to wrong syntax in mypy.ini. (#11332)
This was only checking the __init__ files in modules instead of
all files in a module, which don't pass yet.
2021-11-12 20:10:03 +00:00
David Robertson
bea815cec8
Test room alias deletion (#11327)
* Prefer `HTTPStatus` over plain `int`

This is an Opinion that no-one has seemed to object to yet.

* `--disallow-untyped-defs` for `tests.rest.client.test_directory`
* Improve synapse's annotations for deleting aliases
* Test case for deleting a room alias
* Changelog
2021-11-12 19:56:00 +00:00
Shay
0bcae8ad56
Change display names/avatar URLs to None if they contain null bytes before storing in DB (#11230)
* change display names/avatar URLS to None if they contain null bytes

* add changelog

* add POC test, requested changes

* add a saner test and remove old one

* update test to verify that display name has been changed to None

* make test less fragile
2021-11-12 10:38:24 -08:00
Patrick Cloke
9b90b9454b
Add type hints to media repository storage module (#11311) 2021-11-12 11:05:26 -05:00
David Robertson
6f8f3d4bc5
Attempt to annotate events_forward_extremities (#11314)
* Make DataStore inherit from EventForwardExtremitiesStore before CacheInvalidationWorkerStore

the former implicitly inherits from the latter, so they should be
ordered like this when used.
2021-11-12 15:58:17 +00:00
David Robertson
4c96ce396e
Misc typing fixes for tests, part 1 of N (#11323)
* Annotate HomeserverTestCase.servlets
* Correct annotation of federation_auth_origin
* Use AnyStr custom_headers instead of a Union

This allows (str, str) and (bytes, bytes).
This disallows (str, bytes) and (bytes, str)

* DomainSpecificString.SIGIL is a ClassVar
2021-11-12 15:50:54 +00:00
Patrick Cloke
95547e5300
Generalize the disallowed_untyped_defs in mypy.ini (#11322) 2021-11-12 14:27:45 +00:00
Patrick Cloke
b64b6d12d4
Add more type hints to synapse.util. (#11321) 2021-11-12 13:43:06 +00:00
reivilibre
2fffcb24d8
Suggest using /etc/matrix-synapse/conf.d/ for configuration with Debian packages (#11281) 2021-11-12 13:17:06 +00:00
Dirk Klimpel
8840a7b7f1
Convert delete room admin API to async endpoint (#11223)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-11-12 12:35:31 +00:00
David Robertson
c99da2d079
Annotations for user_erasure_store (#11313)
I'm not sure why this was excluded---it seemed to be passing for me. But
it's easy enough to fixup.
2021-11-11 19:22:19 +00:00
David Robertson
6a605f4a77
Get db signatures file to pass mypy (#11312) 2021-11-11 17:04:44 +00:00
David Robertson
8dc666f785
Correct type hint for room_batch.py (#11310)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-11-11 16:49:28 +00:00
Dirk Klimpel
48278a0d09
Move sql file for remove_deleted_devices_from_device_inbox into v65 (#11303) 2021-11-11 15:01:13 +00:00
Patrick Cloke
64ef25391d
Add type hints to some storage classes (#11307) 2021-11-11 08:47:31 -05:00
Neeeflix
6ce19b94e8
Fix error in thumbnail generation (#11288)
Signed-off-by: Jonas Zeunert <jonas@zeunert.org>
2021-11-10 20:49:43 +00:00
Patrick Cloke
5cace20bf1
Add missing type hints to synapse.app. (#11287) 2021-11-10 15:06:54 -05:00
Patrick Cloke
66c4b774fd
Add type hints to synapse._scripts (#11297) 2021-11-10 17:55:32 +00:00
Andrew Morgan
5f277ffe89
Add documentation page stubs for Single Sign-On, SAML and CAS pages (#11298) 2021-11-10 17:54:56 +00:00
Richard van der Hoff
73cbb284b9
Remove redundant parameters on _check_event_auth (#11292)
as of #11012, these parameters are unused.
2021-11-10 14:16:06 +00:00
Olivier Wilkinson (reivilibre)
68c258a604 Synapse 1.47.0rc2 (2021-11-10)
==============================
 
 This fixes an issue with publishing the Debian packages for 1.47.0rc1.
 It is otherwise identical to 1.47.0rc1.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmGLl4sACgkQLS76LzL7
 4EdpXxAArEBqEWCUCy6wSNRfexzI+qAITzhhqR5BiDDtkt33GzXLWDN83lT3kCS/
 xNZyhiwwUbtt/TqkZ0/Tqu2afI5JZCizpP/kXLVS2WA03jY8+l+eQbKEQR5vEsEV
 752J8OVJ9GUunewOI4Uo4xRqndzvOMQBKaoPzyq44PcFd6lS2rkAnJYstVnow+rB
 JvRIXFjOocaOzpemal5Mh8ToH5Y6yfe7MEE8B0s3PjX1FMAv87475X1oLaK7GeKn
 3yf8XJ1mJcvJfkHuopqX8PxW+pFGorb4N1AFk2BikxB4XV0nCo5VHUejSWaEP4oH
 uHtTYDV5RHwHhZyHYMbBOyPD2bSIKZ6wXqgn/o5io7mfsUS77SdwuGhRDqnupCi/
 Vg+4F5e1nuSPaLEJd5Qfb1NM2xErGm8gfYN3DlRBzpuQ+Jx/034YWds42ZuNv2IO
 qdAOeztiOMdrPkdPTL+XlNNXV80waMsQFt2EaycYkYmPVtgvlKvr/c/Wg06jyD7u
 dll33KlE8d+jM3JbOf6D/Ze5ECQlf0gGtCukXTMIEOorrCdQuLH4R8hb49YqmiLq
 nL8aPXCv1pZOrMbHGbcYfHoZIV+qMhUK04PN7jMLF/+nyBkTI1wRODN0lvBh8gy6
 dNWTzM/dMk5IvFY0FDipkR7Cv5c2U2Wu1xCy8026VHCu/DIzVRw=
 =Baxl
 -----END PGP SIGNATURE-----

Merge tag 'v1.47.0rc2' into develop

Synapse 1.47.0rc2 (2021-11-10)
==============================

This fixes an issue with publishing the Debian packages for 1.47.0rc1.
It is otherwise identical to 1.47.0rc1.
2021-11-10 13:01:08 +00:00
Stanislav Motylkov
b09d90cac9
Fix typos in the username_available admin API documentation. (#11286) 2021-11-09 21:11:05 +00:00
Eric Eastwood
f1d5c2f269
Split out federated PDU retrieval into a non-cached version (#11242)
Context: https://github.com/matrix-org/synapse/pull/11114/files#r741643968
2021-11-09 15:07:57 -06:00
Patrick Cloke
0ef69ddbdc
Ignore missing imports for parameterized. (#11285)
This was due to a conflict between #11282, which changed
mypy configuration, and #11228, a normal change.
2021-11-09 19:04:53 +00:00
Dan Callahan
3b951445a7
Require mypy for synapse/ & tests/ unless excluded (#11282)
Signed-off-by: Dan Callahan <danc@element.io>
2021-11-09 16:22:47 +00:00
Andrew Morgan
a026695083
Clarifications and small fixes to to-device related code (#11247)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-11-09 14:31:15 +00:00
David Robertson
b6f4d122ef
Allow admins to proactively block rooms (#11228)
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-11-09 13:11:47 +00:00
Patrick Cloke
a19d01c3d9
Support filtering by relations per MSC3440 (#11236)
Adds experimental support for `relation_types` and `relation_senders`
fields for filters.
2021-11-09 08:10:58 -05:00
Olivier Wilkinson (reivilibre)
01f61da77f 1.47.0rc1 2021-11-09 12:17:35 +00:00
Andrew Morgan
4b3e30c276
Fix typo in RelationAggregationPaginationServlet error response (#11278) 2021-11-09 12:11:50 +00:00
Erik Johnston
af784644c3
Include cross-signing signatures when syncing remote devices for the first time (#11234)
When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results.

c.f. #11159
2021-11-09 11:45:36 +00:00
rogersheu
820337e6a4
Require body for read receipts with user-agent exceptions (#11157)
Co-authored-by: reivilibre <olivier@librepush.net>
2021-11-09 10:26:07 +00:00
Eric Eastwood
84f235aea4
Rename to more clear get_insertion_event_id_by_batch_id (MSC2716) (#11244)
`get_insertion_event_by_batch_id` -> `get_insertion_event_id_by_batch_id`

Split out from https://github.com/matrix-org/synapse/pull/11114
2021-11-08 21:21:10 -06:00
Erik Johnston
4ee71b9637
Add some background update admin APIs (#11263)
Fixes #11259
2021-11-08 16:08:02 +00:00
Patrick Cloke
0c82d4aabe
Fix typo in comment from #11255. (#11276) 2021-11-08 14:36:49 +00:00
Richard van der Hoff
86a497efaa
Default value for public_baseurl (#11210)
We might as well use a default value for `public_baseurl` based on
`server_name` - in many cases, it will be correct.
2021-11-08 14:13:10 +00:00
Dan Callahan
556a488209
Address review feedback from #11269 (#11273)
Signed-off-by: Dan Callahan <danc@element.io>
2021-11-08 11:57:37 +00:00
Erik Johnston
a55e1ec9af
Blacklist new sytest validation test (#11270) 2021-11-08 10:37:43 +00:00
Erik Johnston
98c8fc6ce8
Handle federation inbound instances being killed more gracefully (#11262)
* Make lock better handle process being killed

If the process gets killed and restarted (so that it didn't have a
chance to drop its locks gracefully) then there may still be locks in
the DB that are for the same instance that haven't yet timed out but are
safe to delete.

We handle this case by a) checking if the current instance already has
taken out the lock, and b) if not then ignoring locks that are for the
same instance.

* Periodically check for old staged events

This is to protect against other instances dying and their locks timing
out.
2021-11-08 09:54:47 +00:00
Dan Callahan
9799c569bb
Minor cleanup to Debian packaging (#11269)
* Remove unused Vagrant scripts

* Change package Architecture to any

* Preinstall the wheel package when building venvs.

Addresses the following warnings during Debian builds:

    Using legacy 'setup.py install' for jaeger-client, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for matrix-synapse-ldap3, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for opentracing, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for psycopg2, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for systemd-python, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for pympler, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for threadloop, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for thrift, since package 'wheel' is not installed.

* Allow /etc/default/matrix-synapse to be missing

Per the systemd.exec manpage, prefixing an EnvironmentFile with "-":

> indicates that if the file does not exist, it will not be read and no
> error or warning message is logged.

Signed-off-by: Dan Callahan <danc@element.io>
2021-11-07 21:18:33 +00:00
Julian
09cb441a04
Add doc to integrate synapse with LemonLDAP OIDC (#11257)
Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
Co-authored-by: Julian Vanden Broeck <julian.vandenbroeck@dalibo.com>
2021-11-05 12:08:02 +00:00
Erik Johnston
a37df1b091
Fix rolling back when using workers (#11255)
Fixes #11252
2021-11-05 11:12:10 +00:00
reivilibre
499c44d696
Make minor correction to type of auth_checkers callbacks (#11253) 2021-11-04 17:10:11 +00:00
Richard van der Hoff
f36434590c
Additional test for cachedList (#11246)
I was trying to understand how `cachedList` works, and ended up writing this
extra test. I figure we may as well keep it.
2021-11-04 14:45:34 +00:00
Sean Quah
8eec25a1d9
Track ongoing event fetches correctly in the presence of failure (#11240)
When an event fetcher aborts due to an exception, `_event_fetch_ongoing`
must be decremented, otherwise the event fetcher would never be
replaced. If enough event fetchers were to fail, no more events would be
fetched and requests would get stuck waiting for events.
2021-11-04 10:33:53 +00:00
Nick Barrett
a271e233e9
Add a linearizer on (appservice, stream) when handling ephemeral events. (#11207)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-11-03 16:51:00 +00:00
Nick Barrett
af54167516
Enable passing typing stream writers as a list. (#11237)
This makes the typing stream writer config match the other stream writers
that only currently support a single worker.
2021-11-03 14:25:47 +00:00
Patrick Cloke
2735b3e6f2
Remove a debug statement from tests. (#11239) 2021-11-03 13:11:16 +00:00
Erik Johnston
bcc115c28d
Add twine and towncrier as dev dependencies (#11233)
We don't pin them as we execute them as commands, rather than use them
as libs.
2021-11-03 11:10:25 +00:00
Eric Eastwood
da0040785e
Support sending no state_events_at_start in the MSC2716 /batch_send endpoint (#11188)
As brought up by @tulir, https://matrix.to/#/!SBYNQlpqkwJzFIdzxI:nevarro.space/$Gwnb2ZvXHc3poYXuBhho0cmoYq4KJ11Jh3m5s8kjNOM?via=nevarro.space&via=beeper.com&via=matrix.org

This use case only works if the user is already joined in the current room state
at the given `?prev_event_id`
2021-11-03 03:13:51 -05:00
Erik Johnston
6250b95efe
Add index to local_group_updates.stream_id (#11231)
This should speed up startup times and generally increase performance of
groups.
2021-11-02 15:46:48 +00:00
Patrick Cloke
c01bc5f43d
Add remaining type hints to synapse.events. (#11098) 2021-11-02 09:55:52 -04:00
Dirk Klimpel
4535532526
Delete messages for hidden devices from device_inbox (#11199) 2021-11-02 13:18:30 +00:00
Andrew Morgan
c9c3aea9b1
Fix providing a RoomStreamToken instance to _notify_app_services_ephemeral (#11137)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-02 10:39:02 +00:00
Dirk Klimpel
7537201840
Add search by room ID and room alias to List Room admin API (#11099)
Fixes: #10874
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-11-02 10:01:13 +00:00
Richard van der Hoff
46d0937447
ObservableDeferred: run observers in order (#11229) 2021-11-02 00:17:35 +00:00
Dirk Klimpel
93aa670642
Update outdated links in PULL_REQUEST_TEMPLATE.md (#11225) 2021-11-01 17:29:51 +00:00
Shay
f5c6a80886
Handle missing Content-Type header when accessing remote media (#11200)
* add code to handle missing content-type header and a test to verify that it works

* add handling for missing content-type in the /upload endpoint as well

* slightly refactor test code to put private method in approriate place

* handle possible null value for content-type when pulling from the local db

* add changelog

* refactor test and add code to handle missing content-type in cached remote media

* requested changes

* Update changelog.d/11200.bugfix

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-11-01 10:26:02 -07:00
Shay
e81fa92648
Add use_float=true to ijson calls in Synapse (#11217)
* add use_float=true to ijson calls

* lints

* add changelog

* Update changelog.d/11217.bugfix

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-11-01 09:28:04 -07:00
Dirk Klimpel
caa706d825
Fix a bug in unit test test_block_room_and_not_purge (#11226) 2021-11-01 16:10:09 +00:00
reivilibre
69ab3dddbc
Make check_event_allowed module API callback not fail open (accept events) when an exception is raised (#11033) 2021-11-01 15:45:56 +00:00
Dirk Klimpel
66bdca3e31
Remove deprecated delete room admin API (#11213)
Remove deprecated delete room admin API,
`POST /_synapse/admin/v1/rooms/<room_id>/delete`
2021-11-01 15:11:24 +00:00
Richard van der Hoff
71f9966f27
Support for serving server well-known files (#11211)
Fixes https://github.com/matrix-org/synapse/issues/8308
2021-11-01 15:10:16 +00:00
Brett Bethke
2014098d01
Add domain specific matching for haproxy config (#11128) 2021-11-01 14:16:02 +00:00
Richard van der Hoff
0b99d4c8d2
Docker: avoid changing userid unnecessarily (#11209)
* Docker image: avoid changing user during `generate`

The intention was always that the config files get written as the initial user
(normally root) - only the data directory needs to be writable by Synapse. This
got changed in https://github.com/matrix-org/synapse/pull/5970, but that seems
to have been a mistake.

* Avoid changing user if no explicit UID is given

* changelog
2021-11-01 13:55:30 +00:00
Aaron R
3ae1464efd
Support Client-Server API r0.6.1 (#11097)
Fixes #11064

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-11-01 13:28:39 +00:00
Sumner Evans
ece84f2c45
Improve code formatting and fix a few typos in docs (#11221)
* Labeled a lot more code blocks with the appropriate type
* Fixed a couple of minor typos (missing/extraneous commas)

Signed-off-by: Sumner Evans <me@sumnerevans.com>
2021-11-01 11:35:55 +00:00
Erik Johnston
82d2168a15
Add metrics to the threadpools (#11178) 2021-11-01 11:21:36 +00:00
Sean Quah
2451003f6f
Test that ClientIpStore combines database and in-memory data correctly (#11179) 2021-11-01 11:20:54 +00:00
JohannesKleine
29ffd680bf
Stop synapse from saving messages in device_inbox for hidden devices. (#10097)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-11-01 10:40:41 +00:00
Brendan Abolivier
e320f5dba3
Deprecate user_may_create_room_with_invites (#11206) 2021-11-01 10:46:08 +01:00
Dirk Klimpel
bfd7a9b65c
Fix comments referencing v1.46.0 from PR #10969. (#11212)
#10969 was merged after 1.46.0rc1 was cut and will be included
in v1.47.0rc1 instead.
2021-10-29 13:43:51 -04:00
Brendan Abolivier
ad4eab9862
Add a module API method to retrieve state from a room (#11204) 2021-10-29 16:28:29 +00:00
Sean Quah
3ed17ff651
Clarify lack of Windows support in documentation (#11198) 2021-10-29 14:03:58 +01:00
Patrick Cloke
56e281bf6c
Additional type hints for relations database class. (#11205) 2021-10-28 14:35:12 -04:00
Rafael Gonçalves
0e16b418f6
Add knock information in admin exported data (#11171)
Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
2021-10-28 18:54:38 +01:00
Shay
e002faee01
Fetch verify key locally rather than trying to do so over federation if origin and host are the same. (#11129)
* add tests for fetching key locally

* add logic to check if origin server is same as host and fetch verify key locally rather than over federation

* add changelog

* slight refactor, add docstring, change changelog entry

* Make changelog entry one line

* remove verify_json_locally and push locality check to process_request, add function process_request_locally

* remove leftover code reference

* refactor to add common call to 'verify_json and associated handling code

* add type hint to process_json

* add some docstrings + very slight refactor
2021-10-28 10:27:17 -07:00
Brendan Abolivier
adc0d35b17
Add a ModuleApi method to update a user's membership in a room (#11147)
Co-authored-by: reivilibre <oliverw@matrix.org>
2021-10-28 16:45:53 +00:00
David Robertson
1bfd141205
Type hints for the remaining two files in synapse.http. (#11164)
* Teach MyPy that the sentinel context is False

This means that if `ctx: LoggingContextOrSentinel`
then `bool(ctx)` narrows us to `ctx:LoggingContext`, which is a really
neat find!

* Annotate RequestMetrics

- Raise errors for sentry if we use the sentinel context
- Ensure we don't raise an error and carry on, but not recording stats
- Include stack trace in the error case to lower Sean's blood pressure

* Make mypy pass for synapse.http.request_metrics

* Make synapse.http.connectproxyclient pass mypy

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-10-28 14:14:42 +01:00
Skyler Mäntysaari
a19bf32a03
docs/openid: Add Authentik documentation. (#11151) 2021-10-28 10:31:22 +00:00
Dan Callahan
0dffa9d0e0
Merge remote-tracking branch 'origin/develop' into shellcheck
Fixes a merge conflict with debian/changelog

Signed-off-by: Dan Callahan <danc@element.io>
2021-10-27 20:04:00 +01:00
reivilibre
75ca0a6168
Annotate log_function decorator (#10943)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-27 17:27:23 +01:00
Samuel Philipp
4e393af52f
Fixed config parse bug in review_recent_signups (#11191) 2021-10-27 17:25:18 +01:00
Patrick Cloke
19d5dc6931
Refactor Filter to handle fields according to data being filtered. (#11194)
This avoids filtering against fields which cannot exist on an
event source. E.g. presence updates don't have a room.
2021-10-27 11:26:30 -04:00
Dirk Klimpel
8d46fac98e
Delete messages from device_inbox table when deleting device (#10969)
Fixes: #9346
2021-10-27 16:01:18 +01:00
Patrick Cloke
a930da3291
Include the stable identifier for MSC3288. (#11187)
Includes both the stable and unstable identifier to store-invite
calls to the identity server. In the future we should remove the
unstable identifier.
2021-10-27 14:19:19 +00:00
Erik Johnston
179dc8ae9e Merge remote-tracking branch 'origin/release-v1.46' into develop 2021-10-27 14:45:40 +01:00
Erik Johnston
b615fc35d6 Update changelog with new changes 2021-10-27 13:22:48 +01:00
Sean Quah
f3a4be8700
Shut down the DNS threadpool (#11190)
The DNS threadpool must be explicitly stopped, otherwise Synapse will
hang indefinitely when asked to shut down.
2021-10-27 13:04:56 +01:00
Erik Johnston
72626b78ef
Fix thread BG update to not seq scan event_json (#11192)
For some reason the query optimiser decided to seq scan both tables,
rather than index scanning `event_json`.
2021-10-27 11:33:21 +00:00
Brendan Abolivier
c7a5e49664
Implement an on_new_event callback (#11126)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-10-26 15:17:36 +02:00
Sean Quah
cc75a6b1b2 1.46.0rc1 2021-10-26 14:04:51 +01:00
Erik Johnston
7004f43da1
Move DNS lookups into separate thread pool (#11177)
This is to stop large bursts of lookups starving out other users of the
thread pools.

Fixes #11049.
2021-10-26 13:45:38 +01:00
Patrick Cloke
d52c58dfa3
Add a background update for updating MSC3440 relation threads. (#11181) 2021-10-26 07:38:45 -04:00
Brendan Abolivier
8c8e36af0d
Document the version each module API method was added to Synapse (#11183) 2021-10-26 11:09:10 +02:00
Jason Robinson
63cbdd8af0
Enable changing user type via users admin API (#11174)
Users admin API can now also modify user
type in addition to allowing it to be
set on user creation.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-26 09:01:06 +00:00
Brendan Abolivier
c1510c97b5
Fix cyclic import in the module API (#11180)
Introduced in #10548

See https://github.com/matrix-org/synapse-email-account-validity/runs/3979337154?check_suite_focus=true for an example of a module's CI choking over this issue.
2021-10-25 16:45:19 +00:00
AndrewFerr
4387b791e0
Don't set new room alias before potential 403 (#10930)
Fixes: #10929 

Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-10-25 15:24:49 +01:00
Richard van der Hoff
da957a60e8
Ensure that we correctly auth events returned by send_join (#11012)
This is the final piece of the jigsaw for #9595. As with other changes before this one (eg #10771), we need to make sure that we auth the auth events in the right order, and actually check that their predecessors haven't been rejected.

To do this I've reused the existing code we use when persisting outliers elsewhere.

I've removed the code for attempting to fetch missing auth_events - the events should have been present in the send_join response, so the likely reason they are missing is that we couldn't verify them, so requesting them again is unlikely to help. Instead, we simply drop any state which relies on those auth events, as we do at a backwards-extremity. See also matrix-org/complement#216 for a test for this.
2021-10-25 15:21:09 +01:00
Sean Quah
85a09f8b8b
Fix module API's get_user_ip_and_agents function when run on workers (#11112) 2021-10-25 13:01:04 +01:00
Dan Callahan
1afc6ecae1
Changelog
Signed-off-by: Dan Callahan <danc@element.io>
2021-10-22 23:21:40 +01:00
Sean Quah
2b82ec425f
Add type hints for most HomeServer parameters (#11095) 2021-10-22 18:15:41 +01:00
Jason Robinson
b9ce53e878
Fix synapse.config module "read" command (#11145)
`synapse.config.__main__` has the possibility to read a config item. This can be used to conveniently also validate the config is valid before trying to start Synapse.

 The "read" command broke in https://github.com/matrix-org/synapse/pull/10916 as it now requires passing in "server.server_name" for example.

 Also made the read command optional so one can just call this with just the confirm file reference and get a "Config parses OK" if things are ok.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-22 12:00:52 +02:00
Dirk Klimpel
b0f03aeb6a
Add more information what happens when a user is deactivated (#11083) 2021-10-22 09:00:28 +01:00
Patrick Cloke
ba00e20234
Add a thread relation type per MSC3440. (#11088)
Adds experimental support for MSC3440's `io.element.thread` relation
type (and the aggregation for it).
2021-10-21 14:39:16 -04:00
David Robertson
2d91b6256e
Fix adding excluded users to the private room sharing tables when joining a room (#11143)
* We only need to fetch users in private rooms

* Filter out `user_id` at the top

* Discard excluded users in the top loop

We weren't doing this in the "First, if they're our user" branch so this
is a bugfix.

* The caller must check that `user_id` is included

This is in the docstring. There are two call sites:
- one in `_handle_room_publicity_change`, which explicitly checks before calling;
- and another in `_handle_room_membership_event`, which returns early if
  the user is excluded.

So this change is safe.

* Test joining a private room with an excluded user

* Tweak an existing test

* Changelog

* test docstring

* lint
2021-10-21 17:48:59 +01:00
Andrew Morgan
6408372234
Improve docstrings for methods related to sending EDUs to application services (#11138) 2021-10-21 17:42:25 +01:00
Patrick Cloke
0f9adc99ad
Add missing type hints to synapse.crypto. (#11146)
And require type hints for this module.
2021-10-21 13:07:07 +00:00
Richard van der Hoff
09eff1b3db
fix relative link in docker readme (#11144)
relative links don't work when it's on dockerhub.
2021-10-21 13:50:43 +01:00
Dirk Klimpel
ef7fe09778
Fix setting a user's external_id via the admin API returns 500 and deletes users existing external mappings if that external ID is already mapped (#11051)
Fixes #10846
2021-10-21 09:52:32 +01:00
Richard van der Hoff
57501d9194
Update sign_json to support inline key config (#11139)
It's been possible to configure a key inline in the homeserver.yaml since
13bc1e0746. Update `sign_json` to work with this.
2021-10-20 18:49:20 +01:00
Robert Edström
62db603fa0
Consider IP whitelist for identity server resolution (#11120)
Signed-off-by: Robert Edström <github@legogris.se>
2021-10-20 18:43:49 +01:00
Richard van der Hoff
0930e9ae12
Clean up _update_auth_events_and_context_for_auth (#11122)
Remove some redundant code, and generally simplify.
2021-10-20 18:22:40 +01:00
Aaron R
2c61a318cc
Show error when timestamp in seconds is provided to the /purge_media_cache API (#11101) 2021-10-20 14:41:48 +00:00
Travis Ralston
106d99b8cd
Remove false warning about copying the log config to a homeserver.yaml (#11092)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-10-20 11:48:15 +00:00
Brendan Abolivier
78d5896d19
Document the version of Synapse each module callback was introduced in (#11132)
* Mention callbacks introduced in v1.37.0

According to the documentation introduced in https://github.com/matrix-org/synapse/pull/10062

* Mention callbacks introduced in v1.39.0

According to https://github.com/matrix-org/synapse/pull/10386 and https://github.com/matrix-org/synapse/pull/9884

* Mention callbacks introduced in v1.42.0

According to https://github.com/matrix-org/synapse/pull/10524

* Mention callbacks introduced in v1.44.0 and v1.45.0

As per https://github.com/matrix-org/synapse/pull/10898, https://github.com/matrix-org/synapse/pull/10910 and https://github.com/matrix-org/synapse/pull/10894

* Mention callbacks introduced in v1.46.0

According to https://github.com/matrix-org/synapse/pull/10548
2021-10-20 11:04:27 +00:00
Patrick Cloke
0dd0c40329
Add missing type hints to event fetching. (#11121)
Updates the event rows returned from the database to be
attrs classes instead of dictionaries.
2021-10-19 14:29:03 +00:00
Andrew Morgan
5e0e683541
Fix instances of [example]{.title-ref} in the upgrade notes (#11118) 2021-10-19 14:13:56 +01:00
Richard van der Hoff
f3efa0036b
Move _persist_auth_tree into FederationEventHandler (#11115)
This is just a lift-and-shift, because it fits more naturally here. We do
rename it to `process_remote_join` at the same time though.
2021-10-19 10:24:09 +01:00
Richard van der Hoff
0170774b19
Rename _auth_and_persist_fetched_events (#11116)
... to `_auth_and_persist_outliers`, since that reflects its purpose better.
2021-10-19 10:23:55 +01:00
Richard van der Hoff
d85bc9a4a7
Include rejected status when we log events. (#11008)
If we find ourselves dealing with rejected events, we proably want to know
about it. Let's include it in the stringification of the event so that it gets
logged.
2021-10-19 10:21:50 +01:00
Patrick Cloke
3ab55d43bd
Add missing type hints to synapse.api. (#11109)
* Convert UserPresenceState to attrs.
* Remove args/kwargs from error classes and explicitly pass msg/errorcode.
2021-10-18 15:01:10 -04:00
Richard van der Hoff
cc33d9eee2
Check auth on received events' auth_events (#11001)
Currently, when we receive an event whose auth_events differ from those we expect, we state-resolve between the two state sets, and check that the event passes auth based on the resolved state.

This means that it's possible for us to accept events which don't pass auth at their declared auth_events (or where the auth events themselves were rejected), leading to problems down the line like #10083.

This change means we will:

 * ignore any events where we cannot find the auth events
 * reject any events whose auth events were rejected
 * reject any events which do not pass auth at their declared auth_events.

Together with a whole raft of previous work, this is a partial fix to #9595.

Fixes #6643.

Based on #11009.
2021-10-18 18:29:37 +01:00
Richard van der Hoff
a5d2ea3d08
Check *all* auth events for room id and rejection (#11009)
This fixes a bug where we would accept an event whose `auth_events` include
rejected events, if the rejected event was shadowed by another `auth_event`
with same `(type, state_key)`.

The approach is to pass a list of auth events into
`check_auth_rules_for_event` instead of a dict, which of course means updating
the call sites.

This is an extension of #10956.
2021-10-18 18:28:30 +01:00
Brendan Abolivier
73743b8ad1
Document Synapse's behaviour when dealing with multiple modules (#11096)
Document Synapse's behaviour when multiple modules register the same
callback/web resource/etc.

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-10-18 18:26:52 +02:00
Richard van der Hoff
e8f24b6c35
_run_push_actions_and_persist_event: handle no min_depth (#11014)
Make sure that we correctly handle rooms where we do not yet have a
`min_depth`, and also add some comments and logging.
2021-10-18 17:17:15 +01:00
Hillery Shay
7d70582eb0
Fix broken export-data admin command and add a test for it to CI (#11078)
Fix broken export-data admin command and add a test for it to CI
2021-10-18 08:14:12 -07:00
David Robertson
37b845dabc
Don't remove local users from dir when the leave their last room (#11103) 2021-10-18 13:20:04 +00:00
David Robertson
e09be0c87a
Correctly exclude users when making a room public or private (#11075)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-15 15:53:05 +01:00
Patrick Cloke
5573133348
Move experimental & retention config out of the server module. (#11070) 2021-10-15 14:30:48 +00:00
Sean Quah
6a67f3786a
Fix logging context warnings when losing replication connection (#10984)
Instead of triggering `__exit__` manually on the replication handler's
logging context, use it as a context manager so that there is an
`__enter__` call to balance the `__exit__`.
2021-10-15 13:10:58 +01:00
Dirk Klimpel
013e0f9cae
Update doc of the allowed characters for registration tokens (#11093)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-15 09:56:39 +00:00
Eric Eastwood
daf498e099
Fix 500 error on /messages when we accumulate more than 5 backward extremities (#11027)
Found while working on the Gitter backfill script and noticed
it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390

When there are more than 5 backward extremities for a given depth,
backfill will throw an error because we sliced the extremity list
to 5 but then try to iterate over the full list. This causes
us to look for state that we never fetched and we get a `KeyError`.

Before when calling `/messages` when there are more than 5 backward extremities:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render
    callback_return = await raw_callback_return
  File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET
    msgs = await self.pagination_handler.get_messages(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages
    await self.hs.get_federation_handler().maybe_backfill(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill
    return await self._maybe_backfill_inner(room_id, current_depth, limit)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner
    likely_extremeties_domains = get_domains_from_state(states[e_id])
KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl'
```
2021-10-14 18:53:45 -05:00
Patrick Cloke
efd0074ab7
Ensure each charset is attempted only once during media preview. (#11089)
There's no point in trying more than once since it is guaranteed to
continually fail.
2021-10-14 18:51:44 +00:00
Patrick Cloke
e2f0b49b3f
Attempt different character encodings when previewing a URL. (#11077)
This follows similar logic to BeautifulSoup where we attempt different
character encodings until we find one which works.
2021-10-14 10:17:20 -04:00
Patrick Cloke
1609ccf8fe
Fix-up some type hints in the relations tests. (#11076) 2021-10-14 09:19:35 -04:00
reivilibre
50d8601581
Add a test for a workaround concerning the behaviour of third-party rule modules and SynapseErrors. (#11071) 2021-10-14 14:14:15 +01:00
David Robertson
b3698f945c Synapse 1.45.0rc2 (2021-10-14)
==============================
 
 **Note:** This release candidate [fixes](https://github.com/matrix-org/synapse/issues/11053) the user directory [bug](https://github.com/matrix-org/synapse/issues/11025) present in 1.45.0rc1. However, the [performance issue](https://github.com/matrix-org/synapse/issues/11049) which appeared in v1.44.0 is yet to be resolved.
 
 Bugfixes
 --------
 
 - Fix a long-standing bug when using multiple event persister workers where events were not correctly sent down `/sync` due to a race. ([\#11045](https://github.com/matrix-org/synapse/issues/11045))
 - Fix a bug introduced in Synapse 1.45.0rc1 where the user directory would stop updating if it processed an event from a
   user not in the `users` table. ([\#11053](https://github.com/matrix-org/synapse/issues/11053))
 - Fix a bug introduced in Synapse v1.44.0 when logging errors during oEmbed processing. ([\#11061](https://github.com/matrix-org/synapse/issues/11061))
 
 Internal Changes
 ----------------
 
 - Add an 'approximate difference' method to `StateFilter`. ([\#10825](https://github.com/matrix-org/synapse/issues/10825))
 - Fix inconsistent behavior of `get_last_client_by_ip` when reporting data that has not been stored in the database yet. ([\#10970](https://github.com/matrix-org/synapse/issues/10970))
 - Fix a bug introduced in Synapse 1.21.0 that causes opentracing and Prometheus metrics for replication requests to be measured incorrectly. ([\#10996](https://github.com/matrix-org/synapse/issues/10996))
 - Ensure that cache config tests do not share state. ([\#11036](https://github.com/matrix-org/synapse/issues/11036))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmFoBHYACgkQkD7OEIo5
 3t2+wRAAlk2GR9Ezi8E/AG6f5Dsq0+guswHK9b34PtI0wcLQFgy76y200g8D/V0F
 XFdby4IZsSLVmP2EVF+vIaU0/zviClHR81b8pvNcIRxORBAhV6fH5JkCRPNAlurS
 aGnoKA1TSSg4HwNHn+dWyg7VCHl76oLEBhixRdz7aWPhLnCNkCM62E+jlwVq19BH
 lPIDcwEvtq2ovHIGyE/sTg6Shrf9ELy0mYMQpPJNfDmxqjWqRhKaSpx5JSEJnzih
 amONFY7Nc8kIUdP6JSR6105IhUDAnItScmyyUowyCvwtAQsXsgMd8Cs0FdiJ0JLy
 BtE9qsBmBpmVWjfk7ESXE1eTHqH9Eexsy5dcB43hiVZ3ihhCCLmtVANej3QaRnRL
 rYAiH+W9WK7n3189hzkmEG/BKk+zu65dLaHSMjBvEDMyiYLoPRlYnmunLeo6mESq
 HHa25o5WiY8+UQ4RgcYqScIqvlgFqODmZT21OlknZfupUd2kDCN7Ga7TY599sBp2
 XSGNB6l/eGWRWtOPcdrandsDnppcoYsfJkXIfAUIxwjnSN/RU4kDPDiYDXhU8d+D
 hW/f59XwfMy3D3zZI4a+pEY8ZhcSHTnLJni95UFyb/7zOwnT+pZXUfVX46727nxg
 imUfXsXIxUYQnDkAGBXdt7QMZPy4gvTGenllqaA987xjfd5/fKs=
 =TTLK
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmFoCZQACgkQkD7OEIo5
 3t2Ygw/9E48TTeW7F5j16aCVVmRYnlAoh+rBlrKiaIE4MSJtwI+TEdIIrwGm+JdB
 lh59N3YmkXrn2No4ew0UY9fjD8LgofX3GAmlP2vN/fHst9kfe/UtoIDHWdQG1Xhh
 xeHnt9vXirQMOLxapkDM4ymBW1WAQdvgdn3Ozy3hQ+ZUQiMqJg4CQVMb7mUpjdJz
 D6Jb0Ub5FGnCGcSrofBk9KvHSZQb9Qrp8Rz3kokib98/+eJGdJDXhf0Yg4junJDy
 mOvTAhbJYG9TS3taFajuw8IfcngJYyBMUOf7HlcqMtyEPdHFW2wClUHzeQIPOurU
 YQmpqpDO7LSrHhFOt3cdmIlCGpGxKgR9AIUn8Volb2pxwH2i/BA7ryP9ZKz9dht3
 VnPFLEHYumBYOAB05TsUkDlpr/0x3YS1JCgzHGABM8spsi8XO/ZPRkwpjdc/jSRB
 3A4I7WGpKddk7pY3mCOnDx8hm2asgTPgTHkvJr2v2wYndIHxW91BDKhhSZ4tzjIY
 aOr2CtJmoFb3Buek/s1S2AGVLRXMYa/IIRHl8FjmL7WhsqebJkcRfcaMfHHtCq5Y
 tLhK2CYavEOu8TrkFWyIPPCCn8Y6Y8qnfg8c8f1ncfHsVHE4sOs9hjsVVz+n4QBd
 LD+ICWBeLvpGQK+0cZu3QFtIyo6XKBJlYuAyLD5saNkuLnupA3Y=
 =A9jl
 -----END PGP SIGNATURE-----

Merge tag 'v1.45.0rc2' into develop

Synapse 1.45.0rc2 (2021-10-14)
==============================

**Note:** This release candidate [fixes](https://github.com/matrix-org/synapse/issues/11053) the user directory [bug](https://github.com/matrix-org/synapse/issues/11025) present in 1.45.0rc1. However, the [performance issue](https://github.com/matrix-org/synapse/issues/11049) which appeared in v1.44.0 is yet to be resolved.

Bugfixes
--------

- Fix a long-standing bug when using multiple event persister workers where events were not correctly sent down `/sync` due to a race. ([\#11045](https://github.com/matrix-org/synapse/issues/11045))
- Fix a bug introduced in Synapse 1.45.0rc1 where the user directory would stop updating if it processed an event from a
  user not in the `users` table. ([\#11053](https://github.com/matrix-org/synapse/issues/11053))
- Fix a bug introduced in Synapse v1.44.0 when logging errors during oEmbed processing. ([\#11061](https://github.com/matrix-org/synapse/issues/11061))

Internal Changes
----------------

- Add an 'approximate difference' method to `StateFilter`. ([\#10825](https://github.com/matrix-org/synapse/issues/10825))
- Fix inconsistent behavior of `get_last_client_by_ip` when reporting data that has not been stored in the database yet. ([\#10970](https://github.com/matrix-org/synapse/issues/10970))
- Fix a bug introduced in Synapse 1.21.0 that causes opentracing and Prometheus metrics for replication requests to be measured incorrectly. ([\#10996](https://github.com/matrix-org/synapse/issues/10996))
- Ensure that cache config tests do not share state. ([\#11036](https://github.com/matrix-org/synapse/issues/11036))
2021-10-14 11:42:16 +01:00
David Robertson
99a4e5222d
1.45.0rc2 2021-10-14 10:59:27 +01:00
Eric Eastwood
35d6b914eb
Resolve and share state_groups for all historical events in batch (MSC2716) (#10975)
Resolve and share `state_groups` for all historical events in batch.  This also helps for showing the appropriate avatar/displayname in Element and will work whenever `/messages` has one of the historical messages as the first message in the batch.

This does have the flaw where if you just insert a single historical event somewhere, it probably won't resolve the state correctly from `/messages` or `/context` since it will grab a non historical event above or below with resolved state which never included the historical state back then. For the same reasions, this also does not work in Element between the transition from actual messages to historical messages. In the Gitter case, this isn't really a problem since all of the historical messages are in one big lump at the beginning of the room.

For a future iteration, might be good to look at `/messages` and `/context` to additionally add the `state` for any historical messages in that batch.

---

How are the `state_groups` shared? To illustrate the `state_group` sharing, see this example:


**Before** (new `state_group` for every event 😬, very inefficient):
```
# Tests from https://github.com/matrix-org/complement/pull/206
$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events
create_new_client_event m.room.member event=$_JXfwUDIWS6xKGG4SmZXjSFrizhARM7QblhATVWWUcA state_group=None
create_new_client_event org.matrix.msc2716.insertion event=$1ZBfmBKEjg94d-vGYymKrVYeghwBOuGJ3wubU1-I9y0 state_group=9
create_new_client_event org.matrix.msc2716.insertion event=$Mq2JvRetTyclPuozRI682SAjYp3GqRuPc8_cH5-ezPY state_group=10
create_new_client_event m.room.message event=$MfmY4rBQkxrIp8jVwVMTJ4PKnxSigpG9E2cn7S0AtTo state_group=11
create_new_client_event m.room.message event=$uYOv6V8wiF7xHwOMt-60d1AoOIbqLgrDLz6ZIQDdWUI state_group=12
create_new_client_event m.room.message event=$PAbkJRMxb0bX4A6av463faiAhxkE3FEObM1xB4D0UG4 state_group=13
create_new_client_event org.matrix.msc2716.batch event=$Oy_S7AWN7rJQe_MYwGPEy6RtbYklrI-tAhmfiLrCaKI state_group=14
```

**After** (all events in batch sharing `state_group=10`) (the base insertion event has `state_group=8` which matches the `prev_event` we're inserting next to):

```
# Tests from https://github.com/matrix-org/complement/pull/206
$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events
create_new_client_event m.room.member event=$PWomJ8PwENYEYuVNoG30gqtybuQQSZ55eldBUSs0i0U state_group=None
create_new_client_event org.matrix.msc2716.insertion event=$e_mCU7Eah9ABF6nQU7lu4E1RxIWccNF05AKaTT5m3lw state_group=9
create_new_client_event org.matrix.msc2716.insertion event=$ui7A3_GdXIcJq0C8GpyrF8X7B3DTjMd_WGCjogax7xU state_group=10
create_new_client_event m.room.message event=$EnTIM5rEGVezQJiYl62uFBl6kJ7B-sMxWqe2D_4FX1I state_group=10
create_new_client_event m.room.message event=$LGx5jGONnBPuNhAuZqHeEoXChd9ryVkuTZatGisOPjk state_group=10
create_new_client_event m.room.message event=$wW0zwoN50lbLu1KoKbybVMxLbKUj7GV_olozIc5i3M0 state_group=10
create_new_client_event org.matrix.msc2716.batch event=$5ZB6dtzqFBCEuMRgpkU201Qhx3WtXZGTz_YgldL6JrQ state_group=10
```
2021-10-13 17:44:00 -05:00
David Robertson
404444260a
Fix upgrade dead links (#11069) 2021-10-13 14:32:43 +01:00
David Robertson
317e9e415c
Rearrange the user_directory's _handle_deltas function (#11035)
* Pull out `_handle_room_membership_event`
* Discard excluded users early
* Rearrange logic so the change is membership is effectively switched over. See PR for rationale.
2021-10-13 12:50:00 +00:00
Sean Quah
b59f3281d5
Remove dead code from MediaFilePaths (#11056) 2021-10-13 13:41:24 +01:00
David Robertson
b3e9b00fb2
Merge remote-tracking branch 'origin/release-v1.45' into develop 2021-10-13 12:46:30 +01:00
Patrick Cloke
1f9d0b8a7a
Add type hints to synapse.events.*. (#11066)
Except `synapse/events/__init__.py`, which will be done in a follow-up.
2021-10-13 07:24:07 -04:00
Azrenbeth
cdd308845b
Port the Password Auth Providers module interface to the new generic interface (#10548)
Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-13 11:21:52 +00:00
Patrick Cloke
732bbf6737
Be more lenient when parsing the version for oEmbed responses. (#11065) 2021-10-13 07:00:07 -04:00
David Robertson
b83e822556
Stop user directory from failing if it encounters users not in the users table. (#11053)
The following scenarios would halt the user directory updater:

- user joins room
- user leaves room
- user present in room which switches from private to public, or vice versa.

for two classes of users:

- appservice senders
- users missing from the user table.

If this happened, the user directory would be stuck, unable to make forward progress.

Exclude both cases from the user directory, so that we ignore them.

Co-authored-by: Eric Eastwood <erice@element.io>
Co-authored-by: reivilibre <oliverw@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-13 09:38:22 +00:00
reivilibre
2a2b189130
Mark Module API error imports as re-exported and mark Synapse as containing type annotations (#11054) 2021-10-13 08:42:41 +01:00
Patrick Cloke
8711e15734
Always dump logs from trial during CI. (#11068)
Instead of only dumping them if trial passes.
2021-10-12 20:09:49 +00:00
Dirk Klimpel
988de0afb0
Simplify the user admin API tests (#11048) 2021-10-12 20:38:48 +01:00
Hillery Shay
5dcacdf6d1
Add support for ubuntu 21.10 "Impish Indri" (#11024)
* support ubuntu 21.10 indri

* add changelog

* update to correct codename

Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>

Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
2021-10-12 11:29:39 -07:00
Patrick Cloke
9abc5f2a05 Merge remote-tracking branch 'origin/release-v1.45' into develop 2021-10-12 14:21:05 -04:00
Sean Quah
84f5d83257
Add tests for MediaFilePaths (#11057) 2021-10-12 18:19:35 +01:00
Sean Quah
8eaffe013c
Update _wrap_in_base_path type hints to preserve function arguments (#11055) 2021-10-12 18:19:21 +01:00
Patrick Cloke
1db9282dfa
Fix formatting string when oEmbed errors occur. (#11061) 2021-10-12 17:15:42 +00:00
Erik Johnston
77ea03086c Merge remote-tracking branch 'origin/release-v1.45' into develop 2021-10-12 14:41:13 +01:00
Erik Johnston
333d6f4e84
Fix race in MultiWriterIdGenerator (#11045)
The race allowed the current position to advance too far when stream IDs
are still being persisted.

This happened when it received a new stream ID from a remote write
between a new stream ID being allocated and it being added to the set of
unpersisted stream IDs.

Fixes #9424.
2021-10-12 14:27:09 +01:00
Patrick Cloke
5c35074d85
Reset global cache state before cache tests. (#11036)
This reverts #11019 and structures the code a bit more like it was before #10985.

The global cache state must be reset before running the tests since other test
cases might have configured caching (and thus touched the global state).
2021-10-12 12:55:33 +00:00
Sean Quah
36224e056a
Add type hints to synapse.storage.databases.main.client_ips (#10972) 2021-10-12 13:50:34 +01:00
Brendan Abolivier
a18c568516 Synapse 1.45.0rc1 (2021-10-12)
==============================
 
 **Note:** We are aware of [a performance issue](https://github.com/matrix-org/synapse/issues/11049) introduced in Synapse v1.44.0, as well as [a bug](https://github.com/matrix-org/synapse/issues/11025) with the user directory when using application services. While this release candidate doesn't fix either of those issues, a second release candidate is expected to come out in a few days to address them.
 
 Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450) for more information.
 
 Features
 --------
 
 - Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655))
 - Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822))
 - Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894))
 - Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910))
 - Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954))
 - Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028))
 
 Bugfixes
 --------
 
 - Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922))
 - Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924))
 - Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](https://github.com/matrix-org/synapse/issues/10927))
 - Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947))
 - Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956))
 - Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960))
 - Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](https://github.com/matrix-org/synapse/issues/10962))
 - Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981))
 - Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](https://github.com/matrix-org/synapse/issues/10982))
 - Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](https://github.com/matrix-org/synapse/issues/10995))
 - Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002))
 - Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](https://github.com/matrix-org/synapse/issues/11003))
 - Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](https://github.com/matrix-org/synapse/issues/11042))
 - Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](https://github.com/matrix-org/synapse/issues/10877))
 
 Improved Documentation
 ----------------------
 
 - Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971))
 - Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973))
 - Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990))
 - Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991))
 
 Internal Changes
 ----------------
 
 - Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888))
 - Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892))
 - Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895))
 - Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902))
 - Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903))
 - Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915))
 - Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916), [\#10959](https://github.com/matrix-org/synapse/issues/10959), [\#10985](https://github.com/matrix-org/synapse/issues/10985))
 - Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926), [\#10940](https://github.com/matrix-org/synapse/issues/10940), [\#10986](https://github.com/matrix-org/synapse/issues/10986), [\#10987](https://github.com/matrix-org/synapse/issues/10987), [\#10988](https://github.com/matrix-org/synapse/issues/10988), [\#11010](https://github.com/matrix-org/synapse/issues/11010), [\#11011](https://github.com/matrix-org/synapse/issues/11011))
 - Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934))
 - Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935))
 - Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936))
 - Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939))
 - Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945))
 - Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958))
 - Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961))
 - Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](https://github.com/matrix-org/synapse/issues/10963))
 - Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966))
 - Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974))
 - Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983))
 - Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992))
 - Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993))
 - Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994))
 - Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004))
 - Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005))
 - Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006))
 - Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017))
 - Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019))
 - Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021))
 - Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023))
 - When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034))
 - Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmFlaJYACgkQHgFcFF8Z
 Fs0OBQ//QuIZHqwajqu6mVdcm5/GzYzerMVqcxByOsGX2Xeq0am1AnAG3MFH6TT/
 f+e7T0ZIed7KTQFGKxyNX9nOi1RmQIaypBn66jt+tMzaCqMmlto3s0w2M5C+x4l4
 lf9I0Shk/0wvrklc1GbmM89wB8eLmSz+UsrNK/ksMsevlP0K8GxA7HE3TrzDAth+
 cLJOine4yfW4VlHus00u09E/2uILOE2wQ8V7YSpDMYOZc0bp0QEB5TGaab+4ESLz
 JVLh1j3+dAmYPAAZMyWAp3PGN430NlNEwqomB1TYPdDRJ5ohlkWe22ACF6PD8eq+
 TlnrANlo2CThZkTFQAecYkVPziETQv25BkGRho1IoBOfS8pTIDMoVwrrTc3okGfj
 L6XlB1Rn+WauHXPq7EBJZnk9iw8qD0Pq8arMZeKYmQ1OMhUQmYTwBOj82fFU+jfu
 a4v2rrMSVf3lmP9nAEHozGN0MAGsD0in+uvQuehdqVmKUbYUD0AWtHRZlO0ca4fH
 BZxE2/CCJWaK/2IdH4I4S1l/AF4kdX0K0tcmAiwUKrm5jSC2Ww8BWeEqjwF9Bd+s
 e1tF2pjw/VXjhAMmLIJVLEysb0yrw+QyZHWJ9fQP1KgaCcSwQjMgUBVrM3gcliIS
 +/qjTl1QgkdTVHis6r5rdWBZrVJ/CeLFUz7pATEfoomf/EiEMHQ=
 =VnsK
 -----END PGP SIGNATURE-----

Merge tag 'v1.45.0rc1' into develop

Synapse 1.45.0rc1 (2021-10-12)
==============================

**Note:** We are aware of [a performance issue](https://github.com/matrix-org/synapse/issues/11049) introduced in Synapse v1.44.0, as well as [a bug](https://github.com/matrix-org/synapse/issues/11025) with the user directory when using application services. While this release candidate doesn't fix either of those issues, a second release candidate is expected to come out in a few days to address them.

Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450) for more information.

Features
--------

- Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655))
- Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822))
- Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894))
- Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910))
- Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954))
- Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028))

Bugfixes
--------

- Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922))
- Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924))
- Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](https://github.com/matrix-org/synapse/issues/10927))
- Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947))
- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956))
- Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960))
- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](https://github.com/matrix-org/synapse/issues/10962))
- Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981))
- Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](https://github.com/matrix-org/synapse/issues/10982))
- Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](https://github.com/matrix-org/synapse/issues/10995))
- Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002))
- Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](https://github.com/matrix-org/synapse/issues/11003))
- Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](https://github.com/matrix-org/synapse/issues/11042))
- Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](https://github.com/matrix-org/synapse/issues/10877))

Improved Documentation
----------------------

- Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971))
- Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973))
- Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990))
- Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991))

Internal Changes
----------------

- Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888))
- Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892))
- Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895))
- Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902))
- Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903))
- Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915))
- Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916), [\#10959](https://github.com/matrix-org/synapse/issues/10959), [\#10985](https://github.com/matrix-org/synapse/issues/10985))
- Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926), [\#10940](https://github.com/matrix-org/synapse/issues/10940), [\#10986](https://github.com/matrix-org/synapse/issues/10986), [\#10987](https://github.com/matrix-org/synapse/issues/10987), [\#10988](https://github.com/matrix-org/synapse/issues/10988), [\#11010](https://github.com/matrix-org/synapse/issues/11010), [\#11011](https://github.com/matrix-org/synapse/issues/11011))
- Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934))
- Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935))
- Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936))
- Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939))
- Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945))
- Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958))
- Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961))
- Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](https://github.com/matrix-org/synapse/issues/10963))
- Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966))
- Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974))
- Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983))
- Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992))
- Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993))
- Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994))
- Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004))
- Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005))
- Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006))
- Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017))
- Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019))
- Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021))
- Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023))
- When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034))
- Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043))
2021-10-12 12:33:41 +01:00
Sean Quah
b8b905c4ea
Fix inconsistent behavior of get_last_client_by_ip (#10970)
Make `get_last_client_by_ip` return the same dictionary structure
regardless of whether the data has been persisted to the database.

This change will allow slightly cleaner type hints to be applied later
on.
2021-10-12 11:24:05 +01:00
Sean Quah
6b18eb4430
Fix opentracing and Prometheus metrics for replication requests (#10996)
This commit fixes two bugs to do with decorators not instrumenting
`ReplicationEndpoint`'s `send_request` correctly. There are two
decorators on `send_request`: Prometheus' `Gauge.track_inprogress()`
and Synapse's `opentracing.trace`.

`Gauge.track_inprogress()` does not have any support for async
functions when used as a decorator. Since async functions behave like
regular functions that return coroutines, only the creation of the
coroutine was covered by the metric and none of the actual body of
`send_request`.

`Gauge.track_inprogress()` returns a regular, non-async function
wrapping `send_request`, which is the source of the next bug.
The `opentracing.trace` decorator would normally handle async functions
correctly, but since the wrapped `send_request` is a non-async function,
the decorator ends up suffering from the same issue as
`Gauge.track_inprogress()`: the opentracing span only measures the
creation of the coroutine and none of the actual function body.

Using `Gauge.track_inprogress()` as a context manager instead of a
decorator resolves both bugs.
2021-10-12 11:23:46 +01:00
Brendan Abolivier
8c5255b664 1.45.0rc1 2021-10-12 10:47:15 +01:00
reivilibre
406f7bfa17
Add an approximate difference method to StateFilters (#10825) 2021-10-12 10:44:59 +01:00
David Robertson
e0f11ae4a5
disallow-untyped-defs for synapse.push (#11023) 2021-10-11 17:42:10 +01:00
reivilibre
5e29d417fc
Include the requirements for [mypy,lint] in [dev] (#11034) 2021-10-11 16:34:31 +01:00
reivilibre
3828dd819b
Pass through SynapseErrors that are raised from experimental check_event_allowed callback of the module API (#11042)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-11 16:29:02 +01:00
reivilibre
4c838112dc
Remove unnecessary list comprehension in synapse_port_db to fix linting in CI (#11043) 2021-10-11 16:28:29 +01:00
reivilibre
b742cb2e4a
Release script improvements (#10966)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-11 14:48:38 +01:00
Eric Eastwood
a7d22c36db
Refactor MSC2716 /batch_send endpoint into separate handler functions (#10974) 2021-10-08 18:35:00 -05:00
Patrick Cloke
1b112840d2
Autodiscover oEmbed endpoint from returned HTML (#10822)
Searches the returned HTML for an oEmbed endpoint using the
autodiscovery mechanism (`<link rel=...>`), and will request it
to generate the preview.
2021-10-08 14:14:42 -04:00
Olivier Wilkinson (reivilibre)
593eeac19e Revert accidental push to develop. 2021-10-08 17:17:22 +01:00
Olivier Wilkinson (reivilibre)
d51a340019 Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2021-10-08 17:12:40 +01:00
David Robertson
51a5da74cc
Annotate synapse.storage.util (#10892)
Also mark `synapse.streams` as having has no untyped defs

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-10-08 14:25:16 +00:00
David Robertson
797ee7812d
Relax ignore-missing-imports for modules that have stubs now and update mypy (#11006)
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py

Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.

The rest of this change consists of fixups to make the newer mypy + stubs pass CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-08 14:49:41 +01:00
David Robertson
670a8d9a1e
Fix overwriting profile when making room public (#11003)
This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
2021-10-08 12:52:48 +01:00
Patrick Cloke
eb9ddc8c2e
Remove the deprecated BaseHandler. (#11005)
The shared ratelimit function was replaced with a dedicated
RequestRatelimiter class (accessible from the HomeServer
object).

Other properties were copied to each sub-class that inherited
from BaseHandler.
2021-10-08 07:44:43 -04:00
Sean Quah
49a683d871
Fix long-standing bug where ReadWriteLock could drop logging contexts (#10993)
Use `PreserveLoggingContext()` to ensure that logging contexts are not
lost when exiting a read/write lock.

When exiting a read/write lock, callbacks on a `Deferred` are triggered
as a signal to any waiting coroutines. Any waiting coroutine that
becomes runnable is likely to follow the Synapse logging context rules
and will restore its own logging context, then either run to completion
or await another `Deferred`, resetting the logging context in the
process.
2021-10-08 12:27:16 +01:00
Nick Barrett
bb228f3523
Include exception in json logging (#11028) 2021-10-08 13:08:25 +02:00
Erik Johnston
0b4d5ce5e3
Fix CI to run the unit tests without optional deps (#11017)
This also turns off calculating code coverage, as we didn't use it and it was a lot of noise
2021-10-08 10:05:48 +01:00
David Robertson
e79ee48313
disallow-untyped-defs for synapse.server_notices (#11021) 2021-10-07 19:55:15 +01:00
Patrick Cloke
7301019d48
Ensure each cache config test uses separate state. (#11019)
Hopefully this fixes these tests sometimes failing in CI.
2021-10-07 09:38:31 -04:00
David Robertson
e0bf34dada
Don't alter directory entries for local users when setting a per-room nickname (#11002)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-07 13:26:11 +01:00
Richard van der Hoff
96fe77c254
Improve the logging in _auth_and_persist_outliers (#11010)
Include the event ids being peristed
2021-10-07 11:43:25 +00:00
Richard van der Hoff
86af6b2f0e
Add a comment in _process_received_pdu (#11011) 2021-10-07 12:20:03 +01:00
Brendan Abolivier
52aefd5086
Catch AttributeErrors when calling registerProducer (#10995)
Looks like the wrong exception type was caught in #10932.
2021-10-07 10:37:10 +00:00
David Robertson
f563676c09
disallow-untyped-defs for synapse.state (#11004)
* `disallow-untyped-defs` for `synapse.state`

Much smaller than I was expecting!
2021-10-06 17:55:25 +00:00
Andrew Morgan
e564bdd127
Add content to the Synapse documentation intro page (#10990)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-06 17:09:35 +00:00
Brendan Abolivier
4e51621064
Add a spamchecker method to allow or deny 3pid invites (#10894)
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.

Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
2021-10-06 17:18:13 +02:00
Patrick Cloke
f4b1a9a527
Require direct references to configuration variables. (#10985)
This removes the magic allowing accessing configurable
variables directly from the config object. It is now required
that a specific configuration class is used (e.g. `config.foo`
must be replaced with `config.server.foo`).
2021-10-06 10:47:41 -04:00
Brendan Abolivier
829f2a82b0
Add a spamchecker callback to allow or deny room joins (#10910)
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-10-06 14:32:16 +00:00
Brendan Abolivier
b0460936c8
Add the synapse-core team as code owners (#10994)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-10-06 16:03:17 +02:00
David Robertson
370bca32e6
Don't drop user dir deltas when server leaves room (#10982)
Fix a long-standing bug where a batch of user directory changes would be
silently dropped if the server left a room early in the batch.

* Pull out `wait_for_background_update` in tests

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-06 12:56:45 +00:00
Max Kratz
38b7db5885
Updated development doc on samling environment for testing. (#10973) 2021-10-06 07:20:41 -04:00
Nick Barrett
c80878d22a
Add --run-background-updates option to update_database script. (#10954)
Signed-off-by: Nick Barrett <nick@beeper.com>
2021-10-06 11:26:18 +01:00
David Robertson
f8d0f72b27
More types for synapse.util, part 1 (#10888)
The following modules now pass `disallow_untyped_defs`:

* synapse.util.caches.cached_call 
* synapse.util.caches.lrucache
* synapse.util.caches.response_cache 
* synapse.util.caches.stream_change_cache
* synapse.util.caches.ttlcache pass
* synapse.util.daemonize
* synapse.util.patch_inline_callbacks pass `no-untyped-defs`
* synapse.util.versionstring

Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch #10847
2021-10-06 11:20:49 +01:00
Max Kratz
6744273f0b
Remove "reference" wording according Synapse homeserver (#10971) 2021-10-06 10:05:07 +00:00
David Robertson
4f00432ce1
Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981)
There are two steps to rebuilding the user directory:

1. a scan over rooms, followed by
2. a scan over local users.

The former reads avatars and display names from the `room_memberships`
table and therefore contains potentially private avatars and
display names. The latter reads from the the `profiles` table which only
contains public data; moreover it will overwrite any private profiles
that the rooms scan may have written to the user directory. This means
that the rebuild could leak private user while the rebuild was in
progress, only to later cover up the leaks once the rebuild had completed.

This change skips over local users when writing user_directory rows
when scanning rooms. Doing so means that it'll take longer for a rebuild
to make local users searchable, which is unfortunate. I think a future
PR can improve this by swapping the order of the two steps above. (And
indeed there's more to do here, e.g. copying from `profiles` without
going via Python.)

Small tidy-ups while I'm here:

* Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`.
* Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`.
* Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-05 18:35:25 +01:00
Eric Eastwood
392863fbf1
Fix logic flaw preventing tracking of MSC2716 events in existing room versions (#10962)
We correctly allowed using the MSC2716 batch endpoint for
the room creator in existing room versions but accidentally didn't track
the events because of a logic flaw.

This prevented you from connecting subsequent chunks together because it would
throw the unknown batch ID error.

We only want to process MSC2716 events when:

 - The room version supports MSC2716
 - Any room where the homeserver has the `msc2716_enabled` experimental feature enabled and the event is from the room creator
2021-10-05 11:51:57 -05:00
Richard van der Hoff
3a5b0cbe7a
Ensure that we reject events which use rejected events for auth (#10956)
When we consider whether to accept events, we should not accept those which
depend on rejected events for their auth events.

This (together with earlier changes such as
https://github.com/matrix-org/synapse/pull/10771 and
https://github.com/matrix-org/synapse/pull/10896) forms a partial fix to
https://github.com/matrix-org/synapse/issues/9595. There still remain code
paths where we do not check the `auth_events` at all.
2021-10-05 13:23:29 +01:00
Richard van der Hoff
787af4a106
Host cache_joined_hosts_for_event to caller (#10986)
`_check_event_auth` is only called in two places, and only one of those sets
`send_on_behalf_of`. Warming the cache isn't really part of auth anyway, so
moving it out makes a lot more sense.
2021-10-05 13:01:41 +01:00