MirrorHub/synapse
0
0
Fork 1
mirror of https://mau.dev/maunium/synapse.git synced 2024-11-19 00:11:47 +01:00
Code Issues Projects Releases Wiki Activity
16569 commits 5 branches 731 tags 91 MiB
Commit graph

11639 commits

Author SHA1 Message Date
Richard van der Hoff
f84700fba8 Pass room version object into FederationClient.get_pdu 2020-02-05 17:25:46 +00:00
Richard van der Hoff
577f460369
Merge pull request #6840 from matrix-org/rav/federation_client_async 
Port much of `synapse.federation.federation_client` to async/await
 2020-02-05 16:56:39 +00:00
Richard van der Hoff
6bbd890f05 make FederationClient._do_send_invite async 2020-02-05 15:50:31 +00:00
Richard van der Hoff
146fec0820
Apply suggestions from code review 
Co-Authored-By: Erik Johnston <erik@matrix.org>
 2020-02-05 15:47:00 +00:00
Erik Johnston
a58860e480
Check sender_key matches on inbound encrypted events. (#6850) 
If they don't then the device lists are probably out of sync.
 2020-02-05 14:02:39 +00:00
Hubert Chathi
60d0672426 Merge pull request #6844 from matrix-org/uhoreg/cross_signing_fix_device_fed 
add device signatures to device key query results
 2020-02-05 10:54:49 +00:00
Michael Kaye
a831d2e4e3
Reduce performance logging to DEBUG (#6833) 
* Reduce tnx performance logging to DEBUG
* Changelog.d
 2020-02-05 08:57:37 +00:00
Richard van der Hoff
d88e0ec080
Database updates to populate rooms.room_version (#6847) 
We're going to need this so that we can figure out how to handle redactions when fetching events from the database.
 2020-02-04 21:31:08 +00:00
Erik Johnston
6475382d80
Fix detecting unknown devices from remote encrypted events. (#6848) 
We were looking at the wrong event type (`m.room.encryption` vs
`m.room.encrypted`).

Also fixup the duplicate `EvenTypes` entries.

Introduced in #6776.
 2020-02-04 17:25:54 +00:00
Hubert Chathi
74bf3fdbb9
Merge pull request #6844 from matrix-org/uhoreg/cross_signing_fix_device_fed 
add device signatures to device key query results
 2020-02-04 12:03:54 -05:00
Richard van der Hoff
5ef91b96f1 Merge remote-tracking branch 'origin/develop' into rav/federation_client_async 2020-02-04 12:07:05 +00:00
Richard van der Hoff
c7d6d5c69e
Merge pull request #6837 from matrix-org/rav/federation_async 
Port much of `synapse.handlers.federation` to async/await.
 2020-02-04 12:06:18 +00:00
Hubert Chathi
23d8a55c7a add device signatures to device key query results 2020-02-04 00:13:12 -05:00
Richard van der Hoff
ea23210b2d make FederationClient.send_invite async 2020-02-03 22:29:49 +00:00
Richard van der Hoff
6deeefb68c make FederationClient.get_missing_events async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
abadf44eb2 make FederationClient._do_send_leave async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
e88b90aaeb make FederationClient.send_leave.send_request async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
638001116d make FederationClient._do_send_join async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
3960527c2e make FederationClient.send_join.send_request async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
ad09ee9262 make FederationClient.make_membership_event.send_request async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
1330c311b7 make FederationClient._try_destination_list async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
a46fabf17b make FederationClient.send_leave async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
8af9f11bea make FederationClient.send_join async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
3f11cbb404 make FederationClient.make_membership_event async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
24d814ca23 make FederationClient.get_event_auth async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
d73683c363 make FederationClient.get_room_state_ids async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
0cb0c7bcd5 make FederationClient.get_pdu async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
0536d0c9be make FederationClient.backfill async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
5d17c31596 make FederationHandler.send_invite async 2020-02-03 22:28:11 +00:00
Richard van der Hoff
e81c093974 make FederationHandler.on_get_missing_events async 2020-02-03 19:15:08 +00:00
Erik Johnston
b9391c9575
Add typing to SyncHandler (#6821) 
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-02-03 18:05:44 +00:00
Erik Johnston
ae5b3104f0
Fix stacktraces when using ObservableDeferred and async/await (#6836) 2020-02-03 17:10:54 +00:00
Richard van der Hoff
f64c96662e make FederationHandler.user_joined_room async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
52642860da make FederationHandler._clean_room_for_join async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
814cc00cb9 make FederationHandler._notify_persisted_event async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
05299599b6 make FederationHandler.persist_events_and_notify async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
3b7e0e002b make FederationHandler._make_and_verify_event async 2020-02-03 16:22:30 +00:00
Richard van der Hoff
4286e429a7 make FederationHandler.do_remotely_reject_invite async 2020-02-03 16:19:18 +00:00
Richard van der Hoff
c3f296af32 make FederationHandler._check_for_soft_fail async 2020-02-03 16:16:31 +00:00
Richard van der Hoff
dbdf843012 make FederationHandler._persist_auth_tree async 2020-02-03 16:14:58 +00:00
Richard van der Hoff
ebd6a15af3 make FederationHandler.do_invite_join async 2020-02-03 16:13:13 +00:00
Richard van der Hoff
94f7b4cd54 make FederationHandler.on_event_auth async 2020-02-03 16:06:46 +00:00
Richard van der Hoff
863087d186 make FederationHandler.on_exchange_third_party_invite_request async 2020-02-03 16:02:50 +00:00
Richard van der Hoff
957129f4a7 make FederationHandler.construct_auth_difference async 2020-02-03 16:00:46 +00:00
Richard van der Hoff
0d5f2f4bb0 make FederationHandler._update_context_for_auth_events async 2020-02-03 15:55:35 +00:00
Richard van der Hoff
a25ddf26a3 make FederationHandler._update_auth_events_and_context_for_auth async 2020-02-03 15:53:54 +00:00
Richard van der Hoff
bc9b75c6f0 make FederationHandler.do_auth async 2020-02-03 15:51:24 +00:00
Richard van der Hoff
8033b257a7 make FederationHandler._prep_event async 2020-02-03 15:49:32 +00:00
Richard van der Hoff
1cdc253e0a make FederationHandler._handle_new_event async 2020-02-03 15:48:33 +00:00
Richard van der Hoff
c556ed9e15 make FederationHandler._handle_new_events async 2020-02-03 15:43:51 +00:00
Richard van der Hoff
6e89ec5e32 make FederationHandler.on_make_leave_request async 2020-02-03 15:40:41 +00:00
Richard van der Hoff
d184cbc031 make FederationHandler.on_send_leave_request async 2020-02-03 15:39:24 +00:00
Richard van der Hoff
98681f90cb make FederationHandler.on_make_join_request async 2020-02-03 15:38:02 +00:00
Richard van der Hoff
af8ba6b525 make FederationHandler.on_invite_request async 2020-02-03 15:33:42 +00:00
Richard van der Hoff
7571bf86f0 make FederationHandler.on_send_join_request async 2020-02-03 15:32:48 +00:00
Richard van der Hoff
b3e44f0bdf make FederationHandler.on_query_auth async 2020-02-03 15:30:23 +00:00
Andrew Morgan
370080531e
Allow URL-encoded user IDs on user admin api paths (#6825) 2020-02-03 13:18:42 +00:00
Richard van der Hoff
b0d112e78b
Fix room_version in on_invite_request flow (#6827) 
I messed this up a bit in #6805, but fortunately we weren't actually doing
anything with the room_version so it didn't matter that it was a str not a RoomVersion.
 2020-02-03 13:15:23 +00:00
Erik Johnston
ac0d45b78b 1.10.0rc1 2020-01-31 15:35:37 +00:00
Erik Johnston
83b0ea047b
Fix deleting of stale marker for device lists (#6819) 
We were in fact only deleting stale marker when we got an incremental
update, rather than when we did a full resync.
 2020-01-31 14:04:15 +00:00
Richard van der Hoff
7f93eb1903
pass room_version into compute_event_signature (#6807) 2020-01-31 13:47:43 +00:00
Richard van der Hoff
a5afdd15e5
Merge pull request #6806 from matrix-org/rav/redact_changes/3 
Pass room_version into add_hashes_and_signatures
 2020-01-31 10:57:03 +00:00
Richard van der Hoff
160522e32c
Merge pull request #6820 from matrix-org/rav/get_room_version_id 
Make `get_room_version` return a RoomVersion object
 2020-01-31 10:56:42 +00:00
Richard van der Hoff
08f41a6f05 Add get_room_version method 
So that we can start factoring out some of this boilerplatey boilerplate.
 2020-01-31 10:28:15 +00:00
Richard van der Hoff
d7bf793cc1 s/get_room_version/get_room_version_id/ 
... to make way for a forthcoming get_room_version which returns a RoomVersion
object.
 2020-01-31 10:06:21 +00:00
Erik Johnston
7d846e8704
Fix bug with getting missing auth event during join 500'ed (#6810) 2020-01-31 09:49:13 +00:00
Richard van der Hoff
2a81393a4b Pass room_version into add_hashes_and_signatures 2020-01-30 22:15:50 +00:00
Richard van der Hoff
54f3f369bd Pass room_version into create_local_event_from_event_dict 2020-01-30 22:15:50 +00:00
Richard van der Hoff
ef6bdafb29 Store the room version in EventBuilder 2020-01-30 22:15:50 +00:00
Richard van der Hoff
46a446828d
pass room version into FederationHandler.on_invite_request (#6805) 2020-01-30 22:13:02 +00:00
Erik Johnston
e0992fcc5b
Log when we delete room in bg update (#6816) 2020-01-30 17:55:34 +00:00
Richard van der Hoff
184303b865
MSC2260: Block direct sends of m.room.aliases events (#6794) 
as per MSC2260
 2020-01-30 17:20:55 +00:00
Erik Johnston
57ad702af0
Backgroud update to clean out rooms from current state (#6802) 2020-01-30 17:17:44 +00:00
Erik Johnston
b660327056
Resync remote device list when detected as stale. (#6786) 2020-01-30 17:06:38 +00:00
Erik Johnston
c3d4ad8afd
Fix sending server up commands from workers (#6811) 
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2020-01-30 16:42:11 +00:00
Erik Johnston
a5bab2d058
When server leaves room check for stale device lists. (#6801) 
When a server leaves a room it may stop sharing a room with remote
users, and thus not get any updates to their device lists. So we need to
check for this case and delete those device lists from the cache.

We don't need to do this if we stop sharing a room because the remote
user leaves the room, because we track that case via looking at
membership changes.
 2020-01-30 16:10:30 +00:00
Erik Johnston
c80a9fe13d
When a client asks for remote keys check if should resync. (#6797) 
If we detect that the remote users' keys may have changed then we should
attempt to resync against the remote server rather than using the
(potentially) stale local cache.
 2020-01-30 15:06:58 +00:00
Richard van der Hoff
5a246611e3
Type defintions for use in refactoring for redaction changes (#6803) 
* Bump signedjson to 1.1

... so that we can use the type definitions

* Fix breakage caused by upgrade to signedjson 1.1

Thanks, @illicitonion...
 2020-01-30 11:25:59 +00:00
Erik Johnston
a855b7c3a8
Remove unused DeviceRow class (#6800) 2020-01-29 12:06:31 +00:00
Richard van der Hoff
dcd85b976d Make /directory/room/<alias> handle restrictive power levels 
Fixes a bug where the alias would be added, but `PUT /directory/room/<alias>`
would return a 403.
 2020-01-29 11:52:52 +00:00
Richard van der Hoff
b36095ae5c Set the PL for aliases events to 0. 2020-01-29 11:52:52 +00:00
Richard van der Hoff
ee42a5513e Factor out a copy_power_levels_contents method 
I'm going to need another copy (hah!) of this.
 2020-01-29 11:52:52 +00:00
Erik Johnston
6b9e1014cf
Fix race in federation sender that delayed device updates. (#6799) 
We were sending device updates down both the federation stream and
device streams. This mean there was a race if the federation sender
worker processed the federation stream first, as when the sender checked
if there were new device updates the slaved ID generator hadn't been
updated with the new stream IDs and so returned nothing.

This situation is correctly handled by events/receipts/etc by not
sending updates down the federation stream and instead having the
federation sender worker listen on the other streams and poke the
transaction queues as appropriate.
 2020-01-29 11:23:01 +00:00
Erik Johnston
611215a49c
Delete current state when server leaves a room (#6792) 
Otherwise its just stale data, which may get deleted later anyway so
can't be relied on. It's also a bit of a shotgun if we're trying to get
the current state of a room we're not in.
 2020-01-29 11:01:32 +00:00
Erik Johnston
2cad8baa70
Fix bug when querying remote user keys that require a resync. (#6796) 
We ended up only returning a single device, rather than all of them.
 2020-01-29 09:56:41 +00:00
Erik Johnston
fcfb591b31
Fix outbound federation request metrics (#6795) 2020-01-28 18:59:48 +00:00
Richard van der Hoff
cc109b79dd
Merge pull request #6787 from matrix-org/rav/msc2260 
Implement updated auth rules from MSC2260
 2020-01-28 15:11:22 +00:00
Richard van der Hoff
a1f307f7d1 fix bad variable ref 2020-01-28 14:55:22 +00:00
Erik Johnston
e17a110661
Detect unknown remote devices and mark cache as stale (#6776) 
We just mark the fact that the cache may be stale in the database for
now.
 2020-01-28 14:43:21 +00:00
Richard van der Hoff
49d3bca37b Implement updated auth rules from MSC2260 2020-01-28 14:20:10 +00:00
Richard van der Hoff
a8ce7aeb43
Pass room version object into event_auth.check and check_redaction (#6788) 
These are easier to work with than the strings and we normally have one around.

This fixes `FederationHander._persist_auth_tree` which was passing a
RoomVersion object into event_auth.check instead of a string.
 2020-01-28 14:18:29 +00:00
Erik Johnston
02b44db922
Warn if postgres database has non-C locale. (#6734) 
As using non-C locale can cause issues on upgrading OS.
 2020-01-28 13:44:21 +00:00
Erik Johnston
33f904835a Merge branch 'master' into develop 2020-01-28 13:39:39 +00:00
Erik Johnston
77d9357226 1.9.1 2020-01-28 13:09:36 +00:00
Erik Johnston
bdbeeb94ec
Fix setting mau_limit_reserved_threepids config (#6793) 
Calling the invalidation function during initialisation of the data
stores introduces a circular dependency, causing Synapse to fail to
start.
 2020-01-28 13:05:24 +00:00
Erik Johnston
8df862e45d
Add rooms.room_version column (#6729) 
This is so that we don't have to rely on pulling it out from `current_state_events` table.
 2020-01-27 14:30:57 +00:00
Erik Johnston
d5275fc55f
Propagate cache invalidates from workers to other workers. (#6748) 
Currently if a worker invalidates a cache it will be streamed to master, which then didn't forward those to other workers.
 2020-01-27 13:47:50 +00:00
Andrew Morgan
9f7aaf90b5
Validate client_secret parameter (#6767) 2020-01-24 14:28:40 +00:00
Erik Johnston
fa4d609e20
Make 'event.redacts' never raise. (#6771) 
There are quite a few places that we assume that a redaction event has a
corresponding `redacts` key, which is not always the case. So lets
cheekily make it so that event.redacts just returns None instead.
 2020-01-23 15:19:03 +00:00
Brendan Abolivier
51fc3f693e Merge branch 'master' into develop 2020-01-23 13:45:23 +00:00
Brendan Abolivier
f3eac2b3e9 1.9.0 2020-01-23 12:57:55 +00:00
Richard van der Hoff
6b7462a13f
a bit of debugging for media storage providers (#6757) 
* a bit of debugging for media storage providers

* changelog
 2020-01-23 12:11:44 +00:00
Richard van der Hoff
5bd3cb7260
Minor fixes to user admin api (#6761) 
* don't insist on a password (this is valid if you have an SSO login)
* fix reference to undefined `requester`
 2020-01-23 12:03:58 +00:00
Andrew Morgan
ce84dd9e20
Remove unnecessary abstractions in admin handler (#6751) 2020-01-22 15:09:57 +00:00
Brendan Abolivier
0d0f32bc53 1.9.0rc1 2020-01-22 14:03:46 +00:00
Andrew Morgan
90a28fb475
Admin API to list, filter and sort rooms (#6720) 2020-01-22 13:36:43 +00:00
Brendan Abolivier
ed83c3a018
Fix typo in _select_thumbnail 2020-01-22 12:27:42 +00:00
Andrew Morgan
aa9b00fb2f
Fix and add test to deprecated quarantine media admin api (#6756) 2020-01-22 11:05:50 +00:00
Neil Johnson
5e52d8563b Allow monthly active user limiting support for worker mode, fixes #4639. (#6742) 2020-01-22 11:05:14 +00:00
Erik Johnston
5d7a6ad223
Allow streaming cache invalidate all to workers. (#6749) 2020-01-22 10:37:00 +00:00
Erik Johnston
0e68760078
Add a DeltaState to track changes to be made to current state (#6716) 2020-01-20 18:07:20 +00:00
Erik Johnston
b0a66ab83c
Fixup synapse.rest to pass mypy (#6732) 2020-01-20 17:38:21 +00:00
Erik Johnston
74b74462f1
Fix /events/:event_id deprecated API. (#6731) 2020-01-20 17:38:09 +00:00
Erik Johnston
0f6e525be3
Fixup synapse.api to pass mypy (#6733) 2020-01-20 17:34:13 +00:00
Erik Johnston
ceecedc68b
Fix changing password via user admin API. (#6730) 2020-01-20 17:23:59 +00:00
Andrew Morgan
198d52da3a Fix empty account_validity config block 2020-01-20 14:05:29 +00:00
Brendan Abolivier
a17f64361c
Add more logging around message retention policies support (#6717) 
So we can debug issues like #6683 more easily
 2020-01-17 20:51:44 +00:00
Richard van der Hoff
0b885d62ef bump version to v1.9.0.dev2 2020-01-17 14:58:58 +00:00
Satsuki Yanagi
722b4f302d Fix syntax error in run_upgrade for schema 57 (#6728) 
Fix #6727
Related #6655

Co-authored-by: Erik Johnston <erikj@jki.re>
 2020-01-17 14:30:35 +00:00
Brendan Abolivier
3b72bb780a
Merge pull request #6714 from matrix-org/babolivier/retention_select_event 
Fix instantiation of message retention purge jobs
 2020-01-17 14:23:51 +00:00
Richard van der Hoff
1dee1e900b bump version to v1.9.0.dev1 2020-01-17 10:44:12 +00:00
Richard van der Hoff
59dc87c618
Merge pull request #6724 from matrix-org/rav/log_saml_attributes 
Log saml assertions rather than the whole response
 2020-01-17 10:33:24 +00:00
Richard van der Hoff
2b6a77fcde
Delegate remote_user_id mapping to the saml mapping provider (#6723) 
Turns out that figuring out a remote user id for the SAML user isn't quite as obvious as it seems. Factor it out to the SamlMappingProvider so that it's easy to control.
 2020-01-17 10:32:47 +00:00
Erik Johnston
a8a50f5b57
Wake up transaction queue when remote server comes back online (#6706) 
This will be used to retry outbound transactions to a remote server if
we think it might have come back up.
 2020-01-17 10:27:19 +00:00
Richard van der Hoff
5ce0b17e38
Clarify the account_validity and email sections of the sample configuration. (#6685) 
Generally try to make this more comprehensible, and make it match the
conventions.

I've removed the documentation for all the settings which allow you to change
the names of the template files, because I can't really see why they are
useful.
 2020-01-17 10:04:15 +00:00
Richard van der Hoff
acc7820574 Log saml assertions rather than the whole response 
... since the whole response is huge.

We even need to break up the assertions, since kibana otherwise truncates them.
 2020-01-16 22:26:34 +00:00
Richard van der Hoff
14d8f342d5 move batch_iter to a separate module 2020-01-16 22:25:32 +00:00
Brendan Abolivier
dac148341b
Fixup diff 2020-01-16 20:25:09 +00:00
Brendan Abolivier
842c2cfbf1
Remove get_room_event_after_stream_ordering entirely 2020-01-16 20:24:17 +00:00
Erik Johnston
d386f2f339
Add StateMap type alias (#6715) 2020-01-16 13:31:22 +00:00
Brendan Abolivier
e601f35d3b
Lint 2020-01-16 09:55:11 +00:00
Neil Johnson
38e0e59f42
Add org.matrix.e2e_cross_signing to unstable_features in /versions as per MSC1756 (#6712) 2020-01-16 09:46:14 +00:00
Erik Johnston
48c3a96886
Port synapse.replication.tcp to async/await (#6666) 
* Port synapse.replication.tcp to async/await

* Newsfile

* Correctly document type of on_<FOO> functions as async

* Don't be overenthusiastic with the asyncing....
 2020-01-16 09:16:12 +00:00
Brendan Abolivier
066b9f52b8
Correctly order when selecting before stream ordering 2020-01-15 19:32:47 +00:00
Brendan Abolivier
8363588237
Fix typo 2020-01-15 19:13:22 +00:00
Brendan Abolivier
855af069a4
Fix instantiation of message retention purge jobs 
When figuring out which topological token to start a purge job at, we
need to do the following:

1. Figure out a timestamp before which events will be purged
2. Select the first stream ordering after that timestamp
3. Select info about the first event after that stream ordering
4. Build a topological token from that info

In some situations (e.g. quiet rooms with a short max_lifetime), there
might not be an event after the stream ordering at step 3, therefore we
abort the purge with the error `No event found`. To mitigate that, this
patch fetches the first event _before_ the stream ordering, instead of
after.
 2020-01-15 18:56:18 +00:00
Erik Johnston
19a1aac48c
Fix purge_room admin API (#6711) 2020-01-15 18:13:47 +00:00
Andrew Morgan
edc244eec4
Remove duplicate session check in web fallback servlet (#6702) 2020-01-15 18:05:18 +00:00
Richard van der Hoff
107f256cd8 Merge branch 'develop' into rav/module_api_extensions 2020-01-15 16:00:24 +00:00
Richard van der Hoff
8f5d7302ac
Implement RedirectException (#6687) 
Allow REST endpoint implemnentations to raise a RedirectException, which will
redirect the user's browser to a given location.
 2020-01-15 15:58:55 +00:00
Erik Johnston
28c98e51ff
Add local_current_membership table (#6655) 
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
 2020-01-15 14:59:33 +00:00
Erik Johnston
b5ce7f5874
Process EDUs in parallel with PDUs. (#6697) 
This means that things like to device messages don't get blocked behind
processing PDUs, which can potentially take *ages*.
 2020-01-14 14:08:35 +00:00
Erik Johnston
e8b68a4e4b
Fixup synapse.replication to pass mypy checks (#6667) 2020-01-14 14:08:06 +00:00
Andrew Morgan
1177d3f3a3
Quarantine media by ID or user ID (#6681) 2020-01-13 18:10:43 +00:00
Richard van der Hoff
326c893d24
Kill off RegistrationError (#6691) 
This is pretty pointless. Let's just use SynapseError.
 2020-01-13 12:48:22 +00:00
Richard van der Hoff
2d07c73777
Don't assign numeric IDs for empty usernames (#6690) 
Fix a bug where we would assign a numeric userid if somebody tried registering
with an empty username
 2020-01-13 12:47:30 +00:00
Richard van der Hoff
3cfac9593c
Merge pull request #6689 from matrix-org/rav/saml_mapping_provider_updates 
Updates to the SAML mapping provider API
 2020-01-13 12:44:55 +00:00
Richard van der Hoff
8039685051
Allow additional_resources to implement Resource directly (#6686) 
AdditionalResource really doesn't add any value, and it gets in the way for
resources which want to support child resources or the like. So, if the
resource object already implements the IResource interface, don't bother
wrapping it.
 2020-01-13 12:42:44 +00:00
Richard van der Hoff
feee819973
Fix exceptions on requests for non-ascii urls (#6682) 
Fixes #6402
 2020-01-13 12:41:51 +00:00
Richard van der Hoff
da4e52544e comment for run_in_background 2020-01-12 21:53:47 +00:00
Richard van der Hoff
dc69a1cf43 Pass client redirect URL into SAML mapping providers 2020-01-12 21:40:49 +00:00
Richard van der Hoff
47e63cc67a Pass the module_api into the SamlMappingProvider 
... for consistency with other modules, and because we'll need it sooner or
later and it will be a pain to introduce later.
 2020-01-12 21:40:49 +00:00
Richard van der Hoff
01243b98e1 Handle config not being set for synapse plugin modules 
Some modules don't need any config, so having to define a `config` property
just to keep the loader happy is a bit annoying.
 2020-01-12 21:34:36 +00:00
Richard van der Hoff
473d3801b6 Cleanups and additions to the module API 
Add some useful things, such as error types and logcontext handling, to the
API.

Make `hs` a private member to dissuade people from using it (hopefully
they aren't already).

Add a couple of new methods (`record_user_external_id` and
`generate_short_term_login_token`).
 2020-01-12 21:31:44 +00:00
Richard van der Hoff
bf46821180 Refuse to start if sqlite is older than 3.11.0 2020-01-09 18:11:04 +00:00
Richard van der Hoff
e48ba84e0b Check postgres version in check_database 
this saves doing it on each connection, and will allow us to pass extra options
in.
 2020-01-09 18:05:59 +00:00
Richard van der Hoff
e97d1cf001 Modify check_database to take a connection rather than a cursor 
We might not need the cursor at all.
 2020-01-09 18:05:50 +00:00
Manuel Stahl
d2906fe666 Allow admin users to create or modify users without a shared secret (#6495) 
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
 2020-01-09 13:31:00 +00:00
Erik Johnston
d773290cb1 Merge branch 'master' into develop 2020-01-09 13:25:48 +00:00
Erik Johnston
24b2c940fb 1.8.0 2020-01-09 11:39:29 +00:00
Erik Johnston
7c232bd98b
Merge pull request #6664 from matrix-org/erikj/media_admin_apis 
Fix media repo admin APIs when using a media worker.
 2020-01-08 15:50:06 +00:00
Erik Johnston
d74054afda Shuffle the code 2020-01-08 14:57:45 +00:00
Erik Johnston
bca3455b38 Comments 2020-01-08 14:27:35 +00:00
Erik Johnston
187dc6ad02 Do not rely on streaming events, as media repo doesn't 2020-01-08 14:24:28 +00:00
Richard van der Hoff
b064a41291 Merge remote-tracking branch 'origin/release-v1.8.0' into develop 2020-01-08 13:27:17 +00:00
Manuel Stahl
7caaa29daa Fix GET request on /_synapse/admin/v2/users endpoint (#6563) 
Fixes #6552
 2020-01-08 13:26:40 +00:00
Erik Johnston
1adf27c82a Import RoomStore in media worker to fix admin APIs 2020-01-08 13:26:20 +00:00
Erik Johnston
3cf7d6d5b6 Move media admin store functions to worker store 2020-01-08 13:26:20 +00:00
Richard van der Hoff
573fee759c
Back out ill-advised notary server hackery (#6657) 
This was ill-advised. We can't modify verify_keys here, because the response
object has already been signed by the requested key.

Furthermore, it's somewhat unnecessary because existing versions of Synapse
(which get upset that the notary key isn't present in verify_keys) will fall
back to a direct fetch via `/key/v2/server`.

Also: more tests for fetching keys via perspectives: it would be nice if we actually tested when our fetcher can't talk to our notary impl.
 2020-01-08 13:24:10 +00:00
Erik Johnston
be29ed7ad8
Correctly proxy remote group HTTP errors. (#6654) 
e.g. if remote returns a 404 then that shouldn't be treated as an error
but should be proxied through.
 2020-01-07 15:36:41 +00:00
Brendan Abolivier
2b6b7f482a
Merge pull request #6621 from matrix-org/babolivier/purge_job_config_typo 
Fix a typo in the purge jobs configuration example
 2020-01-07 16:17:40 +01:00
Brendan Abolivier
391fb47791
Reword 2020-01-07 14:54:32 +00:00
Brendan Abolivier
3a86477162
Change the example from 5min to 12h 
Have a purge job running every 5min is probably not something we want to advise admins to do as a sort-of default.
 2020-01-07 14:53:07 +00:00
Erik Johnston
7f0e706ebf 1.8.0rc1 2020-01-07 14:31:13 +00:00
Brendan Abolivier
0ab5853ec9
Merge pull request #6652 from matrix-org/babolivier/depth_missing_events 
Fix conditions failing if min_depth = 0
 2020-01-07 15:22:07 +01:00
Richard van der Hoff
85db7f73be
Add a background update to clear tombstoned rooms from the directory (#6648) 
* Add a background update to clear tombstoned rooms from the directory

* use the ABC metaclass
 2020-01-07 14:18:43 +00:00
Richard van der Hoff
9824a39d80
Async/await for background updates (#6647) 
so that bg update routines can be async
 2020-01-07 14:12:42 +00:00
Richard van der Hoff
1ff5491117
Merge pull request #6645 from matrix-org/rav/fix_synchrotron_error 
Fix exceptions in the synchrotron worker log when events are rejected.
 2020-01-07 14:02:14 +00:00
Brendan Abolivier
cd428a93e2
Fix conditions failing if min_depth = 0 
This could result in Synapse not fetching prev_events for new events in the room if it has missed some events.
 2020-01-07 12:08:58 +00:00
Richard van der Hoff
1807db5e73
Merge pull request #6629 from matrix-org/rav/kill_event_reference_hashes 
Remove a bunch of unused code from event creation
 2020-01-06 17:46:31 +00:00
Richard van der Hoff
26c5d3d398 Fix exceptions in log when rejected event is replicated 2020-01-06 17:16:28 +00:00
Richard van der Hoff
c74de81bfc async/await for SyncReplicationHandler.process_and_notify 2020-01-06 17:14:28 +00:00
Richard van der Hoff
bc42da4ab8 Clarify documentation on get_event* methods 
Make it clearer how they behave in the face of rejected and/or missing events.
 2020-01-06 17:12:06 +00:00
Richard van der Hoff
ba897a7590
Fix some test failures when frozen_dicts are enabled (#6642) 
Fixes #4026
 2020-01-06 15:22:46 +00:00
Erik Johnston
9f6c1befbb
Add experimental 'databases' config (#6580) 2020-01-06 14:44:01 +00:00
Richard van der Hoff
ab4b4ee6a7
Fix an error which was thrown by the PresenceHandler _on_shutdown handler. (#6640) 2020-01-06 14:34:02 +00:00
Richard van der Hoff
a7d2e5b37f Remove unused get_latest_event_ids_and_hashes_in_room 2020-01-06 13:45:33 +00:00
Richard van der Hoff
dc41fbf0dd Remove unused get_prev_events_and_hashes_for_room 2020-01-06 13:45:33 +00:00
Richard van der Hoff
38e0829a4c Remove unused hashes and depths from _update_membership params 2020-01-06 13:45:33 +00:00
Richard van der Hoff
3bef62488e Remove unused hashes and depths from create_event params 2020-01-06 13:45:33 +00:00
Richard van der Hoff
66ca914dc0 Remove unused hashes and depths from create_new_client_event params 2020-01-06 13:45:33 +00:00
Richard van der Hoff
15720092ac replace get_prev_events_and_hashes_for_room with get_prev_events_for_room in create_new_client_event 2020-01-06 13:45:33 +00:00
Richard van der Hoff
5a04781643 rename get_prev_events_for_room to get_prev_events_and_hashes_for_room 
... to make way for a new method which just returns the event ids
 2020-01-06 13:45:33 +00:00
Richard van der Hoff
4b36b482e0
Fix exception when fetching notary server's old keys (#6625) 
Lift the restriction that *all* the keys used for signing v2 key responses be
present in verify_keys.

Fixes #6596.
 2020-01-06 12:33:56 +00:00
Richard van der Hoff
18674eebb1
Workaround for error when fetching notary's own key (#6620) 
* Kill off redundant SynapseRequestFactory

We already get the Site via the Channel, so there's no need for a dedicated
RequestFactory: we can just use the right constructor.

* Workaround for error when fetching notary's own key

As a notary server, when we return our own keys, include all of our signing
keys in verify_keys.

This is a workaround for #6596.
 2020-01-06 12:28:58 +00:00
Andrew Morgan
01c3c6c929 Fix power levels being incorrectly set in old and new rooms after a room upgrade (#6633) 
Modify a copy of an upgraded room's PL before sending to the new room
 2020-01-06 09:53:07 +00:00
Richard van der Hoff
08815566bc
Automate generation of the sample and debian log configs (#6627) 2020-01-03 17:14:00 +00:00
Richard van der Hoff
e484101306
Raise an error if someone tries to use the log_file config option (#6626) 
This has caused some confusion for people who didn't notice it going away.
 2020-01-03 17:11:29 +00:00
Richard van der Hoff
98247c4a0e
Remove unused, undocumented "content repo" resource (#6628) 
This looks like it got half-killed back in #888.

Fixes #6567.
 2020-01-03 17:10:52 +00:00
Richard van der Hoff
b6b57ecb4e
Kill off redundant SynapseRequestFactory (#6619) 
We already get the Site via the Channel, so there's no need for a dedicated
RequestFactory: we can just use the right constructor.
 2020-01-03 14:19:48 +00:00
Richard van der Hoff
6964ea095b
Reduce the reconnect time when replication fails. (#6617) 2020-01-03 14:19:09 +00:00
Brendan Abolivier
4efe1d4d3f
Fix a typo in the purge jobs configuration example 2020-01-03 12:57:24 +01:00
Richard van der Hoff
dc96943d51 Merge branch 'master' into develop 2019-12-31 11:01:06 +00:00
Richard van der Hoff
77661ce81a 1.7.3 2019-12-31 10:45:12 +00:00
Richard van der Hoff
92eac974b9
Hacks to work around #6605 (#6608) 
When we have an event which refers to non-existent auth_events, ignore said events rather than exploding in a ball of fire.

Fixes #6605.
 2019-12-31 10:41:44 +00:00
Richard van der Hoff
56ad230efa Merge branch 'master' into develop 2019-12-20 11:11:34 +00:00
Richard van der Hoff
29794c6bc8 1.7.2 2019-12-20 10:58:07 +00:00
Erik Johnston
75d8f26ac8
Split state groups into a separate data store (#6296) 2019-12-20 10:48:24 +00:00
Richard van der Hoff
4caab0e95e
Backport fixes to sqlite upgrade from develop (#6578) 
Only run prepare_database on connection for in-memory databases.

Fixes #6569.
 2019-12-20 10:46:46 +00:00
Erik Johnston
fa780e9721
Change EventContext to use the Storage class (#6564) 2019-12-20 10:32:02 +00:00
Richard van der Hoff
03d3792f3c
Fix exceptions when attempting to backfill (#6576) 
Fixes #6575
 2019-12-20 09:55:45 +00:00
Erik Johnston
0b5dbadd96
Explode on duplicate delta file names. (#6565) 2019-12-19 15:07:37 +00:00
Erik Johnston
3d46124ad0
Port some admin handlers to async/await (#6559) 2019-12-19 15:07:28 +00:00
Richard van der Hoff
bca30cefee
Improve diagnostics on database upgrade failure (#6570) 
`Failed to upgrade database` is not helpful, and it's unlikely that UPGRADE.rst
has anything useful.
 2019-12-19 14:53:15 +00:00
Richard van der Hoff
0b794cbd7b
Fix sdnotify with acme enabled (#6571) 
If acme was enabled, the sdnotify startup hook would never be run because we
would try to add it to a hook which had already fired.

There's no need to delay it: we can sdnotify as soon as we've started the
listeners.
 2019-12-19 14:52:52 +00:00
Richard van der Hoff
b95b762560
Add an export_signing_key script (#6546) 
I want to do some key rotation, and it is silly that we don't have a way to do
this.
 2019-12-19 11:11:14 +00:00
Richard van der Hoff
d6752ce5da
Clean up startup for the pusher (#6558) 
* Remove redundant python2 support code

`str.decode()` doesn't exist on python3, so presumably this code was doing
nothing

* Filter out pushers with corrupt data

When we get a row with unparsable json, drop the row, rather than returning a
row with null `data`, which will then cause an explosion later on.

* Improve logging when we can't start a pusher

Log the ID to help us understand the problem

* Make email pusher setup more robust

We know we'll have a `data` member, since that comes from the database. What we
*don't* know is if that is a dict, and if that has a `brand` member, and if
that member is a string.
 2019-12-18 14:26:58 +00:00
Andrew Morgan
7963ca83cb
Add delta file to fix missing default table data (#6555) 2019-12-18 11:13:33 +00:00
Erik Johnston
2284eb3a53
Add database config class (#6513) 
This encapsulates config for a given database and is the way to get new
connections.
 2019-12-18 10:45:12 +00:00
Richard van der Hoff
91ccfe9f37 Merge branch 'master' into develop 2019-12-18 10:16:39 +00:00
Richard van der Hoff
6e8f8e14f2 Merge release-v1.7.1 into develop 2019-12-18 09:51:51 +00:00
Richard van der Hoff
d656e91fc2 1.7.1 2019-12-18 09:38:08 +00:00
Erik Johnston
5029422530
Fix bug where we added duplicate event IDs as auth_events (#6560) 2019-12-17 15:06:08 +00:00
Erik Johnston
02553901ce
Remove unused get_pagination_rows methods. (#6557) 
Remove unused get_pagination_rows methods
 2019-12-17 11:44:32 +00:00
Erik Johnston
5ca2cfadc3 Add auth events as per spec. (#6556) 
Previously we tried to be clever and filter out some unnecessary event
IDs to keep the auth chain small, but that had some annoying
interactions with state res v2 so we stop doing that for now.
 2019-12-16 17:05:01 +00:00
Erik Johnston
3fbe5b7ec3
Add auth events as per spec. (#6556) 
Previously we tried to be clever and filter out some unnecessary event
IDs to keep the auth chain small, but that had some annoying
interactions with state res v2 so we stop doing that for now.
 2019-12-16 16:59:32 +00:00
Brendan Abolivier
6316530366
Merge pull request #6553 from matrix-org/babolivier/fix-context-filter 
Use the filtered version of an event when responding to /context requests for that event
 2019-12-16 16:24:10 +00:00
Will Hunt
bfb95654c9 Add option to allow profile queries without sharing a room (#6523) 2019-12-16 16:11:55 +00:00
Brendan Abolivier
a820069549
Incorporate review 2019-12-16 16:00:18 +00:00
Richard van der Hoff
bbb75ff6ee Exclude rejected state events when calculating state at backwards extrems (#6527) 
This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
 2019-12-16 14:07:29 +00:00
Richard van der Hoff
ff773ff724 Persist auth/state events at backwards extremities when we fetch them (#6526) 
The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
 2019-12-16 14:07:11 +00:00
Richard van der Hoff
83895316d4 sanity-checking for events used in state res (#6531) 
When we perform state resolution, check that all of the events involved are in
the right room.
 2019-12-16 13:53:53 +00:00
Richard van der Hoff
6577f2d887 Sanity-check room ids in event auth (#6530) 
When we do an event auth operation, check that all of the events involved are
in the right room.
 2019-12-16 13:47:34 +00:00
Richard van der Hoff
35bbe4ca79 Check the room_id of events when fetching room state/auth (#6524) 
When we request the state/auth_events to populate a backwards extremity (on
backfill or in the case of missing events in a transaction push), we should
check that the returned events are in the right room rather than blindly using
them in the room state or auth chain.

Given that _get_events_from_store_or_dest takes a room_id, it seems clear that
it should be sanity-checking the room_id of the requested events, so let's do
it there.
 2019-12-16 13:47:07 +00:00
Richard van der Hoff
20d5ba16e6 Add include_event_in_state to _get_state_for_room (#6521) 
Make it return the state *after* the requested event, rather than the one
before it. This is a bit easier and requires fewer calls to
get_events_from_store_or_dest.
 2019-12-16 13:26:12 +00:00
Richard van der Hoff
be294d6fde Move get_state methods into FederationHandler (#6503) 
This is a non-functional refactor as a precursor to some other work.
 2019-12-16 13:20:21 +00:00
Richard van der Hoff
4c7b1bb6cc Refactor get_events_from_store_or_dest to return a dict (#6501) 
There was a bunch of unnecessary conversion back and forth between dict and
list going on here. We can simplify a bunch of the code.
 2019-12-16 13:19:32 +00:00
Richard van der Hoff
6920d88892
Exclude rejected state events when calculating state at backwards extrems (#6527) 
This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
 2019-12-16 13:14:37 +00:00
Richard van der Hoff
bc7de87650
Persist auth/state events at backwards extremities when we fetch them (#6526) 
The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
 2019-12-16 12:26:28 +00:00
Brendan Abolivier
ac87ddb242
Update the documentation of the filtering function 2019-12-16 12:15:37 +00:00
Brendan Abolivier
487f1bb49d
Use the filtered version of an event when responding to /context requests for that event 
Sometimes the filtering function can return a pruned version of an event (on top of either the event itself or an empty list), if it thinks the user should be able to see that there's an event there but not the content of that event. Therefore, the previous logic of 'if filtered is empty then we can use the event we retrieved from the database' is flawed, and we should use the event returned by the filtering function.
 2019-12-16 12:14:12 +00:00
Werner Sembach
9d173b312c Automatically delete empty groups/communities (#6453) 
Signed-off-by: Werner Sembach <werner.sembach@fau.de>
 2019-12-16 12:12:40 +00:00
Richard van der Hoff
1da15f05f5
sanity-checking for events used in state res (#6531) 
When we perform state resolution, check that all of the events involved are in
the right room.
 2019-12-13 12:55:32 +00:00
Richard van der Hoff
971a0702b5
Sanity-check room ids in event auth (#6530) 
When we do an event auth operation, check that all of the events involved are
in the right room.
 2019-12-13 11:44:41 +00:00
Erik Johnston
5cadbd9ebb
Merge pull request #6537 from matrix-org/erikj/bump_mypy_version 
Bump mypy version
 2019-12-13 11:32:53 +00:00
Erik Johnston
f46e05d053 Merge branch 'master' into develop 2019-12-13 10:55:47 +00:00
Erik Johnston
f5aeea9e89 1.7.0 2019-12-13 10:19:53 +00:00
Erik Johnston
caa52836e4
Merge pull request #6496 from matrix-org/erikj/initial_sync_asnyc 
Port synapse.handlers.initial_sync to async/await
 2019-12-13 10:01:51 +00:00
Hubert Chathi
cb2db17994
look up cross-signing keys from the DB in bulk (#6486) 2019-12-12 12:03:28 -05:00
Andrew Morgan
5bfd8855d6
Fix redacted events being returned in search results ordered by "recent" (#6522) 2019-12-12 15:53:49 +00:00