Commit graph

2803 commits

Author SHA1 Message Date
Andrew Morgan
72fe2affb6 Query missing cross-signing keys on local sig upload
Add changelog

Save retrieved keys to the db

lint

Fix and de-brittle remote result dict processing

Use query_user_devices instead, assume only master, self_signing key types

Make changelog more useful

Remove very specific exception handling

Wrap get_verify_key_from_cross_signing_key in a try/except

Note that _get_e2e_cross_signing_verify_key can raise a SynapseError

lint

Add comment explaining why this is useful

Only fetch master and self_signing key types

Fix log statements, docstrings

Remove extraneous items from remote query try/except

lint

Factor key retrieval out into a separate function

Send device updates, modeled after SigningKeyEduUpdater._handle_signing_key_updates

Update method docstring
2020-04-17 15:47:49 +01:00
Richard van der Hoff
67ff7b8ba0
Improve type checking in replication.tcp.Stream (#7291)
The general idea here is to get rid of the type: ignore annotations on all of the current_token and update_function assignments, which would have caught #7290.

After a bit of experimentation, it seems like the least-awful way to do this is to pass the offending functions in as parameters to the Stream constructor. Unfortunately that means that the concrete implementations no longer have the same constructor signature as Stream itself, which means that it gets hard to correctly annotate STREAMS_MAP.

I've also introduced a couple of new types, to take out some duplication.
2020-04-17 14:49:55 +01:00
Tristan Lins
c07fca9e2f
Clarify the comments for media_storage_providers options (#7272) 2020-04-17 07:09:33 -04:00
Patrick Cloke
01294e6b3a
Do not treat display names as globs for push rules. (#7271) 2020-04-16 10:52:55 -04:00
Richard van der Hoff
d7d42387f5
Fix 'generator object is not subscriptable' error (#7290)
Some of the query functions return generators rather than lists, so we can't
index into the result. Happily we already have a copy of the results.

(think this was introduced in #7024)
2020-04-16 14:37:06 +01:00
Patrick Cloke
eed7c5b89e
Convert auth handler to async/await (#7261) 2020-04-15 12:40:18 -04:00
Patrick Cloke
17a2433b0d
Add notes to the changelog about an additional SSO template. (#7259) 2020-04-15 10:36:33 -04:00
Andrew Morgan
a48138784e
Allow specifying the value of Accept-Language header for URL previews (#7265) 2020-04-15 13:35:29 +01:00
Richard van der Hoff
f2049a8d21
Fix a potentially-huge sql query (#7274)
We could end up looking up tens of thousands of events, which could cause large
amounts of data to be logged to the postgres log.
2020-04-15 10:16:35 +01:00
Zay11Zay
f1097e7720
Fix the parameters of a test fixture (#7243) 2020-04-14 15:37:28 -04:00
Andrew Morgan
ac6a84818f Only register devices edu handler on the master process (#7255) 2020-04-14 11:36:24 +01:00
Andrew Morgan
fef82f4e22
Only register devices edu handler on the master process (#7255) 2020-04-14 10:09:58 +01:00
Ryan Hovland
4a0dadafbe
Add setting to nginx configuration to allow larger file uploads (#7251) 2020-04-13 17:23:36 +01:00
PeerD
f41b742161
Check on room creation if the user is allowed to publish the room to the room directory. (#7260) 2020-04-13 07:42:32 -04:00
Andrew Morgan
ac978ab3da
Default PL100 to enable encryption in a room (#7230) 2020-04-09 18:45:38 +01:00
Patrick Cloke
b85d7652ff
Do not allow a deactivated user to login via SSO. (#7240) 2020-04-09 13:28:13 -04:00
Andrew Morgan
967f99b9f8 Unblacklist /upgrade creates a new room (#7228)
This commit was originally merged in #7228 but reverted in #7254 as the
associated sytest was not ready yet. Now that it is, we can merge this
again.
2020-04-09 13:53:35 +01:00
Andrew Morgan
a026bdaab7
Add matrix-synapse-shared-secret-auth as an example password provider (#7248) 2020-04-09 12:49:05 +01:00
Andrew Morgan
e3cd28d8a4
Revert "Unblacklist /upgrade creates a new room (#7228)" (#7254)
This reverts commit 59f0ca8b87.
2020-04-09 12:47:16 +01:00
Andrew Morgan
59f0ca8b87
Unblacklist /upgrade creates a new room (#7228) 2020-04-09 12:44:48 +01:00
Richard van der Hoff
5a709630bf
Fix --help commandline argument (#7249)
I don't really remember why this was so complicated; I think it dates
back to the time when we had to instantiate the Config classes before
we could call `add_arguments` - ie before #5597. In any case, I don't
think there's a good reason for it any more, and the impact of it
being complicated is that `--help` doesn't work correctly.
2020-04-09 12:44:37 +01:00
Andrew Morgan
7f7f9968b3
Make it clearer to the user that they haven't overridden onLogin/Register (#7237) 2020-04-09 12:38:38 +01:00
Andrew Morgan
55d46da59a
Upgrade jQuery to 3.x on fallback login/registration screens (#7236) 2020-04-09 12:23:30 +01:00
Richard van der Hoff
23f8d285eb
Remove redundant checks on daemonize from synctl (#7233)
We pass --daemonize on the commandline, which (since at least #4853) overrides
whatever the config file, so there is no need for it to be set in the config
file.
2020-04-08 11:59:47 +01:00
Richard van der Hoff
cae4121484
Make systemd-with-workers doc official (#7234)
Simplify and update this documentation, and make it part of the core dist.
2020-04-08 11:59:26 +01:00
Brendan Abolivier
c11d24d48c
Fix changelog for #7235 2020-04-08 11:59:51 +02:00
Andrew Morgan
29b7e22b93
Add documentation to password_providers config option (#7238) 2020-04-08 00:46:50 +01:00
Richard van der Hoff
f31e65a749
bg update to clear out duplicate outbound_device_list_pokes (#7193)
We seem to have some duplicates, which could do with being cleared out.
2020-04-07 23:06:39 +01:00
Richard van der Hoff
aedeedc206
Merge pull request #7239 from matrix-org/rav/replication_cleanup
Miscellaneous cleanups to replication code
2020-04-07 22:00:55 +01:00
Patrick Cloke
1722b8a527
Convert delete_url_cache_media to async/await. (#7241) 2020-04-07 21:56:34 +01:00
Patrick Cloke
d78cb31588
Add typing information to federation_server. (#7219) 2020-04-07 15:03:23 -04:00
Richard van der Hoff
bd2ea3432b changelog 2020-04-07 17:44:51 +01:00
Brendan Abolivier
ec5ac8e2b1
Fix typo in the login fallback javascript (#7235)
* Fix typo in the login fallback javascript

* Changelog
2020-04-07 18:31:50 +02:00
Richard van der Hoff
2e105c156b
Remove sent outbound device list pokes from the database (#7192)
They just get in the way.
2020-04-07 15:19:19 +01:00
Erik Johnston
ce72355d7f
Fix race in replication (#7226)
Fixes a race between handling `POSITION` and `RDATA` commands. We do this by simply linearizing handling of them.
2020-04-07 11:01:04 +01:00
Erik Johnston
82498ee901
Move server command handling out of TCP protocol (#7187)
This completes the merging of server and client command processing.
2020-04-07 10:51:07 +01:00
Andrew Morgan
4b0f00ad0c
Remove stream before/after debug log lines (#7207) 2020-04-06 12:40:34 +01:00
Andrew Morgan
b21000a44f
Improve error responses when a remote server doesn't allow you to access its public rooms list (#6899) 2020-04-06 12:35:30 +01:00
Erik Johnston
5016b162fc
Move client command handling out of TCP protocol (#7185)
The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.
2020-04-06 09:58:42 +01:00
Brendan Abolivier
d73bf18d13
Server notices: Dissociate room creation/lookup from invite (#7199)
Fixes #6815

Before figuring out whether we should alert a user on MAU, we call get_notice_room_for_user to get some info on the existing server notices room for this user. This function, if the room doesn't exist, creates it and invites the user in it. This means that, if we decide later that no server notice is needed, the user gets invited in a room with no message in it. This happens at every restart of the server, since the room ID returned by get_notice_room_for_user is cached.

This PR fixes that by moving the inviting bit to a dedicated function, that's only called when the server actually needs to send a notice to the user. A potential issue with this approach is that the room that's created by get_notice_room_for_user doesn't match how that same function looks for an existing room (i.e. it creates a room that doesn't have an invite or a join for the current user in it, so it could lead to a new room being created each time a user syncs), but I'm not sure this is a problem given it's cached until the server restarts, so that function won't run very often.

It also renames get_notice_room_for_user into get_or_create_notice_room_for_user to make what it does clearer.
2020-04-04 17:27:45 +02:00
Patrick Cloke
694d8bed0e
Support CAS in UI Auth flows. (#7186) 2020-04-03 15:35:05 -04:00
Martin Milata
b0db928c63
Extend web_client_location to handle absolute URLs (#7006)
Log warning when filesystem path is used.

Signed-off-by: Martin Milata <martin@martinmilata.cz>
2020-04-03 11:57:34 -04:00
Amber Brown
334bfdbc90
Add some benchmarks for LruCache (#6446) 2020-04-03 16:31:52 +01:00
Andrew Morgan
07b88c546d
Convert http.HTTPStatus objects to their int equivalent (#7188) 2020-04-03 14:26:07 +01:00
Richard van der Hoff
0f05fd1530
Reduce the number of calls to resource.getrusage (#7183)
Let's just call `getrusage` once on each logcontext change, rather than twice.
2020-04-03 13:21:30 +01:00
Richard van der Hoff
fd4c975b5b
Merge pull request #7190 from matrix-org/rav/one_bg_update_at_a_time
Only run one background update at a time
2020-04-03 13:17:30 +01:00
Richard van der Hoff
bae32740da
Remove some run_in_background calls in replication code (#7203)
By running this stuff with `run_in_background`, it won't be correctly reported
against the relevant CPU usage stats.

Fixes #7202
2020-04-03 12:29:30 +01:00
Richard van der Hoff
29ce90358c 1.12.3 2020-04-03 10:57:07 +01:00
Richard van der Hoff
daa1ac89a0
Fix device list update stream ids going backward (#7158)
Occasionally we could get a federation device list update transaction which
looked like:

```
[
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D2', 'prev_id': [], 'stream_id': 12, 'deleted': True}},
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D1', 'prev_id': [12], 'stream_id': 11, 'deleted': True}},
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D3', 'prev_id': [11], 'stream_id': 13, 'deleted': True}}
]
```

Having `stream_ids` which are lower than `prev_ids` looks odd. It might work
(I'm not actually sure), but in any case it doesn't seem like a reasonable
thing to expect other implementations to support.
2020-04-03 10:40:22 +01:00
Richard van der Hoff
6d7cec7a57
Fix the debian build in a better way. (#7212) 2020-04-03 10:23:36 +01:00
Patrick Cloke
b9930d24a0
Support SAML in the user interactive authentication workflow. (#7102) 2020-04-01 08:48:00 -04:00
Patrick Cloke
468dcc767b
Allow admins to create aliases when they are not in the room (#7191) 2020-04-01 08:27:05 -04:00
siroccal
250f87d0de
Update postgres.md (#7119) 2020-04-01 12:44:51 +01:00
Erik Johnston
dfa0782254
Remove connections per replication stream metric. (#7195)
This broke in a recent PR (#7024) and is no longer useful due to all
replication clients implicitly subscribing to all streams, so let's
just remove it.
2020-04-01 10:40:46 +01:00
Richard van der Hoff
b413ab8aa6 changelog 2020-03-31 17:44:36 +01:00
Jostein Kjønigsen
2e826cd80c
Improve TURN documentation. (#7167) 2020-03-31 15:50:48 +01:00
Richard van der Hoff
60adcbed91
Fix "'NoneType' has no attribute start|stop" logcontext errors (#7181)
Fixes #7179.
2020-03-31 15:18:41 +01:00
Karlinde
fe1580bfd9
Fill in the 'default' field for user-defined push rules (#6639)
Signed-off-by: Karl Linderhed <git@karlinde.se>
2020-03-31 15:08:56 +01:00
Andrew Morgan
b994e86e35
Only setdefault for signatures if device has key_json (#7177) 2020-03-31 14:51:22 +01:00
Patrick Cloke
0a7b0882c1
Fix use of async/await in media code (#7184) 2020-03-31 09:33:02 -04:00
Richard van der Hoff
62a7289133
Fix a bug which could cause incorrect 'cyclic dependency' error. (#7178)
If there was an exception setting up one of the attributes of the Homeserver
god object, then future attempts to fetch that attribute would raise a
confusing "Cyclic dependency" error. Let's make sure that we clear the
`building` flag so that we just get the original exception.

Ref: #7169
2020-03-31 13:09:16 +01:00
Andrew Morgan
3fb9fc40f5 1.12.1rc1 2020-03-31 11:49:43 +01:00
Erik Johnston
5d99bde788 Newsfile 2020-03-31 11:30:34 +01:00
Andrew Morgan
2cf115f0ea Rewrite changelog 2020-03-31 11:30:16 +01:00
Andrew Morgan
2cb38ca871 Add changelog 2020-03-31 11:30:05 +01:00
David Vo
5bd2b27525 Only import sqlite3 when type checking
Fixes: #7127
Signed-off-by: David Vo <david@vovo.id.au>
2020-03-31 11:27:17 +01:00
Richard van der Hoff
7966a1cde9
Rewrite prune_old_outbound_device_pokes for efficiency (#7159)
make sure we clear out all but one update for the user
2020-03-30 19:06:52 +01:00
Andrew Morgan
7042840b32
Transfer alias mappings when joining an upgraded room (#6946) 2020-03-30 17:53:25 +01:00
Andrew Morgan
d9f29f8dae
Fix a small typo in the metrics_flags config option. (#7171) 2020-03-30 17:38:21 +01:00
Erik Johnston
4f21c33be3
Remove usage of "conn_id" for presence. (#7128)
* Remove `conn_id` usage for UserSyncCommand.

Each tcp replication connection is assigned a "conn_id", which is used
to give an ID to a remotely connected worker. In a redis world, there
will no longer be a one to one mapping between connection and instance,
so instead we need to replace such usages with an ID generated by the
remote instances and included in the replicaiton commands.

This really only effects UserSyncCommand.

* Add CLEAR_USER_SYNCS command that is sent on shutdown.

This should help with the case where a synchrotron gets restarted
gracefully, rather than rely on 5 minute timeout.
2020-03-30 16:37:24 +01:00
David Baker
07569f25d1
Merge pull request #7160 from matrix-org/dbkr/always_send_own_device_list_updates
Always send the user updates to their own device list
2020-03-30 14:34:28 +01:00
Richard van der Hoff
6486c96b65
Merge pull request #7157 from matrix-org/rev.outbound_device_pokes_tests
Add tests for outbound device pokes
2020-03-30 13:59:07 +01:00
Patrick Cloke
c5f89fba55
Add developer documentation for running a local CAS server (#7147) 2020-03-30 07:28:42 -04:00
Richard van der Hoff
b7da598a61 Always whitelist the login fallback for SSO (#7153)
That fallback sets the redirect URL to itself (so it can process the login
token then return gracefully to the client). This would make it pointless to
ask the user for confirmation, since the URL the confirmation page would be
showing wouldn't be the client's.
2020-03-27 20:24:52 +00:00
Brendan Abolivier
84f7eaed16 Improve the UX of the login fallback when using SSO (#7152)
* Don't show the login forms if we're currently logging in with a
  password or a token.
* Submit directly the SSO login form, showing only a spinner to the
  user, in order to eliminate from the clunkiness of SSO through this
  fallback.
2020-03-27 20:19:54 +00:00
Dirk Klimpel
fb69690761
Admin API to join users to a room. (#7051) 2020-03-27 19:16:43 +00:00
Dirk Klimpel
8327eb9280
Add options to prevent users from changing their profile. (#7096) 2020-03-27 19:15:23 +00:00
txt-file
ae219fb411
update debian installation instructions to recommend installing virtualenv instead of python3-virtualenv (#6892)
* change debian package from python3-virtualenv to virtualenv

The virtualenv package is needed for the virtualenv command. The
virtualenv package depends on python3-virtualenv (at least since
debian jessie) so there is no need to specify python3-virtualenv
additionally.

Signed-off-by: Vieno Hakkerinen <vieno@hakkerinen.eu>

* Add changelog

Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2020-03-27 15:02:00 +00:00
Andrew Morgan
12aa5a7fa7
Ensure is_verified on /_matrix/client/r0/room_keys/keys is a boolean (#7150) 2020-03-27 13:30:22 +00:00
David Vo
fbf0782c63
Only import sqlite3 when type checking (#7155)
Fixes: #7127
Signed-off-by: David Vo <david@vovo.id.au>
2020-03-27 13:20:00 +00:00
David Baker
d9965fb8d6 changelog 2020-03-27 12:30:59 +00:00
Richard van der Hoff
665630fcaa Add tests for outbound device pokes 2020-03-27 12:01:37 +00:00
Jason Robinson
7496d3d2f6
Merge pull request #7151 from matrix-org/jaywink/saml-redirect-fix
Allow RedirectResponse in SAML response handler
2020-03-26 22:10:31 +02:00
Patrick Cloke
fa4f12102d
Refactor the CAS code (move the logic out of the REST layer to a handler) (#7136) 2020-03-26 15:05:26 -04:00
Jason Robinson
55ca6cf88c
Update changelog.d/7151.bugfix
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-03-26 20:35:50 +02:00
Nektarios Katakis
825fb5d0a5
Don't default to an invalid sqlite config if no database configuration is provided (#6573) 2020-03-26 17:13:14 +00:00
Jason Robinson
060e7dce09 Allow RedirectResponse in SAML response handler
Allow custom SAML handlers to redirect after processing an auth response.

Fixes #7149

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2020-03-26 19:02:35 +02:00
Dirk Klimpel
e8e2ddb60a
Allow server admins to define and enforce a password policy (MSC2000). (#7118) 2020-03-26 16:51:13 +00:00
Patrick Cloke
1c1242acba
Validate that the session is not modified during UI-Auth (#7068) 2020-03-26 07:39:34 -04:00
Aaron Raimist
6ca5e56fd1
Remove unused captcha_bypass_secret option (#7137)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-03-25 17:49:34 +00:00
Erik Johnston
4cff617df1
Move catchup of replication streams to worker. (#7024)
This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
2020-03-25 14:54:01 +00:00
Andrew Morgan
7bab642707
Various cleanups to INSTALL.md (#7141) 2020-03-25 13:56:40 +00:00
Erik Johnston
b1cfaf08af
Merge pull request #7133 from matrix-org/erikj/fix_worker_startup
Fix starting workers when federation sending not split out.
2020-03-25 09:42:39 +00:00
Richard van der Hoff
39230d2171
Clean up some LoggingContext stuff (#7120)
* Pull Sentinel out of LoggingContext

... and drop a few unnecessary references to it

* Factor out LoggingContext.current_context

move `current_context` and `set_context` out to top-level functions.

Mostly this means that I can more easily trace what's actually referring to
LoggingContext, but I think it's generally neater.

* move copy-to-parent into `stop`

this really just makes `start` and `stop` more symetric. It also means that it
behaves correctly if you manually `set_log_context` rather than using the
context manager.

* Replace `LoggingContext.alive` with `finished`

Turn `alive` into `finished` and make it a bit better defined.
2020-03-24 14:45:33 +00:00
Naugrimm
1fcf9c6f95
Fix CAS redirect url (#6634)
Build the same service URL when requesting the CAS ticket and when calling the proxyValidate URL.
2020-03-24 11:59:04 +00:00
Erik Johnston
d6828c129f Newsfile 2020-03-24 10:36:44 +00:00
Patrick Cloke
190ab593b7
Use the proper error code when a canonical alias that does not exist is used. (#7109) 2020-03-23 15:21:54 -04:00
Kartikaya Gupta (kats)
e341518f92
Update pre-built package name for FreeBSD (#7107). (#7107)
Signed-off-by: Kartikaya Gupta <kats@trevize.staktrace.com>
2020-03-23 15:31:02 +00:00
Richard van der Hoff
a564b92d37
Convert *StreamRow classes to inner classes (#7116)
This just helps keep the rows closer to their streams, so that it's easier to
see what the format of each stream is.
2020-03-23 13:59:11 +00:00
Richard van der Hoff
b3cee0ce67
Fix processing of groups stream, and use symbolic names for streams (#7117)
`groups` != `receipts`

Introduced in #6964
2020-03-23 11:39:36 +00:00