synapse

mirror of https://mau.dev/maunium/synapse.git synced 2024-11-13 21:41:30 +01:00

Author	SHA1	Message	Date
Richard van der Hoff	7966a1cde9	Rewrite prune_old_outbound_device_pokes for efficiency (#7159 ) make sure we clear out all but one update for the user	2020-03-30 19:06:52 +01:00
Richard van der Hoff	6486c96b65	Merge pull request #7157 from matrix-org/rev.outbound_device_pokes_tests Add tests for outbound device pokes	2020-03-30 13:59:07 +01:00
Richard van der Hoff	b7da598a61	Always whitelist the login fallback for SSO (#7153 ) That fallback sets the redirect URL to itself (so it can process the login token then return gracefully to the client). This would make it pointless to ask the user for confirmation, since the URL the confirmation page would be showing wouldn't be the client's.	2020-03-27 20:24:52 +00:00
Dirk Klimpel	fb69690761	Admin API to join users to a room. (#7051 )	2020-03-27 19:16:43 +00:00
Dirk Klimpel	8327eb9280	Add options to prevent users from changing their profile. (#7096 )	2020-03-27 19:15:23 +00:00
Richard van der Hoff	665630fcaa	Add tests for outbound device pokes	2020-03-27 12:01:37 +00:00
Dirk Klimpel	e8e2ddb60a	Allow server admins to define and enforce a password policy (MSC2000). (#7118 )	2020-03-26 16:51:13 +00:00
Patrick Cloke	1c1242acba	Validate that the session is not modified during UI-Auth (#7068 )	2020-03-26 07:39:34 -04:00
Erik Johnston	4cff617df1	Move catchup of replication streams to worker. (#7024 ) This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.	2020-03-25 14:54:01 +00:00
Richard van der Hoff	28d9d6e8a9	Remove spurious "name" parameter to `default_config` this is never set to anything other than "test", and is a source of unnecessary boilerplate.	2020-03-24 18:33:49 +00:00
Richard van der Hoff	39230d2171	Clean up some LoggingContext stuff (#7120 ) * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.	2020-03-24 14:45:33 +00:00
Richard van der Hoff	a564b92d37	Convert `*StreamRow` classes to inner classes (#7116 ) This just helps keep the rows closer to their streams, so that it's easier to see what the format of each stream is.	2020-03-23 13:59:11 +00:00
Richard van der Hoff	c165c1233b	Improve database configuration docs (#6988 ) Attempts to clarify the sample config for databases, and add some stuff about tcp keepalives to `postgres.md`.	2020-03-20 15:24:22 +00:00
Erik Johnston	a319cb1dd1	Change device list streams to have one row per ID (#7010 ) * Add 'device_lists_outbound_pokes' as extra table. This makes sure we check all the relevant tables to get the current max stream ID. Currently not doing so isn't problematic as the max stream ID in `device_lists_outbound_pokes` is the same as in `device_lists_stream`, however that will change. * Change device lists stream to have one row per id. This will make it possible to process the streams more incrementally, avoiding having to process large chunks at once. * Change device list replication to match new semantics. Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host). * Newsfile * Remove handling of multiple rows per ID * Fix worker handling * Comments from review	2020-03-19 11:36:53 +00:00
Erik Johnston	4a17a647a9	Improve get auth chain difference algorithm. (#7095 ) It was originally implemented by pulling the full auth chain of all state sets out of the database and doing set comparison. However, that can take a lot work if the state and auth chains are large. Instead, lets try and fetch the auth chains at the same time and calculate the difference on the fly, allowing us to bail early if all the auth chains converge. Assuming that the auth chains do converge more often than not, this should improve performance. Hopefully.	2020-03-18 16:46:41 +00:00
Richard van der Hoff	c37db0211e	Share SSL contexts for non-federation requests (#7094 ) Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests. Fixes #7092.	2020-03-17 21:32:25 +00:00
Patrick Cloke	60724c46b7	Remove special casing of `m.room.aliases` events (#7034 )	2020-03-17 07:37:04 -04:00
Richard van der Hoff	6a35046363	Revert "Add options to disable setting profile info for prevent changes. (#7053 )" This reverts commit `54dd28621b`, reversing changes made to `6640460d05`.	2020-03-17 11:25:01 +00:00
Brendan Abolivier	54dd28621b	Add options to disable setting profile info for prevent changes. (#7053 )	2020-03-10 22:23:01 +00:00
dklimpel	39f6595b4a	lint, fix tests	2020-03-09 22:13:20 +01:00
dklimpel	885134529f	updates after review	2020-03-09 22:09:29 +01:00
dklimpel	7e5f40e771	fix tests	2020-03-09 21:00:36 +01:00
dklimpel	50ea178c20	lint	2020-03-09 19:57:04 +01:00
dklimpel	04f4b5f6f8	add tests	2020-03-09 19:51:31 +01:00
Patrick Cloke	06eb5cae08	Remove special auth and redaction rules for aliases events in experimental room ver. (#7037 )	2020-03-09 08:58:25 -04:00
dklimpel	1f5f3ae8b1	Add options to disable setting profile info for prevent changes.	2020-03-08 14:49:33 +01:00
Neil Johnson	1d66dce83e	Break down monthly active users by appservice_id (#7030 ) * Break down monthly active users by appservice_id and emit via prometheus. Co-authored-by: Brendan Abolivier <babolivier@matrix.org>	2020-03-06 18:14:19 +00:00
Patrick Cloke	13892776ef	Allow deleting an alias if the user has sufficient power level (#6986 )	2020-03-04 11:30:46 -05:00
Richard van der Hoff	8ef8fb2c1c	Read the room version from database when fetching events (#6874 ) This is a precursor to giving EventBase objects the knowledge of which room version they belong to.	2020-03-04 13:11:04 +00:00
Brendan Abolivier	43f874055d	Merge branch 'master' into develop	2020-03-03 15:20:49 +00:00
Patrick Cloke	7dcbc33a1b	Validate the alt_aliases property of canonical alias events (#6971 )	2020-03-03 07:12:45 -05:00
Richard van der Hoff	b68041df3d	Add a whitelist for the SSO confirmation step.	2020-03-02 17:05:09 +00:00
Richard van der Hoff	27d099edd6	Merge remote-tracking branch 'origin/release-v1.11.1' into release-v1.11.1	2020-03-02 16:43:33 +00:00
Brendan Abolivier	b2bd54a2e3	Add a confirmation step to the SSO login flow	2020-03-02 16:36:32 +00:00
Dirk Klimpel	bbeee33d63	Fixed set a user as an admin with the new API (#6928 ) Fix #6910	2020-03-02 13:28:50 +00:00
Erik Johnston	f70f44abc7	Remove handling of multiple rows per ID	2020-02-28 11:45:35 +00:00
Dirk Klimpel	9b06d8f8a6	Fixed set a user as an admin with the new API (#6928 ) Fix #6910	2020-02-28 09:58:05 +00:00
Richard van der Hoff	cab4a52535	set worker_app for frontend proxy test (#7003 ) to stop the federationhandler trying to do master stuff	2020-02-27 13:08:43 +00:00
Richard van der Hoff	3e99528f2b	Store room version on invite (#6983 ) When we get an invite over federation, store the room version in the rooms table. The general idea here is that, when we pull the invite out again, we'll want to know what room_version it belongs to (so that we can later redact it if need be). So we need to store it somewhere...	2020-02-26 16:58:33 +00:00
Erik Johnston	1f773eec91	Port PresenceHandler to async/await (#6991 )	2020-02-26 15:33:26 +00:00
Andrew Morgan	8c75b621bf	Ensure 'deactivated' parameter is a boolean on user admin API, Fix error handling of call to deactivate user (#6990 )	2020-02-26 12:22:55 +00:00
Erik Johnston	bbf8886a05	Merge worker apps into one. (#6964 )	2020-02-25 16:56:55 +00:00
Patrick Cloke	509e381afa	Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957 ) Ensure good comprehension hygiene using flake8-comprehensions.	2020-02-21 07:15:07 -05:00
Erik Johnston	2b37eabca1	Reduce auth chains fetched during v2 state res. (#6952 ) The state res v2 algorithm only cares about the difference between auth chains, so we can pass in the known common state to the `get_auth_chain` storage function so that it can ignore those events.	2020-02-19 15:04:47 +00:00
Richard van der Hoff	2fb7794e60	Merge pull request #6949 from matrix-org/rav/list_room_aliases_peekable Make room alias lists peekable	2020-02-19 11:19:11 +00:00
Richard van der Hoff	880aaac1d8	Move MSC2432 stuff onto unstable prefix (#6948 ) it's not in the spec yet, so needs to be unstable. Also add a feature flag for it. Also add a test for admin users.	2020-02-19 10:40:27 +00:00
Richard van der Hoff	709e81f518	Make room alias lists peekable As per https://github.com/matrix-org/matrix-doc/pull/2432#pullrequestreview-360566830, make room alias lists accessible to users outside world_readable rooms.	2020-02-19 08:53:32 +00:00
Richard van der Hoff	b58d17e44f	Refactor the membership check methods in Auth these were getting a bit unwieldy, so let's combine `check_joined_room` and `check_user_was_in_room` into a single `check_user_in_room`.	2020-02-18 23:21:44 +00:00
Richard van der Hoff	adfaea8c69	Implement GET /_matrix/client/r0/rooms/{roomId}/aliases (#6939 ) per matrix-org/matrix-doc#2432	2020-02-18 16:23:25 +00:00
Richard van der Hoff	3f1cd14791	Merge pull request #6872 from matrix-org/rav/dictproperty Rewrite _EventInternalMetadata to back it with a dict	2020-02-18 16:21:02 +00:00
Patrick Cloke	fe3941f6e3	Stop sending events when creating or deleting aliases (#6904 ) Stop sending events when creating or deleting associations (room aliases). Send an updated canonical alias event if one of the alt_aliases is deleted.	2020-02-18 07:29:44 -05:00
Andrew Morgan	3404ad289b	Raise the default power levels for invites, tombstones and server acls (#6834 )	2020-02-17 13:23:37 +00:00
Richard van der Hoff	43b2be9764	Replace _event_dict_property with DictProperty this amounts to the same thing, but replaces `_event_dict` with `_dict`, and removes some of the function layers generated by `property`.	2020-02-14 16:42:37 +00:00
Patrick Cloke	02e89021f5	Convert the directory handler tests to use HomeserverTestCase (#6919 ) Convert directory handler tests to use HomeserverTestCase.	2020-02-14 09:05:43 -05:00
Patrick Cloke	49f877d32e	Filter the results of user directory searching via the spam checker (#6888 ) Add a method to the spam checker to filter the user directory results.	2020-02-14 07:17:54 -05:00
Andrew Morgan	d8994942f2	Return a 404 for admin api user lookup if user not found (#6901 )	2020-02-12 18:14:10 +00:00
Patrick Cloke	a92e703ab9	Reject device display names that are too long (#6882 ) * Reject device display names that are too long. Too long is currently defined as 100 characters in length. * Add a regression test for rejecting a too long device display name.	2020-02-10 16:35:26 -05:00
Richard van der Hoff	799001f2c0	Add a `make_event_from_dict` method (#6858 ) ... and use it in places where it's trivial to do so. This will make it easier to pass room versions into the FrozenEvent constructors.	2020-02-07 15:30:04 +00:00
Erik Johnston	b08b0a22d5	Add typing to synapse.federation.sender (#6871 )	2020-02-07 13:56:38 +00:00
Dirk Klimpel	56ca93ef59	Admin api to add an email address (#6789 )	2020-02-07 10:29:36 +00:00
Richard van der Hoff	928edef979	Pass room_version into `event_from_pdu_json` It's called from all over the shop, so this one's a bit messy.	2020-02-06 16:08:27 +00:00
Erik Johnston	b9391c9575	Add typing to SyncHandler (#6821 ) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2020-02-03 18:05:44 +00:00
Richard van der Hoff	a5afdd15e5	Merge pull request #6806 from matrix-org/rav/redact_changes/3 Pass room_version into add_hashes_and_signatures	2020-01-31 10:57:03 +00:00
Richard van der Hoff	d7bf793cc1	s/get_room_version/get_room_version_id/ ... to make way for a forthcoming get_room_version which returns a RoomVersion object.	2020-01-31 10:06:21 +00:00
Richard van der Hoff	2a81393a4b	Pass room_version into add_hashes_and_signatures	2020-01-30 22:15:50 +00:00
Richard van der Hoff	ef6bdafb29	Store the room version in EventBuilder	2020-01-30 22:15:50 +00:00
Richard van der Hoff	184303b865	MSC2260: Block direct sends of m.room.aliases events (#6794 ) as per MSC2260	2020-01-30 17:20:55 +00:00
Erik Johnston	b660327056	Resync remote device list when detected as stale. (#6786 )	2020-01-30 17:06:38 +00:00
Richard van der Hoff	5a246611e3	Type defintions for use in refactoring for redaction changes (#6803 ) * Bump signedjson to 1.1 ... so that we can use the type definitions * Fix breakage caused by upgrade to signedjson 1.1 Thanks, @illicitonion...	2020-01-30 11:25:59 +00:00
Richard van der Hoff	ee42a5513e	Factor out a `copy_power_levels_contents` method I'm going to need another copy (hah!) of this.	2020-01-29 11:52:52 +00:00
Richard van der Hoff	a8ce7aeb43	Pass room version object into event_auth.check and check_redaction (#6788 ) These are easier to work with than the strings and we normally have one around. This fixes `FederationHander._persist_auth_tree` which was passing a RoomVersion object into event_auth.check instead of a string.	2020-01-28 14:18:29 +00:00
Erik Johnston	8df862e45d	Add `rooms.room_version` column (#6729 ) This is so that we don't have to rely on pulling it out from `current_state_events` table.	2020-01-27 14:30:57 +00:00
Andrew Morgan	9f7aaf90b5	Validate client_secret parameter (#6767 )	2020-01-24 14:28:40 +00:00
Erik Johnston	fa4d609e20	Make 'event.redacts' never raise. (#6771 ) There are quite a few places that we assume that a redaction event has a corresponding `redacts` key, which is not always the case. So lets cheekily make it so that event.redacts just returns None instead.	2020-01-23 15:19:03 +00:00
Andrew Morgan	90a28fb475	Admin API to list, filter and sort rooms (#6720 )	2020-01-22 13:36:43 +00:00
Brendan Abolivier	6ae0c8db33	Lint + changelog	2020-01-22 12:38:18 +00:00
Brendan Abolivier	d9a8728b11	Remove unused import	2020-01-22 12:30:49 +00:00
Brendan Abolivier	67aa18e8dc	Add tests for thumbnailing	2020-01-22 12:28:07 +00:00
Andrew Morgan	aa9b00fb2f	Fix and add test to deprecated quarantine media admin api (#6756 )	2020-01-22 11:05:50 +00:00
Erik Johnston	74b74462f1	Fix `/events/:event_id` deprecated API. (#6731 )	2020-01-20 17:38:09 +00:00
Erik Johnston	ceecedc68b	Fix changing password via user admin API. (#6730 )	2020-01-20 17:23:59 +00:00
Richard van der Hoff	acc7820574	Log saml assertions rather than the whole response ... since the whole response is huge. We even need to break up the assertions, since kibana otherwise truncates them.	2020-01-16 22:26:34 +00:00
Erik Johnston	48c3a96886	Port synapse.replication.tcp to async/await (#6666 ) * Port synapse.replication.tcp to async/await * Newsfile * Correctly document type of on_<FOO> functions as async * Don't be overenthusiastic with the asyncing....	2020-01-16 09:16:12 +00:00
Erik Johnston	19a1aac48c	Fix purge_room admin API (#6711 )	2020-01-15 18:13:47 +00:00
Richard van der Hoff	8f5d7302ac	Implement RedirectException (#6687 ) Allow REST endpoint implemnentations to raise a RedirectException, which will redirect the user's browser to a given location.	2020-01-15 15:58:55 +00:00
Erik Johnston	28c98e51ff	Add `local_current_membership` table (#6655 ) Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.	2020-01-15 14:59:33 +00:00
Andrew Morgan	1177d3f3a3	Quarantine media by ID or user ID (#6681 )	2020-01-13 18:10:43 +00:00
Richard van der Hoff	326c893d24	Kill off RegistrationError (#6691 ) This is pretty pointless. Let's just use SynapseError.	2020-01-13 12:48:22 +00:00
Manuel Stahl	d2906fe666	Allow admin users to create or modify users without a shared secret (#6495 ) Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>	2020-01-09 13:31:00 +00:00
Manuel Stahl	7caaa29daa	Fix GET request on /_synapse/admin/v2/users endpoint (#6563 ) Fixes #6552	2020-01-08 13:26:40 +00:00
Richard van der Hoff	573fee759c	Back out ill-advised notary server hackery (#6657 ) This was ill-advised. We can't modify verify_keys here, because the response object has already been signed by the requested key. Furthermore, it's somewhat unnecessary because existing versions of Synapse (which get upset that the notary key isn't present in verify_keys) will fall back to a direct fetch via `/key/v2/server`. Also: more tests for fetching keys via perspectives: it would be nice if we actually tested when our fetcher can't talk to our notary impl.	2020-01-08 13:24:10 +00:00
Richard van der Hoff	d20c346544	port BackgroundUpdateTestCase to HomeserverTestCase (#6653 )	2020-01-07 14:09:07 +00:00
Richard van der Hoff	1807db5e73	Merge pull request #6629 from matrix-org/rav/kill_event_reference_hashes Remove a bunch of unused code from event creation	2020-01-06 17:46:31 +00:00
Richard van der Hoff	dc41fbf0dd	Remove unused get_prev_events_and_hashes_for_room	2020-01-06 13:45:33 +00:00
Richard van der Hoff	3bef62488e	Remove unused hashes and depths from create_event params	2020-01-06 13:45:33 +00:00
Richard van der Hoff	5a04781643	rename get_prev_events_for_room to get_prev_events_and_hashes_for_room ... to make way for a new method which just returns the event ids	2020-01-06 13:45:33 +00:00
Richard van der Hoff	4b36b482e0	Fix exception when fetching notary server's old keys (#6625 ) Lift the restriction that all the keys used for signing v2 key responses be present in verify_keys. Fixes #6596.	2020-01-06 12:33:56 +00:00
Richard van der Hoff	18674eebb1	Workaround for error when fetching notary's own key (#6620 ) * Kill off redundant SynapseRequestFactory We already get the Site via the Channel, so there's no need for a dedicated RequestFactory: we can just use the right constructor. * Workaround for error when fetching notary's own key As a notary server, when we return our own keys, include all of our signing keys in verify_keys. This is a workaround for #6596.	2020-01-06 12:28:58 +00:00
Richard van der Hoff	b6b57ecb4e	Kill off redundant SynapseRequestFactory (#6619 ) We already get the Site via the Channel, so there's no need for a dedicated RequestFactory: we can just use the right constructor.	2020-01-03 14:19:48 +00:00
Erik Johnston	75d8f26ac8	Split state groups into a separate data store (#6296 )	2019-12-20 10:48:24 +00:00
Erik Johnston	fa780e9721	Change EventContext to use the Storage class (#6564 )	2019-12-20 10:32:02 +00:00
Richard van der Hoff	d6752ce5da	Clean up startup for the pusher (#6558 ) * Remove redundant python2 support code `str.decode()` doesn't exist on python3, so presumably this code was doing nothing * Filter out pushers with corrupt data When we get a row with unparsable json, drop the row, rather than returning a row with null `data`, which will then cause an explosion later on. * Improve logging when we can't start a pusher Log the ID to help us understand the problem * Make email pusher setup more robust We know we'll have a `data` member, since that comes from the database. What we don't know is if that is a dict, and if that has a `brand` member, and if that member is a string.	2019-12-18 14:26:58 +00:00
Erik Johnston	2284eb3a53	Add database config class (#6513 ) This encapsulates config for a given database and is the way to get new connections.	2019-12-18 10:45:12 +00:00
Richard van der Hoff	6e8f8e14f2	Merge release-v1.7.1 into develop	2019-12-18 09:51:51 +00:00
Brendan Abolivier	6316530366	Merge pull request #6553 from matrix-org/babolivier/fix-context-filter Use the filtered version of an event when responding to /context requests for that event	2019-12-16 16:24:10 +00:00
Will Hunt	bfb95654c9	Add option to allow profile queries without sharing a room (#6523 )	2019-12-16 16:11:55 +00:00
Brendan Abolivier	a820069549	Incorporate review	2019-12-16 16:00:18 +00:00
Brendan Abolivier	a29420f9f4	Lint	2019-12-16 14:55:50 +00:00
Brendan Abolivier	596dd9914d	Add test case	2019-12-16 14:53:21 +00:00
Richard van der Hoff	83895316d4	sanity-checking for events used in state res (#6531 ) When we perform state resolution, check that all of the events involved are in the right room.	2019-12-16 13:53:53 +00:00
Richard van der Hoff	1da15f05f5	sanity-checking for events used in state res (#6531 ) When we perform state resolution, check that all of the events involved are in the right room.	2019-12-13 12:55:32 +00:00
Hubert Chathi	cb2db17994	look up cross-signing keys from the DB in bulk (#6486 )	2019-12-12 12:03:28 -05:00
Erik Johnston	b8e4b39b69	Merge pull request #6511 from matrix-org/erikj/remove_db_config_from_apps Move database config from apps into HomeServer object	2019-12-12 10:37:56 +00:00
Richard van der Hoff	894d2addac	Merge pull request #6517 from matrix-org/rav/event_auth/13 Port some of FederationHandler to async/await	2019-12-11 16:36:06 +00:00
Erik Johnston	31905a518e	Merge pull request #6504 from matrix-org/erikj/account_validity_async_await Port handlers.account_validity to async/await.	2019-12-11 14:49:26 +00:00
Richard van der Hoff	e77237b935	convert to async: FederationHandler.on_receive_pdu and associated functions: * on_receive_pdu * handle_queued_pdus * get_missing_events_for_pdu	2019-12-11 14:39:26 +00:00
Richard van der Hoff	40eda84933	Fix race which caused deleted devices to reappear (#6514 ) Stop the `update_client_ips` background job from recreating deleted devices.	2019-12-10 16:22:29 +00:00
Erik Johnston	257ef2c727	Port handlers.account_validity to async/await.	2019-12-10 14:40:15 +00:00
Erik Johnston	bc5cb8bfe8	Remove database config parsing from apps.	2019-12-10 14:34:17 +00:00
Erik Johnston	35f3c366ef	Merge pull request #6505 from matrix-org/erikj/make_deferred_yiedable Fix `make_deferred_yieldable` to work with coroutines	2019-12-10 14:20:26 +00:00
Erik Johnston	e3f528c544	Merge pull request #6506 from matrix-org/erikj/remove_snapshot_cache Remove SnapshotCache in favour of ResponseCache	2019-12-10 13:04:50 +00:00
Erik Johnston	9a2223d4c8	Fix make_deferred_yieldable to work with coroutines	2019-12-10 11:22:12 +00:00
Neil Johnson	adfdd82b21	Back out perf regression from get_cross_signing_keys_from_cache. (#6494 ) Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression.	2019-12-09 13:59:27 +00:00
Erik Johnston	30e9adf32f	Merge pull request #6487 from matrix-org/erikj/pass_in_db Pass in Database object to data stores.	2019-12-09 13:53:21 +00:00
Erik Johnston	f166a8d1f5	Remove SnapshotCache in favour of ResponseCache	2019-12-09 13:42:49 +00:00
Erik Johnston	a9b393340f	Merge pull request #6484 from matrix-org/erikj/port_sync_handler Port SyncHandler to async/await	2019-12-09 11:32:44 +00:00
Erik Johnston	852f80d8a6	Fixup tests	2019-12-06 16:02:50 +00:00
Erik Johnston	9a4fb457cf	Change DataStores to accept 'database' param.	2019-12-06 13:30:06 +00:00
Erik Johnston	2ace775d88	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_database_class	2019-12-06 11:33:34 +00:00
Erik Johnston	b3a4e35ca8	Fixup functions to consistently return deferreds	2019-12-06 10:40:05 +00:00
Erik Johnston	8437e2383e	Port SyncHandler to async/await	2019-12-05 17:58:25 +00:00
Erik Johnston	4ca3ef10b9	Fixup tests	2019-12-05 15:57:28 +00:00
Erik Johnston	4a33a6dd19	Move background update handling out of store	2019-12-05 11:11:26 +00:00
Erik Johnston	756d4942f5	Move DB pool and helper functions into dedicated Database class	2019-12-05 10:46:37 +00:00
Erik Johnston	ee86abb2d6	Remove underscore from SQLBaseStore functions	2019-12-04 16:23:43 +00:00
Brendan Abolivier	c530f9af4d	Merge pull request #6329 from matrix-org/babolivier/context_filters Filter state, events_before and events_after in /context requests	2019-12-04 15:24:16 +00:00
Brendan Abolivier	65c6aee621	Un-remove room purge test	2019-12-04 14:37:04 +00:00
Brendan Abolivier	aeda1b3b94	Merge branch 'babolivier/context_filters' of github.com:matrix-org/synapse into babolivier/context_filters	2019-12-04 14:29:03 +00:00
Brendan Abolivier	9dc84b7989	Merge branch 'develop' into babolivier/context_filters	2019-12-04 14:23:44 +00:00
Neil Johnson	cb0aeb147e	privacy by default for room dir (#6355 ) Ensure that the the default settings for the room directory are that the it is hidden from public view by default.	2019-12-04 09:46:16 +00:00
Brendan Abolivier	54dd5dc12b	Add ephemeral messages support (MSC2228) (#6409 ) Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are: * the feature is hidden behind a configuration flag (`enable_ephemeral_messages`) * self-destruction doesn't happen for state events * only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one) * doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database	2019-12-03 19:19:45 +00:00
Erik Johnston	8c9a713f8d	Add tests	2019-11-28 11:32:06 +00:00
Amber Brown	0f87b912ab	Implementation of MSC2314 (#6176 )	2019-11-28 08:54:07 +11:00
Hubert Chathi	0d27aba900	add etag and count to key backup endpoints (#5858 )	2019-11-27 16:14:44 -05:00
Brendan Abolivier	d31f69afa0	Merge pull request #6358 from matrix-org/babolivier/message_retention Implement message retention policies (MSC1763)	2019-11-27 15:04:38 +00:00
Andrew Morgan	ce578031f4	Remove assertion and provide a clear warning on startup for missing public_baseurl (#6379 )	2019-11-26 18:42:27 +00:00
Brendan Abolivier	9e937c28ee	Merge branch 'develop' into babolivier/message_retention	2019-11-26 17:53:57 +00:00
Brendan Abolivier	f0ef970824	Don't restrict the tests to v1 rooms	2019-11-26 17:49:12 +00:00
Brendan Abolivier	4c1b799e1b	Merge branch 'develop' into babolivier/context_filters	2019-11-26 10:53:48 +00:00
Amber Brown	9eebd46048	Improve the performance of structured logging (#6322 )	2019-11-26 03:45:50 +11:00
Brendan Abolivier	e2a20326e8	Lint	2019-11-20 15:08:47 +00:00
Brendan Abolivier	6356f2088f	Test if a purge can make /messages return 500 responses	2019-11-20 12:09:06 +00:00
Brendan Abolivier	bf9a11c54d	Lint again	2019-11-19 13:30:04 +00:00
Brendan Abolivier	7c24d0f443	Lint	2019-11-19 13:22:37 +00:00
Andrew Morgan	c350bc2f92	Blacklist PurgeRoomTestCase (#6361 )	2019-11-13 19:09:20 +00:00
Erik Johnston	f713c01e2b	Merge pull request #6295 from matrix-org/erikj/split_purge_history Split purge API into events vs state and add PurgeEventsStorage	2019-11-08 10:19:15 +00:00
Brendan Abolivier	f03c9d3444	Don't apply retention policy based filtering on state events As per MSC1763, 'Retention is only considered for non-state events.', so don't filter out state events based on the room's retention policy.	2019-11-06 18:40:04 +00:00
Erik Johnston	5c3363233c	Fix deleting state groups during room purge. And fix the tests to actually test that things got deleted.	2019-11-06 17:02:08 +00:00
Richard van der Hoff	807ec3bd99	Fix bug which caused rejected events to be stored with the wrong room state (#6320 ) Fixes a bug where rejected events were persisted with the wrong state group. Also fixes an occasional internal-server-error when receiving events over federation which are rejected and (possibly because they are backwards-extremities) have no prev_group. Fixes #6289.	2019-11-06 10:01:39 +00:00
Richard van der Hoff	55a7da247a	Merge branch 'develop' into rav/url_preview_limit_title	2019-11-05 17:08:07 +00:00
Richard van der Hoff	e78167c94b	Apply suggestions from code review Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-Authored-By: Erik Johnston <erik@matrix.org>	2019-11-05 16:46:39 +00:00
Richard van der Hoff	e9bfe719ba	Strip overlong OpenGraph data from url preview ... to stop people causing DoSes with malicious web pages	2019-11-05 15:51:18 +00:00
Brendan Abolivier	a6863da249	Lint	2019-11-05 14:50:19 +00:00
Brendan Abolivier	8822b33111	Update copyrights	2019-11-05 14:46:35 +00:00
Brendan Abolivier	037360e6cf	Add tests for /search	2019-11-05 14:33:18 +00:00
Brendan Abolivier	c9e4748cb7	Merge labels tests for /context and /messages	2019-11-05 13:47:47 +00:00
Brendan Abolivier	a7c818c79b	Add test case	2019-11-05 13:21:26 +00:00
Erik Johnston	ffe595381d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-11-05 10:27:41 +00:00
Amber Brown	4e1c7b79fa	Remove the psutil dependency (#6318 ) * remove psutil and replace with resource	2019-11-05 05:05:48 +11:00
Brendan Abolivier	09957ce0e4	Implement per-room message retention policies	2019-11-04 17:09:22 +00:00
Erik Johnston	6a0092d371	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-11-04 13:29:35 +00:00
Brendan Abolivier	f496d25877	Merge pull request #6301 from matrix-org/babolivier/msc2326 Implement MSC2326 (label based filtering)	2019-11-01 17:04:45 +00:00
Brendan Abolivier	988d8d6507	Incorporate review	2019-11-01 16:22:44 +00:00
Richard van der Hoff	c6516adbe0	Factor out an _AsyncEventContextImpl (#6298 ) The intention here is to make it clearer which fields we can expect to be populated when: notably, that the _event_type etc aren't used for the synchronous impl of EventContext.	2019-11-01 16:19:09 +00:00
Richard van der Hoff	1cb84c6486	Support for routing outbound HTTP requests via a proxy (#6239 ) The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will not be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.	2019-11-01 14:07:44 +00:00
Brendan Abolivier	c6dbca2422	Incorporate review	2019-11-01 10:30:51 +00:00
Hubert Chathi	6f4bc6d01d	Merge branch 'develop' into cross-signing_federation	2019-10-31 22:38:21 -04:00
Erik Johnston	cd581338cf	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-10-31 15:19:26 +00:00
Erik Johnston	dfe0cd71b6	Merge pull request #6294 from matrix-org/erikj/add_state_storage Add StateGroupStorage interface	2019-10-31 16:17:53 +01:00
Erik Johnston	97c60ccaa3	Add unit test for /purge_room API	2019-10-31 11:30:25 +00:00
Hubert Chathi	bb6cec27a5	rename get_devices_by_remote to get_device_updates_by_remote	2019-10-30 14:57:34 -04:00
Brendan Abolivier	dcc069a2e2	Lint	2019-10-30 18:01:56 +00:00
Brendan Abolivier	d8c9109aee	Add integration tests for /messages	2019-10-30 17:48:22 +00:00
Brendan Abolivier	fe51d6cacf	Add more integration testing	2019-10-30 17:28:41 +00:00
Brendan Abolivier	395683add1	Add integration tests for sync	2019-10-30 16:47:37 +00:00
Brendan Abolivier	e7943f660a	Add unit tests	2019-10-30 16:15:04 +00:00
Erik Johnston	7c8c97e635	Split purge API into events vs state	2019-10-30 15:23:37 +00:00
Erik Johnston	69f0054ce6	Port to use state storage	2019-10-30 14:46:54 +00:00
Hubert Chathi	7d7eac61be	Merge branch 'develop' into cross-signing_federation	2019-10-30 10:17:10 -04:00
Erik Johnston	ec6de1cc7d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store	2019-10-30 13:37:04 +00:00
Erik Johnston	a8d16f6c00	Review comments	2019-10-30 13:36:12 +00:00
Erik Johnston	326b3dace7	Make ObservableDeferred.observe() always return deferred. This makes it easier to use in an async/await world. Also fixes a bug where cache descriptors would occaisonally return a raw value rather than a deferred.	2019-10-30 11:35:46 +00:00
Erik Johnston	3f33879be4	Port federation_server to async/await	2019-10-29 14:13:08 +00:00
Erik Johnston	d0d8a22c13	Quick fix to ensure cache descriptors always return deferreds	2019-10-28 13:33:04 +00:00
Hubert Chathi	da78f61778	Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keys delete keys when deleting backup versions	2019-10-25 11:28:11 -04:00
Hubert Chathi	4697c0de0b	remove unneeded imports	2019-10-25 10:47:02 -04:00
Hubert Chathi	4cf3a30a20	switch to using HomeserverTestCase	2019-10-25 10:42:07 -04:00
Erik Johnston	7e7a1461f6	Fix tests	2019-10-25 10:57:37 +01:00
Hubert Chathi	c40d7244f8	Merge branch 'develop' into cross-signing_federation	2019-10-24 22:31:25 -04:00
Hubert Chathi	29a0bc5637	remove some unnecessary lines	2019-10-24 21:43:02 -04:00
Hubert Chathi	848cd388d9	delete keys when deleting backups	2019-10-24 21:21:51 -04:00
Neil Johnson	2794b79052	Option to suppress resource exceeded alerting (#6173 ) The expected use case is to suppress MAU limiting on small instances	2019-10-24 11:48:46 +01:00
Erik Johnston	3ca4c7c516	Use new EventPersistenceStore	2019-10-23 16:15:03 +01:00
Hubert Chathi	3e3f9b684e	fix unit test	2019-10-22 22:26:30 -04:00
Erik Johnston	c17efdc01c	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores	2019-10-22 18:44:29 +01:00
Erik Johnston	336eeea3ff	Fix postgres unit tests to use prepare_database	2019-10-22 11:02:01 +01:00
Erik Johnston	bb6264be0b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores	2019-10-22 10:41:18 +01:00
Erik Johnston	c66a06ac6b	Move storage classes into a main "data store". This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.	2019-10-21 16:05:06 +01:00
Hubert Chathi	36adfaedab	Merge branch 'develop' into cross-signing_sig_upload	2019-10-18 18:34:42 +01:00
Hubert Chathi	cfc28325a6	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-10-18 11:39:40 +01:00
Erik Johnston	5859a5c569	Fix presence timeouts when synchrotron restarts. (#6212 ) * Fix presence timeouts when synchrotron restarts. Handling timeouts would fail if there was an external process that had timed out, e.g. a synchrotron restarting. This was due to a couple of variable name typoes. Fixes #3715.	2019-10-18 06:42:26 +01:00
Hubert Chathi	06fc66c81e	Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage make storage layer in charge of interpreting the device key data	2019-10-11 15:19:06 -04:00
Hubert Chathi	691dd67fcd	Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version make version optional in body of e2e backup version update	2019-10-11 10:11:59 -04:00
Neil Johnson	a0d0ba7862	Fix MAU reaping where reserved users are specified. (#6168 )	2019-10-11 09:38:26 +01:00
Hubert Chathi	4908fb3b30	make storage layer in charge of interpreting the device key data	2019-10-10 19:15:30 -04:00
Erik Johnston	bc244627ac	Fix postgres unit tests	2019-10-10 15:37:53 +01:00
Hubert Chathi	5373de6cce	change test name to be unique	2019-10-10 08:54:07 -04:00
Erik Johnston	933034e2fe	Merge pull request #6127 from matrix-org/erikj/patch_inner Add more log context checks when patching inlineCallbacks	2019-10-10 13:47:50 +01:00
werner291	b5b03b7079	Add domain validation when creating room with list of invitees (#6121 )	2019-10-10 13:05:48 +01:00
krombel	2efd050c9d	send 404 as http-status when filter-id is unknown to the server (#2380 ) This fixed the weirdness of 400 vs 404 as http status code in the case the filter id is not known by the server. As e.g. matrix-js-sdk expects 404 to catch this situation this leads to unwanted behaviour.	2019-10-10 12:59:55 +01:00
Erik Johnston	9970f955ce	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner	2019-10-10 11:51:50 +01:00
Richard van der Hoff	a139420a3c	Fix races in room stats (and other) updates. (#6187 ) Hopefully this will fix the occasional failures we were seeing in the room directory. The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted before current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8. We can solve this easily by only processing up to the stream_id where we know all events have been persisted.	2019-10-10 11:29:01 +01:00
Erik Johnston	1d6dd1c294	Move patch_inline_callbacks into synapse/	2019-10-10 10:53:06 +01:00
Amber Brown	f743108a94	Refactor HomeserverConfig so it can be typechecked (#6137 )	2019-10-10 09:39:35 +01:00
Hubert Chathi	4535a07f4a	make version optional in body of e2e backup version update to agree with latest version of the MSC	2019-10-09 17:54:03 -04:00
Erik Johnston	5c1f886c75	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner	2019-10-09 16:52:21 +01:00
Brendan Abolivier	ae0b78cb1f	Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked Don't 500 when trying to exchange a revoked 3PID invite	2019-10-04 12:09:05 +01:00
Brendan Abolivier	8a5e8e829b	Lint (again)	2019-10-03 11:30:43 +01:00
Brendan Abolivier	ebcb6a30d7	Lint	2019-10-03 11:29:07 +01:00
Brendan Abolivier	6527fa18c1	Add test case	2019-10-03 11:24:36 +01:00
Erik Johnston	a5166e4d5f	Land improved room list based on room stats (#6019 ) Use room_stats and room_state for room directory search	2019-10-02 14:08:35 +01:00
Erik Johnston	ecd254bc49	Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop	2019-10-02 11:08:07 +01:00
Erik Johnston	b4fe7e19c0	Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings Fix errors storing large retry intervals.	2019-10-02 11:05:22 +01:00
Erik Johnston	f44f1d2e83	Fix errors storing large retry intervals. We have set the max retry interval to a value larger than a postgres or sqlite int can hold, which caused exceptions when updating the destinations table. To fix postgres we need to change the column to a bigint, and for sqlite we lower the max interval to 2**62 (which is still incredibly long).	2019-10-02 10:36:27 +01:00
Erik Johnston	ce7a3e7e27	Fix fetching censored redactions from DB Fetching a censored redactions caused an exception due to the code expecting redactions to have a `redact` key, which redacted redactions don't have.	2019-10-02 10:14:01 +01:00
Erik Johnston	6374ca40c2	Update	2019-09-27 15:58:14 +01:00
Brendan Abolivier	fbb8ff3088	ok	2019-09-27 15:23:07 +01:00
Brendan Abolivier	873fe7883c	Lint	2019-09-27 15:21:03 +01:00
Brendan Abolivier	e94ff67903	Add test to validate the change	2019-09-27 15:14:02 +01:00
Erik Johnston	132279a46f	Patch inlinecallbacks for log contexts	2019-09-27 15:11:14 +01:00
Neil Johnson	034db2ba21	Fix dummy event insertion consent bug (#6053 ) Fixes #5905	2019-09-26 11:47:53 +01:00
Erik Johnston	4fb3c129aa	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2	2019-09-25 17:53:13 +01:00
Erik Johnston	d2bd0bc6b1	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips	2019-09-25 17:16:28 +01:00
Richard van der Hoff	990928abde	Stop advertising unsupported flows for registration (#6107 ) If email or msisdn verification aren't supported, let's stop advertising them for registration. Fixes #6100.	2019-09-25 12:10:26 +01:00
Richard van der Hoff	2cd98812ba	Refactor the user-interactive auth handling (#6105 ) Pull the checkers out to their own classes, rather than having them lost in a massive 1000-line class which does everything. This is also preparation for some more intelligent advertising of flows, as per #6100	2019-09-25 11:33:03 +01:00
Richard van der Hoff	8004d6ca2f	Refactor code for calculating registration flows (#6106 ) because, frankly, it looked like it was written by an axe-murderer. This should be a non-functional change, except that where `m.login.dummy` was previously advertised before `m.login.terms`, it will now be advertised afterwards. AFAICT that should have no effect, and will be more consistent with the flows that involve passing a 3pid.	2019-09-25 11:32:05 +01:00
Erik Johnston	faac453f08	Test that pruning of old user IPs works	2019-09-24 15:53:17 +01:00
Erik Johnston	acb62a7cc6	Test background update	2019-09-23 16:59:45 +01:00
Erik Johnston	51d28272e2	Query devices table for last seen info. This is a) simpler than querying user_ips directly and b) means we can purge older entries from user_ips without losing the required info. The storage functions now no longer return the access_token, since it was unused.	2019-09-23 16:59:45 +01:00
Richard van der Hoff	1e19ce00bf	Add 'failure_ts' column to 'destinations' table (#6016 ) Track the time that a server started failing at, for general analysis purposes.	2019-09-17 11:41:54 +01:00
Amber Brown	850dcfd2d3	Fix well-known lookups with the federation certificate whitelist (#5997 )	2019-09-14 04:58:38 +10:00
Amber Brown	b617864cd9	Fix for structured logging tests stomping on logs (#6023 )	2019-09-13 02:29:55 +10:00
Jason Robinson	6d847d8ce6	Ensure support users can be registered even if MAU limit is reached This allows support users to be created even on MAU limits via the admin API. Support users are excluded from MAU after creation, so it makes sense to exclude them in creation - except if the whole host is in disabled state. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-11 20:48:31 +03:00
Erik Johnston	57dd41a45b	Fix comments Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2019-09-11 13:54:50 +01:00
Erik Johnston	caa9d6fed7	Add test for admin redaction ratelimiting.	2019-09-11 11:18:04 +01:00
Jason Robinson	63f9317b8e	Merge pull request #6004 from matrix-org/jaywink/autojoin-create-real-users Only count real users when checking for auto-creation of auto-join room	2019-09-09 17:37:52 +03:00
Erik Johnston	470dc621ae	Merge pull request #5934 from matrix-org/erikj/censor_redactions Censor redactions in DB after a month	2019-09-09 15:29:39 +01:00
Amber Brown	aeb9b2179e	Add a build info metric to Prometheus (#6005 )	2019-09-10 00:14:58 +10:00
Jason Robinson	be618e0551	Only count real users when checking for auto-creation of auto-join room Previously if the first registered user was a "support" or "bot" user, when the first real user registers, the auto-join rooms were not created. Fix to exclude non-real (ie users with a special user type) users when counting how many users there are to determine whether we should auto-create a room. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 14:48:08 +03:00
Hubert Chathi	8e86f5b65c	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-09-07 13:20:34 -04:00
Hubert Chathi	0d61d1d735	Merge branch 'develop' into cross-signing_sig_upload	2019-09-07 13:14:45 -04:00
Hubert Chathi	ab729e31cf	use something that's the right type for user_id	2019-09-06 17:52:37 -04:00
Amber Brown	55d5b3af88	Servers-known-about statistic (#5981 )	2019-09-07 01:45:51 +10:00
Hubert Chathi	e47af0f086	fix test	2019-09-05 17:03:14 -04:00
Erik Johnston	591d82f06b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions	2019-09-05 17:27:46 +01:00
Erik Johnston	ad9b64b496	Fix test	2019-09-05 17:17:47 +01:00
Erik Johnston	3ff0422d2d	Make redaction retention period configurable	2019-09-05 17:16:45 +01:00
Hubert Chathi	5914fd09c7	add test	2019-09-04 22:05:02 -04:00
Hubert Chathi	9061b4198a	make isort happy	2019-09-04 22:04:41 -04:00
Hubert Chathi	7d6c70fc7a	make black happy	2019-09-04 22:04:12 -04:00
Hubert Chathi	ac4746ac4b	allow uploading signatures of master key signed by devices	2019-09-04 22:03:41 -04:00
Hubert Chathi	faf72a4c40	Merge branch 'develop' into cross-signing_keys	2019-09-04 19:12:29 -04:00
Erik Johnston	6e834e94fc	Fix and refactor room and user stats (#5971 ) Previously the stats were not being correctly populated.	2019-09-04 13:04:27 +01:00
Andrew Morgan	4548d1f87e	Remove unnecessary parentheses around return statements (#5931 ) Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)	2019-08-30 16:28:26 +01:00
Erik Johnston	a4bf72c30c	Censor redactions in DB after a month	2019-08-30 11:29:17 +01:00
Hubert Chathi	e3d3fbf63f	Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys	2019-08-28 17:36:46 -07:00
Will Hunt	c8fa620d7a	Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent Exempt support users from consent	2019-08-28 16:31:40 +01:00
Jorik Schellekens	6d97843793	Config templating (#5900 ) Template config files * Imagine a system composed entirely of x, y, z etc and the basic operations.. Wait George, why XOR? Why not just neq? George: Eh, I didn't think of that.. Co-Authored-By: Erik Johnston <erik@matrix.org>	2019-08-28 13:12:22 +01:00
Amber Brown	7dc398586c	Implement a structured logging output system. (#5680 )	2019-08-28 21:18:53 +10:00
Half-Shot	0fb5189072	Fix registration test	2019-08-23 09:25:35 +01:00
Erik Johnston	1e4b4d85e7	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reliable_lookups	2019-08-22 13:41:57 +01:00
Erik Johnston	1b09cf8658	Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_fail Retry well known on fail	2019-08-22 13:17:05 +01:00
Richard van der Hoff	5019945828	Refactor the Appservice scheduler code Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code (it seemed to be previously inexplicably split between `ApplicationServiceScheduler.start` and `_Recoverer.start`). Add some docstrings too.	2019-08-20 11:50:23 +01:00
Erik Johnston	7777d353bf	Remove test debugs	2019-08-20 11:46:59 +01:00
Erik Johnston	1dec31560e	Change jitter to be a factor rather than absolute value	2019-08-20 11:46:00 +01:00
Erik Johnston	f299c5414c	Refactor MatrixFederationAgent to retry SRV. This refactors MatrixFederationAgent to move the SRV lookup into the endpoint code, this has two benefits: 1. Its easier to retry different host/ports in the same way as HostnameEndpoint. 2. We avoid SRV lookups if we have a free connection in the pool	2019-08-15 15:43:22 +01:00
Erik Johnston	e6e136decc	Retry well known on fail. If we have recently seen a valid well-known for a domain we want to retry on (non-final) errors a few times, to handle temporary blips in networking/etc.	2019-08-15 09:28:58 +01:00
Erik Johnston	17e1e80726	Retry well-known lookup before expiry. This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.	2019-08-13 16:20:38 +01:00
Erik Johnston	e9906b0772	Merge pull request #5836 from matrix-org/erikj/lower_bound_ttl_well_known Add a lower bound to well-known TTL.	2019-08-13 12:41:16 +01:00
Richard van der Hoff	fb86217553	Merge pull request #5788 from matrix-org/rav/metaredactions Fix handling of redactions of redactions	2019-08-12 12:25:19 +01:00
Erik Johnston	107ad133fc	Move well known lookup into a separate clas	2019-08-07 15:36:38 +01:00
Erik Johnston	af9f1c0764	Add a lower bound for TTL on well known results. It costs both us and the remote server for us to fetch the well known for every single request we send, so we add a minimum cache period. This is set to 5m so that we still honour the basic premise of "refetch frequently".	2019-08-06 17:01:23 +01:00
Hubert Chathi	8c9adcc95d	fix formatting	2019-08-01 22:09:05 -04:00
Hubert Chathi	336c546d6a	Merge branch 'cross-signing_hidden' into cross-signing_keys	2019-08-01 16:31:40 -04:00
Brendan Abolivier	8ed9e63432	Account validity: allow defining HTML templates to serve the us… (#5807 ) Account validity: allow defining HTML templates to serve the user on account renewal attempt	2019-08-01 16:09:25 +02:00
Brendan Abolivier	bc35503528	Add tests	2019-08-01 12:00:08 +02:00
Erik Johnston	8fde611a8c	Merge pull request #5794 from matrix-org/erikj/share_ssl_options_for_well_known Share SSL options for well-known requests	2019-07-31 11:40:02 +01:00
Amber Brown	8f15832950	Remove DelayedCall debugging from test runs (#5787 )	2019-07-31 20:39:22 +10:00
Erik Johnston	a9bcae9f50	Share SSL options for well-known requests	2019-07-31 10:39:24 +01:00
Erik Johnston	63eb4a1b62	Merge pull request #5746 from matrix-org/erikj/test_bg_update_currnet_state Add unit test for current state membership bg update	2019-07-30 10:00:02 +01:00
Richard van der Hoff	8c97f6414c	Remove non-functional 'expire_access_token' setting (#5782 ) The `expire_access_token` didn't do what it sounded like it should do. What it actually did was make Synapse enforce the 'time' caveat on macaroons used as access tokens, but since our access token macaroons never contained such a caveat, it was always a no-op. (The code to add 'time' caveats was removed back in v0.18.5, in #1656)	2019-07-30 08:25:02 +01:00
Richard van der Hoff	4e97eb89e5	Handle loops in redaction events	2019-07-30 00:00:34 +01:00
Amber Brown	865077f1d1	Room Complexity Client Implementation (#5783 )	2019-07-30 02:47:27 +10:00
Richard van der Hoff	1cad8d7b6f	Convert RedactionTestCase to modern test style (#5768 )	2019-07-26 07:38:55 +01:00
Hubert Chathi	c659b9f94f	allow uploading keys for cross-signing	2019-07-25 11:08:24 -04:00
Richard van der Hoff	618bd1ee76	Fix some error cases in the caching layer. (#5749 ) There was some inconsistent behaviour in the caching layer around how exceptions were handled - particularly synchronously-thrown ones. This seems to be most easily handled by pushing the creation of ObservableDeferreds down from CacheDescriptor to the Cache.	2019-07-25 15:59:45 +01:00
Andrew Morgan	baf081cd3b	Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734)) -----BEGIN PGP SIGNATURE----- iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl04Ur0THGFuZHJld0Bh bW9yZ2FuLnh5egAKCRCIhIgNLv5f9F4oD/0TY6S/SEd2uAmzor64ojmbX5BOwPzf j/wzUTrfvuf40EvkNPDpnejNZSvy/ysbaGQaQusv0SQKlV3xrvdn4RuMvnOWVWck kBsO+lvzOaUTR0KHDxN4y9F5eI2NdPbub4847PPVzyqSIHAd+kolxXS8kSBBhwpL yfaICWV/AOy5L7xN+JZ9IQpnegVAvUj5DmgXzDHd6VdeiHDVJuARaBgrR5uCkwVS ZoLRqZ95XV/qiguMAUvPOwyEqht2mwO64989MswP16YYm8oMkB5QA6I5nYnACsTP qk9YcN/oNvEfQXUhttku6MxK1/4yUMPUhEoDBDH7ebc0440QDtWN+IHTdA6oPVZB IuStL9YGY16m7Ltx37ZUA4URfNMiSeLHo3zKc/mCAcwxN4HyOjJewtxbG5zKQAOZ SMs8UcDwGR4zL1hnt8ZDNYtWwfzJBQIdGjoHvjXJEY7/1csTv2lmAwewFTXiqSAr 30GW5ews94kotqBK53zZT6V0F5gHNqgGHniOz1ZpqLLxYLqO3LSAGe97CrqlWUdX GkhA9tZyweknociD9fyyBmKdcFJ4mL4a+oGI5CMnSMph8UvCY8Y5XMb1T+iYEABI tA9G3mBvgkLPj+5V+8QggNkBafSigW2Q4FX7enGsDmiiskZOtfeKrAcVkapD4ooi 3I7IW5aetZr2IQ== =+JBn -----END PGP SIGNATURE----- Merge tag 'v1.2.0rc2' into develop Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))	2019-07-24 13:47:51 +01:00
Jorik Schellekens	cf2972c818	Fix servlet metric names (#5734 ) * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths	2019-07-24 13:07:35 +01:00
Erik Johnston	73bbaf2bc6	Add unit test for current state membership bg update	2019-07-23 17:00:22 +01:00
Amber Brown	4806651744	Replace returnValue with return (#5736 )	2019-07-23 23:00:55 +10:00
Richard van der Hoff	c7095be913	Refactor Keyring._start_key_lookups There's an awful lot of deferreds and dictionaries flying around here. The whole thing can be made much simpler and achieve the same effect.	2019-07-19 17:49:19 +01:00
Amber Brown	7ad1d76356	Support Prometheus_client 0.4.0+ (#5636 )	2019-07-18 23:57:15 +10:00
Andrew Morgan	b2a382efdb	Remove the ability to query relations when the original event was redacted. (#5629 ) Fixes #5594 Forbid viewing relations on an event once it has been redacted.	2019-07-18 14:41:42 +01:00
Richard van der Hoff	9c70a02a9c	Ignore redactions of m.room.create events (#5701 )	2019-07-17 19:08:02 +01:00
Richard van der Hoff	2091c91fde	More refactoring in `get_events_as_list` (#5707 ) We can now use `_get_events_from_cache_or_db` rather than going right back to the database, which means that (a) we can benefit from caching, and (b) it opens the way forward to more extensive checks on the original event. We now always require the original event to exist before we will serve up a redaction.	2019-07-17 17:34:13 +01:00
Erik Johnston	d86321300a	Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data Add basic function to get all data for a user out of synapse	2019-07-15 10:04:02 +01:00
Richard van der Hoff	5f158ec039	Implement access token expiry (#5660 ) Record how long an access token is valid for, and raise a soft-logout once it expires.	2019-07-12 17:26:02 +01:00
Richard van der Hoff	6bb0357c94	Add a mechanism for per-test configs (#5657 ) It's useful to be able to tweak the homeserver config to be used for each test. This PR adds a mechanism to do so.	2019-07-12 10:16:23 +01:00
Richard van der Hoff	0a4001eba1	Clean up exception handling for access_tokens (#5656 ) First of all, let's get rid of `TOKEN_NOT_FOUND_HTTP_STATUS`. It was a hack we did at one point when it was possible to return either a 403 or a 401 if the creds were missing. We always return a 401 in these cases now (thankfully), so it's not needed. Let's also stop abusing `AuthError` for these cases. Honestly they have nothing that relates them to the other places that `AuthError` is used, other than the fact that they are loosely under the 'Auth' banner. It makes no sense for them to share exception classes. Instead, let's add a couple of new exception classes: `InvalidClientTokenError` and `MissingClientTokenError`, for the `M_UNKNOWN_TOKEN` and `M_MISSING_TOKEN` cases respectively - and an `InvalidClientCredentialsError` base class for the two of them.	2019-07-11 11:06:23 +01:00
Richard van der Hoff	1890cfcf82	Inline issue_access_token (#5659 ) this is only used in one place, so it's clearer if we inline it and reduce the API surface. Also, fixes a buglet where we would create an access token even if we were about to block the user (we would never return the AT, so the user could never use it, but it was still created and added to the db.)	2019-07-11 04:10:07 +10:00
Richard van der Hoff	953dbb7980	Remove access-token support from RegistrationStore.register (#5642 ) The 'token' param is no longer used anywhere except the tests, so let's kill that off too.	2019-07-10 16:26:49 +01:00
Andrew Morgan	d88421ab03	Include the original event in /relations (#5626 ) When asking for the relations of an event, include the original event in the response. This will mostly be used for efficiently showing edit history, but could be useful in other circumstances.	2019-07-09 13:43:08 +01:00
Brendan Abolivier	af67c7c1de	Merge pull request #5644 from matrix-org/babolivier/profile-allow-self Allow newly-registered users to lookup their own profiles	2019-07-09 10:25:40 +01:00
Richard van der Hoff	824707383b	Remove access-token support from RegistrationHandler.register (#5641 ) Nothing uses this now, so we can remove the dead code, and clean up the API. Since we're changing the shape of the return value anyway, we take the opportunity to give the method a better name.	2019-07-08 19:01:08 +01:00
Brendan Abolivier	73cb716b3c	Lint	2019-07-08 17:44:20 +01:00
Brendan Abolivier	5e01e9ac19	Add test case	2019-07-08 17:41:16 +01:00
Richard van der Hoff	1af2fcd492	Move get_or_create_user to test code (#5628 ) This is only used in tests, so...	2019-07-08 23:52:26 +10:00
Andrew Morgan	ad8b909ce9	Add origin_server_ts and sender fields to m.replace (#5613 ) Riot team would like some extra fields as part of m.replace, so here you go. Fixes: #5598	2019-07-05 17:20:02 +01:00
Erik Johnston	7f0d8e4288	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data	2019-07-05 14:08:21 +01:00
Richard van der Hoff	9481707a52	Fixes to the federation rate limiter (#5621 ) - Put the default window_size back to 1000ms (broken by #5181) - Make the `rc_federation` config actually do something - fix an off-by-one error in the 'concurrent' limit - Avoid creating an unused `_PerHostRatelimiter` object for every single incoming request	2019-07-05 11:10:19 +01:00
Erik Johnston	c061d4f237	Fixup from review comments.	2019-07-04 11:41:06 +01:00
Amber Brown	463b072b12	Move logging utilities out of the side drawer of util/ and into logging/ (#5606 )	2019-07-04 00:07:04 +10:00
Amber Brown	0ee9076ffe	Fix media repo breaking (#5593 )	2019-07-02 19:01:28 +01:00
Erik Johnston	8ee69f299c	Add basic function to get all data for a user out of synapse	2019-07-02 12:09:04 +01:00
Amber Brown	f40a7dc41f	Make the http server handle coroutine-making REST servlets (#5475 )	2019-06-29 17:06:55 +10:00
Amber Brown	be3b901ccd	Update the TLS cipher string and provide configurability for TLS on outgoing federation (#5550 )	2019-06-28 18:19:09 +10:00
Brendan Abolivier	bfe84e051e	Split public rooms directory auth config in two	2019-06-24 15:42:31 +01:00
Richard van der Hoff	c3c6b00d95	Pass config_dir_path and data_dir_path into Config.read_config. (#5522 ) * Pull config_dir_path and data_dir_path calculation out of read_config_files * Pass config_dir_path and data_dir_path into read_config	2019-06-24 11:34:45 +01:00
Amber Brown	32e7c9e7f2	Run Black. (#5482 )	2019-06-20 19:32:02 +10:00
Erik Johnston	554609288b	Run as background process and fix comments	2019-06-19 11:33:03 +01:00
Erik Johnston	b42f90470f	Add experimental option to reduce extremities. Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.	2019-06-18 15:02:18 +01:00
Erik Johnston	dd927b29e1	Merge pull request #5388 from matrix-org/erikj/fix_email_push Fix email notifications for unnamed rooms with multiple people	2019-06-17 13:54:35 +01:00
Brendan Abolivier	f12e1f029c	Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules Allow server admins to define implementations of extra rules for allowing or denying incoming events	2019-06-14 19:37:59 +01:00
Erik Johnston	9ca4ae7131	Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitive Prometheus histograms are cumalative	2019-06-14 18:21:42 +01:00
Brendan Abolivier	f874b16b2e	Add plugin APIs for implementations of custom event rules.	2019-06-14 18:16:03 +01:00
Brendan Abolivier	6d56a694f4	Don't send renewal emails to deactivated users	2019-06-14 15:05:56 +01:00
Erik Johnston	3ed595e327	Prometheus histograms are cumalative	2019-06-14 14:07:32 +01:00
Brendan Abolivier	d0530382ee	Track deactivated accounts in the database (#5378 )	2019-06-14 13:18:24 +01:00
Amber H. Brown	a10c8dae85	fix prometheus rendering error	2019-06-14 21:09:33 +10:00
Amber Brown	6312d6cc7c	Expose statistics on extrems to prometheus (#5384 )	2019-06-13 22:40:52 +10:00
Erik Johnston	7e68691ce9	Merge branch 'master' of github.com:matrix-org/synapse into develop	2019-06-11 17:25:16 +01:00
Erik Johnston	a766c41d25	Bump bleach version so that tests can run on old deps.	2019-06-11 12:34:18 +01:00
Erik Johnston	10383e6e6f	Change password reset links to /_matrix.	2019-06-11 11:34:33 +01:00
Erik Johnston	abce00fc6a	Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into develop	2019-06-10 18:33:55 +01:00
Erik Johnston	a6b1817940	Merge pull request #5417 from matrix-org/rav/shared_ssl_context Share an SSL context object between SSL connections	2019-06-10 18:16:12 +01:00
Richard van der Hoff	db74c4fc6c	fix ci on py2, again	2019-06-10 17:55:01 +01:00
Richard van der Hoff	19780a521e	fix CI on python 2.7	2019-06-10 17:41:10 +01:00
Richard van der Hoff	e016681221	Tests for SSL certs for federation connections Add some tests for bad certificates for federation and .well-known connections	2019-06-10 16:14:20 +01:00
Erik Johnston	0382b0ffee	Merge pull request #5415 from matrix-org/erikj/fix_null_valid_until_ms Fix key verification when key stored with null valid_until_ms	2019-06-10 15:56:36 +01:00
Erik Johnston	43badd2cd4	Fix key verification when key stored with null valid_until_ms Some keys are stored in the synapse database with a null valid_until_ms which caused an exception to be thrown when using that key. We fix this by treating nulls as zeroes, i.e. they keys will match verification requests with a minimum_valid_until_ms of zero (i.e. don't validate ts) but will not match requests with a non-zero minimum_valid_until_ms. Fixes #5391.	2019-06-10 15:37:10 +01:00
Brendan Abolivier	26b62796c2	Merge pull request #5363 from matrix-org/babolivier/account_validity_send_mail_auth Don't check whether the user's account is expired on /send_mail requests	2019-06-10 11:57:02 +01:00
Erik Johnston	2ebeda48b2	Add test	2019-06-07 12:15:33 +01:00
Erik Johnston	8182a1cfb5	Refactor email tests	2019-06-07 12:15:33 +01:00
Andrew Morgan	2d1d7b7e6f	Prevent multiple device list updates from breaking a batch send (#5156 ) fixes #5153	2019-06-06 23:54:00 +01:00
Neil Johnson	a11865016e	Set default room version to v4. (#5379 ) Set default room version to v4.	2019-06-06 20:13:47 +01:00
Andrew Morgan	3719680ee4	Add ability to perform password reset via email without trusting the identity server (#5377 ) Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368	2019-06-06 17:34:07 +01:00
Richard van der Hoff	9fbb20a531	Stop hardcoding trust of old matrix.org key (#5374 ) There are a few changes going on here: * We make checking the signature on a key server response optional: if no verify_keys are specified, we trust to TLS to validate the connection. * We change the default config so that it does not require responses to be signed by the old key. * We replace the old 'perspectives' config with 'trusted_key_servers', which is also formatted slightly differently. * We emit a warning to the logs every time we trust a key server response signed by the old key.	2019-06-06 17:33:11 +01:00
Richard van der Hoff	cb3b381fcb	Merge pull request #5359 from matrix-org/rav/enable_tls_verification Validate federation server TLS certificates by default.	2019-06-06 10:50:42 +01:00
Richard van der Hoff	2eb47e5ee7	Merge pull request #5353 from matrix-org/rav/verify_key_logging Associate a request_name with each verify request, for logging	2019-06-06 09:33:09 +01:00
Brendan Abolivier	fe13bd52ac	Don't check whether the user's account is expired on /send_mail requests	2019-06-05 16:35:05 +01:00
Richard van der Hoff	7603a706eb	Merge branch 'rav/fix_custom_ca' into rav/enable_tls_verification	2019-06-05 16:32:35 +01:00
Erik Johnston	75538813fc	Fix background updates to handle redactions/rejections (#5352 ) * Fix background updates to handle redactions/rejections In background updates based on current state delta stream we need to handle that we may not have all the events (or at least that `get_events` may raise an exception).	2019-06-06 00:45:46 +10:00
Richard van der Hoff	e2dfb922e1	Validate federation server TLS certificates by default.	2019-06-05 14:17:50 +01:00
Richard van der Hoff	cea9750d11	Associate a request_name with each verify request, for logging Also: * rename VerifyKeyRequest->VerifyJsonRequest * calculate key_ids on VerifyJsonRequest construction * refactor things to pass around VerifyJsonRequests instead of 4-tuples	2019-06-05 10:46:26 +01:00
Richard van der Hoff	def5ea4062	Don't bomb out on direct key fetches as soon as one fails	2019-06-04 00:16:56 +01:00
Richard van der Hoff	fec2dcb1a5	Enforce validity period on server_keys for fed requests. (#5321 ) When handling incoming federation requests, make sure that we have an up-to-date copy of the signing key. We do not yet enforce the validity period for event signatures.	2019-06-03 22:59:51 +01:00
Amber Brown	2889b05554	Unify v1 and v2 REST client APIs (#5226 )	2019-06-03 21:28:59 +10:00
Richard van der Hoff	d828d1dc57	Merge pull request #5309 from matrix-org/rav/limit_displayname_length Limit displaynames and avatar URLs	2019-06-01 11:34:50 +01:00
Richard van der Hoff	93003aa172	add some tests	2019-06-01 11:14:37 +01:00
Erik Johnston	58cce39f3a	Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta Allow configuring a range for the account validity startup job	2019-05-31 12:11:56 +01:00
Brendan Abolivier	4d794dae21	Move delta from +10% to -10%	2019-05-31 11:09:39 +01:00
Brendan Abolivier	847b9dcd1c	Make max_delta equal to period * 10%	2019-05-31 09:54:46 +01:00
Erik Johnston	e9e5d3392d	Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities Add DB bg update to cleanup extremities.	2019-05-30 14:28:26 +01:00
Amber Brown	46c8f7a517	Implement the SHHS complexity API (#5216 )	2019-05-30 01:47:16 +10:00
Erik Johnston	d7add713a8	Add test	2019-05-29 15:59:06 +01:00
Richard van der Hoff	540f40f0cd	Merge pull request #5251 from matrix-org/rav/server_keys/01-check_sig Ensure that server_keys fetched via a notary server are correctly signed.	2019-05-28 21:32:17 +01:00
Brendan Abolivier	7e1c7cc274	Typo	2019-05-28 17:13:26 +01:00
Brendan Abolivier	52839886d6	Allow configuring a range for the account validity startup job When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal. In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.	2019-05-28 16:52:45 +01:00
Richard van der Hoff	fa1b293da2	Simplification to Keyring.wait_for_previous_lookups. (#5250 ) The list of server names was redundant, since it was equivalent to the keys on the server_to_deferred map. This reduces the number of large lists being passed around, and has the benefit of deduplicating the entries in `wait_on`.	2019-05-24 22:17:18 +01:00
Erik Johnston	8c41c04ee4	Merge pull request #5244 from matrix-org/rav/server_keys/00-factor-out-fetchers Factor out KeyFetchers from KeyRing	2019-05-23 16:28:51 +01:00
Richard van der Hoff	753b1270da	Require sig from origin server on perspectives responses	2019-05-23 15:01:09 +01:00
Andrew Morgan	6368150a74	Add config option for setting homeserver's default room version (#5223 ) Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present. That hardcoded value is now located in the server.py config file.	2019-05-23 15:00:20 +01:00
Richard van der Hoff	895b79ac2e	Factor out KeyFetchers from KeyRing Rather than have three methods which have to have the same interface, factor out a separate interface which is provided by three implementations. I find it easier to grok the code this way.	2019-05-23 13:46:47 +01:00
Richard van der Hoff	b75537beaf	Store key validity time in the storage layer This is a first step to checking that the key is valid at the required moment. The idea here is that, rather than passing VerifyKey objects in and out of the storage layer, we instead pass FetchKeyResult objects, which simply wrap the VerifyKey and add a valid_until_ts field.	2019-05-23 11:52:22 +01:00
Richard van der Hoff	2e052110ee	Rewrite store_server_verify_key to store several keys at once (#5234 ) Storing server keys hammered the database a bit. This replaces the implementation which stored a single key, with one which can do many updates at once.	2019-05-23 11:45:39 +01:00
Amber Brown	4a30e4acb4	Room Statistics (#4338 )	2019-05-21 11:36:50 -05:00
Erik Johnston	9526aa96a6	Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions Block attempts to annotate the same event twice	2019-05-21 17:08:14 +01:00
Brendan Abolivier	6a5a70edf0	Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date Add startup background job for account validity	2019-05-21 14:55:15 +01:00
Erik Johnston	c7ec06e8a6	Block attempts to annotate the same event twice	2019-05-20 17:39:05 +01:00
Erik Johnston	1dff859d6a	Rename relation types to match MSC	2019-05-20 14:31:19 +01:00
Erik Johnston	57ba3451b6	Merge pull request #5209 from matrix-org/erikj/reactions_base Land basic reaction and edit support.	2019-05-20 14:06:40 +01:00
Erik Johnston	2ac9c965dd	Fixup comments	2019-05-20 12:32:26 +01:00
Erik Johnston	935af0da38	Correctly update aggregation counts after redaction	2019-05-20 12:09:27 +01:00
ReidAnderson	3787133c9e	Limit UserIds to a length that fits in a state key (#5198 )	2019-05-20 11:20:08 +01:00
Brendan Abolivier	ad5b4074e1	Add startup background job for account validity If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.	2019-05-17 19:37:31 +01:00
Erik Johnston	d4ca533d70	Make tests use different user for each reaction it sends As users aren't allowed to react with the same emoji more than once.	2019-05-17 18:03:05 +01:00
Brendan Abolivier	fdeac1e984	Merge pull request #5196 from matrix-org/babolivier/per_room_profiles Add an option to disable per-room profiles	2019-05-17 12:10:49 +01:00
David Baker	07cff7b121	Merge pull request #5174 from matrix-org/dbkr/add_dummy_flow_to_recaptcha_only Re-order registration stages to do msisdn & email auth last	2019-05-16 17:27:39 +01:00
Erik Johnston	d46aab3fa8	Add basic editing support	2019-05-16 16:54:45 +01:00
Brendan Abolivier	8f9ce1a8a2	Lint	2019-05-16 15:25:54 +01:00
Brendan Abolivier	cc8c139a39	Lint	2019-05-16 15:20:59 +01:00
Brendan Abolivier	efdc55db75	Forgot copyright	2019-05-16 15:10:24 +01:00
Brendan Abolivier	54a582ed44	Add test case	2019-05-16 15:09:16 +01:00
Erik Johnston	95f3fcda3c	Check that event is visible in new APIs	2019-05-16 14:19:06 +01:00
Erik Johnston	a0603523d2	Add aggregations API	2019-05-16 09:37:20 +01:00
Amber Brown	f1e5b41388	Make all the rate limiting options more consistent (#5181 )	2019-05-15 12:06:04 -05:00
Erik Johnston	b50641e357	Add simple pagination API	2019-05-15 13:36:51 +01:00
Erik Johnston	efe3c7977a	Add simple send_relation API and track in DB	2019-05-15 13:36:51 +01:00
Erik Johnston	d94544051b	Merge pull request #5184 from matrix-org/erikj/expose_get_events_as_array Expose DataStore._get_events as get_events_as_list	2019-05-15 10:17:38 +01:00
Erik Johnston	495e859e58	Merge branch 'erikj/fix_config_ratelimiting' into erikj/test	2019-05-14 14:42:47 +01:00
Erik Johnston	dc4f6d1b01	Use correct config option for ratelimiting in tests	2019-05-14 14:37:40 +01:00
Erik Johnston	4fb44fb5b9	Expose DataStore._get_events as get_events_as_list This is in preparation for reaction work which requires it.	2019-05-14 13:37:44 +01:00
Amber Brown	df2ebd75d3	Migrate all tests to use the dict-based config format instead of hanging items off HomeserverConfig (#5171 )	2019-05-13 15:01:14 -05:00
Andrew Morgan	5a4b328f52	Add ability to blacklist ip ranges for federation traffic (#5043 )	2019-05-13 19:05:06 +01:00
David Baker	822072b1bb	Terms might not be the last stage	2019-05-13 16:10:26 +01:00
David Baker	9e99143c47	Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only	2019-05-13 15:37:03 +01:00
David Baker	8782bfb783	And now I realise why the test is failing...	2019-05-13 15:34:11 +01:00
Andrew Morgan	2f48c4e1ae	URL preview blacklisting fixes (#5155 ) Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.	2019-05-10 10:32:44 -07:00
Amber Brown	b36c82576e	Run Black on the tests again (#5170 )	2019-05-10 00:12:11 -05:00
Brendan Abolivier	d216a36b37	Fix bogus imports in tests (#5154 )	2019-05-08 21:57:03 +01:00
Matthew Hodgson	c0e0740bef	add options to require an access_token to GET /profile and /publicRooms on CS API (#5083 ) This commit adds two config options: * `restrict_public_rooms_to_local_users` Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API. * `require_auth_for_profile_requests` When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301. MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though. Groups have been intentionally omitted from this commit.	2019-05-08 18:26:56 +01:00
Brendan Abolivier	1473058b5e	Do checks on aliases for incoming m.room.aliases events (#5128 ) Follow-up to #5124 Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.	2019-05-08 17:01:30 +01:00
Richard van der Hoff	59e2d2694d	Remove the requirement to authenticate for /admin/server_version. (#5122 ) This endpoint isn't much use for its intended purpose if you first need to get yourself an admin's auth token. I've restricted it to the `/_synapse/admin` path to make it a bit easier to lock down for those concerned about exposing this information. I don't imagine anyone is using it in anger currently.	2019-05-07 09:29:30 +01:00
Richard van der Hoff	12f9d51e82	Add admin api for sending server_notices (#5121 )	2019-05-02 11:59:16 +01:00
Richard van der Hoff	40e576e29c	Move admin api impl to its own package It doesn't really belong under rest/client/v1 any more.	2019-05-01 15:44:30 +01:00
Andrew Morgan	6824ddd93d	Config option for verifying federation certificates (MSC 1711) (#4967 )	2019-04-25 14:22:49 +01:00
Erik Johnston	ca90336a69	Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration	2019-04-17 19:44:40 +01:00
Brendan Abolivier	eaf41a943b	Add management endpoints for account validity	2019-04-17 19:34:45 +01:00
Brendan Abolivier	91934025b9	Merge pull request #5047 from matrix-org/babolivier/account_expiration Send out emails with links to extend an account's validity period	2019-04-17 14:57:39 +01:00
Brendan Abolivier	20f0617e87	Send out emails with links to extend an account's validity period	2019-04-17 14:42:20 +01:00
Andrew Morgan	caa76e6021	Remove periods from copyright headers (#5046 )	2019-04-11 17:08:13 +01:00
Brendan Abolivier	bfc8fdf1fc	Merge pull request #5027 from matrix-org/babolivier/account_expiration Add time-based account expiration	2019-04-09 17:02:41 +01:00
Brendan Abolivier	747aa9f8ca	Add account expiration feature	2019-04-09 16:46:04 +01:00
Richard van der Hoff	18b69be00f	Rewrite Datastore.get_server_verify_keys Rewrite this so that it doesn't hammer the database.	2019-04-09 00:00:10 +01:00
Richard van der Hoff	0084309cd2	Rewrite test_keys as a HomeserverTestCase	2019-04-08 23:59:57 +01:00
Richard van der Hoff	7d2a0c848e	Fix from_server buglet in get_keys_from_perspectives make sure we store the name of the server the keys came from, rather than the origin server, after doing a fetch-from-perspectives.	2019-04-08 12:51:16 +01:00
Richard van der Hoff	7fc1e17f4c	Merge pull request #5001 from matrix-org/rav/keyring_cleanups Cleanups in the Keyring	2019-04-08 12:47:09 +01:00
Richard van der Hoff	b2d574f126	test for get_keys_from_perspectives	2019-04-04 19:12:54 +01:00
Richard van der Hoff	30805237fa	add a test for get_keys_from_server	2019-04-04 19:12:54 +01:00
Brendan Abolivier	8e85493b0c	Add config option to block users from looking up 3PIDs (#5010 )	2019-04-04 17:25:47 +01:00
Erik Johnston	c6a233a936	Add unit test for deleting groups	2019-04-03 16:29:52 +01:00
Andrew Morgan	4a4d5c4fd6	Fix grammar and document get_current_users_in_room (#4998 )	2019-04-03 14:32:20 +01:00
Richard van der Hoff	e4d473d855	Rewrite KeyringTestCase as a HomeServerTestCase (#4986 ) This is a bit fiddly due to the keyring doing weird things with logcontexts.	2019-04-03 14:11:27 +01:00
Neil Johnson	e8419554ff	Remove presence lists (#4989 ) Remove presence list support as per MSC 1819	2019-04-03 11:11:15 +01:00
Richard van der Hoff	297bf2547e	Fix sync bug when accepting invites (#4956 ) Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated before the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.	2019-04-02 12:42:39 +01:00
Neil Johnson	4c552ed78a	Neilj/fix threepid auth check (with tests) (#4474 ) test threepid checking	2019-04-01 17:42:18 +01:00
Richard van der Hoff	54a87a7b08	Collect room-version variations into one place (#4969 ) Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.	2019-04-01 10:24:38 +01:00
Erik Johnston	248014379e	Merge pull request #4942 from matrix-org/erikj/fix_presence Use event streams to calculate presence	2019-03-28 14:38:31 +00:00
Erik Johnston	40e56997bc	Review comments	2019-03-28 13:48:41 +00:00
Richard van der Hoff	a5798de067	Move replication.tcp.streams into a package	2019-03-27 21:13:14 +00:00
Erik Johnston	b7fa834c40	Add unit tests	2019-03-27 13:41:36 +00:00
Andrew Morgan	4a125be138	Make federation endpoints more tolerant of trailing slashes v2 (#4935 ) Redo of https://github.com/matrix-org/synapse/pull/4840	2019-03-26 11:35:29 +00:00
Richard van der Hoff	9bde730ef8	Fix bug where read-receipts lost their timestamps (#4927 ) Make sure that they are sent correctly over the replication stream. Fixes: #4898	2019-03-25 16:38:05 +00:00
Richard van der Hoff	a54a44734f	Use an explicit dbname for postgres connections in the tests. (#4928 ) I don't have a database with the same name as my user, so leaving the database name unset fails. While we're at it, clear out some unused stuff in the test setup.	2019-03-25 16:36:56 +00:00
Erik Johnston	5fee9d8067	Merge pull request #4869 from matrix-org/erikj/yaml_load Fix yaml warnings by using safe_load	2019-03-22 11:58:13 +00:00
Erik Johnston	3677548a82	Use yaml safe_load	2019-03-22 10:20:17 +00:00
Amber Brown	a68e00fca8	Some more porting to HomeserverTestCase and remove old RESTHelper (#4913 )	2019-03-22 02:10:21 +11:00
Andrew Morgan	7bef97dfb7	Remove trailing slashes from outbound federation requests and retry on 400 (#4840 ) As per #3622, we remove trailing slashes from outbound federation requests. However, to ensure that we remain backwards compatible with previous versions of Synapse, if we receive a HTTP 400 with `M_UNRECOGNIZED`, then we are likely talking to an older version of Synapse in which case we retry with a trailing slash appended to the request path.	2019-03-21 15:07:28 +00:00
Erik Johnston	01e6b405be	Merge pull request #4908 from matrix-org/erikj/block_peek_on_blocked_rooms Deny peeking into rooms that have been blocked	2019-03-21 14:07:17 +00:00
Erik Johnston	d3f640f0ac	isort	2019-03-21 11:29:48 +00:00
Erik Johnston	3959858eaa	Merge pull request #4904 from matrix-org/erikj/fix_shutdown Fixup shutdown room API	2019-03-21 11:24:42 +00:00
Erik Johnston	cd80cbffea	Fix typo and add description	2019-03-21 11:24:04 +00:00
Erik Johnston	536a266520	Deny peeking into rooms that have been blocked	2019-03-21 11:20:13 +00:00
Erik Johnston	4a8a1ac962	Rejig testcase to make it more extensible	2019-03-21 11:02:11 +00:00
Erik Johnston	9c9e618b93	Remove debug	2019-03-21 10:58:56 +00:00
Erik Johnston	5c6f61f81c	Add tests	2019-03-21 10:51:21 +00:00
Erik Johnston	09f991a63d	Merge pull request #4896 from matrix-org/erikj/disable_room_directory Add option to disable search room lists	2019-03-21 10:16:54 +00:00
Richard van der Hoff	a6f2d3053d	Log requests which are simulated by the unit tests. (#4905 ) Rather than stubbing out the access_log, make it actually log the requests, which makes it a lot more obvious what is going on during tests.	2019-03-20 18:00:02 +00:00
Erik Johnston	263f2c9ce1	Merge pull request #4895 from matrix-org/erikj/disable_user_search Add option to disable searching in the user dir	2019-03-20 16:47:15 +00:00
Richard van der Hoff	a902d13180	Batch up outgoing read-receipts to reduce federation traffic. (#4890 ) Rate-limit outgoing read-receipts as per #4730.	2019-03-20 16:02:25 +00:00
Erik Johnston	3660d24ebe	Add test	2019-03-20 15:16:36 +00:00
Erik Johnston	cc09685830	Add test	2019-03-20 14:53:44 +00:00
Andrew Morgan	bb52a2e653	lint	2019-03-20 14:08:57 +00:00
Andrew Morgan	cd36a1283b	New test, fix issues	2019-03-20 14:00:39 +00:00
Andrew Morgan	c69df5d5d3	Fix comments. v0.99.2 -> v0.99.3	2019-03-20 11:27:18 +00:00
Andrew Morgan	94cb7939e4	Federation test fixed!	2019-03-20 10:50:44 +00:00
Richard van der Hoff	b5d48560c7	Fix RegistrationTestCase turns out this relies on there being a `user_consent_version` set.	2019-03-19 12:05:05 +00:00
Richard van der Hoff	13bc1e0746	Use a regular HomeServerConfig object for unit tests Rather than using a Mock for the homeserver config, use a genuine HomeServerConfig object. This makes for a more realistic test, and means that we don't have to keep remembering to add things to the mock config every time we add a new config setting.	2019-03-19 11:44:43 +00:00
Richard van der Hoff	053c50bcb3	Fix resource limits tests Make sure that we have a `server_notices_mxid` set, given that we are relying on it.	2019-03-19 11:44:43 +00:00
Richard van der Hoff	45bb54a6c6	Fix registration test * Set allow_guest_access = True, since we rely on it * config doesn't have a `hostname` attribute; it is `server_name`	2019-03-19 11:44:43 +00:00
Richard van der Hoff	8c1774e821	Fix email test The Mailer expects the config object to have `email_smtp_pass` and `email_riot_base_url` attributes (and it won't by default, because the default config impl doesn't set any of the attributes unless email_enable_notifs is set).	2019-03-19 11:44:43 +00:00
Richard van der Hoff	45c4e19c74	Merge remote-tracking branch 'origin/develop' into HEAD	2019-03-19 11:43:46 +00:00
Neil Johnson	88f0675967	fix test_auto_create_auto_join_where_no_consent (#4886 )	2019-03-19 11:38:59 +00:00
Richard van der Hoff	0dbfae03f9	Enforce hs_disabled_message correctly Fixes a bug where hs_disabled_message was not enforced for 3pid-based requests if there was no server_notices_mxid configured.	2019-03-19 11:30:54 +00:00
Amber Brown	282c97327f	Migrate the user directory initial population to a background task (#4864 )	2019-03-19 04:50:24 +11:00
Andrew Morgan	621e7f37f1	Better exception handling	2019-03-18 17:45:54 +00:00
Brendan Abolivier	651ad8bc96	Add ratelimiting on failed login attempts (#4865 )	2019-03-18 12:57:20 +00:00
Brendan Abolivier	899e523d6d	Add ratelimiting on login (#4821 ) Add two ratelimiters on login (per-IP address and per-userID).	2019-03-15 17:46:16 +00:00
Erik Johnston	271cb1998b	Revert "Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793 )" This reverts commit `290552fd83`.	2019-03-14 14:30:54 +00:00
Erik Johnston	9073cfc8bd	Merge pull request #4846 from matrix-org/hawkowl/userdir-search Improve searching in the userdir	2019-03-14 13:54:15 +00:00
Andrew Morgan	ecea5af491	Correct var name	2019-03-13 21:21:03 +00:00
Andrew Morgan	86c60bda15	i should have given up x3	2019-03-13 20:19:07 +00:00
Andrew Morgan	45524f2f5e	i should have given up x2	2019-03-13 20:17:39 +00:00
Andrew Morgan	09626bfd39	Switch to wrapper function around _send_request	2019-03-13 18:26:06 +00:00
Andrew Morgan	97653ef1f4	Correct argument name	2019-03-12 14:30:26 +00:00
Amber Brown	797b6a63fc	fixup	2019-03-13 01:17:51 +11:00
Andrew Morgan	0ea8582f8b	Cleaner way of implementing trailing slashes	2019-03-12 14:11:11 +00:00
Amber Brown	d306bd1b26	fixup	2019-03-12 22:38:01 +11:00
Amber Brown	10480c4348	fixup	2019-03-12 21:47:14 +11:00
Andrew Morgan	4868b12029	and again	2019-03-11 18:22:26 +00:00
Andrew Morgan	802cb5dcf0	Fix syntax error	2019-03-11 18:08:28 +00:00
Andrew Morgan	66f205e93d	We're calling different functions now	2019-03-11 18:01:58 +00:00
Andrew Morgan	290552fd83	Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793 ) Server side of a solution towards #3622.	2019-03-11 17:44:03 +00:00
Amber Brown	5ba8ceab4c	fixes	2019-03-12 00:35:31 +11:00
Amber Brown	26eefca3b7	setup master	2019-03-11 21:16:10 +11:00
Amber Brown	30a8deeb68	Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search	2019-03-11 21:12:30 +11:00
Amber Brown	4abc988c6a	initial	2019-03-11 21:11:36 +11:00
Andrew Morgan	d6e0be92fe	Disable captcha registration by default in tests	2019-03-08 15:49:38 +00:00
Amber Brown	f6135d06cf	Rewrite userdir to be faster (#4537 )	2019-03-07 01:22:53 -08:00
Erik Johnston	16c8b4ecbd	Merge pull request #4772 from jbweston/jbweston/server-version-api Add 'server_version' endpoint to admin API	2019-03-05 16:31:00 +00:00
Brendan Abolivier	a4c3a361b7	Add rate-limiting on registration (#4735 ) * Rate-limiting for registration * Add unit test for registration rate limiting * Add config parameters for rate limiting on auth endpoints * Doc * Fix doc of rate limiting function Co-Authored-By: babolivier <contact@brendanabolivier.com> * Incorporate review * Fix config parsing * Fix linting errors * Set default config for auth rate limiting * Fix tests * Add changelog * Advance reactor instead of mocked clock * Move parameters to registration specific config and give them more sensible default values * Remove unused config options * Don't mock the rate limiter un MAU tests * Rename _register_with_store into register_with_store * Make CI happy * Remove unused import * Update sample config * Fix ratelimiting test for py2 * Add non-guest test	2019-03-05 14:25:33 +00:00
Richard van der Hoff	2c3548d9d8	Update test_typing to use HomeserverTestCase. (#4771 )	2019-03-04 10:05:39 +00:00
Joseph Weston	1e8388b311	Add 'server_version' endpoint to admin API This is required because the 'Server' HTTP header is not always passed through proxies.	2019-03-01 09:56:58 +01:00
Joseph Weston	b136ee10df	Import 'admin' module rather than 'register_servlets' directly We will later need also to import 'register_servlets' from the 'login' module, so we un-pollute the namespace now to keep the logical changes separate.	2019-03-01 09:46:25 +01:00
Richard van der Hoff	68f47d6744	Fix parsing of Content-Disposition headers (#4763 ) * Fix parsing of Content-Disposition headers TIL: filenames in content-dispostion headers can contain semicolons, and aren't %-encoded. * fix python2 incompatibility * Fix docstrings	2019-02-27 14:29:10 -08:00
Amber Brown	f2a753ea38	Move from TravisCI to BuildKite (#4752 )	2019-02-27 13:03:14 -08:00
Erik Johnston	d4dc527a1a	Fix unit tests	2019-02-27 14:24:45 +00:00
Amber Brown	a06614bd2a	UPSERT many functionality (#4644 )	2019-02-20 23:03:30 +11:00
Erik Johnston	dbdc565dfd	Fix registration on workers (#4682 ) * Move RegistrationHandler init to HomeServer * Move post registration actions to RegistrationHandler * Add post regisration replication endpoint * Newsfile	2019-02-20 18:47:31 +11:00
Erik Johnston	af691e415c	Move register_device into handler	2019-02-18 16:49:38 +00:00
Erik Johnston	41c3f21c3b	Fix unit tests	2019-02-18 13:43:16 +00:00
Erik Johnston	cb12a37708	Clarify and fix behaviour when there are multiple aliases	2019-02-14 18:16:32 +00:00
Erik Johnston	eaf4d11af9	Add configurable room list publishing rules This allows specifying who and what is allowed to be published onto the public room list	2019-02-14 16:02:23 +00:00
Richard van der Hoff	32b781bfe2	Fix error when loading cert if tls is disabled (#4618 ) If TLS is disabled, it should not be an error if no cert is given. Fixes #4554.	2019-02-12 10:51:31 +00:00
Richard van der Hoff	0ca2908653	fix tests	2019-02-11 22:01:27 +00:00
Richard van der Hoff	5d27730a73	Move ClientTLSOptionsFactory init out of refresh_certificates (#4611 ) It's nothing to do with refreshing the certificates. No idea why it was here.	2019-02-11 18:03:30 +00:00
Erik Johnston	719e073f00	Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating add updating of backup versions	2019-02-11 13:45:49 +00:00
Erik Johnston	b201149c7e	Merge pull request #4420 from matrix-org/jaywink/openid-listener New listener resource for the federation API "openid/userinfo" endpoint	2019-02-11 09:44:00 +00:00
Hubert Chathi	afae8442b5	make sure version is in body and wrap in linearizer queue also add tests	2019-02-08 01:32:45 -05:00
Richard van der Hoff	8a21b03fba	Treat an invalid .well-known the same as an absent one ... basically, carry on and fall back to SRV etc.	2019-02-01 11:37:31 +00:00
Richard van der Hoff	d428b46346	Update federation routing logic to check .well-known before SRV	2019-01-31 23:14:18 +00:00
Erik Johnston	6587b0b89b	Merge pull request #4472 from matrix-org/neilj/room_capabilities Server capabilities support	2019-01-30 14:26:56 +00:00
Richard van der Hoff	c7b24ac3d0	Follow redirects on .well-known (#4520 )	2019-01-30 11:43:33 +00:00
Richard van der Hoff	bc5f6e1797	Add a caching layer to .well-known responses (#4516 )	2019-01-30 10:55:25 +00:00
Amber Brown	3f189c902e	Fix flake8 (#4519 )	2019-01-30 10:53:17 +00:00
Neil Johnson	ee4df7fd7a	Merge branch 'develop' into neilj/room_capabilities	2019-01-30 10:28:08 +00:00
Neil Johnson	e4bef9d470	rework format of change password capability	2019-01-29 18:04:56 +00:00
Richard van der Hoff	cc2d650ef7	Relax requirement for a content-type on .well-known (#4511 )	2019-01-29 16:49:17 +00:00
Neil Johnson	f03b3a7a3a	support change_password in capabilities end-point	2019-01-29 15:58:37 +00:00
Amber Brown	6bd4374636	Do not generate self-signed TLS certificates by default. (#4509 )	2019-01-29 14:09:10 +00:00
Erik Johnston	b8d75ef53e	Merge pull request #4481 from matrix-org/erikj/event_builder Refactor event building into EventBuilder	2019-01-29 14:07:23 +00:00
Richard van der Hoff	99e36d5e24	Implement MSC1708 (.well-known lookups for server routing) (#4489 )	2019-01-29 13:53:02 +00:00
Neil Johnson	4eeb2fb215	isort	2019-01-29 12:44:10 +00:00
Richard van der Hoff	5488cadaae	Enable configuring test log level via env var (#4506 ) I got fed up with always adding '@unittest.DEBUG' every time I needed to debug a test.	2019-01-29 12:07:00 +00:00
Neil Johnson	a124025dab	enforce auth for capabilities endpoint	2019-01-29 11:37:56 +00:00
Erik Johnston	aee39f7de8	Fix test to use valid event format	2019-01-29 11:19:50 +00:00
Erik Johnston	554ca58ea1	Make add_hashes_and_signatures operate on dicts	2019-01-29 11:12:38 +00:00
Richard van der Hoff	f2b553d656	Use SimpleResolverComplexifier in tests (#4497 ) two reasons for this. One, it saves a bunch of boilerplate. Two, it squashes unicode to IDNA-in-a-`str` (even on python 3) in a way that it turns out we rely on to give consistent behaviour between python 2 and 3.	2019-01-29 09:38:29 +00:00
Erik Johnston	f1a04462eb	Merge pull request #4482 from matrix-org/erikj/event_auth_room_version Pass through room version to event auth	2019-01-28 20:09:38 +00:00
Amber Brown	7072fe3084	Fix UPSERTs on SQLite 3.24+ (#4477 )	2019-01-28 15:43:32 +00:00
Richard van der Hoff	0fd5b3b53e	Handle IP literals explicitly We don't want to be doing .well-known lookups on these guys.	2019-01-28 10:34:30 +00:00
Richard van der Hoff	d840019192	Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487 ) Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.	2019-01-28 09:56:59 +00:00
Erik Johnston	ae2a957dba	Pass through room version to event auth	2019-01-25 18:31:41 +00:00
Erik Johnston	b6dce9b9fd	Merge pull request #4470 from matrix-org/erikj/require_format_version Require event format version to parse or create events	2019-01-25 15:59:36 +00:00
Richard van der Hoff	8520bc3109	Fix Host header sent by MatrixFederationAgent (#4468 ) Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.	2019-01-25 12:38:16 +00:00
Neil Johnson	95f871fc0d	Support room version capabilities in CS API (MSC1804)	2019-01-25 11:16:29 +00:00
Erik Johnston	9770ed91c2	Fix tests	2019-01-25 10:32:26 +00:00
Erik Johnston	be6a7e47fa	Revert "Require event format version to parse or create events"	2019-01-25 10:23:51 +00:00
Erik Johnston	efb8ed1d45	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version	2019-01-24 18:52:34 +00:00
Richard van der Hoff	afd69a0920	Look up the right SRV record	2019-01-24 13:31:43 +00:00
Richard van der Hoff	e1c8440e0c	lots more tests for MatrixFederationAgent	2019-01-24 13:28:07 +00:00
Neil Johnson	10b89d5c2e	Merge pull request #4435 from matrix-org/neilj/fix_threepid_auth_check Neilj/fix threepid auth check	2019-01-24 13:02:50 +00:00
Amber Brown	58f6c48183	Use native UPSERTs where possible (#4306 )	2019-01-24 21:31:54 +11:00
Richard van der Hoff	97fd29c019	Don't send IP addresses as SNI (#4452 ) The problem here is that we have cut-and-pasted an impl from Twisted, and then failed to maintain it. It was fixed in Twisted in https://github.com/twisted/twisted/pull/1047/files; let's do the same here.	2019-01-24 09:34:44 +00:00
Erik Johnston	e79ba9eb34	Fix tests	2019-01-24 09:28:16 +00:00
Richard van der Hoff	6b574f3df7	fix python2 test failure	2019-01-23 11:25:36 +00:00
Jason Robinson	6f680241bd	Fix flake8 issues Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:53:48 +02:00
Jason Robinson	1838ef1ac3	Fix openid tests after rebase Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:38:13 +02:00
Jason Robinson	a47fac9af6	Fix sorting of imports in tests. Remove an unnecessary mock Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	5336e49b39	Add tests for the openid lister for SynapseHomeServer Check all possible variants of openid and federation listener on/off possibilities. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	6d25599098	Add tests for the openid lister for FederationReaderServer Check all possible variants of openid and federation listener on/off possibilities. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	ab97b6e33c	Fix a test docstring in frontend proxy tests Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Richard van der Hoff	d02c4532c0	Add a test for MatrixFederationAgent	2019-01-22 20:35:12 +00:00
Richard van der Hoff	7021784d46	put resolve_service in an object this makes it easier to stub things out for tests.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	53a327b4d5	Require that service_name be a byte string it is only ever a bytes now, so let's enforce that.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	7871146667	Make MatrixFederationClient use MatrixFederationAgent ... instead of the matrix_federation_endpoint	2019-01-22 20:34:35 +00:00
Neil Johnson	d619b113ed	Fix None guard in config.server.is_threepid_reserved	2019-01-22 16:52:29 +00:00
Richard van der Hoff	33a55289cb	Refactor and bugfix for resove_service (#4427 )	2019-01-22 10:59:27 +00:00
Amber Brown	23b0813599	Require ECDH key exchange & remove dh_params (#4429 ) * remove dh_params and set better cipher string	2019-01-22 21:58:50 +11:00
Richard van der Hoff	de6888e7ce	Remove redundant WrappedConnection (#4409 ) * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.	2019-01-18 23:07:38 +11:00
Richard van der Hoff	676cf2ee26	Fix incorrect logcontexts after a Deferred was cancelled (#4407 )	2019-01-17 14:00:23 +00:00
Amber Brown	522dada206	Merge remote-tracking branch 'origin/master' into develop	2019-01-11 02:22:48 +11:00
Richard van der Hoff	e0910d0145	Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1 Fixes #4371	2019-01-10 14:12:50 +00:00
Richard van der Hoff	aa70d24125	Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34	2019-01-10 12:58:33 +00:00
Richard van der Hoff	566947ff34	Skip macaroon check for access tokens in the db	2019-01-10 12:57:21 +00:00
Amber Brown	7960c26fda	Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369 )	2019-01-09 22:26:25 +11:00
Erik Johnston	b970cb0e96	Refactor request sending to have better excpetions (#4358 ) * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.	2019-01-08 11:04:28 +00:00
Amber Brown	d7843f47b6	Remove v1 only REST APIs now we don't ship matrix console (#4334 )	2018-12-29 23:12:30 +11:00
Richard van der Hoff	f85676cc93	Return well_known in /login response (#4319 ) ... as per MSC1730.	2018-12-24 10:44:33 +01:00
Richard van der Hoff	3355c1a9ec	Merge pull request #4317 from de-vri-es/test-metric-prometheus-0.5 Fix test_metrics.py compatibility prometheus_client 0.5	2018-12-23 00:14:47 +01:00

... 10 11 12 13 14 ...

2272 commits