MirrorHub/synapse
0
0
Fork 1
mirror of https://mau.dev/maunium/synapse.git synced 2024-12-30 03:04:00 +01:00
Code Issues Projects Releases Wiki Activity
15459 commits 5 branches 731 tags 91 MiB
Commit graph

1527 commits

Author SHA1 Message Date
Neil Johnson
ee4df7fd7a
Merge branch 'develop' into neilj/room_capabilities 2019-01-30 10:28:08 +00:00
Neil Johnson
e4bef9d470 rework format of change password capability 2019-01-29 18:04:56 +00:00
Richard van der Hoff
cc2d650ef7
Relax requirement for a content-type on .well-known (#4511) 2019-01-29 16:49:17 +00:00
Neil Johnson
f03b3a7a3a support change_password in capabilities end-point 2019-01-29 15:58:37 +00:00
Amber Brown
6bd4374636
Do not generate self-signed TLS certificates by default. (#4509) 2019-01-29 14:09:10 +00:00
Erik Johnston
b8d75ef53e
Merge pull request #4481 from matrix-org/erikj/event_builder 
Refactor event building into EventBuilder
 2019-01-29 14:07:23 +00:00
Richard van der Hoff
99e36d5e24
Implement MSC1708 (.well-known lookups for server routing) (#4489) 2019-01-29 13:53:02 +00:00
Neil Johnson
4eeb2fb215 isort 2019-01-29 12:44:10 +00:00
Richard van der Hoff
5488cadaae
Enable configuring test log level via env var (#4506) 
I got fed up with always adding '@unittest.DEBUG' every time I needed to debug a test.
 2019-01-29 12:07:00 +00:00
Neil Johnson
a124025dab enforce auth for capabilities endpoint 2019-01-29 11:37:56 +00:00
Erik Johnston
aee39f7de8 Fix test to use valid event format 2019-01-29 11:19:50 +00:00
Erik Johnston
554ca58ea1 Make add_hashes_and_signatures operate on dicts 2019-01-29 11:12:38 +00:00
Richard van der Hoff
f2b553d656 Use SimpleResolverComplexifier in tests (#4497) 
two reasons for this. One, it saves a bunch of boilerplate. Two, it squashes
unicode to IDNA-in-a-`str` (even on python 3) in a way that it turns out we
rely on to give consistent behaviour between python 2 and 3.
 2019-01-29 09:38:29 +00:00
Erik Johnston
f1a04462eb
Merge pull request #4482 from matrix-org/erikj/event_auth_room_version 
Pass through room version to event auth
 2019-01-28 20:09:38 +00:00
Amber Brown
7072fe3084
Fix UPSERTs on SQLite 3.24+ (#4477) 2019-01-28 15:43:32 +00:00
Richard van der Hoff
0fd5b3b53e Handle IP literals explicitly 
We don't want to be doing .well-known lookups on these guys.
 2019-01-28 10:34:30 +00:00
Richard van der Hoff
d840019192
Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487) 
Turns out that the library does a better job of parsing URIs than our
reinvented wheel. Who knew.

There are two things going on here. The first is that, unlike
parse_server_name, URI.fromBytes will strip off square brackets from IPv6
literals, which means that it is valid input to ClientTLSOptionsFactory and
HostnameEndpoint.

The second is that we stay in `bytes` throughout (except for the argument to
ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up
with idna-encoded values being held in `unicode` variables. TBH it probably
would have been ok but it made the tests fragile.
 2019-01-28 09:56:59 +00:00
Erik Johnston
ae2a957dba Pass through room version to event auth 2019-01-25 18:31:41 +00:00
Erik Johnston
b6dce9b9fd
Merge pull request #4470 from matrix-org/erikj/require_format_version 
Require event format version to parse or create events
 2019-01-25 15:59:36 +00:00
Richard van der Hoff
8520bc3109
Fix Host header sent by MatrixFederationAgent (#4468) 
Move the Host header logic down here so that (a) it is used if we reuse the
agent elsewhere, and (b) we can mess about with it with .well-known.
 2019-01-25 12:38:16 +00:00
Neil Johnson
95f871fc0d Support room version capabilities in CS API (MSC1804) 2019-01-25 11:16:29 +00:00
Erik Johnston
9770ed91c2 Fix tests 2019-01-25 10:32:26 +00:00
Erik Johnston
be6a7e47fa
Revert "Require event format version to parse or create events" 2019-01-25 10:23:51 +00:00
Erik Johnston
efb8ed1d45 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version 2019-01-24 18:52:34 +00:00
Richard van der Hoff
afd69a0920 Look up the right SRV record 2019-01-24 13:31:43 +00:00
Richard van der Hoff
e1c8440e0c lots more tests for MatrixFederationAgent 2019-01-24 13:28:07 +00:00
Neil Johnson
10b89d5c2e
Merge pull request #4435 from matrix-org/neilj/fix_threepid_auth_check 
Neilj/fix threepid auth check
 2019-01-24 13:02:50 +00:00
Amber Brown
58f6c48183
Use native UPSERTs where possible (#4306) 2019-01-24 21:31:54 +11:00
Richard van der Hoff
97fd29c019
Don't send IP addresses as SNI (#4452) 
The problem here is that we have cut-and-pasted an impl from Twisted, and then
failed to maintain it. It was fixed in Twisted in
https://github.com/twisted/twisted/pull/1047/files; let's do the same here.
 2019-01-24 09:34:44 +00:00
Erik Johnston
e79ba9eb34 Fix tests 2019-01-24 09:28:16 +00:00
Richard van der Hoff
6b574f3df7 fix python2 test failure 2019-01-23 11:25:36 +00:00
Jason Robinson
6f680241bd Fix flake8 issues 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:53:48 +02:00
Jason Robinson
1838ef1ac3 Fix openid tests after rebase 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:38:13 +02:00
Jason Robinson
a47fac9af6 Fix sorting of imports in tests. Remove an unnecessary mock 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Jason Robinson
5336e49b39 Add tests for the openid lister for SynapseHomeServer 
Check all possible variants of openid and federation listener on/off
possibilities.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Jason Robinson
6d25599098 Add tests for the openid lister for FederationReaderServer 
Check all possible variants of openid and federation listener on/off
possibilities.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Jason Robinson
ab97b6e33c Fix a test docstring in frontend proxy tests 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Richard van der Hoff
d02c4532c0 Add a test for MatrixFederationAgent 2019-01-22 20:35:12 +00:00
Richard van der Hoff
7021784d46 put resolve_service in an object 
this makes it easier to stub things out for tests.
 2019-01-22 20:35:12 +00:00
Richard van der Hoff
53a327b4d5 Require that service_name be a byte string 
it is only ever a bytes now, so let's enforce that.
 2019-01-22 20:35:12 +00:00
Richard van der Hoff
7871146667 Make MatrixFederationClient use MatrixFederationAgent 
... instead of the matrix_federation_endpoint
 2019-01-22 20:34:35 +00:00
Neil Johnson
d619b113ed Fix None guard in config.server.is_threepid_reserved 2019-01-22 16:52:29 +00:00
Richard van der Hoff
33a55289cb
Refactor and bugfix for resove_service (#4427) 2019-01-22 10:59:27 +00:00
Amber Brown
23b0813599
Require ECDH key exchange & remove dh_params (#4429) 
* remove dh_params and set better cipher string
 2019-01-22 21:58:50 +11:00
Richard van der Hoff
de6888e7ce Remove redundant WrappedConnection (#4409) 
* Remove redundant WrappedConnection

The matrix federation client uses an HTTP connection pool, which times out its
idle HTTP connections, so there is no need for any of this business.
 2019-01-18 23:07:38 +11:00
Richard van der Hoff
676cf2ee26
Fix incorrect logcontexts after a Deferred was cancelled (#4407) 2019-01-17 14:00:23 +00:00
Amber Brown
522dada206 Merge remote-tracking branch 'origin/master' into develop 2019-01-11 02:22:48 +11:00
Richard van der Hoff
e0910d0145 Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1 
Fixes #4371
 2019-01-10 14:12:50 +00:00
Richard van der Hoff
aa70d24125 Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34 2019-01-10 12:58:33 +00:00
Richard van der Hoff
566947ff34 Skip macaroon check for access tokens in the db 2019-01-10 12:57:21 +00:00
Amber Brown
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369) 2019-01-09 22:26:25 +11:00
Erik Johnston
b970cb0e96 Refactor request sending to have better excpetions (#4358) 
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
 2019-01-08 11:04:28 +00:00
Amber Brown
d7843f47b6
Remove v1 only REST APIs now we don't ship matrix console (#4334) 2018-12-29 23:12:30 +11:00
Richard van der Hoff
f85676cc93
Return well_known in /login response (#4319) 
... as per MSC1730.
 2018-12-24 10:44:33 +01:00
Richard van der Hoff
3355c1a9ec
Merge pull request #4317 from de-vri-es/test-metric-prometheus-0.5 
Fix test_metrics.py compatibility prometheus_client 0.5
 2018-12-23 00:14:47 +01:00
Amber Brown
ea6abf6724
Fix IP URL previews on Python 3 (#4215) 2018-12-22 01:56:13 +11:00
Maarten de Vries
48b7ff7a35 Fix test_metrics.py compatibility prometheus_client 0.5 
prometheus_client 0.5 has a named-tuple Sample type with more member
than the old plain tuple had. This commit makes sure the unit test
detects this and changes the way it reads the sample.

Signed-off-by: Maarten de Vries <maarten@de-vri.es>
 2018-12-21 01:53:57 +01:00
Neil Johnson
d2f7c4e6b1
create support user (#4141) 
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
 2018-12-14 18:20:59 +00:00
Richard van der Hoff
30da50a5b8
Initialise user displayname from SAML2 data (#4272) 
When we register a new user from SAML2 data, initialise their displayname
correctly.
 2018-12-07 14:44:46 +01:00
Richard van der Hoff
c7401a697f
Implement SAML2 authentication (#4267) 
This implements both a SAML2 metadata endpoint (at
`/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at
`/_matrix/saml2/authn_response`). If the SAML2 response matches what's been
configured, we complete the SSO login flow by redirecting to the client url
(aka `RelayState` in SAML2 jargon) with a login token.

What we don't yet have is anything to build a SAML2 request and redirect the
user to the identity provider. That is left as an exercise for the reader.
 2018-12-07 13:11:11 +01:00
Richard van der Hoff
c588b9b9e4
Factor SSO success handling out of CAS login (#4264) 
This is mostly factoring out the post-CAS-login code to somewhere we can reuse
it for other SSO flows, but it also fixes the userid mapping while we're at it.
 2018-12-07 13:10:07 +01:00
Richard van der Hoff
b0c24a66ec Rip out half-implemented m.login.saml2 support (#4265) 
* Rip out half-implemented m.login.saml2 support

This was implemented in an odd way that left most of the work to the client, in
a way that I really didn't understand. It's going to be a pain to maintain, so
let's start by ripping it out.

* drop undocumented dependency on dateutil

It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 19:44:38 +11:00
Richard van der Hoff
e8d98466b0
Implement .well-known handling (#4262) 
Sometimes it's useful for synapse to generate its own .well-known file.
 2018-12-05 14:38:58 +01:00
Richard van der Hoff
48972ce9d1
Patch defer.inlineCallbacks to check logcontexts in tests (#4205) 2018-12-04 11:30:32 +01:00
Neil Johnson
7039ece8fb Neilj/fix autojoin (#4223) 
* Fix auto join failures for servers that require user consent

* Fix auto join failures for servers that require user consent
 2018-11-28 22:24:57 +11:00
Richard van der Hoff
a44c0a096f Check logcontexts before and after each test (#4190) 
* Add better diagnostics to flakey keyring test

* fix interpolation fail

* Check logcontexts before and after each test

* update changelog

* update changelog
 2018-11-27 13:47:18 +11:00
Richard van der Hoff
80527b568d Fix more logcontext leaks in tests (#4209) 2018-11-27 13:01:04 +11:00
Richard van der Hoff
de8772a655 Do a GC after each test to fix logcontext leaks (#4227) 
* Some words about garbage collections and logcontexts

* Do a GC after each test to fix logcontext leaks

This feels like an awful hack, but...

* changelog
 2018-11-27 13:00:33 +11:00
Richard van der Hoff
416c671474
Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes 
Fix some logcontext leaks
 2018-11-20 12:19:19 +01:00
Amber Brown
80cac86b2c
Fix fallback auth on Python 3 (#4197) 2018-11-19 12:27:33 -06:00
Richard van der Hoff
828f18bd8b Fix logcontext leak in test_url_preview 2018-11-19 17:07:01 +00:00
Richard van der Hoff
a267c2e3ed Fix logcontext leak in http pusher test 2018-11-19 17:07:01 +00:00
Richard van der Hoff
884a561447 Fix some tests which leaked logcontexts 2018-11-19 17:07:01 +00:00
Travis Ralston
3da9781c98 Fix the terms UI auth tests 
By setting the config value directly, we skip the block that adds the slash automatically for us.
 2018-11-15 23:00:28 -07:00
Amber Brown
8b1affe7d5
Fix Content-Disposition in media repository (#4176) 2018-11-15 15:55:58 -06:00
Travis Ralston
835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
Amber Brown
df758e155d
Use <meta> tags to discover the per-page encoding of html previews (#4183) 2018-11-15 11:05:08 -06:00
Richard van der Hoff
c1efcd7c6a Add a test for the public T&Cs form 2018-11-14 10:46:27 +00:00
Erik Johnston
abaa93c158 Add test to assert set_e2e_device_keys correctly returns False on no-op 2018-11-08 14:06:44 +00:00
Amber Brown
b3708830b8
Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157) 2018-11-08 01:37:43 +11:00
Amber Brown
e62f7f17b3
Remove some boilerplate in tests (#4156) 2018-11-07 03:00:00 +11:00
Travis Ralston
0f5e51f726 Add config variables for enabling terms auth and the policy name (#4142) 
So people can still collect consent the old way if they want to.
 2018-11-06 10:32:34 +00:00
Hubert Chathi
f1087106cf
handle empty backups according to latest spec proposal (#4123) 
fixes #4056
 2018-11-05 17:59:29 -05:00
Amber Brown
efdcbbe46b
Tests for user consent resource (#4140) 2018-11-06 05:53:44 +11:00
Amber Brown
5a63589e80
Add some tests for the HTTP pusher (#4149) 2018-11-06 05:53:24 +11:00
Erik Johnston
bc80b3f454 Add helpers for getting prev and auth events (#4139) 
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
 2018-11-06 00:35:15 +11:00
Erik Johnston
90d713b8c6
Merge pull request #4137 from matrix-org/erikj/clean_up_events 
Clean up event accesses and tests
 2018-11-02 14:12:49 +00:00
Erik Johnston
b86d05a279 Clean up event accesses and tests 
This is in preparation to refactor FrozenEvent to support different
event formats for different room versions
 2018-11-02 13:44:14 +00:00
Amber Brown
cb7a6b2379
Fix typing being reset causing infinite syncs (#4127) 2018-11-03 00:19:23 +11:00
Erik Johnston
b199534518
Merge pull request #4135 from matrix-org/erikj/fix_state_res_none 
Fix None exception in state res v2
 2018-11-02 10:45:57 +00:00
Erik Johnston
54aec35867 Fix None exception in state res v2 2018-11-02 10:29:19 +00:00
Travis Ralston
a8c9faa9a2 The tests also need a version parameter 2018-10-31 13:28:08 -06:00
Travis Ralston
d1e7b9c44c Merge branch 'develop' into travis/login-terms 2018-10-31 13:15:14 -06:00
Amber Brown
3bade14ec0
Fix search 500ing (#4122) 2018-10-31 04:33:41 +11:00
Amber Brown
0dce9e1379
Write some tests for the email pusher (#4095) 2018-10-30 23:55:43 +11:00
Richard van der Hoff
bf33eed609
Merge pull request #4091 from matrix-org/rav/room_version_upgrades 
Room version upgrade support
 2018-10-29 12:47:20 +00:00
Amber Brown
77d70a7646
Port register_new_matrix_user to Python 3 and add tests (#4085) 2018-10-26 22:05:22 +11:00
Richard van der Hoff
474810d9d5 fix broken test 
This test stubbed out some stuff in a very weird way. I have no idea why. It broke.
 2018-10-25 23:15:03 +01:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011) 
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
 2018-10-25 17:49:55 +01:00
Erik Johnston
c85e063302
Merge pull request #4051 from matrix-org/erikj/alias_disallow_list 
Add config option to control alias creation
 2018-10-25 17:04:59 +01:00
Neil Johnson
95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init 
fix race condiftion in calling initialise_reserved_users
 2018-10-25 16:33:40 +01:00
Erik Johnston
b94a43d5b5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list 2018-10-25 15:25:31 +01:00
Erik Johnston
e5481b22aa Use allow/deny 2018-10-25 15:25:21 +01:00
Neil Johnson
f7f487e14c Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin 2018-10-25 14:40:06 +01:00
Travis Ralston
a5468eaadf pep8 2018-10-24 13:54:38 -06:00
Travis Ralston
81880beff4 It helps to import things 2018-10-24 13:32:13 -06:00
Travis Ralston
4acb6fe8a3 Move test to where the other integration tests are 2018-10-24 13:24:24 -06:00
Travis Ralston
9283987f7e Fix test 
Debug tests

Try printing the channel

fix

Import and use six

Remove debugging

Disable captcha

Add some mocks

Define the URL

Fix the clock?

Less rendering?

use the other render

Complete the dummy auth stage

Fix last stage of the test

Remove mocks we don't need
 2018-10-24 13:23:08 -06:00
Travis Ralston
54def42c19 Merge branch 'develop' into travis/login-terms 2018-10-24 13:22:59 -06:00
Neil Johnson
ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson
07126e43a4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init 2018-10-24 16:25:39 +01:00
Richard van der Hoff
3ad359e5be Merge remote-tracking branch 'origin/develop' into rav/fix_event_filter_validation 2018-10-24 11:23:49 +01:00
Richard van der Hoff
7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8 
Fix a number of flake8 errors
 2018-10-24 11:23:35 +01:00
Erik Johnston
3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase 
Add v2 state resolution algorithm
 2018-10-24 11:12:12 +01:00
Richard van der Hoff
7e07d25ed6 Allow backslashes in event field filters 
Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which
meant that single backslashes were not allowed in event field filters.

The intention here is to allow single-backslashes, but disallow
double-backslashes.
 2018-10-24 11:11:24 +01:00
Richard van der Hoff
ef771cc4c2 Fix a number of flake8 errors 
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
 2018-10-24 10:39:03 +01:00
Erik Johnston
b313b9b009 isort 2018-10-24 10:02:41 +01:00
Erik Johnston
810715f79a Rename resolve_events_with_factory 2018-10-24 09:44:22 +01:00
Neil Johnson
6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Erik Johnston
3c580c2b47 Add tests for alias creation rules 2018-10-19 10:22:45 +01:00
Erik Johnston
1b4bf232b9 Add tests for config generation 2018-10-19 10:22:45 +01:00
Travis Ralston
dba84fa69c Fix terms UI auth test 2018-10-18 12:45:21 -06:00
Travis Ralston
88c5ffec33 Test for terms UI auth 2018-10-18 12:35:30 -06:00
Will Hunt
d6a7797dd1 Fix roomlist since tokens on Python 3 (#4046) 
Thanks @Half-Shot !!!
 2018-10-17 23:04:55 +11:00
Erik Johnston
947c7443eb Add some state res v2 tests 2018-10-16 16:28:39 +01:00
Neil Johnson
a2bfb778c8 improve auto room join logic, comments and tests 2018-10-12 18:17:36 +01:00
David Baker
dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
Neil Johnson
ed82043efb
Merge branch 'develop' into matthew/autocreate_autojoin 2018-10-04 17:26:59 +01:00
Neil Johnson
2dadc092b8 move logic into register, fix room alias localpart bug, tests 2018-10-04 17:00:27 +01:00
Erik Johnston
9c834b8ee9 Add tests 2018-10-02 16:22:39 +01:00
Amber Brown
7232917f12
Disable frozen dicts by default (#3987) 2018-10-02 22:53:47 +10:00
Richard van der Hoff
b5b93f45d5
Merge pull request #3968 from matrix-org/rav/fix_federation_errors 
Fix exceptions when handling incoming transactions
 2018-10-01 15:54:24 +01:00
Amber Brown
6e05fd032c
Fix userconsent on Python 3 (#3938) 2018-10-02 00:11:58 +10:00
Richard van der Hoff
f094f715cf Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors 2018-09-27 15:18:21 +01:00
Amber Brown
a512e637ac
Merge pull request #3970 from schnuffle/develop-py3 
Replaced all occurences of e.message with str(e)
 2018-09-27 23:39:45 +10:00
Richard van der Hoff
484a9b8c81 Remove redundant, failing, test 
This test didn't do what it claimed to do, and what it claimed to do was the
same as test_cant_hide_direct_ancestors anyway.

This stuff is tested by sytest anyway.
 2018-09-27 13:11:23 +01:00
Schnuffle
dc5db01ff2 Replaced all occurences of e.message with str(e) 
Signed-off-by: Schnuffle  <schnuffle@github.com>
 2018-09-27 13:38:50 +02:00
Richard van der Hoff
4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions (#3959) 
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.

This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
 2018-09-27 11:25:34 +01:00
Richard van der Hoff
ae6ad4cf41
docstrings and unittests for storage.state (#3958) 
I spent ages trying to figure out how I was going mad...
 2018-09-27 11:22:25 +01:00
Richard van der Hoff
4c3e7eeec5
Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches 
Fix some instances of ExpiringCache not expiring cache items
 2018-09-25 12:02:57 +01:00
Amber Brown
fbe5ba25f6 Merge branch 'master' into develop 2018-09-25 03:10:01 +10:00
Oleg Girko
7d3f639844 Fix compatibility issue with older Twisted in tests. 
Older Twisted (18.4.0) returns TimeoutError instead of
ConnectingCancelledError when connection times out.
This change allows tests to be compatible with this behaviour.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
 2018-09-25 02:51:18 +10:00
Erik Johnston
8601c24287 Fix some instances of ExpiringCache not expiring cache items 
ExpiringCache required that `start()` be called before it would actually
start expiring entries. A number of places didn't do that.

This PR removes `start` from ExpiringCache, and automatically starts
backround reaping process on creation instead.
 2018-09-21 14:19:46 +01:00
Amber Brown
1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Amber Brown
aeca5a5ed5
Add a regression test for logging on failed connections (#3912) 2018-09-20 16:28:18 +10:00
Richard van der Hoff
31c15dcb80
Refactor matrixfederationclient to fix logging (#3906) 
We want to wait until we have read the response body before we log the request
as complete, otherwise a confusing thing happens where the request appears to
have completed, but we later fail it.

To do this, we factor the salient details of a request out to a separate
object, which can then keep track of the txn_id, so that it can be logged.
 2018-09-18 18:17:15 +01:00
Erik Johnston
f6e82dcddb Tests 2018-09-14 15:08:37 +01:00
Amber Brown
bfa0b759e0
Attempt to figure out what's going on with timeouts (#3857) 2018-09-14 00:15:51 +10:00
Neil Johnson
8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Neil Johnson
0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown
52ec6e9dfa
Port tests/ to Python 3 (#3808) 2018-09-07 02:58:18 +10:00
Neil Johnson
c5440b2ca0
Merge pull request #3800 from matrix-org/neilj/remove-guests-from-mau-count 
guest users should not be part of mau total
 2018-09-06 17:45:55 +01:00
Neil Johnson
84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Hubert Chathi
3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Richard van der Hoff
4f8baab0c4 Merge branch 'master' into develop 2018-09-06 13:05:22 +01:00
Neil Johnson
61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Richard van der Hoff
c127c8d042 Fix origin handling for pushed transactions 
Use the actual origin for push transactions, rather than whatever the remote
server claimed.
 2018-09-05 13:08:07 +01:00
Amber Brown
77055dba92
Fix tests on postgresql (#3740) 2018-09-04 02:21:48 +10:00
Neil Johnson
0b01281e77 move threepid checker to config, add missing yields 2018-08-31 17:11:11 +01:00
Neil Johnson
ea068d6f3c fix bug where preserved threepid user comes to sign up and server is mau blocked 2018-08-31 10:49:14 +01:00
Amber Brown
14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Richard van der Hoff
414fa36f3e Fix up tests 2018-08-28 17:21:05 +01:00
Hubert Chathi
16a31c6fce update to newer Synapse APIs 2018-08-24 22:51:25 -04:00
Erik Johnston
9db2476991 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_contact 2018-08-24 17:00:37 +01:00
Erik Johnston
05077e06fa Change admin_uri to admin_contact in config and errors 2018-08-24 16:51:27 +01:00
Hubert Chathi
83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Erik Johnston
01a5a8b9e3 Fix checking if service notice room is already tagged 
This manifested in synapse repeatedly setting the tag for the room
 2018-08-24 16:22:37 +01:00
Erik Johnston
f2cbbda956 Unit tests 2018-08-23 19:17:19 +01:00
Erik Johnston
cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston
7e6e588e60 Fix bug where we resent "limit exceeded" server notices 
This was due to a bug where we mutated a cached event's contents
 2018-08-23 16:21:20 +01:00
Erik Johnston
c5842dff1a Actually run the tests 2018-08-23 10:35:54 +01:00
Erik Johnston
fd2dbf1836 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-22 17:06:10 +01:00
Erik Johnston
9643a6f7f2 Update notice format 2018-08-22 17:00:29 +01:00
Erik Johnston
764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles 
Allow profile updates to happen on workers
 2018-08-22 11:35:55 +01:00
Erik Johnston
3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler 
Refactor state module to support multiple room versions
 2018-08-22 10:04:55 +01:00
Matthew Hodgson
bb81e78ec6
Split the state_group_cache in two (#3726) 
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
 2018-08-22 00:56:37 +02:00
Erik Johnston
4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Neil Johnson
e07970165f rename error code 2018-08-18 14:39:45 +01:00
Neil Johnson
ba1fbf7d5b special case server_notices_mxid 2018-08-18 12:31:08 +01:00
Amber Brown
c334ca67bb
Integrate presence from hotfixes (#3694) 2018-08-18 01:08:45 +10:00
Neil Johnson
d49b77404b clean up, no functional changes 2018-08-17 15:21:34 +01:00
Richard van der Hoff
63260397c6
Merge pull request #3701 from matrix-org/rav/use_producer_for_responses 
Use a producer to stream back responses
 2018-08-17 14:58:45 +01:00
Neil Johnson
3ee57bdcbb Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:34:10 +01:00
Erik Johnston
782689bd40 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-17 14:15:48 +01:00
Erik Johnston
ca87ad1def Split ProfileHandler into master and worker 2018-08-17 14:15:14 +01:00
Neil Johnson
b5f638f1f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:04:15 +01:00
Neil Johnson
69c49d3fa3
Merge branch 'develop' into neilj/limit_exceeded_error 2018-08-17 12:44:26 +00:00
Neil Johnson
bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Neil Johnson
13ad9930c8 add new error type ResourceLimit 2018-08-16 18:02:02 +01:00
Neil Johnson
3c1080b6e4 refactor for readability, and reuse caching for setting tags 2018-08-16 17:02:04 +01:00
Neil Johnson
a675f9c556 check for room state before deciding on action 2018-08-16 14:53:35 +01:00
Neil Johnson
8cfad2e686 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 17:20:38 +01:00
Neil Johnson
81d727efa9
Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus 
Fix Mau off by one errors
 2018-08-15 16:19:41 +00:00
Matthew Hodgson
2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Neil Johnson
fc5d937550 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 16:31:40 +01:00
Neil Johnson
86a00e05e1 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-15 16:27:08 +01:00
Amber Brown
a87af25fbb Fix the tests 2018-08-15 15:12:23 +01:00
Neil Johnson
75c663c7b9 update error codes 2018-08-15 11:27:48 +01:00
Neil Johnson
b8429c7c81 update error codes for resource limiting 2018-08-15 10:19:48 +01:00
Neil Johnson
ab035bdeac replace admin_email with admin_uri for greater flexibility 2018-08-15 10:16:41 +01:00
Neil Johnson
19b433e3f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email 2018-08-14 17:44:46 +01:00
Neil Johnson
b586b8b986 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 17:43:22 +01:00
Neil Johnson
7277216d01 fix setup_test_homeserver to be postgres compatible 2018-08-14 17:14:39 +01:00
Neil Johnson
1522ed9c07 in case max_mau is less than I think 2018-08-14 16:52:30 +01:00
Neil Johnson
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block 
Block ability to read via sync if mau limit exceeded
 2018-08-14 15:21:31 +00:00
Neil Johnson
06b331ff40 fix off by 1 errors 2018-08-14 15:28:15 +01:00
Neil Johnson
8f9a7eb58d support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-14 15:11:54 +01:00
Neil Johnson
ed4bc3d2fc fix off by 1s on mau 2018-08-14 15:04:48 +01:00
Neil Johnson
9b5bf3d858 Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 14:51:38 +01:00
Neil Johnson
e25d87d97b Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 14:32:18 +01:00
Amber Brown
bdfbd934d6
Implement a new test baseclass to cut down on boilerplate (#3684) 2018-08-14 20:53:43 +10:00
Neil Johnson
f4b49152e2 support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-13 21:09:47 +01:00
Neil Johnson
ce7de9ae6b Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases" 
This reverts commit 0d43f991a1.
 2018-08-13 18:06:18 +01:00
Neil Johnson
0d43f991a1 support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-13 18:00:23 +01:00
Amber Brown
99dd975dae
Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
Matthew Hodgson
66a4ca1d28 404 nicely if you try to interact with a missing current version 2018-08-12 19:14:31 -04:00
Matthew Hodgson
edc427a351 flake8 2018-08-12 19:14:31 -04:00
Matthew Hodgson
fe87890b18 implement remaining tests and make them work 2018-08-12 19:14:31 -04:00
Matthew Hodgson
f6a3067868 linting 2018-08-12 19:14:31 -04:00
Matthew Hodgson
15d513f16f fix idiocies and so make tests pass 2018-08-12 19:14:31 -04:00
Matthew Hodgson
174be586e5 first cut at a UT 2018-08-12 19:14:31 -04:00
Neil Johnson
6c6aba76e1 implementation of server notices to alert on hitting resource limits 2018-08-10 15:12:59 +01:00
Amber Brown
807449d8f2 fix up a forced long line 2018-08-11 00:01:43 +10:00
black
8b3d9b6b19 Run black. 2018-08-10 23:54:09 +10:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Richard van der Hoff
3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests 
send SNI for federation requests
 2018-08-10 12:23:54 +01:00
Neil Johnson
885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Neil Johnson
04df714259 fix imports 2018-08-09 17:41:52 +01:00
Neil Johnson
09cf130898 only block on sync where user is not part of the mau cohort 2018-08-09 17:39:12 +01:00
Erik Johnston
3e19beb941 Fix tests 2018-08-09 14:58:49 +01:00
Neil Johnson
69ce057ea6 block sync if auth checks fail 2018-08-09 12:26:27 +01:00
Neil Johnson
3dce9050cf Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block 2018-08-09 11:42:02 +01:00
Neil Johnson
0ad98e38d0
Merge pull request #3655 from matrix-org/neilj/disable_hs 
Flag to disable HS without disabling federation
 2018-08-09 10:41:43 +00:00
Neil Johnson
a5ef110749 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block 2018-08-09 11:40:37 +01:00
Amber Brown
2511f3f8a0
Test fixes for Python 3 (#3647) 2018-08-09 12:22:01 +10:00
Neil Johnson
e92fb00f32 sync auth blocking 2018-08-08 17:54:49 +01:00
Neil Johnson
5298d79fb5
Merge branch 'develop' into neilj/disable_hs 2018-08-08 16:13:03 +00:00
Richard van der Hoff
8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions 
Support for room versioning
 2018-08-08 17:10:53 +01:00
Neil Johnson
d2f3ef98ac
Merge branch 'develop' into neilj/disable_hs 2018-08-08 15:55:47 +00:00
Neil Johnson
7f3d897e7a mock config.max_mau_value 2018-08-08 11:46:23 +01:00
Neil Johnson
e8eba2b4e3 implement reserved users for mau limits 2018-08-07 17:49:43 +01:00
Neil Johnson
fbe255f9a4 add default mau_limits_reserved_threepids 2018-08-06 23:24:54 +01:00
Neil Johnson
42c6823827 disable HS from config 2018-08-04 22:07:04 +01:00
Neil Johnson
d08296f9f2 remove unused import 2018-08-03 23:00:16 +01:00
Neil Johnson
886be75ad1 bug fixes 2018-08-03 22:29:03 +01:00
Neil Johnson
e10830e976 wip commit - tests failing 2018-08-03 17:55:50 +01:00
Richard van der Hoff
0ca459ea33 Basic support for room versioning 
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
 2018-08-03 16:08:32 +01:00
Neil Johnson
5593ff6773 fix (lots of) py3 test failures 2018-08-03 14:59:17 +01:00
Neil Johnson
b2aab04d2c fix py3 test failure 2018-08-03 14:12:56 +01:00
Neil Johnson
950807d93a fix caching and tests 2018-08-03 13:49:53 +01:00
Neil Johnson
74b1d46ad9 do mau checks based on monthly_active_users table 2018-08-02 16:57:35 +01:00
Neil Johnson
9180061b49 remove unused count_monthly_users 2018-08-02 15:55:29 +01:00
Neil Johnson
c4ffbecb68 fix test, update constructor call 2018-08-02 13:51:05 +01:00
Neil Johnson
00f99f74b1 insertion into monthly_active_users 2018-08-02 13:47:19 +01:00
Neil Johnson
165e067033 Revert "change monthly_active_users table to be a single column" 
This reverts commit ec716a35b2.
 2018-08-02 10:59:58 +01:00
Neil Johnson
ec716a35b2 change monthly_active_users table to be a single column 2018-08-01 17:54:37 +01:00
Neil Johnson
d766f26de9 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-01 17:49:41 +01:00
Neil Johnson
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits 
Initial impl of capping MAU
 2018-08-01 15:58:45 +00:00
Amber Brown
da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Neil Johnson
c480c4c962 fix isort 2018-08-01 14:25:58 +01:00
Neil Johnson
6eed16d8a2 fix test for py3 2018-08-01 14:02:10 +01:00
Neil Johnson
303f1c851f Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits 2018-08-01 13:42:50 +01:00
Neil Johnson
4e6e00152c fix known broken test 2018-08-01 11:48:37 +01:00
Richard van der Hoff
cab782c17e
Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator 
Rewrite cache list decorator
 2018-08-01 10:28:56 +01:00
Neil Johnson
7931393495 make count_monthly_users async synapse/handlers/auth.py 2018-08-01 10:21:56 +01:00
Neil Johnson
6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Neil Johnson
df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Neil Johnson
7d05406a07 fix user_ips counting 2018-07-31 12:03:23 +01:00
Travis Ralston
e908b86832 Remove pdu_failures from transactions 
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
 2018-07-30 16:28:47 -06:00
Neil Johnson
cefac79c10 monthly_active_tests 2018-07-30 22:08:09 +01:00
Neil Johnson
251e6c1210 limit register and sign in on number of monthly users 2018-07-30 15:55:57 +01:00
Richard van der Hoff
a8cbce0ced fix invalidation 2018-07-27 16:17:17 +01:00
Richard van der Hoff
f102c05856 Rewrite cache list decorator 
Because it was complicated and annoyed me. I suspect this will be more
efficient too.
 2018-07-27 13:47:04 +01:00
Richard van der Hoff
7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests 2018-07-27 09:17:11 +01:00
Matthew Hodgson
2565804030 Merge branch 'develop' into matthew/filter_members 2018-07-25 17:27:49 +01:00
Matthew Hodgson
0620d27f4d flake8 2018-07-25 17:21:17 +01:00
Matthew Hodgson
0a7ee0ab8b add tests for _get_some_state_from_cache 2018-07-25 16:33:52 +01:00
Richard van der Hoff
f59be4eb0e Fix unit tests 
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
 2018-07-25 10:30:36 +01:00
Matthew Hodgson
cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Matthew Hodgson
adfe29ec0b Merge branch 'develop' into matthew/filter_members 2018-07-23 19:21:37 +01:00
Erik Johnston
8fbe418777 Fix unit tests 2018-07-23 13:33:49 +01:00
Amber Brown
3132b89f12
Make the rest of the .iterwhatever go away (#3562) 2018-07-21 15:47:18 +10:00
Richard van der Hoff
3d6df84658 Test and fix support for cancellation in Linearizer 2018-07-20 13:59:55 +01:00
Richard van der Hoff
4f67623674
Merge pull request #3571 from matrix-org/rav/limiter_fixes 
A set of improvements to the Limiter
 2018-07-20 13:53:03 +01:00
Amber Brown
e1a237eaab
Admin API for creating new users (#3415) 2018-07-20 22:41:13 +10:00
Richard van der Hoff
7c712f95bb Combine Limiter and Linearizer 
Linearizer was effectively a Limiter with max_count=1, so rather than
maintaining two sets of code, let's combine them.
 2018-07-20 13:11:43 +01:00
Richard van der Hoff
d7275eecf3 Add a sleep to the Limiter to fix stack overflows. 
Fixes #3570
 2018-07-20 12:37:12 +01:00
Matthew Hodgson
650daf5628 make test work 2018-07-19 20:49:44 +01:00
Matthew Hodgson
1fa4f7e03e first cut of a UT for testing state store (untested) 2018-07-19 20:19:32 +01:00
Richard van der Hoff
c754e006f4
Merge pull request #3556 from matrix-org/rav/background_processes 
Run things as background processes
 2018-07-19 11:04:18 +01:00
Amber Brown
a97c845271
Move v1-only APIs into their own module & isolate deprecated ones (#3460) 2018-07-19 20:03:33 +10:00
Richard van der Hoff
8c69b735e3 Make Distributor run its processes as a background process 
This is more involved than it might otherwise be, because the current
implementation just drops its logcontexts and runs everything in the sentinel
context.

It turns out that we aren't actually using a bunch of the functionality here
(notably suppress_failures and the fact that Distributor.fire returns a
deferred), so the easiest way to fix this is actually by simplifying a bunch of
code.
 2018-07-18 20:55:05 +01:00
Richard van der Hoff
d897be6a98 Fix visibility of events from erased users over federation 2018-07-17 14:02:07 +01:00
Richard van der Hoff
9c04b4abf9
Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server 
Refactor and optimze filter_events_for_server
 2018-07-17 14:01:39 +01:00
Richard van der Hoff
94440ae994 fix imports 2018-07-17 11:51:26 +01:00
Amber Brown
bc006b3c9d
Refactor REST API tests to use explicit reactors (#3351) 2018-07-17 20:43:18 +10:00
Erik Johnston
850238b4ef Add unit test 2018-07-17 10:59:02 +01:00
Amber Brown
3fe0938b76
Merge pull request #3530 from matrix-org/erikj/stream_cache 
Don't return unknown entities in get_entities_changed
 2018-07-17 13:44:46 +10:00
Richard van der Hoff
15b13b537f Add a test which profiles filter_events_for_server in a large room 2018-07-16 14:06:09 +01:00
Richard van der Hoff
ea69d35651 Move filter_events_for_server out of FederationHandler 
for easier unit testing.
 2018-07-16 13:06:24 +01:00
Jeroen
505530f36a Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests 
# Conflicts:
#	synapse/crypto/context_factory.py
 2018-07-14 20:24:46 +02:00
Amber Brown
33b60c01b5
Make auth & transactions more testable (#3499) 2018-07-14 07:34:49 +10:00
Erik Johnston
bc832f822f Fixup unit test 2018-07-13 17:03:04 +01:00
Richard van der Hoff
3b391d9c45 Fix unit tests 2018-07-13 16:28:04 +01:00
Oleg Girko
6c1ec5a1bd Use more portable syntax using attrs package. 
Newer syntax

    attr.ib(factory=dict)

is just a syntactic sugar for

    attr.ib(default=attr.Factory(dict))

It was introduced in newest version of attrs package (18.1.0)
and doesn't work with older versions.

We should either require minimum version of attrs to be 18.1.0,
or use older (slightly more verbose) syntax.
Requiring newest version is not a good solution because
Linux distributions may have older version of attrs (17.4.0 in Fedora 28),
and requiring to build (and package)
newer version just to use newer syntactic sugar in only one test
is just too much.
It's much better to fix that test to use older syntax.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
 2018-07-10 00:38:49 +01:00
Jeroen
b5e157d895 Merge branch 'develop' into send_sni_for_federation_requests 
# Conflicts:
#	synapse/http/endpoint.py
 2018-07-09 08:51:11 +02:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Neil Johnson
feef8461d1 Merge remote-tracking branch 'hera/rav/server_acls' into develop 2018-07-05 11:04:01 +01:00
Erik Johnston
1a88640677
Merge pull request #3483 from matrix-org/rav/more_server_name_validation 
More server_name validation
 2018-07-05 10:04:20 +01:00
Richard van der Hoff
3cf3e08a97 Implementation of server_acls 
... as described at
https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.
 2018-07-04 19:06:20 +01:00
Richard van der Hoff
546bc9e28b More server_name validation 
We need to do a bit more validation when we get a server name, but don't want
to be re-doing it all over the shop, so factor out a separate
parse_and_validate_server_name, and do the extra validation.

Also, use it to verify the server name in the config file.
 2018-07-04 18:59:51 +01:00
Richard van der Hoff
ea555d5633 Reinstate lost run_on_reactor in unit test 
a61738b removed a call to run_on_reactor from a unit test, but that call was
doing something useful, in making the function in question asynchronous.

Reinstate the call and add a check that we are testing what we wanted to be
testing.
 2018-07-04 09:40:01 +01:00
Richard van der Hoff
508196e08a
Reject invalid server names (#3480) 
Make sure that server_names used in auth headers are sane, and reject them with
a sensible error code, before they disappear off into the depths of the system.
 2018-07-03 14:36:14 +01:00
Erik Johnston
e3b4043800
Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking 
Check the state of prev_events a bit more thoroughly when coming over federation
 2018-06-29 13:55:02 +01:00
Matthew Hodgson
f82cf3c7df add test 2018-06-28 21:14:16 +01:00
Matthew Hodgson
e72234f6bd fix tests 2018-06-28 20:56:07 +01:00
Amber Brown
f03a5d1a17 pep8 2018-06-27 11:38:14 +01:00
Amber Brown
94f09618e5 cleanups 2018-06-27 11:38:03 +01:00
Amber Brown
35cc3e8b14 stylistic cleanup 2018-06-27 11:32:09 +01:00
Amber Brown
8d62baa48c cleanups 2018-06-27 11:31:48 +01:00
Amber Brown
77078d6c8e handle federation not telling us about prev_events 2018-06-27 11:27:32 +01:00
Amber Brown
cd6bcdaf87
Better testing framework for homeserver-using things (#3446) 2018-06-27 10:37:24 +01:00
Jeroen
3d605853c8 send SNI for federation requests 2018-06-24 22:38:43 +02:00
Erik Johnston
f79abda87f
Merge pull request #3427 from matrix-org/erikj/remove_filters 
remove dead filter_events_for_clients
 2018-06-22 14:45:42 +01:00
Richard van der Hoff
43e02c409d Disable partial state group caching for wildcard lookups 
When _get_state_for_groups is given a wildcard filter, just do a complete
lookup. Hopefully this will give us the best of both worlds by not filling up
the ram if we only need one or two keys, but also making the cache still work
for the federation reader usecase.
 2018-06-22 11:52:07 +01:00
Amber Brown
77ac14b960
Pass around the reactor explicitly (#3385) 2018-06-22 09:37:10 +01:00
Richard van der Hoff
9a793f861c Merge branch 'master' into develop 2018-06-14 16:36:01 +01:00
Richard van der Hoff
1e77ac66e3 Fix broken unit test 
We need power levels for this test to do what it is supposed to do.
 2018-06-14 14:21:29 +01:00
Richard van der Hoff
a502cfec00 remove spurious debug 2018-06-14 14:20:53 +01:00
Richard van der Hoff
5c9afd6f80 Make default state_default 50 
Make it so that, before there is a power-levels event in the room, you need a
power level of at least 50 to send state.

Partially addresses https://github.com/matrix-org/matrix-doc/issues/1192
 2018-06-14 12:38:09 +01:00
Amber Brown
a61738b316
Remove run_on_reactor (#3395) 2018-06-14 18:27:37 +10:00
Richard van der Hoff
bd348f0af6 remove dead filter_events_for_clients 
This is only used by filter_events_for_client, so we can simplify the whole
thing by just doing one user at a time, and removing a dead storage function to
boot.
 2018-06-12 09:51:31 +01:00
Amber Brown
f7869f8f8b
Port to sortedcontainers (with tests!) (#3332) 2018-06-06 00:13:57 +10:00
Amber Brown
5dbf305444
Put python's logs into Trial when running unit tests (#3319) 2018-06-04 16:06:06 +10:00
Richard van der Hoff
a995fdae39 fix tests 2018-05-29 20:19:29 +01:00
Matthew Hodgson
adb6bac4d5 fix another dumb typo 2018-05-29 02:29:22 +01:00
Matthew Hodgson
284e36ad04 fix dumb typo 2018-05-29 02:23:11 +01:00
Amber Brown
a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Richard van der Hoff
a5e2941aad Reject attempts to send event before privacy consent is given 
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet
given.
 2018-05-22 12:00:47 +01:00
Richard van der Hoff
d5dca9a04f Move consent config parsing into ConsentConfig 
turns out we need to reuse this, so it's better in the config class.
 2018-05-22 11:54:51 +01:00
Richard van der Hoff
9ea219c514 Send users a server notice about consent 
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
 2018-05-22 11:54:51 +01:00
Amber Brown
fcc525b0b7 rest of the changes 2018-05-21 19:48:57 -05:00
Amber Brown
df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Richard van der Hoff
6e1cb54a05 Fix logcontext leak in HttpTransactionCache 
ONE DAY I WILL PURGE THE WORLD OF THIS EVIL
 2018-05-21 16:58:20 +01:00
Richard van der Hoff
6d6e7288fe Stop the transaction cache caching failures 
The transaction cache has some code which tries to stop it caching failures,
but if the callback function failed straight away, then things would happen
backwards and we'd end up with the failure stuck in the cache.
 2018-05-21 16:49:59 +01:00
Erik Johnston
6406b70aeb Use stream rather depth ordering for push actions 
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
 2018-05-11 15:30:11 +01:00
Richard van der Hoff
902673e356
Merge pull request #3161 from NotAFile/remove-v1auth 
Make Client-Server API return 403 for invalid token
 2018-05-03 10:10:57 +01:00
Erik Johnston
53a5fdf312
Merge pull request #3175 from matrix-org/erikj/escape_metric_values 
Escape label values in prometheus metrics
 2018-05-03 10:01:04 +01:00
Erik Johnston
32015e1109 Escape label values in prometheus metrics 2018-05-02 16:52:42 +01:00
Richard van der Hoff
415c6b672e Merge branch 'develop' into rav/more_logcontext_leaks 2018-05-02 16:16:01 +01:00
Richard van der Hoff
11607006d9 Remove spurious unittest.DEBUG 2018-05-02 15:48:47 +01:00
Richard van der Hoff
46beeb9a30 Fix a couple of logcontext leaks in unit tests 
... which were making other, innocent, tests, fail.

Plus remove a spurious unittest.DEBUG which was making the output noisy.
 2018-05-02 15:46:22 +01:00
Richard van der Hoff
f22e7cda2c Fix a class of logcontext leaks 
So, it turns out that if you have a first `Deferred` `D1`, you can add a
callback which returns another `Deferred` `D2`, and `D2` must then complete
before any further callbacks on `D1` will execute (and later callbacks on `D1`
get the *result* of `D2` rather than `D2` itself).

So, `D1` might have `called=True` (as in, it has started running its
callbacks), but any new callbacks added to `D1` won't get run until `D2`
completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield`
will 'block'.

In conclusion: some of our assumptions in `logcontext` were invalid. We need to
make sure that we don't optimise out the logcontext juggling when this
situation happens. Fortunately, it is easy to detect by checking `D1.paused`.
 2018-05-02 11:58:00 +01:00
Richard van der Hoff
e482f8cd85 Fix incorrect reference to StringIO 
This was introduced in 4f2f5171
 2018-05-02 09:12:26 +01:00
Adrian Tschira
6495dbb326 Burminate v1auth 
This closes #2602

v1auth was created to account for the differences in status code between
the v1 and v2_alpha revisions of the protocol (401 vs 403 for invalid
tokens). However since those protocols were merged, this makes the r0
version/endpoint internally inconsistent, and violates the
specification for the r0 endpoint.

This might break clients that rely on this inconsistency with the
specification. This is said to affect the legacy angular reference
client. However, I feel that restoring parity with the spec is more
important. Either way, it is critical to inform developers about this
change, in case they rely on the illegal behaviour.

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-30 22:20:43 +02:00
Richard van der Hoff
9e2601f830
Merge pull request #3108 from NotAFile/py3-six-urlparse 
Use six.moves.urlparse
 2018-04-30 00:33:05 +01:00
Richard van der Hoff
3d1ae61399 Merge branch 'develop' into rav/deferred_timeout 2018-04-27 12:54:43 +01:00
Richard van der Hoff
286e20f2bc
Merge pull request #3109 from NotAFile/py3-tests-fix 
Make tests py3 compatible
 2018-04-23 11:59:03 +01:00
Richard van der Hoff
1ea904b9f0 Use deferred.addTimeout instead of time_bound_deferred 
This doesn't feel like a wheel we need to reinvent.
 2018-04-23 00:53:18 +01:00
Richard van der Hoff
d06a9ea5f7
Merge pull request #3104 from NotAFile/py3-unittest-config 
Add some more variables to the unittest config
 2018-04-20 15:35:58 +01:00
Richard van der Hoff
692a3cc806
Merge pull request #3103 from NotAFile/py3-baseexcepton-message 
Use str(e) instead of e.message
 2018-04-20 15:34:49 +01:00
Richard van der Hoff
639480e14a Avoid creating events with huge numbers of prev_events 
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
 2018-04-16 18:41:37 +01:00
Adrian Tschira
a1a3c9660f Make tests py3 compatible 
This is a mixed commit that fixes various small issues

 * print parentheses
 * 01 is invalid syntax (it was octal in py2)
 * [x for i in 1, 2] is invalid syntax
 * six moves

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-16 00:39:32 +02:00
Adrian Tschira
2a3c33ff03 Use six.moves.urlparse 
The imports were shuffled around a bunch in py3

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-15 21:22:43 +02:00
Adrian Tschira
cb9cdfecd0 Add some more variables to the unittest config 
These worked accidentally before (python2 doesn't complain if you
compare incompatible types) but under py3 this blows up spectacularly

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-15 20:36:39 +02:00
Adrian Tschira
1515560f5c Use str(e) instead of e.message 
Doing this I learned e.message was pretty shortlived, added in 2.6,
they realized it was a bad idea and deprecated it in 2.7

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-15 20:32:42 +02:00
Erik Johnston
e939f3bca6 Fix tests 2018-04-11 14:37:11 +01:00
Richard van der Hoff
a3599dda97
Merge pull request #2996 from krombel/allow_auto_join_rooms 
move handling of auto_join_rooms to RegisterHandler
 2018-04-10 01:11:00 +01:00
Richard van der Hoff
87478c5a60
Merge pull request #3061 from NotAFile/add-some-byte-strings 
Add b prefixes to some strings that are bytes in py3
 2018-04-09 23:54:05 +01:00
Richard van der Hoff
664adb4236
Merge pull request #3016 from silkeh/improve-service-lookups 
Improve handling of SRV records for federation connections
 2018-04-09 23:40:06 +01:00
Richard van der Hoff
01afc563c3 Fix overzealous cache invalidation 
Fixes an issue where a cache invalidation would invalidate *all* pending
entries, rather than just the entry that we intended to invalidate.
 2018-04-05 16:24:04 +01:00
Adrian Tschira
6168351877 Add b prefixes to some strings that are bytes in py3 
This has no effect on python2

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-04 13:48:51 +02:00
Silke
72251d1b97 Remove address resolution of hosts in SRV records 
Signed-off-by: Silke Hofstra <silke@slxh.eu>
 2018-04-04 12:26:50 +02:00
Krombel
91ea0202e6 move handling of auto_join_rooms to RegisterHandler 
Currently the handling of auto_join_rooms only works when a user
registers itself via public register api. Registrations via
registration_shared_secret and ModuleApi do not work

This auto_joins the users in the registration handler which enables
the auto join feature for all 3 registration paths.

This is related to issue #2725

Signed-Off-by: Matthias Kesler <krombel@krombel.de>
 2018-03-14 16:45:37 +01:00
Erik Johnston
cb9f8e527c s/replication_client/federation_client/ 2018-03-13 13:26:52 +00:00
Erik Johnston
cea462e285 s/replication_server/federation_server 2018-03-13 13:22:21 +00:00
Erik Johnston
6ea27fafad Fix tests 2018-03-13 10:55:47 +00:00
Erik Johnston
631a73f7ef Fix tests 2018-03-13 10:39:19 +00:00
Richard van der Hoff
2c911d75e8 Fix comment typo 2018-03-05 12:24:49 +00:00
Richard van der Hoff
c818fcab11 Test and fix find_first_stream_ordering_after_ts 
It seemed to suffer from a bunch of off-by-one errors.
 2018-03-05 12:04:02 +00:00
Erik Johnston
784f036306 Move RoomMemberHandler out of Handlers 2018-03-01 14:36:50 +00:00
Erik Johnston
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions 
Batch inserts into event_push_actions_staging
 2018-02-26 14:45:40 +00:00
Erik Johnston
25634ed152 Fix test 2018-02-20 12:40:44 +00:00
Erik Johnston
e440e28456 Fix unit tests 2018-02-20 11:41:40 +00:00
Erik Johnston
3a061cae26 Fix unit test 2018-02-15 16:31:59 +00:00
Erik Johnston
4810f7effd Remove context.push_actions 2018-02-15 15:47:06 +00:00
Erik Johnston
617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation 
Split event creation into a separate handler
 2018-02-06 17:01:17 +00:00
Erik Johnston
770b2252ca s/_create_new_client_event/create_new_client_event/ 2018-02-06 16:40:30 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events (#2784) 
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
 2018-02-06 14:31:24 +00:00
Erik Johnston
25c0a020f4 Updates tests 2018-02-05 16:01:48 +00:00
Richard van der Hoff
db91e72ade
Merge pull request #2844 from matrix-org/rav/evicted_metrics 
montoring metrics for number of cache evictions
 2018-02-05 16:34:36 +01:00
Richard van der Hoff
bc496df192 report metrics on number of cache evictions 2018-02-05 15:34:01 +00:00
Erik Johnston
a1beca0e25 Fix broken unit test for media storage 2018-02-05 12:44:03 +00:00
Erik Johnston
1f881e0746
Merge pull request #2791 from matrix-org/erikj/media_storage_refactor 
Ensure media is in local cache before thumbnailing
 2018-02-05 11:28:52 +00:00
Richard van der Hoff
6da4c4d3bd Factor out resolve_state_groups to a separate handler 
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
 2018-02-01 12:40:04 +00:00
Richard van der Hoff
d1f3490e75 Add tests for user directory search 2018-01-27 17:21:57 +00:00
Richard van der Hoff
88b9c5cbf0 Make it possible to run tests against postgres 2018-01-27 17:15:24 +00:00
Richard van der Hoff
d7eacc4f87 Create dbpool as normal in tests 
... instead of creating our own special SQLiteMemoryDbPool, whose purpose was a
bit of a mystery.

For some reason this makes one of the tests run slightly slower, so bump the
sleep(). Sorry.
 2018-01-27 17:15:15 +00:00
Richard van der Hoff
b178eca261 Run on_new_connection for unit tests 
Configure the connectionpool used for unit tests to run the `on_new_connection`
function.
 2018-01-27 17:06:04 +00:00
Richard van der Hoff
6e9bf67f18 Remove unused/bitrotted MemoryDataStore 
This isn't used, and looks thoroughly bitrotted.
 2018-01-26 00:35:15 +00:00