Amber H. Brown
a10c8dae85
fix prometheus rendering error
2019-06-14 21:09:33 +10:00
Amber Brown
6312d6cc7c
Expose statistics on extrems to prometheus ( #5384 )
2019-06-13 22:40:52 +10:00
Andrew Morgan
2d1d7b7e6f
Prevent multiple device list updates from breaking a batch send ( #5156 )
...
fixes #5153
2019-06-06 23:54:00 +01:00
Neil Johnson
a11865016e
Set default room version to v4. ( #5379 )
...
Set default room version to v4.
2019-06-06 20:13:47 +01:00
Erik Johnston
d7add713a8
Add test
2019-05-29 15:59:06 +01:00
Richard van der Hoff
b75537beaf
Store key validity time in the storage layer
...
This is a first step to checking that the key is valid at the required moment.
The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Richard van der Hoff
2e052110ee
Rewrite store_server_verify_key to store several keys at once ( #5234 )
...
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
Erik Johnston
4fb44fb5b9
Expose DataStore._get_events as get_events_as_list
...
This is in preparation for reaction work which requires it.
2019-05-14 13:37:44 +01:00
Amber Brown
b36c82576e
Run Black on the tests again ( #5170 )
2019-05-10 00:12:11 -05:00
Richard van der Hoff
12f9d51e82
Add admin api for sending server_notices ( #5121 )
2019-05-02 11:59:16 +01:00
Richard van der Hoff
40e576e29c
Move admin api impl to its own package
...
It doesn't really belong under rest/client/v1 any more.
2019-05-01 15:44:30 +01:00
Richard van der Hoff
18b69be00f
Rewrite Datastore.get_server_verify_keys
...
Rewrite this so that it doesn't hammer the database.
2019-04-09 00:00:10 +01:00
Richard van der Hoff
0084309cd2
Rewrite test_keys as a HomeserverTestCase
2019-04-08 23:59:57 +01:00
Neil Johnson
e8419554ff
Remove presence lists ( #4989 )
...
Remove presence list support as per MSC 1819
2019-04-03 11:11:15 +01:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place ( #4969 )
...
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task ( #4864 )
2019-03-19 04:50:24 +11:00
Amber Brown
d306bd1b26
fixup
2019-03-12 22:38:01 +11:00
Amber Brown
f6135d06cf
Rewrite userdir to be faster ( #4537 )
2019-03-07 01:22:53 -08:00
Amber Brown
a06614bd2a
UPSERT many functionality ( #4644 )
2019-02-20 23:03:30 +11:00
Amber Brown
3f189c902e
Fix flake8 ( #4519 )
2019-01-30 10:53:17 +00:00
Amber Brown
7072fe3084
Fix UPSERTs on SQLite 3.24+ ( #4477 )
2019-01-28 15:43:32 +00:00
Erik Johnston
9770ed91c2
Fix tests
2019-01-25 10:32:26 +00:00
Erik Johnston
be6a7e47fa
Revert "Require event format version to parse or create events"
2019-01-25 10:23:51 +00:00
Erik Johnston
efb8ed1d45
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version
2019-01-24 18:52:34 +00:00
Amber Brown
58f6c48183
Use native UPSERTs where possible ( #4306 )
2019-01-24 21:31:54 +11:00
Erik Johnston
e79ba9eb34
Fix tests
2019-01-24 09:28:16 +00:00
Amber Brown
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. ( #4369 )
2019-01-09 22:26:25 +11:00
Neil Johnson
d2f7c4e6b1
create support user ( #4141 )
...
Allow for the creation of a support user.
A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00
Richard van der Hoff
30da50a5b8
Initialise user displayname from SAML2 data ( #4272 )
...
When we register a new user from SAML2 data, initialise their displayname
correctly.
2018-12-07 14:44:46 +01:00
Travis Ralston
835779f7fb
Add option to track MAU stats (but not limit people) ( #3830 )
2018-11-15 18:08:27 +00:00
Erik Johnston
abaa93c158
Add test to assert set_e2e_device_keys correctly returns False on no-op
2018-11-08 14:06:44 +00:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits ( #4011 )
...
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
Neil Johnson
ea69a84bbb
fix style inconsistencies
2018-10-24 17:18:08 +01:00
Neil Johnson
6105c6101f
fix race condiftion in calling initialise_reserved_users
2018-10-23 15:24:58 +01:00
Erik Johnston
9c834b8ee9
Add tests
2018-10-02 16:22:39 +01:00
Amber Brown
6e05fd032c
Fix userconsent on Python 3 ( #3938 )
2018-10-02 00:11:58 +10:00
Richard van der Hoff
ae6ad4cf41
docstrings and unittests for storage.state ( #3958 )
...
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
Amber Brown
1f3f5fcf52
Fix client IPs being broken on Python 3 ( #3908 )
2018-09-20 20:14:34 +10:00
Neil Johnson
8decd6233d
improve naming
2018-09-12 16:22:15 +01:00
Neil Johnson
0ddf486724
expose number of real reserved users
2018-09-12 11:58:52 +01:00
Amber Brown
52ec6e9dfa
Port tests/ to Python 3 ( #3808 )
2018-09-07 02:58:18 +10:00
Neil Johnson
84a750e0c3
ensure guests never enter mau list
2018-09-06 17:22:53 +01:00
Neil Johnson
61b05727fa
guest users should not be part of mau total
2018-09-05 22:30:36 +01:00
Amber Brown
77055dba92
Fix tests on postgresql ( #3740 )
2018-09-04 02:21:48 +10:00
Amber Brown
14e4d4f4bf
Port storage/ to Python 3 ( #3725 )
2018-08-31 00:19:58 +10:00
Richard van der Hoff
414fa36f3e
Fix up tests
2018-08-28 17:21:05 +01:00
Erik Johnston
cd77270a66
Implement trail users
2018-08-23 19:17:19 +01:00
Erik Johnston
3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler
...
Refactor state module to support multiple room versions
2018-08-22 10:04:55 +01:00
Matthew Hodgson
bb81e78ec6
Split the state_group_cache in two ( #3726 )
...
Splits the state_group_cache in two.
One half contains normal state events; the other contains member events.
The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache. This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.
Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow. Hopefully fixes https://github.com/matrix-org/synapse/issues/3720 .
2018-08-22 00:56:37 +02:00
Erik Johnston
4d664278af
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler
2018-08-20 14:49:43 +01:00
Neil Johnson
bcfeb44afe
call reap on start up and fix under reaping bug
2018-08-16 22:55:32 +01:00
Matthew Hodgson
2f78f432c4
speed up /members and add at= and membership params ( #3568 )
2018-08-15 16:35:22 +01:00
Neil Johnson
1522ed9c07
in case max_mau is less than I think
2018-08-14 16:52:30 +01:00
Neil Johnson
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
...
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
Amber Brown
99dd975dae
Run tests under PostgreSQL ( #3423 )
2018-08-13 16:47:46 +10:00
black
8b3d9b6b19
Run black.
2018-08-10 23:54:09 +10:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
Neil Johnson
885ea9c602
rename _user_last_seen_monthly_active
2018-08-09 18:02:12 +01:00
Erik Johnston
3e19beb941
Fix tests
2018-08-09 14:58:49 +01:00
Amber Brown
2511f3f8a0
Test fixes for Python 3 ( #3647 )
2018-08-09 12:22:01 +10:00
Neil Johnson
e8eba2b4e3
implement reserved users for mau limits
2018-08-07 17:49:43 +01:00
Neil Johnson
e10830e976
wip commit - tests failing
2018-08-03 17:55:50 +01:00
Neil Johnson
5593ff6773
fix (lots of) py3 test failures
2018-08-03 14:59:17 +01:00
Neil Johnson
b2aab04d2c
fix py3 test failure
2018-08-03 14:12:56 +01:00
Neil Johnson
950807d93a
fix caching and tests
2018-08-03 13:49:53 +01:00
Neil Johnson
9180061b49
remove unused count_monthly_users
2018-08-02 15:55:29 +01:00
Neil Johnson
c4ffbecb68
fix test, update constructor call
2018-08-02 13:51:05 +01:00
Neil Johnson
00f99f74b1
insertion into monthly_active_users
2018-08-02 13:47:19 +01:00
Neil Johnson
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
2018-08-02 10:59:58 +01:00
Neil Johnson
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
Neil Johnson
d766f26de9
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-01 17:49:41 +01:00
Neil Johnson
c480c4c962
fix isort
2018-08-01 14:25:58 +01:00
Neil Johnson
4e6e00152c
fix known broken test
2018-08-01 11:48:37 +01:00
Neil Johnson
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
Neil Johnson
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
Neil Johnson
7d05406a07
fix user_ips counting
2018-07-31 12:03:23 +01:00
Neil Johnson
cefac79c10
monthly_active_tests
2018-07-30 22:08:09 +01:00
Matthew Hodgson
0620d27f4d
flake8
2018-07-25 17:21:17 +01:00
Matthew Hodgson
0a7ee0ab8b
add tests for _get_some_state_from_cache
2018-07-25 16:33:52 +01:00
Matthew Hodgson
cd241d6bda
incorporate more review
2018-07-24 12:39:40 +01:00
Matthew Hodgson
650daf5628
make test work
2018-07-19 20:49:44 +01:00
Matthew Hodgson
1fa4f7e03e
first cut of a UT for testing state store (untested)
2018-07-19 20:19:32 +01:00
Amber Brown
49af402019
run isort
2018-07-09 16:09:20 +10:00
Richard van der Hoff
a995fdae39
fix tests
2018-05-29 20:19:29 +01:00
Richard van der Hoff
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Erik Johnston
6406b70aeb
Use stream rather depth ordering for push actions
...
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
Richard van der Hoff
46beeb9a30
Fix a couple of logcontext leaks in unit tests
...
... which were making other, innocent, tests, fail.
Plus remove a spurious unittest.DEBUG which was making the output noisy.
2018-05-02 15:46:22 +01:00
Richard van der Hoff
692a3cc806
Merge pull request #3103 from NotAFile/py3-baseexcepton-message
...
Use str(e) instead of e.message
2018-04-20 15:34:49 +01:00
Richard van der Hoff
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
Adrian Tschira
1515560f5c
Use str(e) instead of e.message
...
Doing this I learned e.message was pretty shortlived, added in 2.6,
they realized it was a bad idea and deprecated it in 2.7
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:32:42 +02:00
Erik Johnston
cb9f8e527c
s/replication_client/federation_client/
2018-03-13 13:26:52 +00:00
Erik Johnston
6ea27fafad
Fix tests
2018-03-13 10:55:47 +00:00
Richard van der Hoff
2c911d75e8
Fix comment typo
2018-03-05 12:24:49 +00:00
Richard van der Hoff
c818fcab11
Test and fix find_first_stream_ordering_after_ts
...
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
Erik Johnston
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
Erik Johnston
25634ed152
Fix test
2018-02-20 12:40:44 +00:00
Erik Johnston
e440e28456
Fix unit tests
2018-02-20 11:41:40 +00:00
Erik Johnston
3a061cae26
Fix unit test
2018-02-15 16:31:59 +00:00
Erik Johnston
770b2252ca
s/_create_new_client_event/create_new_client_event/
2018-02-06 16:40:30 +00:00
Erik Johnston
25c0a020f4
Updates tests
2018-02-05 16:01:48 +00:00
Richard van der Hoff
d1f3490e75
Add tests for user directory search
2018-01-27 17:21:57 +00:00
Richard van der Hoff
63ef607f1f
Fix tests for Store.__init__ update
...
Fix the test to pass the right number of args to the Store constructors
2017-11-13 10:46:08 +00:00
Richard van der Hoff
02237ce725
Fix tests for refresh_token removal
2017-11-01 10:19:42 +00:00
Richard van der Hoff
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff
9b714abf35
Remove dead class
...
This isn't used anywhere.
2017-10-17 10:43:36 +01:00
Erik Johnston
ed9a7f5436
Merge pull request #2309 from matrix-org/erikj/user_ip_repl
...
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston
29a4066a4d
Update test
2017-07-04 10:21:25 +01:00
Erik Johnston
8c23221666
Fix up
2017-06-27 15:53:45 +01:00
Erik Johnston
a0a561ae85
Fix up client ips to read from pending data
2017-06-27 14:46:12 +01:00
Erik Johnston
93e7a38370
Remove unhelpful test
2017-06-15 09:30:54 +01:00
Erik Johnston
3e5a62ecd8
Add more granular event send metrics
2017-05-02 14:23:26 +01:00
Erik Johnston
eefd9fee81
Fix up tests
2017-03-30 14:14:46 +01:00
Erik Johnston
00957d1aa4
User Cursor.__iter__ instead of fetchall
...
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Richard van der Hoff
95f21c7a66
Fix caching of remote servers' signature keys
...
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.
By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
2017-03-22 15:11:30 +00:00
Erik Johnston
095b45c165
Aggregate event push actions
2017-02-14 13:39:41 +00:00
Erik Johnston
c7a26b7c32
Fix unit tests
2017-01-30 17:11:24 +00:00
Erik Johnston
b3e1f2aa7a
Fix unit tests
2017-01-26 17:16:24 +00:00
Erik Johnston
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
Erik Johnston
f85b6ca494
Speed up cache size calculation
...
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.
This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
2017-01-17 11:18:13 +00:00
Richard van der Hoff
12f3b9000c
fix imports
2016-11-30 17:45:49 +00:00
Richard van der Hoff
aa09d6b8f0
Rip out more refresh_token code
...
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.
Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Erik Johnston
524d61bf7e
Fix tests
2016-11-21 11:53:02 +00:00
Erik Johnston
35e2cc8b52
Merge pull request #1155 from matrix-org/erikj/pluggable_pwd_auth
...
Implement pluggable password auth
2016-10-12 11:41:20 +01:00
Patrik Oldsberg
9bfc617791
storage/appservice: make appservice methods only relying on the cache synchronous
2016-10-06 15:24:59 +02:00
Erik Johnston
850b103b36
Implement pluggable password auth
...
Allows delegating the password auth to an external module. This also
moves the LDAP auth to using this system, allowing it to be removed from
the synapse tree entirely in the future.
2016-10-03 10:36:40 +01:00
Erik Johnston
93b32d4515
Fix unit tests
2016-08-26 15:40:27 +01:00
Erik Johnston
45fd2c8942
Ensure invalidation list does not grow unboundedly
2016-08-19 16:09:16 +01:00
Erik Johnston
c0d7d9d642
Rename to on_invalidate
2016-08-19 15:13:58 +01:00
Erik Johnston
dc76a3e909
Make cache_context an explicit option
2016-08-19 15:02:38 +01:00
Erik Johnston
ba214a5e32
Remove lru option
2016-08-19 14:17:11 +01:00
Erik Johnston
4161ff2fc4
Add concept of cache contexts
2016-08-19 14:17:07 +01:00
Richard van der Hoff
98385888b8
PEP8
2016-08-03 15:42:08 +01:00
Richard van der Hoff
68264d7404
Include device name in /keys/query response
...
Add an 'unsigned' section which includes the device display name.
2016-08-03 15:42:08 +01:00
Mark Haines
0a7d3cd00f
Create separate methods for getting messages to push
...
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.
The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.
The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
Richard van der Hoff
8e02494166
Delete refresh tokens when deleting devices
2016-07-26 11:10:37 +01:00
Richard van der Hoff
012b4c1913
Implement updating devices
...
You can update the displayname of devices now.
2016-07-26 07:35:48 +01:00
Richard van der Hoff
42f4feb2b7
PEP8
2016-07-25 12:25:06 +01:00
Richard van der Hoff
465117d7ca
Fix background_update tests
...
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
2016-07-25 12:10:42 +01:00
Richard van der Hoff
40a1c96617
Fix PEP8 errors
2016-07-20 18:06:28 +01:00
Richard van der Hoff
bc8f265f0a
GET /devices endpoint
...
implement a GET /devices endpoint which lists all of the user's devices.
It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
2016-07-20 16:42:32 +01:00
Erik Johnston
1e2a7f18a1
Merge pull request #922 from matrix-org/erikj/file_api2
...
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
Richard van der Hoff
f863a52cea
Add device_id support to /login
...
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.
Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.
Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
2016-07-18 16:39:44 +01:00
Erik Johnston
a98d215204
Add filter param to /messages API
2016-07-14 16:30:56 +01:00
Erik Johnston
84379062f9
Fix AS retries, but with correct ordering
2016-06-07 10:24:50 +01:00
Erik Johnston
310197bab5
Fix AS retries
2016-06-07 09:34:50 +01:00
Erik Johnston
05e01f21d7
Remove event fetching from DB threads
2016-06-03 17:22:13 +01:00
Mark Haines
816df9f267
get_room_members is unused now
2016-05-16 19:51:43 +01:00
Mark Haines
7e2f971c08
Remove some unused functions ( #711 )
...
* Remove some unused functions
* get_room_events_stream is only used in tests
* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
Erik Johnston
8aab9d87fa
Don't require config to create database
2016-04-06 14:15:45 +01:00
Mark Haines
673c96ce97
Remove dead code left over from presence changes
2016-03-17 11:03:47 +00:00
