Commit graph

157 commits

Author SHA1 Message Date
David Robertson
313581e4e9
Use importlib.metadata to read requirements (#12088)
* Pull runtime dep checks into their own module
* Reimplement `check_requirements` using `importlib`

I've tried to make this clearer. We start by working out which of
Synapse's requirements we need to be installed here and now. I was
surprised that there wasn't an easier way to see which packages were
installed by a given extra.

I've pulled out the error messages into functions that deal with "is
this for an extra or not". And I've rearranged the loop over two
different sets of requirements into one loop with a "must be instaled"
flag.

I hope you agree that this is clearer.

* Test cases
2022-03-01 17:44:41 +00:00
Sean Quah
4d6b6c17c8
Fix rare error in ReadWriteLock when writers complete immediately (#12105)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-01 15:27:15 +00:00
Sean Quah
91bc15c772
Add stop_cancellation utility function (#12106) 2022-03-01 13:51:03 +00:00
Richard van der Hoff
5458eb8551
Fix 'Unhandled error in Deferred' (#12089)
* Fix 'Unhandled error in Deferred'

Fixes a CRITICAL "Unhandled error in Deferred" log message which happened when
a function wrapped with `@cachedList` failed

* Minor optimisation to cachedListDescriptor

we can avoid re-using `missing`, which saves looking up entries in
`deferreds_map`, and means we don't need to copy it.

* Improve type annotation on CachedListDescriptor
2022-03-01 09:51:38 +00:00
Richard van der Hoff
9d11fee8f2
Improve exception handling for concurrent execution (#12109)
* fix incorrect unwrapFirstError import

this was being imported from the wrong place

* Refactor `concurrently_execute` to use `yieldable_gather_results`

* Improve exception handling in `yieldable_gather_results`

Try to avoid swallowing so many stack traces.

* mark unwrapFirstError deprecated

* changelog
2022-03-01 09:34:30 +00:00
Patrick Cloke
02d708568b
Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
Richard van der Hoff
e24ff8ebe3
Remove HomeServer.get_datastore() (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
reivilibre
84bfe47b01
Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common #11505 (#11687)
Co-authored-by: Sean Quah <seanq@element.io>
2022-01-05 11:41:49 +00:00
Richard van der Hoff
c3e38b88f2
Improve opentracing support for ResponseCache (#11607)
This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
2021-12-20 18:12:08 +00:00
Sean Quah
0147b3de20
Add missing type hints to synapse.logging.context (#11556) 2021-12-14 17:35:28 +00:00
Sean Quah
088d748f2c
Revert "Move glob_to_regex and re_word_boundary to matrix-python-common (#11505) (#11527)
This reverts commit a77c369897.
2021-12-07 13:51:11 +00:00
Sean Quah
a77c369897
Move glob_to_regex and re_word_boundary to matrix-python-common (#11505) 2021-12-06 11:36:08 +00:00
Sean Quah
7ff22d6da4
Fix LruCache corruption bug with a size_callback that can return 0 (#11454)
When all entries in an `LruCache` have a size of 0 according to the
provided `size_callback`, and `drop_from_cache` is called on a cache
node, the node would be unlinked from the LRU linked list but remain in
the cache dictionary. An assertion would be later be tripped due to the
inconsistency.

Avoid unintentionally calling `__len__` and use a strict `is None`
check instead when unwrapping the weak reference.
2021-11-30 16:28:02 +00:00
Richard van der Hoff
f36434590c
Additional test for cachedList (#11246)
I was trying to understand how `cachedList` works, and ended up writing this
extra test. I figure we may as well keep it.
2021-11-04 14:45:34 +00:00
Richard van der Hoff
46d0937447
ObservableDeferred: run observers in order (#11229) 2021-11-02 00:17:35 +00:00
Patrick Cloke
47854c71e9
Use direct references for configuration variables (part 4). (#10893) 2021-09-23 12:03:01 -04:00
Jonathan de Jong
93729719b8
Use inline type hints in tests/ (#10350)
This PR is tantamount to running:

    python3.8 -m com2ann -v 6 tests/

(com2ann requires python 3.8 to run)
2021-07-13 11:52:58 +01:00
Erik Johnston
7a5873277e
Add support for evicting cache entries based on last access time. (#10205) 2021-07-05 16:32:12 +01:00
Richard van der Hoff
d7808a2dde
Extend ResponseCache to pass a context object into the callback (#10157)
This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug.

The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
2021-06-14 10:26:09 +01:00
Erik Johnston
fc3d2dc269
Rewrite the KeyRing (#10035) 2021-06-02 16:37:59 +01:00
Erik Johnston
78b5102ae7
Fix up BatchingQueue (#10078)
Fixes #10068
2021-05-27 14:32:31 +01:00
Richard van der Hoff
224f2f949b
Combine LruCache.invalidate and invalidate_many (#9973)
* Make `invalidate` and `invalidate_many` do the same thing

... so that we can do either over the invalidation replication stream, and also
because they always confused me a bit.

* Kill off `invalidate_many`

* changelog
2021-05-27 10:33:56 +01:00
Patrick Cloke
7adcb20fc0
Add missing type hints to synapse.util (#9982) 2021-05-24 15:32:01 -04:00
Richard van der Hoff
c0df6bae06
Remove keylen from LruCache. (#9993)
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.

The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).

Commits should be independently reviewable.
2021-05-24 14:02:01 +01:00
Erik Johnston
3e831f24ff
Don't hammer the database for destination retry timings every ~5mins (#10036) 2021-05-21 17:57:08 +01:00
Erik Johnston
7958eadcd1
Add a batching queue implementation. (#10017) 2021-05-21 11:20:51 +01:00
Richard van der Hoff
5090f26b63
Minor @cachedList enhancements (#9975)
- use a tuple rather than a list for the iterable that is passed into the
  wrapped function, for performance

- test that we can pass an iterable and that keys are correctly deduped.
2021-05-14 11:12:36 +01:00
Richard van der Hoff
7562d887e1
Change the format of access tokens away from macaroons (#5588) 2021-05-12 15:04:51 +01:00
Richard van der Hoff
03318a766c
Merge pull request from GHSA-x345-32rc-8h85
* tests for push rule pattern matching

* tests for acl pattern matching

* factor out common `re.escape`

* Factor out common re.compile

* Factor out common anchoring code

* add word_boundary support to `glob_to_regex`

* Use `glob_to_regex` in push rule evaluator

NB that this drops support for character classes. I don't think anyone ever
used them.

* Improve efficiency of globs with multiple wildcards

The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.

* add assertion on regex pattern

* Fix mypy

* Simplify glob_to_regex

* Inline the glob_to_regex helper function

Signed-off-by: Dan Callahan <danc@element.io>

* Moar comments

Signed-off-by: Dan Callahan <danc@element.io>

Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
Andrew Morgan
6982db9651 Merge branch 'master' into develop 2021-04-20 14:55:16 +01:00
Patrick Cloke
b076bc276e
Always use the name as the log ID. (#9829)
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.

This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20 14:19:00 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke
0b3112123d
Use mock from the stdlib. (#9772) 2021-04-09 13:44:38 -04:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Patrick Cloke
48d44ab142
Record more information into structured logs. (#9654)
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
2021-04-08 08:01:14 -04:00
Patrick Cloke
01dd90b0f0
Add type hints to DictionaryCache and TTLCache. (#9442) 2021-03-29 12:15:33 -04:00
Jonathan de Jong
d6196efafc
Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Richard van der Hoff
3b754aea27
Clean up caching/locking of OIDC metadata load (#9362)
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
2021-02-16 16:27:38 +00:00
Erik Johnston
056327457f
Fix chain cover update to handle events with duplicate auth events (#9210) 2021-01-22 19:44:08 +00:00
Erik Johnston
1a08e0cdab
Fix event chain bg update. (#9118)
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
2021-01-14 18:57:32 +00:00
Erik Johnston
1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00
Patrick Cloke
1b4d5d6acf
Empty iterables should count towards cache usage. (#9028) 2021-01-06 12:33:20 -05:00
Richard van der Hoff
cbc82aa09f
Implement and use an @lru_cache decorator (#8595)
We don't always need the full power of a DeferredCache.
2020-10-30 11:43:17 +00:00
Richard van der Hoff
6d3905c7c7 Add some more tests 2020-10-21 15:39:25 +01:00
Richard van der Hoff
1f4269700c Push some deferred wrangling down into DeferredCache 2020-10-21 15:39:25 +01:00
Richard van der Hoff
7b71695388 Combine the two sets of tests for CacheDescriptor 2020-10-21 15:38:29 +01:00
Richard van der Hoff
96e7d3c4a0
Fix 'LruCache' object has no attribute '_on_resize' (#8591)
We need to make sure we are readu for the `set_cache_factor` callback.
2020-10-19 21:13:50 +01:00
Richard van der Hoff
903d11c43a
Add DeferredCache.get_immediate method (#8568)
* Add `DeferredCache.get_immediate` method

A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.

* Remove unused 'default' parameter to DeferredCache.get()

* another get_immediate instance
2020-10-19 15:00:12 +01:00
Richard van der Hoff
3ee17585cd
Make LruCache register its own metrics (#8561)
rather than have everything that instantiates an LruCache manage metrics
separately, have LruCache do it itself.
2020-10-16 15:51:57 +01:00