Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes
This commit is contained in:
Erik Johnston
commit
c02f26319d
|
|
@ -220,8 +220,10 @@ steps:
|
|||
env:
|
||||
POSTGRES: "1"
|
||||
WORKERS: "1"
|
||||
BLACKLIST: "synapse-blacklist-with-workers"
|
||||
command:
|
||||
- "bash .buildkite/merge_base_branch.sh"
|
||||
- "bash -c 'cat /src/sytest-blacklist /src/.buildkite/worker-blacklist > /src/synapse-blacklist-with-workers'"
|
||||
- "bash /synapse_sytest.sh"
|
||||
plugins:
|
||||
- docker#v3.0.1:
|
||||
|
|
@ -229,7 +231,6 @@ steps:
|
|||
propagate-environment: true
|
||||
always-pull: true
|
||||
workdir: "/src"
|
||||
soft_fail: true
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: -1
|
||||
|
|
|
|||
28
.buildkite/worker-blacklist
Normal file
28
.buildkite/worker-blacklist
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# This file serves as a blacklist for SyTest tests that we expect will fail in
|
||||
# Synapse when run under worker mode. For more details, see sytest-blacklist.
|
||||
|
||||
Message history can be paginated
|
||||
|
||||
m.room.history_visibility == "world_readable" allows/forbids appropriately for Guest users
|
||||
|
||||
m.room.history_visibility == "world_readable" allows/forbids appropriately for Real users
|
||||
|
||||
Can re-join room if re-invited
|
||||
|
||||
/upgrade creates a new room
|
||||
|
||||
The only membership state included in an initial sync is for all the senders in the timeline
|
||||
|
||||
Local device key changes get to remote servers
|
||||
|
||||
If remote user leaves room we no longer receive device updates
|
||||
|
||||
Forgotten room messages cannot be paginated
|
||||
|
||||
Inbound federation can get public room list
|
||||
|
||||
Members from the gap are included in gappy incr LL sync
|
||||
|
||||
Leaves are present in non-gapped incremental syncs
|
||||
|
||||
Old leaves are present in gapped incremental syncs
|
||||
81
CHANGES.md
81
CHANGES.md
|
|
@ -1,3 +1,84 @@
|
|||
Synapse 1.2.0rc1 (2019-07-22)
|
||||
=============================
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
- Add support for opentracing. ([\#5544](https://github.com/matrix-org/synapse/issues/5544), [\#5712](https://github.com/matrix-org/synapse/issues/5712))
|
||||
- Add ability to pull all locally stored events out of synapse that a particular user can see. ([\#5589](https://github.com/matrix-org/synapse/issues/5589))
|
||||
- Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance. ([\#5597](https://github.com/matrix-org/synapse/issues/5597))
|
||||
- Add `sender` and `origin_server_ts` fields to `m.replace`. ([\#5613](https://github.com/matrix-org/synapse/issues/5613))
|
||||
- Add default push rule to ignore reactions. ([\#5623](https://github.com/matrix-org/synapse/issues/5623))
|
||||
- Include the original event when asking for its relations. ([\#5626](https://github.com/matrix-org/synapse/issues/5626))
|
||||
- Implement `session_lifetime` configuration option, after which access tokens will expire. ([\#5660](https://github.com/matrix-org/synapse/issues/5660))
|
||||
- Return "This account has been deactivated" when a deactivated user tries to login. ([\#5674](https://github.com/matrix-org/synapse/issues/5674))
|
||||
- Enable aggregations support by default ([\#5714](https://github.com/matrix-org/synapse/issues/5714))
|
||||
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix 'utime went backwards' errors on daemonization. ([\#5609](https://github.com/matrix-org/synapse/issues/5609))
|
||||
- Various minor fixes to the federation request rate limiter. ([\#5621](https://github.com/matrix-org/synapse/issues/5621))
|
||||
- Forbid viewing relations on an event once it has been redacted. ([\#5629](https://github.com/matrix-org/synapse/issues/5629))
|
||||
- Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format. ([\#5638](https://github.com/matrix-org/synapse/issues/5638))
|
||||
- Fix newly-registered users not being able to lookup their own profile without joining a room. ([\#5644](https://github.com/matrix-org/synapse/issues/5644))
|
||||
- Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`. ([\#5654](https://github.com/matrix-org/synapse/issues/5654))
|
||||
- Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated. ([\#5658](https://github.com/matrix-org/synapse/issues/5658))
|
||||
- Fix some problems with authenticating redactions in recent room versions. ([\#5699](https://github.com/matrix-org/synapse/issues/5699), [\#5700](https://github.com/matrix-org/synapse/issues/5700), [\#5707](https://github.com/matrix-org/synapse/issues/5707))
|
||||
- Ignore redactions of m.room.create events. ([\#5701](https://github.com/matrix-org/synapse/issues/5701))
|
||||
|
||||
|
||||
Updates to the Docker image
|
||||
---------------------------
|
||||
|
||||
- Base Docker image on a newer Alpine Linux version (3.8 -> 3.10). ([\#5619](https://github.com/matrix-org/synapse/issues/5619))
|
||||
- Add missing space in default logging file format generated by the Docker image. ([\#5620](https://github.com/matrix-org/synapse/issues/5620))
|
||||
|
||||
|
||||
Improved Documentation
|
||||
----------------------
|
||||
|
||||
- Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks! ([\#5397](https://github.com/matrix-org/synapse/issues/5397))
|
||||
- --no-pep517 should be --no-use-pep517 in the documentation to setup the development environment. ([\#5651](https://github.com/matrix-org/synapse/issues/5651))
|
||||
- Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks! ([\#5661](https://github.com/matrix-org/synapse/issues/5661))
|
||||
- Minor tweaks to postgres documentation. ([\#5675](https://github.com/matrix-org/synapse/issues/5675))
|
||||
|
||||
|
||||
Deprecations and Removals
|
||||
-------------------------
|
||||
|
||||
- Remove support for the `invite_3pid_guest` configuration setting. ([\#5625](https://github.com/matrix-org/synapse/issues/5625))
|
||||
|
||||
|
||||
Internal Changes
|
||||
----------------
|
||||
|
||||
- Move logging code out of `synapse.util` and into `synapse.logging`. ([\#5606](https://github.com/matrix-org/synapse/issues/5606), [\#5617](https://github.com/matrix-org/synapse/issues/5617))
|
||||
- Add a blacklist file to the repo to blacklist certain sytests from failing CI. ([\#5611](https://github.com/matrix-org/synapse/issues/5611))
|
||||
- Make runtime errors surrounding password reset emails much clearer. ([\#5616](https://github.com/matrix-org/synapse/issues/5616))
|
||||
- Remove dead code for persiting outgoing federation transactions. ([\#5622](https://github.com/matrix-org/synapse/issues/5622))
|
||||
- Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI. ([\#5627](https://github.com/matrix-org/synapse/issues/5627))
|
||||
- Move RegistrationHandler.get_or_create_user to test code. ([\#5628](https://github.com/matrix-org/synapse/issues/5628))
|
||||
- Add some more common python virtual-environment paths to the black exclusion list. ([\#5630](https://github.com/matrix-org/synapse/issues/5630))
|
||||
- Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details. ([\#5636](https://github.com/matrix-org/synapse/issues/5636))
|
||||
- Unblacklist some user_directory sytests. ([\#5637](https://github.com/matrix-org/synapse/issues/5637))
|
||||
- Factor out some redundant code in the login implementation. ([\#5639](https://github.com/matrix-org/synapse/issues/5639))
|
||||
- Update ModuleApi to avoid register(generate_token=True). ([\#5640](https://github.com/matrix-org/synapse/issues/5640))
|
||||
- Remove access-token support from `RegistrationHandler.register`, and rename it. ([\#5641](https://github.com/matrix-org/synapse/issues/5641))
|
||||
- Remove access-token support from `RegistrationStore.register`, and rename it. ([\#5642](https://github.com/matrix-org/synapse/issues/5642))
|
||||
- Improve logging for auto-join when a new user is created. ([\#5643](https://github.com/matrix-org/synapse/issues/5643))
|
||||
- Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure. ([\#5645](https://github.com/matrix-org/synapse/issues/5645))
|
||||
- Fix a small typo in a code comment. ([\#5655](https://github.com/matrix-org/synapse/issues/5655))
|
||||
- Clean up exception handling around client access tokens. ([\#5656](https://github.com/matrix-org/synapse/issues/5656))
|
||||
- Add a mechanism for per-test homeserver configuration in the unit tests. ([\#5657](https://github.com/matrix-org/synapse/issues/5657))
|
||||
- Inline issue_access_token. ([\#5659](https://github.com/matrix-org/synapse/issues/5659))
|
||||
- Update the sytest BuildKite configuration to checkout Synapse in `/src`. ([\#5664](https://github.com/matrix-org/synapse/issues/5664))
|
||||
- Add a `docker` type to the towncrier configuration. ([\#5673](https://github.com/matrix-org/synapse/issues/5673))
|
||||
- Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces. ([\#5689](https://github.com/matrix-org/synapse/issues/5689))
|
||||
- Documentation for opentracing. ([\#5703](https://github.com/matrix-org/synapse/issues/5703))
|
||||
|
||||
|
||||
Synapse 1.1.0 (2019-07-04)
|
||||
==========================
|
||||
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks!
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Add support for opentracing.
|
||||
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add ability to pull all locally stored events out of synapse that a particular user can see.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Move logging code out of `synapse.util` and into `synapse.logging`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix 'utime went backwards' errors on daemonization.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add a blacklist file to the repo to blacklist certain sytests from failing CI.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add `sender` and `origin_server_ts` fields to `m.replace`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Make runtime errors surrounding password reset emails much clearer.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Move logging code out of `synapse.util` and into `synapse.logging`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Base Docker image on a newer Alpine Linux version (3.8 -> 3.10).
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add missing space in default logging file format generated by the Docker image.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Various minor fixes to the federation request rate limiter.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove dead code for persiting outgoing federation transactions.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add default push rule to ignore reactions.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove support for the `invite_3pid_guest` configuration setting.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Include the original event when asking for its relations.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Move RegistrationHandler.get_or_create_user to test code.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Forbid viewing relations on an event once it has been redacted.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add some more common python virtual-environment paths to the black exclusion list.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Unblacklist some user_directory sytests.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Factor out some redundant code in the login implementation.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Update ModuleApi to avoid register(generate_token=True).
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove access-token support from RegistrationHandler.register, and rename it.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove access-token support from `RegistrationStore.register`, and rename it.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Improve logging for auto-join when a new user is created.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix newly-registered users not being able to lookup their own profile without joining a room.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure.
|
||||
|
|
@ -1 +0,0 @@
|
|||
--no-pep517 should be --no-use-pep517 in the documentation to setup the development environment.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix a small typo in a code comment.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Clean up exception handling around client access tokens.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add a mechanism for per-test homeserver configuration in the unit tests.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Inline issue_access_token.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Implement `session_lifetime` configuration option, after which access tokens will expire.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks!
|
||||
|
|
@ -1 +0,0 @@
|
|||
Update the sytest BuildKite configuration to checkout Synapse in `/src`.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Add a `docker` type to the towncrier configuration.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Return "This account has been deactivated" when a deactivated user tries to login.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Minor tweaks to postgres documentation.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix some problems with authenticating redactions in recent room versions.
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Fix some problems with authenticating redactions in recent room versions.
|
||||
|
||||
|
|
@ -1 +0,0 @@
|
|||
Ignore redactions of m.room.create events.
|
||||
|
|
@ -1 +0,0 @@
|
|||
Fix some problems with authenticating redactions in recent room versions.
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Add support for opentracing.
|
||||
|
||||
|
|
@ -1 +0,0 @@
|
|||
Enable aggregations support by default
|
||||
1
changelog.d/5724.bugfix
Normal file
1
changelog.d/5724.bugfix
Normal file
|
|
@ -0,0 +1 @@
|
|||
Fix stack overflow in server key lookup code.
|
||||
1
changelog.d/5725.bugfix
Normal file
1
changelog.d/5725.bugfix
Normal file
|
|
@ -0,0 +1 @@
|
|||
start.sh no longer uses deprecated cli option.
|
||||
1
changelog.d/5729.removal
Normal file
1
changelog.d/5729.removal
Normal file
|
|
@ -0,0 +1 @@
|
|||
Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration.
|
||||
1
changelog.d/5730.misc
Normal file
1
changelog.d/5730.misc
Normal file
|
|
@ -0,0 +1 @@
|
|||
Cache result of get_version_string to reduce overhead of `/version` federation requests.
|
||||
1
changelog.d/5731.misc
Normal file
1
changelog.d/5731.misc
Normal file
|
|
@ -0,0 +1 @@
|
|||
Return 'user_type' in admin API user endpoints results.
|
||||
|
|
@ -29,7 +29,7 @@ for port in 8080 8081 8082; do
|
|||
|
||||
if ! grep -F "Customisation made by demo/start.sh" -q $DIR/etc/$port.config; then
|
||||
printf '\n\n# Customisation made by demo/start.sh\n' >> $DIR/etc/$port.config
|
||||
|
||||
|
||||
echo 'enable_registration: true' >> $DIR/etc/$port.config
|
||||
|
||||
# Warning, this heredoc depends on the interaction of tabs and spaces. Please don't
|
||||
|
|
@ -43,7 +43,7 @@ for port in 8080 8081 8082; do
|
|||
tls: true
|
||||
resources:
|
||||
- names: [client, federation]
|
||||
|
||||
|
||||
- port: $port
|
||||
tls: false
|
||||
bind_addresses: ['::1', '127.0.0.1']
|
||||
|
|
@ -68,7 +68,7 @@ for port in 8080 8081 8082; do
|
|||
|
||||
# Generate tls keys
|
||||
openssl req -x509 -newkey rsa:4096 -keyout $DIR/etc/localhost\:$https_port.tls.key -out $DIR/etc/localhost\:$https_port.tls.crt -days 365 -nodes -subj "/O=matrix"
|
||||
|
||||
|
||||
# Ignore keys from the trusted keys server
|
||||
echo '# Ignore keys from the trusted keys server' >> $DIR/etc/$port.config
|
||||
echo 'trusted_key_servers:' >> $DIR/etc/$port.config
|
||||
|
|
@ -120,7 +120,6 @@ for port in 8080 8081 8082; do
|
|||
python3 -m synapse.app.homeserver \
|
||||
--config-path "$DIR/etc/$port.config" \
|
||||
-D \
|
||||
-vv \
|
||||
|
||||
popd
|
||||
done
|
||||
|
|
|
|||
100
docs/opentracing.rst
Normal file
100
docs/opentracing.rst
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
===========
|
||||
OpenTracing
|
||||
===========
|
||||
|
||||
Background
|
||||
----------
|
||||
|
||||
OpenTracing is a semi-standard being adopted by a number of distributed tracing
|
||||
platforms. It is a common api for facilitating vendor-agnostic tracing
|
||||
instrumentation. That is, we can use the OpenTracing api and select one of a
|
||||
number of tracer implementations to do the heavy lifting in the background.
|
||||
Our current selected implementation is Jaeger.
|
||||
|
||||
OpenTracing is a tool which gives an insight into the causal relationship of
|
||||
work done in and between servers. The servers each track events and report them
|
||||
to a centralised server - in Synapse's case: Jaeger. The basic unit used to
|
||||
represent events is the span. The span roughly represents a single piece of work
|
||||
that was done and the time at which it occurred. A span can have child spans,
|
||||
meaning that the work of the child had to be completed for the parent span to
|
||||
complete, or it can have follow-on spans which represent work that is undertaken
|
||||
as a result of the parent but is not depended on by the parent to in order to
|
||||
finish.
|
||||
|
||||
Since this is undertaken in a distributed environment a request to another
|
||||
server, such as an RPC or a simple GET, can be considered a span (a unit or
|
||||
work) for the local server. This causal link is what OpenTracing aims to
|
||||
capture and visualise. In order to do this metadata about the local server's
|
||||
span, i.e the 'span context', needs to be included with the request to the
|
||||
remote.
|
||||
|
||||
It is up to the remote server to decide what it does with the spans
|
||||
it creates. This is called the sampling policy and it can be configured
|
||||
through Jaeger's settings.
|
||||
|
||||
For OpenTracing concepts see
|
||||
https://opentracing.io/docs/overview/what-is-tracing/.
|
||||
|
||||
For more information about Jaeger's implementation see
|
||||
https://www.jaegertracing.io/docs/
|
||||
|
||||
=====================
|
||||
Seting up OpenTracing
|
||||
=====================
|
||||
|
||||
To receive OpenTracing spans, start up a Jaeger server. This can be done
|
||||
using docker like so:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker run -d --name jaeger
|
||||
-p 6831:6831/udp \
|
||||
-p 6832:6832/udp \
|
||||
-p 5778:5778 \
|
||||
-p 16686:16686 \
|
||||
-p 14268:14268 \
|
||||
jaegertracing/all-in-one:1.13
|
||||
|
||||
Latest documentation is probably at
|
||||
https://www.jaegertracing.io/docs/1.13/getting-started/
|
||||
|
||||
|
||||
Enable OpenTracing in Synapse
|
||||
-----------------------------
|
||||
|
||||
OpenTracing is not enabled by default. It must be enabled in the homeserver
|
||||
config by uncommenting the config options under ``opentracing`` as shown in
|
||||
the `sample config <./sample_config.yaml>`_. For example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
opentracing:
|
||||
tracer_enabled: true
|
||||
homeserver_whitelist:
|
||||
- "mytrustedhomeserver.org"
|
||||
- "*.myotherhomeservers.com"
|
||||
|
||||
Homeserver whitelisting
|
||||
-----------------------
|
||||
|
||||
The homeserver whitelist is configured using regular expressions. A list of regular
|
||||
expressions can be given and their union will be compared when propagating any
|
||||
spans contexts to another homeserver.
|
||||
|
||||
Though it's mostly safe to send and receive span contexts to and from
|
||||
untrusted users since span contexts are usually opaque ids it can lead to
|
||||
two problems, namely:
|
||||
|
||||
- If the span context is marked as sampled by the sending homeserver the receiver will
|
||||
sample it. Therefore two homeservers with wildly different sampling policies
|
||||
could incur higher sampling counts than intended.
|
||||
- Sending servers can attach arbitrary data to spans, known as 'baggage'. For safety this has been disabled in Synapse
|
||||
but that doesn't prevent another server sending you baggage which will be logged
|
||||
to OpenTracing's logs.
|
||||
|
||||
==================
|
||||
Configuring Jaeger
|
||||
==================
|
||||
|
||||
Sampling strategies can be set as in this document:
|
||||
https://www.jaegertracing.io/docs/1.13/sampling/
|
||||
|
|
@ -1422,18 +1422,8 @@ opentracing:
|
|||
#enabled: true
|
||||
|
||||
# The list of homeservers we wish to send and receive span contexts and span baggage.
|
||||
#
|
||||
# Though it's mostly safe to send and receive span contexts to and from
|
||||
# untrusted users since span contexts are usually opaque ids it can lead to
|
||||
# two problems, namely:
|
||||
# - If the span context is marked as sampled by the sending homeserver the receiver will
|
||||
# sample it. Therefore two homeservers with wildly disparaging sampling policies
|
||||
# could incur higher sampling counts than intended.
|
||||
# - Span baggage can be arbitrary data. For safety this has been disabled in synapse
|
||||
# but that doesn't prevent another server sending you baggage which will be logged
|
||||
# to opentracing logs.
|
||||
#
|
||||
# This a list of regexes which are matched against the server_name of the
|
||||
# See docs/opentracing.rst
|
||||
# This is a list of regexes which are matched against the server_name of the
|
||||
# homeserver.
|
||||
#
|
||||
# By defult, it is empty, so no servers are matched.
|
||||
|
|
|
|||
|
|
@ -35,4 +35,4 @@ try:
|
|||
except ImportError:
|
||||
pass
|
||||
|
||||
__version__ = "1.1.0"
|
||||
__version__ = "1.2.0rc1"
|
||||
|
|
|
|||
|
|
@ -48,18 +48,8 @@ class TracerConfig(Config):
|
|||
#enabled: true
|
||||
|
||||
# The list of homeservers we wish to send and receive span contexts and span baggage.
|
||||
#
|
||||
# Though it's mostly safe to send and receive span contexts to and from
|
||||
# untrusted users since span contexts are usually opaque ids it can lead to
|
||||
# two problems, namely:
|
||||
# - If the span context is marked as sampled by the sending homeserver the receiver will
|
||||
# sample it. Therefore two homeservers with wildly disparaging sampling policies
|
||||
# could incur higher sampling counts than intended.
|
||||
# - Span baggage can be arbitrary data. For safety this has been disabled in synapse
|
||||
# but that doesn't prevent another server sending you baggage which will be logged
|
||||
# to opentracing logs.
|
||||
#
|
||||
# This a list of regexes which are matched against the server_name of the
|
||||
# See docs/opentracing.rst
|
||||
# This is a list of regexes which are matched against the server_name of the
|
||||
# homeserver.
|
||||
#
|
||||
# By defult, it is empty, so no servers are matched.
|
||||
|
|
|
|||
|
|
@ -31,6 +31,7 @@ class WorkerConfig(Config):
|
|||
self.worker_listeners = config.get("worker_listeners", [])
|
||||
self.worker_daemonize = config.get("worker_daemonize")
|
||||
self.worker_pid_file = config.get("worker_pid_file")
|
||||
self.worker_log_config = config.get("worker_log_config")
|
||||
|
||||
# The host used to connect to the main synapse
|
||||
self.worker_replication_host = config.get("worker_replication_host", None)
|
||||
|
|
|
|||
|
|
@ -238,27 +238,9 @@ class Keyring(object):
|
|||
"""
|
||||
|
||||
try:
|
||||
# create a deferred for each server we're going to look up the keys
|
||||
# for; we'll resolve them once we have completed our lookups.
|
||||
# These will be passed into wait_for_previous_lookups to block
|
||||
# any other lookups until we have finished.
|
||||
# The deferreds are called with no logcontext.
|
||||
server_to_deferred = {
|
||||
rq.server_name: defer.Deferred() for rq in verify_requests
|
||||
}
|
||||
ctx = LoggingContext.current_context()
|
||||
|
||||
# We want to wait for any previous lookups to complete before
|
||||
# proceeding.
|
||||
yield self.wait_for_previous_lookups(server_to_deferred)
|
||||
|
||||
# Actually start fetching keys.
|
||||
self._get_server_verify_keys(verify_requests)
|
||||
|
||||
# When we've finished fetching all the keys for a given server_name,
|
||||
# resolve the deferred passed to `wait_for_previous_lookups` so that
|
||||
# any lookups waiting will proceed.
|
||||
#
|
||||
# map from server name to a set of request ids
|
||||
# map from server name to a set of outstanding request ids
|
||||
server_to_request_ids = {}
|
||||
|
||||
for verify_request in verify_requests:
|
||||
|
|
@ -266,40 +248,61 @@ class Keyring(object):
|
|||
request_id = id(verify_request)
|
||||
server_to_request_ids.setdefault(server_name, set()).add(request_id)
|
||||
|
||||
def remove_deferreds(res, verify_request):
|
||||
# Wait for any previous lookups to complete before proceeding.
|
||||
yield self.wait_for_previous_lookups(server_to_request_ids.keys())
|
||||
|
||||
# take out a lock on each of the servers by sticking a Deferred in
|
||||
# key_downloads
|
||||
for server_name in server_to_request_ids.keys():
|
||||
self.key_downloads[server_name] = defer.Deferred()
|
||||
logger.debug("Got key lookup lock on %s", server_name)
|
||||
|
||||
# When we've finished fetching all the keys for a given server_name,
|
||||
# drop the lock by resolving the deferred in key_downloads.
|
||||
def drop_server_lock(server_name):
|
||||
d = self.key_downloads.pop(server_name)
|
||||
d.callback(None)
|
||||
|
||||
def lookup_done(res, verify_request):
|
||||
server_name = verify_request.server_name
|
||||
request_id = id(verify_request)
|
||||
server_to_request_ids[server_name].discard(request_id)
|
||||
if not server_to_request_ids[server_name]:
|
||||
d = server_to_deferred.pop(server_name, None)
|
||||
if d:
|
||||
d.callback(None)
|
||||
server_requests = server_to_request_ids[server_name]
|
||||
server_requests.remove(id(verify_request))
|
||||
|
||||
# if there are no more requests for this server, we can drop the lock.
|
||||
if not server_requests:
|
||||
with PreserveLoggingContext(ctx):
|
||||
logger.debug("Releasing key lookup lock on %s", server_name)
|
||||
|
||||
# ... but not immediately, as that can cause stack explosions if
|
||||
# we get a long queue of lookups.
|
||||
self.clock.call_later(0, drop_server_lock, server_name)
|
||||
|
||||
return res
|
||||
|
||||
for verify_request in verify_requests:
|
||||
verify_request.key_ready.addBoth(remove_deferreds, verify_request)
|
||||
verify_request.key_ready.addBoth(lookup_done, verify_request)
|
||||
|
||||
# Actually start fetching keys.
|
||||
self._get_server_verify_keys(verify_requests)
|
||||
except Exception:
|
||||
logger.exception("Error starting key lookups")
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def wait_for_previous_lookups(self, server_to_deferred):
|
||||
def wait_for_previous_lookups(self, server_names):
|
||||
"""Waits for any previous key lookups for the given servers to finish.
|
||||
|
||||
Args:
|
||||
server_to_deferred (dict[str, Deferred]): server_name to deferred which gets
|
||||
resolved once we've finished looking up keys for that server.
|
||||
The Deferreds should be regular twisted ones which call their
|
||||
callbacks with no logcontext.
|
||||
server_names (Iterable[str]): list of servers which we want to look up
|
||||
|
||||
Returns: a Deferred which resolves once all key lookups for the given
|
||||
servers have completed. Follows the synapse rules of logcontext
|
||||
preservation.
|
||||
Returns:
|
||||
Deferred[None]: resolves once all key lookups for the given servers have
|
||||
completed. Follows the synapse rules of logcontext preservation.
|
||||
"""
|
||||
loop_count = 1
|
||||
while True:
|
||||
wait_on = [
|
||||
(server_name, self.key_downloads[server_name])
|
||||
for server_name in server_to_deferred.keys()
|
||||
for server_name in server_names
|
||||
if server_name in self.key_downloads
|
||||
]
|
||||
if not wait_on:
|
||||
|
|
@ -314,19 +317,6 @@ class Keyring(object):
|
|||
|
||||
loop_count += 1
|
||||
|
||||
ctx = LoggingContext.current_context()
|
||||
|
||||
def rm(r, server_name_):
|
||||
with PreserveLoggingContext(ctx):
|
||||
logger.debug("Releasing key lookup lock on %s", server_name_)
|
||||
self.key_downloads.pop(server_name_, None)
|
||||
return r
|
||||
|
||||
for server_name, deferred in server_to_deferred.items():
|
||||
logger.debug("Got key lookup lock on %s", server_name)
|
||||
self.key_downloads[server_name] = deferred
|
||||
deferred.addBoth(rm, server_name)
|
||||
|
||||
def _get_server_verify_keys(self, verify_requests):
|
||||
"""Tries to find at least one key for each verify request
|
||||
|
||||
|
|
|
|||
|
|
@ -24,6 +24,131 @@
|
|||
# this move the methods have work very similarly to opentracing's and it should only
|
||||
# be a matter of few regexes to move over to opentracing's access patterns proper.
|
||||
|
||||
"""
|
||||
============================
|
||||
Using OpenTracing in Synapse
|
||||
============================
|
||||
|
||||
Python-specific tracing concepts are at https://opentracing.io/guides/python/.
|
||||
Note that Synapse wraps OpenTracing in a small module (this one) in order to make the
|
||||
OpenTracing dependency optional. That means that the access patterns are
|
||||
different to those demonstrated in the OpenTracing guides. However, it is
|
||||
still useful to know, especially if OpenTracing is included as a full dependency
|
||||
in the future or if you are modifying this module.
|
||||
|
||||
|
||||
OpenTracing is encapsulated so that
|
||||
no span objects from OpenTracing are exposed in Synapse's code. This allows
|
||||
OpenTracing to be easily disabled in Synapse and thereby have OpenTracing as
|
||||
an optional dependency. This does however limit the number of modifiable spans
|
||||
at any point in the code to one. From here out references to `opentracing`
|
||||
in the code snippets refer to the Synapses module.
|
||||
|
||||
Tracing
|
||||
-------
|
||||
|
||||
In Synapse it is not possible to start a non-active span. Spans can be started
|
||||
using the ``start_active_span`` method. This returns a scope (see
|
||||
OpenTracing docs) which is a context manager that needs to be entered and
|
||||
exited. This is usually done by using ``with``.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from synapse.logging.opentracing import start_active_span
|
||||
|
||||
with start_active_span("operation name"):
|
||||
# Do something we want to tracer
|
||||
|
||||
Forgetting to enter or exit a scope will result in some mysterious and grievous log
|
||||
context errors.
|
||||
|
||||
At anytime where there is an active span ``opentracing.set_tag`` can be used to
|
||||
set a tag on the current active span.
|
||||
|
||||
Tracing functions
|
||||
-----------------
|
||||
|
||||
Functions can be easily traced using decorators. There is a decorator for
|
||||
'normal' function and for functions which are actually deferreds. The name of
|
||||
the function becomes the operation name for the span.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from synapse.logging.opentracing import trace, trace_deferred
|
||||
|
||||
# Start a span using 'normal_function' as the operation name
|
||||
@trace
|
||||
def normal_function(*args, **kwargs):
|
||||
# Does all kinds of cool and expected things
|
||||
return something_usual_and_useful
|
||||
|
||||
# Start a span using 'deferred_function' as the operation name
|
||||
@trace_deferred
|
||||
@defer.inlineCallbacks
|
||||
def deferred_function(*args, **kwargs):
|
||||
# We start
|
||||
yield we_wait
|
||||
# we finish
|
||||
defer.returnValue(something_usual_and_useful)
|
||||
|
||||
Operation names can be explicitly set for functions by using
|
||||
``trace_using_operation_name`` and
|
||||
``trace_deferred_using_operation_name``
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from synapse.logging.opentracing import (
|
||||
trace_using_operation_name,
|
||||
trace_deferred_using_operation_name
|
||||
)
|
||||
|
||||
@trace_using_operation_name("A *much* better operation name")
|
||||
def normal_function(*args, **kwargs):
|
||||
# Does all kinds of cool and expected things
|
||||
return something_usual_and_useful
|
||||
|
||||
@trace_deferred_using_operation_name("Another exciting operation name!")
|
||||
@defer.inlineCallbacks
|
||||
def deferred_function(*args, **kwargs):
|
||||
# We start
|
||||
yield we_wait
|
||||
# we finish
|
||||
defer.returnValue(something_usual_and_useful)
|
||||
|
||||
Contexts and carriers
|
||||
---------------------
|
||||
|
||||
There are a selection of wrappers for injecting and extracting contexts from
|
||||
carriers provided. Unfortunately OpenTracing's three context injection
|
||||
techniques are not adequate for our inject of OpenTracing span-contexts into
|
||||
Twisted's http headers, EDU contents and our database tables. Also note that
|
||||
the binary encoding format mandated by OpenTracing is not actually implemented
|
||||
by jaeger_client v4.0.0 - it will silently noop.
|
||||
Please refer to the end of ``logging/opentracing.py`` for the available
|
||||
injection and extraction methods.
|
||||
|
||||
Homeserver whitelisting
|
||||
-----------------------
|
||||
|
||||
Most of the whitelist checks are encapsulated in the modules's injection
|
||||
and extraction method but be aware that using custom carriers or crossing
|
||||
unchartered waters will require the enforcement of the whitelist.
|
||||
``logging/opentracing.py`` has a ``whitelisted_homeserver`` method which takes
|
||||
in a destination and compares it to the whitelist.
|
||||
|
||||
=======
|
||||
Gotchas
|
||||
=======
|
||||
|
||||
- Checking whitelists on span propagation
|
||||
- Inserting pii
|
||||
- Forgetting to enter or exit a scope
|
||||
- Span source: make sure that the span you expect to be active across a
|
||||
function call really will be that one. Does the current function have more
|
||||
than one caller? Will all of those calling functions have be in a context
|
||||
with an active span?
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import logging
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -469,7 +469,7 @@ class DataStore(
|
|||
return self._simple_select_list(
|
||||
table="users",
|
||||
keyvalues={},
|
||||
retcols=["name", "password_hash", "is_guest", "admin"],
|
||||
retcols=["name", "password_hash", "is_guest", "admin", "user_type"],
|
||||
desc="get_users",
|
||||
)
|
||||
|
||||
|
|
@ -494,7 +494,7 @@ class DataStore(
|
|||
orderby=order,
|
||||
start=start,
|
||||
limit=limit,
|
||||
retcols=["name", "password_hash", "is_guest", "admin"],
|
||||
retcols=["name", "password_hash", "is_guest", "admin", "user_type"],
|
||||
)
|
||||
count = yield self.runInteraction("get_users_paginate", self.get_user_count_txn)
|
||||
retval = {"users": users, "total": count}
|
||||
|
|
@ -514,7 +514,7 @@ class DataStore(
|
|||
table="users",
|
||||
term=term,
|
||||
col="name",
|
||||
retcols=["name", "password_hash", "is_guest", "admin"],
|
||||
retcols=["name", "password_hash", "is_guest", "admin", "user_type"],
|
||||
desc="search_users",
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -22,6 +22,23 @@ logger = logging.getLogger(__name__)
|
|||
|
||||
|
||||
def get_version_string(module):
|
||||
"""Given a module calculate a git-aware version string for it.
|
||||
|
||||
If called on a module not in a git checkout will return `__verison__`.
|
||||
|
||||
Args:
|
||||
module (module)
|
||||
|
||||
Returns:
|
||||
str
|
||||
"""
|
||||
|
||||
cached_version = getattr(module, "_synapse_version_string_cache", None)
|
||||
if cached_version:
|
||||
return cached_version
|
||||
|
||||
version_string = module.__version__
|
||||
|
||||
try:
|
||||
null = open(os.devnull, "w")
|
||||
cwd = os.path.dirname(os.path.abspath(module.__file__))
|
||||
|
|
@ -80,8 +97,10 @@ def get_version_string(module):
|
|||
s for s in (git_branch, git_tag, git_commit, git_dirty) if s
|
||||
)
|
||||
|
||||
return "%s (%s)" % (module.__version__, git_version)
|
||||
version_string = "%s (%s)" % (module.__version__, git_version)
|
||||
except Exception as e:
|
||||
logger.info("Failed to check for git repository: %s", e)
|
||||
|
||||
return module.__version__
|
||||
module._synapse_version_string_cache = version_string
|
||||
|
||||
return version_string
|
||||
|
|
|
|||
|
|
@ -86,35 +86,6 @@ class KeyringTestCase(unittest.HomeserverTestCase):
|
|||
getattr(LoggingContext.current_context(), "request", None), expected
|
||||
)
|
||||
|
||||
def test_wait_for_previous_lookups(self):
|
||||
kr = keyring.Keyring(self.hs)
|
||||
|
||||
lookup_1_deferred = defer.Deferred()
|
||||
lookup_2_deferred = defer.Deferred()
|
||||
|
||||
# we run the lookup in a logcontext so that the patched inlineCallbacks can check
|
||||
# it is doing the right thing with logcontexts.
|
||||
wait_1_deferred = run_in_context(
|
||||
kr.wait_for_previous_lookups, {"server1": lookup_1_deferred}
|
||||
)
|
||||
|
||||
# there were no previous lookups, so the deferred should be ready
|
||||
self.successResultOf(wait_1_deferred)
|
||||
|
||||
# set off another wait. It should block because the first lookup
|
||||
# hasn't yet completed.
|
||||
wait_2_deferred = run_in_context(
|
||||
kr.wait_for_previous_lookups, {"server1": lookup_2_deferred}
|
||||
)
|
||||
|
||||
self.assertFalse(wait_2_deferred.called)
|
||||
|
||||
# let the first lookup complete (in the sentinel context)
|
||||
lookup_1_deferred.callback(None)
|
||||
|
||||
# now the second wait should complete.
|
||||
self.successResultOf(wait_2_deferred)
|
||||
|
||||
def test_verify_json_objects_for_server_awaits_previous_requests(self):
|
||||
key1 = signedjson.key.generate_signing_key(1)
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue