maxmustermann/synapse
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Merge branch 'release-v1.22.0' into matrix-org-hotfixes

Browse source
This commit is contained in:
Erik Johnston 2020-10-26 15:03:36 +00:00
parent ab4cd7f802 9e0f5a0ac4
commit 32457baa40
96 changed files with 198 additions and 109 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

100
CHANGES.md
View file

@ -1,3 +1,103 @@
Synapse 1.22.0rc1 (2020-10-22)
==============================
Features
--------
- Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. ([\#7658](https://github.com/matrix-org/synapse/issues/7658))
- Add ability for `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory. ([\#8292](https://github.com/matrix-org/synapse/issues/8292), [\#8467](https://github.com/matrix-org/synapse/issues/8467))
- Add support for olm fallback keys ([MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732)). ([\#8312](https://github.com/matrix-org/synapse/issues/8312), [\#8501](https://github.com/matrix-org/synapse/issues/8501))
- Add support for running background tasks in a separate worker process. ([\#8369](https://github.com/matrix-org/synapse/issues/8369), [\#8458](https://github.com/matrix-org/synapse/issues/8458), [\#8489](https://github.com/matrix-org/synapse/issues/8489), [\#8513](https://github.com/matrix-org/synapse/issues/8513), [\#8544](https://github.com/matrix-org/synapse/issues/8544), [\#8599](https://github.com/matrix-org/synapse/issues/8599))
- Add support for device dehydration ([MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697)). ([\#8380](https://github.com/matrix-org/synapse/issues/8380))
- Add support for [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409), which allows sending typing, read receipts, and presence events to appservices. ([\#8437](https://github.com/matrix-org/synapse/issues/8437), [\#8590](https://github.com/matrix-org/synapse/issues/8590))
- Change default room version to "6", per [MSC2788](https://github.com/matrix-org/matrix-doc/pull/2788). ([\#8461](https://github.com/matrix-org/synapse/issues/8461))
- Add the ability to send non-membership events into a room via the `ModuleApi`. ([\#8479](https://github.com/matrix-org/synapse/issues/8479))
- Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. ([\#8502](https://github.com/matrix-org/synapse/issues/8502))
- Add support for modifying event content in `ThirdPartyRules` modules. ([\#8535](https://github.com/matrix-org/synapse/issues/8535), [\#8564](https://github.com/matrix-org/synapse/issues/8564))
Bugfixes
--------
- Fix a longstanding bug where invalid ignored users in account data could break clients. ([\#8454](https://github.com/matrix-org/synapse/issues/8454))
- Fix a bug where backfilling a room with an event that was missing the `redacts` field would break. ([\#8457](https://github.com/matrix-org/synapse/issues/8457))
- Don't attempt to respond to some requests if the client has already disconnected. ([\#8465](https://github.com/matrix-org/synapse/issues/8465))
- Fix message duplication if something goes wrong after persisting the event. ([\#8476](https://github.com/matrix-org/synapse/issues/8476))
- Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. ([\#8486](https://github.com/matrix-org/synapse/issues/8486))
- Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. ([\#8504](https://github.com/matrix-org/synapse/issues/8504))
- Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`. ([\#8517](https://github.com/matrix-org/synapse/issues/8517))
- Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table. ([\#8527](https://github.com/matrix-org/synapse/issues/8527))
- Fix not sending events over federation when using sharded event writers. ([\#8536](https://github.com/matrix-org/synapse/issues/8536))
- Fix a long standing bug where email notifications for encrypted messages were blank. ([\#8545](https://github.com/matrix-org/synapse/issues/8545))
- Fix increase in the number of `There was no active span...` errors logged when using OpenTracing. ([\#8567](https://github.com/matrix-org/synapse/issues/8567))
- Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed. ([\#8585](https://github.com/matrix-org/synapse/issues/8585))
- Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. ([\#8606](https://github.com/matrix-org/synapse/issues/8606))
Updates to the Docker image
---------------------------
- Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. ([\#7921](https://github.com/matrix-org/synapse/issues/7921))
- Add support for passing commandline args to the synapse process. Contributed by @samuel-p. ([\#8390](https://github.com/matrix-org/synapse/issues/8390))
Improved Documentation
----------------------
- Update the directions for using the manhole with coroutines. ([\#8462](https://github.com/matrix-org/synapse/issues/8462))
- Improve readme by adding new shield.io badges. ([\#8493](https://github.com/matrix-org/synapse/issues/8493))
- Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @Maquis196. ([\#8526](https://github.com/matrix-org/synapse/issues/8526))
- Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration. ([\#8529](https://github.com/matrix-org/synapse/issues/8529))
Deprecations and Removals
-------------------------
- Drop unused `device_max_stream_id` table. ([\#8589](https://github.com/matrix-org/synapse/issues/8589))
Internal Changes
----------------
- Check for unreachable code with mypy. ([\#8432](https://github.com/matrix-org/synapse/issues/8432))
- Add unit test for event persister sharding. ([\#8433](https://github.com/matrix-org/synapse/issues/8433))
- Allow events to be sent to clients sooner when using sharded event persisters. ([\#8439](https://github.com/matrix-org/synapse/issues/8439), [\#8488](https://github.com/matrix-org/synapse/issues/8488), [\#8496](https://github.com/matrix-org/synapse/issues/8496), [\#8499](https://github.com/matrix-org/synapse/issues/8499))
- Configure `public_baseurl` when using demo scripts. ([\#8443](https://github.com/matrix-org/synapse/issues/8443))
- Add SQL logging on queries that happen during startup. ([\#8448](https://github.com/matrix-org/synapse/issues/8448))
- Speed up unit tests when using PostgreSQL. ([\#8450](https://github.com/matrix-org/synapse/issues/8450))
- Remove redundant database loads of stream_ordering for events we already have. ([\#8452](https://github.com/matrix-org/synapse/issues/8452))
- Reduce inconsistencies between codepaths for membership and non-membership events. ([\#8463](https://github.com/matrix-org/synapse/issues/8463))
- Combine `SpamCheckerApi` with the more generic `ModuleApi`. ([\#8464](https://github.com/matrix-org/synapse/issues/8464))
- Additional testing for `ThirdPartyEventRules`. ([\#8468](https://github.com/matrix-org/synapse/issues/8468))
- Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit. ([\#8472](https://github.com/matrix-org/synapse/issues/8472))
- Unblacklist some sytests. ([\#8474](https://github.com/matrix-org/synapse/issues/8474))
- Include the log level in the phone home stats. ([\#8477](https://github.com/matrix-org/synapse/issues/8477))
- Remove outdated sphinx documentation, scripts and configuration. ([\#8480](https://github.com/matrix-org/synapse/issues/8480))
- Clarify error message when plugin config parsers raise an error. ([\#8492](https://github.com/matrix-org/synapse/issues/8492))
- Remove the deprecated `Handlers` object. ([\#8494](https://github.com/matrix-org/synapse/issues/8494))
- Fix a threadsafety bug in unit tests. ([\#8497](https://github.com/matrix-org/synapse/issues/8497))
- Add user agent to user_daily_visits table. ([\#8503](https://github.com/matrix-org/synapse/issues/8503))
- Add type hints to various parts of the code base. ([\#8407](https://github.com/matrix-org/synapse/issues/8407), [\#8505](https://github.com/matrix-org/synapse/issues/8505), [\#8507](https://github.com/matrix-org/synapse/issues/8507), [\#8547](https://github.com/matrix-org/synapse/issues/8547), [\#8562](https://github.com/matrix-org/synapse/issues/8562), [\#8609](https://github.com/matrix-org/synapse/issues/8609))
- Remove unused code from the test framework. ([\#8514](https://github.com/matrix-org/synapse/issues/8514))
- Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable. ([\#8515](https://github.com/matrix-org/synapse/issues/8515))
- Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`. ([\#8537](https://github.com/matrix-org/synapse/issues/8537))
- Improve database performance by executing more queries without starting transactions. ([\#8542](https://github.com/matrix-org/synapse/issues/8542))
- Rename `Cache` to `DeferredCache`, to better reflect its purpose. ([\#8548](https://github.com/matrix-org/synapse/issues/8548))
- Move metric registration code down into `LruCache`. ([\#8561](https://github.com/matrix-org/synapse/issues/8561), [\#8591](https://github.com/matrix-org/synapse/issues/8591))
- Replace `DeferredCache` with the lighter-weight `LruCache` where possible. ([\#8563](https://github.com/matrix-org/synapse/issues/8563))
- Add virtualenv-generated folders to `.gitignore`. ([\#8566](https://github.com/matrix-org/synapse/issues/8566))
- Add `get_immediate` method to `DeferredCache`. ([\#8568](https://github.com/matrix-org/synapse/issues/8568))
- Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`. ([\#8569](https://github.com/matrix-org/synapse/issues/8569))
- Fix `synmark` benchmark runner. ([\#8571](https://github.com/matrix-org/synapse/issues/8571))
- Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s. ([\#8572](https://github.com/matrix-org/synapse/issues/8572))
- Adjust a protocol-type definition to fit `sqlite3` assertions. ([\#8577](https://github.com/matrix-org/synapse/issues/8577))
- Support macOS on the `synmark` benchmark runner. ([\#8578](https://github.com/matrix-org/synapse/issues/8578))
- Update `mypy` static type checker to 0.790. ([\#8583](https://github.com/matrix-org/synapse/issues/8583), [\#8600](https://github.com/matrix-org/synapse/issues/8600))
- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8587](https://github.com/matrix-org/synapse/issues/8587))
- Remove extraneous unittest logging decorators from unit tests. ([\#8592](https://github.com/matrix-org/synapse/issues/8592))
- Minor optimisations in caching code. ([\#8593](https://github.com/matrix-org/synapse/issues/8593), [\#8594](https://github.com/matrix-org/synapse/issues/8594))
Synapse 1.21.2 (2020-10-15)
===========================

1
changelog.d/7658.feature
View file

@ -1 +0,0 @@
Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch.

1
changelog.d/7921.docker
View file

@ -1 +0,0 @@
Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196.

1
changelog.d/8292.feature
View file

@ -1 +0,0 @@
Allow `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory.

1
changelog.d/8312.feature
View file

@ -1 +0,0 @@
Add support for olm fallback keys ([MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732)).

1
changelog.d/8369.feature
View file

@ -1 +0,0 @@
Allow running background tasks in a separate worker process.

1
changelog.d/8380.feature
View file

@ -1 +0,0 @@
Add support for device dehydration ([MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697)).

1
changelog.d/8390.docker
View file

@ -1 +0,0 @@
Add support for passing commandline args to the synapse process. Contributed by @samuel-p.

1
changelog.d/8407.misc
View file

@ -1 +0,0 @@
Add typing information to the device handler.

1
changelog.d/8432.misc
View file

@ -1 +0,0 @@
Check for unreachable code with mypy.

1
changelog.d/8433.misc
View file

@ -1 +0,0 @@
Add unit test for event persister sharding.

1
changelog.d/8437.feature
View file

@ -1 +0,0 @@
Implement [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409) to send typing, read receipts, and presence events to appservices.

1
changelog.d/8439.misc
View file

@ -1 +0,0 @@
Allow events to be sent to clients sooner when using sharded event persisters.

1
changelog.d/8443.misc
View file

@ -1 +0,0 @@
Configure `public_baseurl` when using demo scripts.

1
changelog.d/8448.misc
View file

@ -1 +0,0 @@
Add SQL logging on queries that happen during startup.

1
changelog.d/8450.misc
View file

@ -1 +0,0 @@
Speed up unit tests when using PostgreSQL.

1
changelog.d/8452.misc
View file

@ -1 +0,0 @@
Remove redundant databae loads of stream_ordering for events we already have.

1
changelog.d/8454.bugfix
View file

@ -1 +0,0 @@
Fix a longstanding bug where invalid ignored users in account data could break clients.

1
changelog.d/8457.bugfix
View file

@ -1 +0,0 @@
Fix a bug where backfilling a room with an event that was missing the `redacts` field would break.

1
changelog.d/8458.feature
View file

@ -1 +0,0 @@
Allow running background tasks in a separate worker process.

1
changelog.d/8461.feature
View file

@ -1 +0,0 @@
Change default room version to "6", per [MSC2788](https://github.com/matrix-org/matrix-doc/pull/2788).

1
changelog.d/8462.doc
View file

@ -1 +0,0 @@
Update the directions for using the manhole with coroutines.

1
changelog.d/8463.misc
View file

@ -1 +0,0 @@
Reduce inconsistencies between codepaths for membership and non-membership events.

1
changelog.d/8464.misc
View file

@ -1 +0,0 @@
Combine `SpamCheckerApi` with the more generic `ModuleApi`.

1
changelog.d/8465.bugfix
View file

@ -1 +0,0 @@
Don't attempt to respond to some requests if the client has already disconnected.

1
changelog.d/8467.feature
View file

@ -1 +0,0 @@
Allow `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory.

1
changelog.d/8468.misc
View file

@ -1 +0,0 @@
Additional testing for `ThirdPartyEventRules`.

1
changelog.d/8472.misc
View file

@ -1 +0,0 @@
Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit.

1
changelog.d/8474.misc
View file

@ -1 +0,0 @@
Unblacklist some sytests.

1
changelog.d/8476.bugfix
View file

@ -1 +0,0 @@
Fix message duplication if something goes wrong after persisting the event.

1
changelog.d/8477.misc
View file

@ -1 +0,0 @@
Include the log level in the phone home stats.

1
changelog.d/8479.feature
View file

@ -1 +0,0 @@
Add the ability to send non-membership events into a room via the `ModuleApi`.

1
changelog.d/8480.misc
View file

@ -1 +0,0 @@
Remove outdated sphinx documentation, scripts and configuration.

1
changelog.d/8486.bugfix
View file

@ -1 +0,0 @@
Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0.

1
changelog.d/8488.misc
View file

@ -1 +0,0 @@
Allow events to be sent to clients sooner when using sharded event persisters.

1
changelog.d/8489.feature
View file

@ -1 +0,0 @@
Allow running background tasks in a separate worker process.

1
changelog.d/8492.misc
View file

@ -1 +0,0 @@
Clarify error message when plugin config parsers raise an error.

1
changelog.d/8493.doc
View file

@ -1 +0,0 @@
Improve readme by adding new shield.io badges.

1
changelog.d/8494.misc
View file

@ -1 +0,0 @@
Remove the deprecated `Handlers` object.

1
changelog.d/8496.misc
View file

@ -1 +0,0 @@
Allow events to be sent to clients sooner when using sharded event persisters.

1
changelog.d/8497.misc
View file

@ -1 +0,0 @@
Fix a threadsafety bug in unit tests.

1
changelog.d/8499.misc
View file

@ -1 +0,0 @@
Allow events to be sent to clients sooner when using sharded event persisters.

1
changelog.d/8501.feature
View file

@ -1 +0,0 @@
Add support for olm fallback keys ([MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732)).

1
changelog.d/8502.feature
View file

@ -1 +0,0 @@
Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz.

1
changelog.d/8503.misc
View file

@ -1 +0,0 @@
Add user agent to user_daily_visits table.

1
changelog.d/8504.bugfix
View file

@ -1 +0,0 @@
Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow.

1
changelog.d/8505.misc
View file

@ -1 +0,0 @@
Add type hints to various parts of the code base.

1
changelog.d/8507.misc
View file

@ -1 +0,0 @@
Add type hints to various parts of the code base.

1
changelog.d/8513.feature
View file

@ -1 +0,0 @@
Allow running background tasks in a separate worker process.

1
changelog.d/8514.misc
View file

@ -1 +0,0 @@
Remove unused code from the test framework.

1
changelog.d/8515.misc
View file

@ -1 +0,0 @@
Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable.

1
changelog.d/8517.bugfix
View file

@ -1 +0,0 @@
Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`.

1
changelog.d/8526.doc
View file

@ -1 +0,0 @@
Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @Maquis196.

1
changelog.d/8527.bugfix
View file

@ -1 +0,0 @@
Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table.

1
changelog.d/8529.doc
View file

@ -1 +0,0 @@
Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration.

1
changelog.d/8535.feature
View file

@ -1 +0,0 @@
Support modifying event content in `ThirdPartyRules` modules.

1
changelog.d/8536.bugfix
View file

@ -1 +0,0 @@
Fix not sending events over federation when using sharded event writers.

1
changelog.d/8537.misc
View file

@ -1 +0,0 @@
Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`.

1
changelog.d/8542.misc
View file

@ -1 +0,0 @@
Improve database performance by executing more queries without starting transactions.

1
changelog.d/8544.feature
View file

@ -1 +0,0 @@
Allow running background tasks in a separate worker process.

1
changelog.d/8545.bugfix
View file

@ -1 +0,0 @@
Fix a long standing bug where email notifications for encrypted messages were blank.

1
changelog.d/8547.misc
View file

@ -1 +0,0 @@
Enable mypy type checking for `synapse.util.caches`.

1
changelog.d/8548.misc
View file

@ -1 +0,0 @@
Rename `Cache` to `DeferredCache`, to better reflect its purpose.

1
changelog.d/8561.misc
View file

@ -1 +0,0 @@
Move metric registration code down into `LruCache`.

1
changelog.d/8562.misc
View file

@ -1 +0,0 @@
Add type annotations for `LruCache`.

1
changelog.d/8563.misc
View file

@ -1 +0,0 @@
Replace `DeferredCache` with the lighter-weight `LruCache` where possible.

1
changelog.d/8564.feature
View file

@ -1 +0,0 @@
Support modifying event content in `ThirdPartyRules` modules.

1
changelog.d/8566.misc
View file

@ -1 +0,0 @@
Add virtualenv-generated folders to `.gitignore`.

1
changelog.d/8567.bugfix
View file

@ -1 +0,0 @@
Fix increase in the number of `There was no active span...` errors logged when using OpenTracing.

1
changelog.d/8568.misc
View file

@ -1 +0,0 @@
Add `get_immediate` method to `DeferredCache`.

1
changelog.d/8569.misc
View file

@ -1 +0,0 @@
Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`.

1
changelog.d/8571.misc
View file

@ -1 +0,0 @@
Fix `synmark` benchmark runner.

1
changelog.d/8572.misc
View file

@ -1 +0,0 @@
Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s.

1
changelog.d/8577.misc
View file

@ -1 +0,0 @@
Adjust a protocol-type definition to fit `sqlite3` assertions.

1
changelog.d/8578.misc
View file

@ -1 +0,0 @@
Support macOS on the `synmark` benchmark runner.

1
changelog.d/8583.misc
View file

@ -1 +0,0 @@
Update `mypy` static type checker to 0.790.

1
changelog.d/8585.bugfix
View file

@ -1 +0,0 @@
Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed.

1
changelog.d/8587.misc
View file

@ -1 +0,0 @@
Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting.

1
changelog.d/8589.removal
View file

@ -1 +0,0 @@
Drop unused `device_max_stream_id` table.

1
changelog.d/8590.misc
View file

@ -1 +0,0 @@
Implement [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409) to send typing, read receipts, and presence events to appservices.

1
changelog.d/8591.misc
View file

@ -1 +0,0 @@
Move metric registration code down into `LruCache`.

1
changelog.d/8592.misc
View file

@ -1 +0,0 @@
Remove extraneous unittest logging decorators from unit tests.

1
changelog.d/8593.misc
View file

@ -1 +0,0 @@
Minor optimisations in caching code.

1
changelog.d/8594.misc
View file

@ -1 +0,0 @@
Minor optimisations in caching code.

1
changelog.d/8599.feature
View file

@ -1 +0,0 @@
Allow running background tasks in a separate worker process.

1
changelog.d/8600.misc
View file

@ -1 +0,0 @@
Update `mypy` static type checker to 0.790.

1
changelog.d/8606.feature
View file

@ -1 +0,0 @@
Limit appservice transactions to 100 persistent and 100 ephemeral events.

1
changelog.d/8609.misc
View file

@ -1 +0,0 @@
Add type hints to profile and base handler.

1
changelog.d/8648.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a bug introduced in v1.22.0rc1 which would cause ephemeral events to not be sent to appservices.

1
changelog.d/8654.bugfix Normal file
View file

@ -0,0 +1 @@
Fix `user_daily_visits` to not have duplicate rows for UA. Broke in v1.22.0rc1.

1
changelog.d/8656.bugfix Normal file
View file

@ -0,0 +1 @@
Fix a bug introduced in v1.22.0rc1 where presence events were not properly passed to application services.

2
synapse/__init__.py
View file

@ -48,7 +48,7 @@ try:
except ImportError:
pass
__version__ = "1.21.2"
__version__ = "1.22.0rc1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when

21
synapse/handlers/appservice.py
View file

@ -203,16 +203,16 @@ class ApplicationServicesHandler:
events = await self._handle_receipts(service)
if events:
self.scheduler.submit_ephemeral_events_for_as(service, events)
await self.store.set_type_stream_id_for_appservice(
service, "read_receipt", new_token
)
await self.store.set_type_stream_id_for_appservice(
service, "read_receipt", new_token
)
elif stream_key == "presence_key":
events = await self._handle_presence(service, users)
if events:
self.scheduler.submit_ephemeral_events_for_as(service, events)
await self.store.set_type_stream_id_for_appservice(
service, "presence", new_token
)
await self.store.set_type_stream_id_for_appservice(
service, "presence", new_token
)
async def _handle_typing(self, service: ApplicationService, new_token: int):
typing_source = self.event_sources.sources["typing"]
@ -238,7 +238,7 @@ class ApplicationServicesHandler:
async def _handle_presence(
self, service: ApplicationService, users: Collection[UserID]
):
) -> List[JsonDict]:
events = [] # type: List[JsonDict]
presence_source = self.event_sources.sources["presence"]
from_key = await self.store.get_type_stream_id_for_appservice(
@ -252,7 +252,7 @@ class ApplicationServicesHandler:
user=user, service=service, from_key=from_key,
)
time_now = self.clock.time_msec()
presence_events = [
events.extend(
{
"type": "m.presence",
"sender": event.user_id,
@ -261,8 +261,9 @@ class ApplicationServicesHandler:
),
}
for event in presence_events
]
events = events + presence_events
)
return events
async def query_user_exists(self, user_id):
"""Check if any application service knows this user_id exists.

29
synapse/storage/databases/main/appservice.py
View file

@ -369,17 +369,25 @@ class ApplicationServiceTransactionWorkerStore(
async def get_type_stream_id_for_appservice(
self, service: ApplicationService, type: str
) -> int:
if type not in ("read_receipt", "presence"):
raise ValueError(
"Expected type to be a valid application stream id type, got %s"
% (type,)
)
def get_type_stream_id_for_appservice_txn(txn):
stream_id_type = "%s_stream_id" % type
txn.execute(
"SELECT ? FROM application_services_state WHERE as_id=?",
(stream_id_type, service.id,),
# We do NOT want to escape `stream_id_type`.
"SELECT %s FROM application_services_state WHERE as_id=?"
% stream_id_type,
(service.id,),
)
last_txn_id = txn.fetchone()
if last_txn_id is None or last_txn_id[0] is None: # no row exists
last_stream_id = txn.fetchone()
if last_stream_id is None or last_stream_id[0] is None: # no row exists
return 0
else:
return int(last_txn_id[0])
return int(last_stream_id[0])
return await self.db_pool.runInteraction(
"get_type_stream_id_for_appservice", get_type_stream_id_for_appservice_txn
@ -388,11 +396,18 @@ class ApplicationServiceTransactionWorkerStore(
async def set_type_stream_id_for_appservice(
self, service: ApplicationService, type: str, pos: int
) -> None:
if type not in ("read_receipt", "presence"):
raise ValueError(
"Expected type to be a valid application stream id type, got %s"
% (type,)
)
def set_type_stream_id_for_appservice_txn(txn):
stream_id_type = "%s_stream_id" % type
txn.execute(
"UPDATE ? SET device_list_stream_id = ? WHERE as_id=?",
(stream_id_type, pos, service.id),
"UPDATE application_services_state SET %s = ? WHERE as_id=?"
% stream_id_type,
(pos, service.id),
)
await self.db_pool.runInteraction(

9
synapse/storage/databases/main/metrics.py
View file

@ -282,9 +282,10 @@ class ServerMetricsStore(EventPushActionsWorkerStore, SQLBaseStore):
now = self._clock.time_msec()
# A note on user_agent. Technically a given device can have multiple
# user agents, so we need to decide which one to pick. We could have handled this
# in number of ways, but given that we don't _that_ much have gone for MAX()
# For more details of the other options considered see
# user agents, so we need to decide which one to pick. We could have
# handled this in number of ways, but given that we don't care
# _that_ much we have gone for MAX(). For more details of the other
# options considered see
# https://github.com/matrix-org/synapse/pull/8503#discussion_r502306111
sql = """
INSERT INTO user_daily_visits (user_id, device_id, timestamp, user_agent)
@ -299,7 +300,7 @@ class ServerMetricsStore(EventPushActionsWorkerStore, SQLBaseStore):
WHERE last_seen > ? AND last_seen <= ?
AND udv.timestamp IS NULL AND users.is_guest=0
AND users.appservice_id IS NULL
GROUP BY u.user_id, u.device_id, u.user_agent
GROUP BY u.user_id, u.device_id
"""
# This means that the day has rolled over but there could still

56
tests/storage/test_appservice.py
View file

@ -410,6 +410,62 @@ class ApplicationServiceTransactionStoreTestCase(unittest.TestCase):
)
class ApplicationServiceStoreTypeStreamIds(unittest.HomeserverTestCase):
def make_homeserver(self, reactor, clock):
hs = self.setup_test_homeserver()
return hs
def prepare(self, hs, reactor, clock):
self.service = Mock(id="foo")
self.store = self.hs.get_datastore()
self.get_success(self.store.set_appservice_state(self.service, "up"))
def test_get_type_stream_id_for_appservice_no_value(self):
value = self.get_success(
self.store.get_type_stream_id_for_appservice(self.service, "read_receipt")
)
self.assertEquals(value, 0)
value = self.get_success(
self.store.get_type_stream_id_for_appservice(self.service, "presence")
)
self.assertEquals(value, 0)
def test_get_type_stream_id_for_appservice_invalid_type(self):
self.get_failure(
self.store.get_type_stream_id_for_appservice(self.service, "foobar"),
ValueError,
)
def test_set_type_stream_id_for_appservice(self):
read_receipt_value = 1024
self.get_success(
self.store.set_type_stream_id_for_appservice(
self.service, "read_receipt", read_receipt_value
)
)
result = self.get_success(
self.store.get_type_stream_id_for_appservice(self.service, "read_receipt")
)
self.assertEqual(result, read_receipt_value)
self.get_success(
self.store.set_type_stream_id_for_appservice(
self.service, "presence", read_receipt_value
)
)
result = self.get_success(
self.store.get_type_stream_id_for_appservice(self.service, "presence")
)
self.assertEqual(result, read_receipt_value)
def test_set_type_stream_id_for_appservice_invalid_type(self):
self.get_failure(
self.store.set_type_stream_id_for_appservice(self.service, "foobar", 1024),
ValueError,
)
# required for ApplicationServiceTransactionStoreTestCase tests
class TestTransactionStore(ApplicationServiceTransactionStore, ApplicationServiceStore):
def __init__(self, database: DatabasePool, db_conn, hs):