Merge remote-tracking branch 'origin/release-v1.43' into matrix-org-hotfixes

This commit is contained in:
Erik Johnston 2021-09-16 11:12:52 +01:00
commit bb0a4bd6d5
56 changed files with 231 additions and 191 deletions

View file

@ -1,3 +1,72 @@
Synapse 1.43.0rc1 (2021-09-14)
==============================
This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
Features
--------
- Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566))
- Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643))
- Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712))
- Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714), [\#10759](https://github.com/matrix-org/synapse/issues/10759))
- Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772))
Bugfixes
--------
- Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658))
- Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730))
- Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743))
- Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757))
- Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733))
- Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10760](https://github.com/matrix-org/synapse/issues/10760))
- Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774))
- Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783))
Improved Documentation
----------------------
- Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556))
- Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648))
- Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735))
- Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758))
- Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html). ([\#10795](https://github.com/matrix-org/synapse/issues/10795))
- Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804))
Deprecations and Removals
-------------------------
- Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693))
Internal Changes
----------------
- Add OpenTracing logging to help debug stuck messages (as described by issue [#9424](https://github.com/matrix-org/synapse/issues/9424)). ([\#10704](https://github.com/matrix-org/synapse/issues/10704))
- Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601))
- Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697))
- Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707), [\#10728](https://github.com/matrix-org/synapse/issues/10728), [\#10736](https://github.com/matrix-org/synapse/issues/10736))
- Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727))
- Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738))
- Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744), [\#10745](https://github.com/matrix-org/synapse/issues/10745), [\#10746](https://github.com/matrix-org/synapse/issues/10746), [\#10771](https://github.com/matrix-org/synapse/issues/10771), [\#10773](https://github.com/matrix-org/synapse/issues/10773), [\#10781](https://github.com/matrix-org/synapse/issues/10781))
- Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748))
- Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750))
- Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752))
- Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753))
- Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754), [\#10755](https://github.com/matrix-org/synapse/issues/10755), [\#10756](https://github.com/matrix-org/synapse/issues/10756), [\#10780](https://github.com/matrix-org/synapse/issues/10780), [\#10784](https://github.com/matrix-org/synapse/issues/10784))
- Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775))
- Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778))
- Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779))
- Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788))
- Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789))
- Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798))
- Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799))
Synapse 1.42.0 (2021-09-07)
===========================

View file

@ -1 +0,0 @@
Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109.

View file

@ -1 +0,0 @@
Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) in existing room versions.

View file

@ -1 +0,0 @@
Add type annotations to the synapse.util package.

View file

@ -1 +0,0 @@
Add config option to use non-default manhole password and keys.

View file

@ -1 +0,0 @@
Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers.

View file

@ -1 +0,0 @@
Fix a long-standing bug where room avatars were not included in email notifications.

View file

@ -1 +0,0 @@
Remove [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so.

View file

@ -1 +0,0 @@
Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) usage later.

View file

@ -1 +0,0 @@
Added opentrace logging to help debug #9424.

View file

@ -1 +0,0 @@
Add missing type hints to REST servlets.

View file

@ -1 +0,0 @@
Skip final GC at shutdown to improve restart performance.

View file

@ -1 +0,0 @@
Allow configuration of the oEmbed URLs used for URL previews.

View file

@ -1 +0,0 @@
Do not include rooms with unknown room versions in the spaces summary results.

View file

@ -1 +0,0 @@
Add missing type hints to REST servlets.

View file

@ -1 +0,0 @@
Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings.

View file

@ -1 +0,0 @@
Allow user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr.

View file

@ -1 +0,0 @@
Clarify admin API documentation on undoing room deletions.

View file

@ -1 +0,0 @@
Add missing type hints to REST servlets.

View file

@ -1 +0,0 @@
Additional error checking for the `preset` field when creating a room.

View file

@ -1 +0,0 @@
Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated).

View file

@ -1 +0,0 @@
Clean up some of the federation event authentication code for clarity.

View file

@ -1 +0,0 @@
Clean up some of the federation event authentication code for clarity.

View file

@ -1 +0,0 @@
Clean up some of the federation event authentication code for clarity.

View file

@ -1 +0,0 @@
Add an index to `presence_stream` to hopefully speed up startups a little.

View file

@ -1 +0,0 @@
Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted.

View file

@ -1 +0,0 @@
Move tests relating to rooms having encryption out of the user_directory tests.

View file

@ -1 +0,0 @@
Use `attrs` internally for the URL preview code & update documentation.

View file

@ -1 +0,0 @@
Minor speed ups when joining large rooms over federation.

View file

@ -1 +0,0 @@
Minor speed ups when joining large rooms over federation.

View file

@ -1 +0,0 @@
Minor speed ups when joining large rooms over federation.

View file

@ -1 +0,0 @@
Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes.

View file

@ -1 +0,0 @@
Split up the modules documentation and add examples for module developers.

View file

@ -1 +0,0 @@
Allow configuration of the oEmbed URLs used for URL previews.

View file

@ -1 +0,0 @@
Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).

View file

@ -1 +0,0 @@
Clean up some of the federation event authentication code for clarity.

View file

@ -1 +0,0 @@
Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375) for restricted rooms per the [room version caapabilities](https://github.com/matrix-org/matrix-doc/pull/3244) API.

View file

@ -1 +0,0 @@
Clean up some of the federation event authentication code for clarity.

View file

@ -1 +0,0 @@
Properly handle room upgrades of spaces.

View file

@ -1 +0,0 @@
Add a constant for `m.federate`.

View file

@ -1 +0,0 @@
Add a script to update the Debian changelog in a Docker container for systems that aren't Debian-based.

View file

@ -1 +0,0 @@
Change the format of authenticated users in logs when a user is being puppeted by and admin user.

View file

@ -1 +0,0 @@
Minor speed ups when joining large rooms over federation.

View file

@ -1 +0,0 @@
Clean up some of the federation event authentication code for clarity.

View file

@ -1 +0,0 @@
Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image.

View file

@ -1 +0,0 @@
Minor speed ups when joining large rooms over federation.

View file

@ -1 +0,0 @@
Remove fixed and flakey tests from the sytest-blacklist.

View file

@ -1 +0,0 @@
Improve internal details of the user directory code.

View file

@ -1 +0,0 @@
Correct 2 typographical errors in the *Log Contexts* documentation.

View file

@ -1 +0,0 @@
Use direct references to config flags.

View file

@ -1 +0,0 @@
Add a max version for the `jaeger-client` dependency for an incompatibility with the rust reporter.

View file

@ -1 +0,0 @@
Fixed a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net.

1
changelog.d/10828.bugfix Normal file
View file

@ -0,0 +1 @@
Added opentrace logging to help debug #9424.

6
debian/changelog vendored
View file

@ -1,3 +1,9 @@
matrix-synapse-py3 (1.43.0~rc1) stable; urgency=medium
* New synapse release 1.43.0~rc1.
-- Synapse Packaging team <packages@matrix.org> Tue, 14 Sep 2021 11:39:46 +0100
matrix-synapse-py3 (1.42.0) stable; urgency=medium
* New synapse release 1.42.0.

View file

@ -47,7 +47,7 @@ try:
except ImportError:
pass
__version__ = "1.42.0"
__version__ = "1.43.0rc1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when

View file

@ -1533,21 +1533,18 @@ class SyncHandler:
newly_left_rooms = room_changes.newly_left_rooms
async def handle_room_entries(room_entry: "RoomSyncResultBuilder"):
with start_active_span("generate_room_entry"):
set_tag("room_id", room_entry.room_id)
log_kv({"events": len(room_entry.events or [])})
logger.debug("Generating room entry for %s", room_entry.room_id)
res = await self._generate_room_entry(
sync_result_builder,
ignored_users,
room_entry,
ephemeral=ephemeral_by_room.get(room_entry.room_id, []),
tags=tags_by_room.get(room_entry.room_id),
account_data=account_data_by_room.get(room_entry.room_id, {}),
always_include=sync_result_builder.full_state,
)
logger.debug("Generated room entry for %s", room_entry.room_id)
return res
logger.debug("Generating room entry for %s", room_entry.room_id)
res = await self._generate_room_entry(
sync_result_builder,
ignored_users,
room_entry,
ephemeral=ephemeral_by_room.get(room_entry.room_id, []),
tags=tags_by_room.get(room_entry.room_id),
account_data=account_data_by_room.get(room_entry.room_id, {}),
always_include=sync_result_builder.full_state,
)
logger.debug("Generated room entry for %s", room_entry.room_id)
return res
await concurrently_execute(handle_room_entries, room_entries, 10)
@ -1960,139 +1957,157 @@ class SyncHandler:
room_id = room_builder.room_id
since_token = room_builder.since_token
upto_token = room_builder.upto_token
log_kv(
{
"since_token": since_token,
"upto_token": upto_token,
}
)
batch = await self._load_filtered_recents(
room_id,
sync_config,
now_token=upto_token,
since_token=since_token,
potential_recents=events,
newly_joined_room=newly_joined,
)
log_kv(
{
"batch_events": len(batch.events),
"prev_batch": batch.prev_batch,
"batch_limited": batch.limited,
}
)
with start_active_span("generate_room_entry"):
set_tag("room_id", room_id)
log_kv({"events": len(events or ())})
# Note: `batch` can be both empty and limited here in the case where
# `_load_filtered_recents` can't find any events the user should see
# (e.g. due to having ignored the sender of the last 50 events).
log_kv(
{
"since_token": since_token,
"upto_token": upto_token,
}
)
if newly_joined:
# debug for https://github.com/matrix-org/synapse/issues/4422
issue4422_logger.debug(
"Timeline events after filtering in newly-joined room %s: %r",
batch = await self._load_filtered_recents(
room_id,
sync_config,
now_token=upto_token,
since_token=since_token,
potential_recents=events,
newly_joined_room=newly_joined,
)
log_kv(
{
"batch_events": len(batch.events),
"prev_batch": batch.prev_batch,
"batch_limited": batch.limited,
}
)
# Note: `batch` can be both empty and limited here in the case where
# `_load_filtered_recents` can't find any events the user should see
# (e.g. due to having ignored the sender of the last 50 events).
if newly_joined:
# debug for https://github.com/matrix-org/synapse/issues/4422
issue4422_logger.debug(
"Timeline events after filtering in newly-joined room %s: %r",
room_id,
batch,
)
# When we join the room (or the client requests full_state), we should
# send down any existing tags. Usually the user won't have tags in a
# newly joined room, unless either a) they've joined before or b) the
# tag was added by synapse e.g. for server notice rooms.
if full_state:
user_id = sync_result_builder.sync_config.user.to_string()
tags = await self.store.get_tags_for_room(user_id, room_id)
# If there aren't any tags, don't send the empty tags list down
# sync
if not tags:
tags = None
account_data_events = []
if tags is not None:
account_data_events.append({"type": "m.tag", "content": {"tags": tags}})
for account_data_type, content in account_data.items():
account_data_events.append(
{"type": account_data_type, "content": content}
)
account_data_events = (
sync_config.filter_collection.filter_room_account_data(
account_data_events
)
)
ephemeral = sync_config.filter_collection.filter_room_ephemeral(ephemeral)
if not (
always_include
or batch
or account_data_events
or ephemeral
or full_state
):
return
state = await self.compute_state_delta(
room_id,
batch,
sync_config,
since_token,
now_token,
full_state=full_state,
)
# When we join the room (or the client requests full_state), we should
# send down any existing tags. Usually the user won't have tags in a
# newly joined room, unless either a) they've joined before or b) the
# tag was added by synapse e.g. for server notice rooms.
if full_state:
user_id = sync_result_builder.sync_config.user.to_string()
tags = await self.store.get_tags_for_room(user_id, room_id)
summary: Optional[JsonDict] = {}
# If there aren't any tags, don't send the empty tags list down
# sync
if not tags:
tags = None
account_data_events = []
if tags is not None:
account_data_events.append({"type": "m.tag", "content": {"tags": tags}})
for account_data_type, content in account_data.items():
account_data_events.append({"type": account_data_type, "content": content})
account_data_events = sync_config.filter_collection.filter_room_account_data(
account_data_events
)
ephemeral = sync_config.filter_collection.filter_room_ephemeral(ephemeral)
if not (
always_include or batch or account_data_events or ephemeral or full_state
):
return
state = await self.compute_state_delta(
room_id, batch, sync_config, since_token, now_token, full_state=full_state
)
summary: Optional[JsonDict] = {}
# we include a summary in room responses when we're lazy loading
# members (as the client otherwise doesn't have enough info to form
# the name itself).
if sync_config.filter_collection.lazy_load_members() and (
# we recalculate the summary:
# if there are membership changes in the timeline, or
# if membership has changed during a gappy sync, or
# if this is an initial sync.
any(ev.type == EventTypes.Member for ev in batch.events)
or (
# XXX: this may include false positives in the form of LL
# members which have snuck into state
batch.limited
and any(t == EventTypes.Member for (t, k) in state)
)
or since_token is None
):
summary = await self.compute_summary(
room_id, sync_config, batch, state, now_token
)
if room_builder.rtype == "joined":
unread_notifications: Dict[str, int] = {}
room_sync = JoinedSyncResult(
room_id=room_id,
timeline=batch,
state=state,
ephemeral=ephemeral,
account_data=account_data_events,
unread_notifications=unread_notifications,
summary=summary,
unread_count=0,
)
if room_sync or always_include:
notifs = await self.unread_notifs_for_room_id(room_id, sync_config)
unread_notifications["notification_count"] = notifs["notify_count"]
unread_notifications["highlight_count"] = notifs["highlight_count"]
room_sync.unread_count = notifs["unread_count"]
sync_result_builder.joined.append(room_sync)
if batch.limited and since_token:
user_id = sync_result_builder.sync_config.user.to_string()
logger.debug(
"Incremental gappy sync of %s for user %s with %d state events"
% (room_id, user_id, len(state))
# we include a summary in room responses when we're lazy loading
# members (as the client otherwise doesn't have enough info to form
# the name itself).
if sync_config.filter_collection.lazy_load_members() and (
# we recalculate the summary:
# if there are membership changes in the timeline, or
# if membership has changed during a gappy sync, or
# if this is an initial sync.
any(ev.type == EventTypes.Member for ev in batch.events)
or (
# XXX: this may include false positives in the form of LL
# members which have snuck into state
batch.limited
and any(t == EventTypes.Member for (t, k) in state)
)
elif room_builder.rtype == "archived":
archived_room_sync = ArchivedSyncResult(
room_id=room_id,
timeline=batch,
state=state,
account_data=account_data_events,
)
if archived_room_sync or always_include:
sync_result_builder.archived.append(archived_room_sync)
else:
raise Exception("Unrecognized rtype: %r", room_builder.rtype)
or since_token is None
):
summary = await self.compute_summary(
room_id, sync_config, batch, state, now_token
)
if room_builder.rtype == "joined":
unread_notifications: Dict[str, int] = {}
room_sync = JoinedSyncResult(
room_id=room_id,
timeline=batch,
state=state,
ephemeral=ephemeral,
account_data=account_data_events,
unread_notifications=unread_notifications,
summary=summary,
unread_count=0,
)
if room_sync or always_include:
notifs = await self.unread_notifs_for_room_id(room_id, sync_config)
unread_notifications["notification_count"] = notifs["notify_count"]
unread_notifications["highlight_count"] = notifs["highlight_count"]
room_sync.unread_count = notifs["unread_count"]
sync_result_builder.joined.append(room_sync)
if batch.limited and since_token:
user_id = sync_result_builder.sync_config.user.to_string()
logger.debug(
"Incremental gappy sync of %s for user %s with %d state events"
% (room_id, user_id, len(state))
)
elif room_builder.rtype == "archived":
archived_room_sync = ArchivedSyncResult(
room_id=room_id,
timeline=batch,
state=state,
account_data=account_data_events,
)
if archived_room_sync or always_include:
sync_result_builder.archived.append(archived_room_sync)
else:
raise Exception("Unrecognized rtype: %r", room_builder.rtype)
async def get_rooms_for_user_at(
self, user_id: str, room_key: RoomStreamToken