tileradotorg/synapse
1
0
Fork 0
forked from MirrorHub/synapse
Code Pull requests Activity

Deduplicate device updates sent over replication

Browse source 
We currently send several kHz of device list updates over replication
occisonally, which often causes the replications streams to lag and then
get dropped.

A lot of those updates will actually be duplicates, since we don't send
e.g. device_ids across replication, so let's deduplicate it when we pull
them out of the database.
This commit is contained in:
Erik Johnston 2018-10-29 17:34:34 +00:00
parent a163b748a5
commit 88e5ffe6fe
1 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
synapse/storage/devices.py
View file

@ -589,10 +589,14 @@ class DeviceStore(SQLBaseStore):
combined list of changes to devices, and which destinations need to be combined list of changes to devices, and which destinations need to be
poked. `destination` may be None if no destinations need to be poked. poked. `destination` may be None if no destinations need to be poked.
""" """
# We do a group by here as there can be a large number of duplicate
# entries, since we throw away device IDs.
sql = """ sql = """
SELECT stream_id, user_id, destination FROM device_lists_stream SELECT MAX(stream_id) AS stream_id, user_id, destination
FROM device_lists_stream
LEFT JOIN device_lists_outbound_pokes USING (stream_id, user_id, device_id) LEFT JOIN device_lists_outbound_pokes USING (stream_id, user_id, device_id)
WHERE ? < stream_id AND stream_id <= ? WHERE ? < stream_id AND stream_id <= ?
GROUP BY user_id, destination
""" """
return self._execute( return self._execute(
"get_all_device_list_changes_for_remotes", None, "get_all_device_list_changes_for_remotes", None,