synapse/synapse/storage
Erik Johnston 52a50e8686
Use vector clocks for room stream tokens. (#8439)
Currently when using multiple event persisters we (in the worst case) don't tell clients about events until all event persisters have persisted new events after the original event. This is a suboptimal, especially if one of the event persisters goes down.

To handle this, we encode the position of each event persister in the room tokens so that we can send events to clients immediately. To reduce the size of the token we do two things:

1. We create a unique immutable persistent mapping between instance names and a generated small integer ID, which we can encode in the tokens instead of the instance name; and
2. We encode the "persisted upto position" of the room token and then only explicitly include instances that have positions strictly greater than that.

The new tokens look something like: `m3478~1.3488~2.3489`, where the first number is the min position, and the subsequent `-` separated pairs are the instance ID to positions map. (We use `.` and `~` as separators as they're URL safe and not already used by `StreamToken`).
2020-10-07 15:15:33 +01:00
..
databases Use vector clocks for room stream tokens. (#8439) 2020-10-07 15:15:33 +01:00
engines Use SequenceGenerator for state group ID allocation 2020-07-16 11:25:08 +01:00
schema Only run one background update at a time 2020-03-31 17:43:58 +01:00
util Add logging on startup/shutdown (#8448) 2020-10-02 15:20:45 +01:00
__init__.py Make StreamToken.room_key be a RoomStreamToken instance. (#8281) 2020-09-11 12:22:55 +01:00
_base.py Be stricter about JSON that is accepted by Synapse (#8106) 2020-08-19 07:26:03 -04:00
background_updates.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
database.py Add logging on startup/shutdown (#8448) 2020-10-02 15:20:45 +01:00
keys.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
persist_events.py Remove stream ordering from Metadata dict (#8452) 2020-10-05 14:43:14 +01:00
prepare_database.py Add logging on startup/shutdown (#8448) 2020-10-02 15:20:45 +01:00
purge_events.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
push_rule.py Move storage classes into a main "data store". 2019-10-21 16:05:06 +01:00
relations.py Use slots in attrs classes where possible (#8296) 2020-09-14 12:50:06 -04:00
roommember.py Add EventStreamPosition type (#8388) 2020-09-24 13:24:17 +01:00
state.py Mypy fixes for synapse.handlers.federation (#8422) 2020-09-29 15:57:36 +01:00
types.py Add logging on startup/shutdown (#8448) 2020-10-02 15:20:45 +01:00