construct/modules/client/sync/README.md

Client Sync

Organization

This directory contains modules which compose the /sync endpoint content. First note that even though this is a directory of modules, the /sync resource itself is a single endpoint and not a directory. The sync.cc module alone services the /sync resource and invokes the functionality offered by the modules in this directory.

Each module in this directory creates content within some property of the /sync JSON response to fulfill a certain feature (refer to the matrix c2s spec). Some modules have child modules attached to them in the same way the response JSON has objects with child objects and arrays, etc; forming a tree.

There are at least two ways to organize these modules, and this directory represents one of those ways. Another way is to disperse them throughout /client/ near the endpoints related to their feature suite. That still remains a viable option for debate as this system further matures.

Methodology

As documented elsewhere, events processed by the server receive a unique monotonic sequence integer (or index number, type: m::event::idx). The next_batch and since tokens are these sequence integers. This single number is fundamental for the /sync to achieve stateless and stable operation:

• Nothing is saved about a request being made. The since token alone has enough information to fulfill a /sync request without hidden server-side state.
• The same request (same since token) produces a similar result each time, again without any server-side state describing that token.

When a /sync request is made, the since token is compared with the server's current sequence number. One of three branches is then taken:

• Longpoll: When the since token is 1 greater than the current sequence number, the client enters longpoll sync: It waits for the next appropriate event which is then sent immediately. The next_batch will then be 1 greater than the sequence number of that event.

• Linear: When the since token's "delta" from the current sequence number ranges from 0 to some configurable limit (i.e 1024 or 4096), the client enters linear sync: an iteration of events between the since token and the current sequence is made where each event is tested for relevance to the client and a response is then made. If nothing was found for the client in this iteration, it falls back to longpoll sync until an event comes through or timeout.

• Polylog: When the since token's "delta" exceeds the threshold for a linear sync the client enters polylog sync. This is common when no since token is supplied at all (equal to 0) which is known as an initial sync. In this mode, a series of point-lookups are made to compose the response content. This involves iterating the rooms relevant to a user and checking if the sequence numbers for events in the timeline fall within the since window so they can be included. This requires a lot of random access in contrast to linear sync; each room has to be queried at least once for every invocation of polylog sync. The goal for the threshold between polylog and linear is to invoke the cheaper mode: Even though polylog usually involves a minimum of many queries, it is more efficient than a linear iteration of all events on the server.

Implementation

Each /sync module implements two primary functions:

• A simple boolean test of an event which determines if the client should receive that event. This is intended to be invoked for the longpoll and linear modes.

• A complex composer for a response in polylog mode. This requires the function itself to find all the events to include for a satisfying response.