* Merge user API databases into one
* Remove DeviceDatabase from config
* Fix tests
* Try that again
* Clean up keyserver device keys when the devices no longer exist in the user API
* Tweak ordering
* Fix UserExists flag, device check
* Allow including empty entries so we can clean them up
* Remove logging
* Remove dependency on saramajetstream & sarama
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Remove internal.ContinualConsumer from federationapi
* Remove internal.ContinualConsumer from syncapi
* Remove internal.ContinualConsumer from keyserver
* Move to new Prepare function
* Remove saramajetstream & sarama dependency
* Delete unneeded file
* Remove duplicate import
* Log error instead of silently irgnoring it
* Move `OffsetNewest` and `OffsetOldest` into keyserver types, change them to be more sane values
* Fix comments
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Remodel how device list change IDs are created
Previously we made them using the offset Kafka supplied.
We don't run Kafka anymore, so now we make the SQL table assign
the change ID via an AUTOINCREMENTing ID. Redesign the
`keyserver_key_changes` table to have `UNIQUE(user_id)` so we
don't accumulate key changes forevermore, we now have at most 1
row per user which contains the highest change ID.
This needs a SQL migration.
* Ensure we bump the change ID on sqlite
* Actually read the DeviceChangeID not the Offset in synapi
* Add SQL migrations
* Prepare after migration; fixup dendrite-upgrade-test logging
* Use higher version numbers; fix sqlite query to increment better
* Default 0 on postgres
* fixup postgres migration on fresh dendrite instances
* Delete device keys/signatures from key server when deleting device from user API
* Move loop to within database transaction
* Don't fall over deleting no rows
* WIP: Eagerly sync device lists on /user/keys/query requests
Also notify servers when a user's device display name changes. Few
caveats:
- sytest `Device deletion propagates over federation` fails
- `populateResponseWithDeviceKeysFromDatabase` is called from multiple
goroutines and hence is unsafe.
* Handle deleted devices correctly over federation
* Add tests for device list updates
* Add stale_device_lists table and use db before asking remote for device keys
* Fetch remote keys if all devices are requested
* Add display_name col to store remote device names
Few other tweaks to make `Server correctly handles incoming m.device_list_update`
pass.
* Fix sqlite otk bug
* Unbuffered channel to block /send causing sytest to not race anymore
* Linting and fix bug whereby we didn't send updated dl tokens to the client causing a tightloop on /sync sometimes
* No longer assert staleness as Update blocks on workers now
* Back out tweaks
* Bugfixes
* Add InputDeviceListUpdate
* Unbreak unit tests
* Process inbound device list updates from federation
- Persist the keys in the keyserver and produce key changes
- Does not currently fetch keys from the remote server if the prev IDs are missing
* Linting
* Persist partition|offset|user_id in the keyserver
Required for a query API which will be used by the syncapi which
will be called when a `/sync` request comes in which will return
a list of user IDs of people who have changed their device keys
between two tokens.
* Add tests and fix maxOffset bug
* s/offset/log_offset/g because 'offset' is a reserved word in postgres
* Add storage layer for postgres/sqlite
* Return OTK counts when inserting new keys
* Hook up the key DB and make a test pass
* Convert postgres queries to be sqlite queries
* Blacklist test due to requiring rejected events
* Unbreak tests
* Update blacklist
