* Add NATS JetStream support
Update shopify/sarama
* Fix addresses
* Don't change Addresses in Defaults
* Update saramajetstream
* Add missing error check
Keep typing events for at least one minute
* Use all configured NATS addresses
* Update saramajetstream
* Try setting up with NATS
* Make sure NATS uses own persistent directory (TODO: make this configurable)
* Update go.mod/go.sum
* Jetstream package
* Various other refactoring
* Build fixes
* Config tweaks, make random jetstream storage path for CI
* Disable interest policies
* Try to sane default on jetstream base path
* Try to use in-memory for CI
* Restore storage/retention
* Update nats.go dependency
* Adapt changes to config
* Remove unneeded TopicFor
* Dep update
* Revert "Remove unneeded TopicFor"
This reverts commit f5a4e4a339.
* Revert changes made to streams
* Fix build problems
* Update nats-server
* Update go.mod/go.sum
* Roomserver input API queuing using NATS
* Fix topic naming
* Prometheus metrics
* More refactoring to remove saramajetstream
* Add missing topic
* Don't try to populate map that doesn't exist
* Roomserver output topic
* Update go.mod/go.sum
* Message acknowledgements
* Ack tweaks
* Try to resume transaction re-sends
* Try to resume transaction re-sends
* Update to matrix-org/gomatrixserverlib@91dadfb
* Remove internal.PartitionStorer from components that don't consume keychanges
* Try to reduce re-allocations a bit in resolveConflictsV2
* Tweak delivery options on RS input
* Publish send-to-device messages into correct JetStream subject
* Async and sync roomserver input
* Update dendrite-config.yaml
* Remove roomserver tests for now (they need rewriting)
* Remove roomserver test again (was merged back in)
* Update documentation
* Docker updates
* More Docker updates
* Update Docker readme again
* Fix lint issues
* Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)
* Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that
* Go 1.16 instead of Go 1.13 for upgrade tests and Complement
* Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"
This reverts commit 368675283f.
* Don't report any errors on `/send` to see what fun that creates
* Fix panics on closed channel sends
* Enforce state key matches sender
* Do the same for leave
* Various tweaks to make tests happier
Squashed commit of the following:
commit 13f9028e7a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 15:47:14 2022 +0000
Do the same for leave
commit e6be7f05c3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 15:33:42 2022 +0000
Enforce state key matches sender
commit 85ede6d64b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 14:07:04 2022 +0000
Fix panics on closed channel sends
commit 9755494a98
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 13:38:22 2022 +0000
Don't report any errors on `/send` to see what fun that creates
commit 3bb4f87b5d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 13:00:26 2022 +0000
Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"
This reverts commit 368675283f.
commit fe2673ed7b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 12:09:34 2022 +0000
Go 1.16 instead of Go 1.13 for upgrade tests and Complement
commit 368675283f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 11:51:45 2022 +0000
Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that
commit b028dfc085
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Tue Jan 4 10:29:08 2022 +0000
Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)
* Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork
* Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers
* Fix consumer component name in federation API
* Add comment explaining where streams are defined
* Tweaks to roomserver input with comments
* Finish that sentence that I apparently forgot to finish in INSTALL.md
* Bump version number of config to 2
* Add comments around asynchronous sends to roomserver in processEventWithMissingState
* More useful error message when the config version does not match
* Set version in generate-config
* Fix version in config.Defaults
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Initial federation sender -> federation API refactoring
* Move base into own package, avoids import cycle
* Fix build errors
* Fix tests
* Add signing key server tables
* Try to fold signing key server into federation API
* Fix dendritejs builds
* Update embedded interfaces
* Fix panic, fix lint error
* Update configs, docker
* Rename some things
* Reuse same keyring on the implementing side
* Fix federation tests, `NewBaseDendrite` can accept freeform options
* Fix build
* Update create_db, configs
* Name tables back
* Don't rename federationsender consumer for now
* Cross-signing groundwork
* Update to matrix-org/gomatrixserverlib#274
* Fix gobind builds, which stops unit tests in CI from yelling
* Some changes from review comments
* Fix build by passing in UIA
* Update to matrix-org/gomatrixserverlib@bec8d22
* Process master/self-signing keys from devices call
* nolint
* Enum-ify the key type in the database
* Process self-signing key too
* Fix sanity check in device list updater
* Fix check
* Fix sytest, hopefully
* Fix build
* Implement /_synapse/admin/v1/register
This is implemented identically to Synapse, so scripts which work
with Synapse should work with Dendrite.
```
Test 27 POST /_synapse/admin/v1/register with shared secret... OK
Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK
Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK
Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK
```
Sytest however has `implementation_specific => "synapse"` which stops these
tests from running.
* Add missing muxes to gobind
* Linting
This fixes the issue found in #1759 which broke due to go changes. The new command allows you to generate keys with docker, and drop them in the current working directory.
* Allow configuring DNS cache
* Update sample configs
* Fix build errors
* Fix time resolution
* Default 5m
* In seconds
* Use WithDNScache
* Correct field name
* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
Squashed commit of the following:
commit 4a61aa711473deece2adf415cfd65501dbca63b2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Jan 18 12:19:24 2021 +0000
Set back to matrixdotorg on published releases
commit 6d1ac53f2c0c9b30e1e70c0bb1559e1b8ec874a2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Jan 18 11:55:28 2021 +0000
Rename
commit 258999f7fb7b655b3a02a06a7ea05e66fb7740fb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Jan 18 11:52:26 2021 +0000
Refactor multi-stage builds
commit c7ab8e476939899571e7b5668860dec372b9b60f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Jan 18 11:13:19 2021 +0000
Let's try this again
* Initial work oon multipersonality binary
* Remove old binaries
* Monolith and polylith binaries
* Better logging
* dendrite-poly-multi
* Fix path
* Copyright notices etc
* Tweaks
* Update Docker, INSTALL.md
* Take first argument if flags package doesn't find any args
* Postgres 9.6 or later, fix some more Docker stuff
* Don't create unnecessary e2ekey DB
* Run go mod tidy
* Start Kafka connection for each component that needs one
* Fix roomserver unit tests
* Rename to naffkaInstance (@Kegsay review comment)
* Fix import cycle
* Rename serverkeyapi to signingkeyserver
We use "api" for public facing stuff and "server" for internal stuff.
As the server key API is internal only, we call it 'signing key server',
which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc)
* Convert docker/scripts to use signing-key-server
* Rename missed bits
* Use federation sender for backfill and getting missing events
* Fix internal URL paths
* Update go.mod/go.sum for matrix-org/gomatrixserverlib#218
* Add missing server implementations in HTTP interface
- New package `perform` which contains all `Perform` functions
- New package `helpers` which contains helper functions used by both
perform and query/input functions.
- Perform invite/leave have no idea how to `WriteOutputEvents` and this
is now returned from `PerformInvite` or `PerformLeave` respectively.
Still to do:
- RSAPI is fed into the inviter/joiner/leaver - this introduces circular
logic so will need to be removed.
- Put query operations in a `query` package.
- Put input operations (and output) in an `input` package.
- Factor out helper functions as much as possible, possibly rejigging the
storage layer in the process.
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Add QueryDeviceMessages to serve up device keys and stream IDs
* Consume key change events in fedsender
Don't yet send them to destinations as we haven't worked them out yet
* Send device list updates to all required servers
* Glue it all together
* Recheck device lists when join/leave events come in
* Add PerformDeviceDeletion
* Notify clients when devices are deleted
* Unbreak things
* Remove debug logging
* Produce kafka events when keys are added
* Consume key changes in syncapi with TODO markers for handling them and catching up
* unbreak tests
* Linting
* Initial persistence of blacklists
* Move statistics folder
* Make MaxFederationRetries configurable
* Set lower failure thresholds for Yggdrasil demos
* Still write events into database for blacklisted hosts (they can be tidied up later)
* Review comments
Squashed commit of the following:
commit 86c2388e13ffdbabdd50cea205652dccc40e1860
Merge: b0a3ee6c f5e7e751
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:47:10 2020 +0100
Merge branch 'master' into neilalexander/yggbarequic
commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:42:22 2020 +0100
Add support for broadcasting wake-up EDUs to known hosts
commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:42:10 2020 +0100
Bare QUIC demo working
commit d3939b3d6568cf4262c0391486a5203873b68bfc
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jul 15 11:42:43 2020 +0100
Support bare Yggdrasil sessions with encrypted QUIC
* Initial QUIC work
* Update Yggdrasil demo
* Make sure that the federation sender knows how many pending events are in the database when the worker starts
* QUIC tunables
* pprof
* Don't spin
* Set build info for Yggdrasil
* Per-user-per-device sync streams
* Tweaks
* Tweaks
* Pass full device into CompleteSync
* Set user IDs and device IDs properly in tests
* Add new test, fix TestNewEventAndWasPreviouslyJoinedToRoom
* nolint a function that is not used yet
* Add test for waking up single device
* Hopefully unstick test
* Try to ensure that TestCorrectStreamWakeup doesn't block forever
* Update tests
* Server key API (works for monolith but not for polylith yet)
* Re-enable caching on server key API component
* Groundwork for HTTP APIs for server key API
* Hopefully implement HTTP for server key API
* Simplify public key request marshalling from map keys
* Update gomatrixserverlib
* go mod tidy
* Common -> internal
* remove keyring.go
* Update Docker Hub for server key API
* YAML is funny about indentation
* Wire in new server key API into hybrid monolith mode
* Create maps
* Route server key API endpoints on internal API mux
* Fix server key API URLs
* Add fetcher behaviour into server key API implementation
* Return error if we failed to fetch some keys
* Return results anyway
* Move things about a bit
* Remove unused code
* Fix comments, don't use federation sender URL in polylith mode
* Add server_key_api to sample config
* Review comments
* HTTP API to cache keys that have been requested
* Overwrite server_key_api listen in monolith hybrid mode
