0
0
Fork 0
mirror of https://github.com/matrix-org/dendrite synced 2024-10-31 21:19:04 +01:00
Commit graph

150 commits

Author SHA1 Message Date
Neil Alexander
ba66b5a3b9
Allow multiple static peers in Pinecone iOS/Android demos 2022-10-03 14:43:38 +01:00
Neil Alexander
34451d21b8
P2P demo tweaks 2022-10-03 14:35:10 +01:00
devonh
90f1985bf3
Add network interface registration to pinecone demo (#2750)
### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

No additional tests have been added as this simply extends the API for
the embedded pinecone demo.

Signed-off-by: `Devon Hudson <devonhudson@librem.one>`
2022-09-29 16:05:16 +00:00
Neil Alexander
6f3dbee5f4
Fix Go 1.18 2022-09-29 16:08:00 +01:00
Neil Alexander
68d6eb0a6f
P2P demo fixes 2022-09-29 13:06:55 +01:00
Neil Alexander
d531202b0e
Build Docker images using Go 1.19 (related to #2714) 2022-09-27 10:52:03 +01:00
Dov Alperin
b5bfff47d2
Use /usr/bin/env bash in shebangs to make them universal (#2735)
Some systems (like nixos) don't have bash living at `/bin/bash` so using
`/usr/bin/env bash` we can make these scripts universal.

### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

* [X] I have added added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Dov Alperin <git@dov.dev>`

Signed-off-by: `Dov Alperin <git@dov.dev>`
2022-09-27 09:42:08 +01:00
networkException
40fec70d13
Add pinecone demo container image (#2710)
This pull request adds the configuration and CI steps to build and
publish a container wrapping the `dendrite-demo-pinecone` command as
well as fixes a sentence structure issue in the pull request template.

As this does not touch any go source code no tests have been added

### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

* [x] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: networkException <git@nwex.de> (by private sign-off)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 09:39:39 +01:00
Neil Alexander
0ea948c705
Fix Pinecone demo build errors after Pinecone update 2022-09-14 14:26:24 +01:00
Neil Alexander
3a9dde28fd
P2P demo tweaks 2022-09-12 10:19:02 +01:00
Neil Alexander
fea869b41f
Update P2P demos 2022-09-01 17:12:27 +01:00
Neil Alexander
51d229b025
Configuration tweaks (#2567)
This makes the following changes:

* The various `Defaults` functions are now responsible for setting sane defaults if `generate` is specified, rather than hiding them in `generate-config`
* Some configuration options have been marked as `omitempty` so that they don't appear in generated configs unnecessarily (monolith-specific vs. polylith-specific options)
* A new option `-polylith` has been added to `generate-config` to create a config that makes sense for polylith deployments (i.e. including the internal/external API listeners and per-component database sections)
* A new option `-normalise` has been added to `generate-config` to take an existing file and add any missing options and/or defaults
2022-09-01 14:15:41 +01:00
Neil Alexander
ad6b902b84
Refactor appservices component (#2687)
This PR refactors the app services component. It makes the following changes:

* Each appservice now gets its own NATS JetStream consumer
* The appservice database is now removed entirely, since we just use JetStream as a data source instead
* The entire component is now much simpler and we deleted lots of lines of code 💅

The result is that it should be much lighter and hopefully much more performant.
2022-09-01 09:20:40 +01:00
Till
95a509757a
Complement QoL changes (#2663)
This PR does the following:
- adds a `keysize` parameter to `generate-keys`, so we can use lower sized keys when running in CI
- updates the Complement docker files to use BuildKit (requires Docker >18.09)
- uses `exec` when executing `dendrite-monotlith-server`, making it PID 1 inside docker, which results in Dendrite actually receiving the `SIGTERM` signal send by Docker. (Making it faster when running tests with Complement, as we don't take 10 seconds to timeout)
2022-08-23 13:10:29 +02:00
Brian Meek
de78eab63a
Add race testing to tests, and fix a few small race conditions in the tests (#2587)
* Add race testing to tests, and fix a few small race conditions in the tests

* Enable run-sytest on MacOS

* Remove deadlock detecting mutex, per code review feedback

* Remove autoformatting related changes and a closure that is not needed

* Adjust to importing nats client as 'natsclient'

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Clarify the use of gooseMutex to proect goose internal state

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Remove no longer needed mutex for guarding goose

Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-08-05 09:19:33 +01:00
Neil Alexander
497ab4e1b7
Remove goose from Dockerfiles 2022-07-25 14:46:35 +01:00
Neil Alexander
3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Neil Alexander
eb8dc50a97
Update Pinecone to matrix-org/pinecone@1ce778f 2022-07-08 14:54:08 +01:00
Neil Alexander
aafb7bf120
Avoid panicking unnecessarily when shutting down the gobind P2P demos (#2520) 2022-06-07 10:46:21 +01:00
Neil Alexander
b40b548432
The Pinecone gobind demo must listen on localhost for baseURL to be correct 2022-05-13 12:06:47 +01:00
Neil Alexander
fc670f03a2
Separate sample configs for monolith and polylith (#2456)
* Update sample configs

* Update references

* Remove sections that are dead in the monolith sample
2022-05-12 12:05:55 +01:00
Neil Alexander
dd061a172e
Tidy up AddPublicRoutes (#2412)
* Simplify federation API `AddPublicRoutes`

* Simplify client API `AddPublicRoutes`

* Simplify media API `AddPublicRoutes`

* Simplify sync API `AddPublicRoutes`

* Simplify `AddAllPublicRoutes`
2022-05-03 17:17:02 +01:00
Neil Alexander
4ad5f9c982
Global database connection pool (for monolith mode) (#2411)
* Allow monolith components to share a single database pool

* Don't yell about missing connection strings

* Rename field

* Setup tweaks

* Fix panic

* Improve configuration checks

* Update config

* Fix lint errors

* Update comments
2022-05-03 16:35:06 +01:00
Brian Meek
1e083794ef
Update golangci-lint, how it's installed, and added to the PATH (#2403)
* switch to dendrite server

* minor refactor to merge store code

* Fix issue where m.room.name is being filtered by the dendrite server

* refresh dendrite main

* refresh dendrite main

* missing merges from the last dendrite refresh

* revert unwanted changes in dendrite.yaml

* Update golangci-lint, how it's installed, and added to the PATH

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: tak-slashtalk <64229756+tak-slashtalk@users.noreply.github.com>
2022-04-29 08:55:35 +01:00
Till
26a1512808
Add restrictions for open registration (#2402)
* Add restrications for open registration

* Make enable open registration a parameter

* Enable registration for CI

* Update error message

* Shuffle things around a bit

* Add a warning at every startup just to be extra annoying

* Ignore shared secret when warning about open registration, since it's not strictly required when it is set if registration is otherwise enabled

* Make CI happy?

* Add missing parameter; try new parameter in upgrade-test

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-29 09:31:11 +02:00
Neil Alexander
c6ea2c9ff2
Add /_dendrite/admin/evacuateRoom/{roomID} (#2401)
* Add new endpoint to allow admins to evacuate the local server from the room

* Guard endpoint

* Use right prefix

* Auth API

* More useful return error rather than a panic

* More useful return value again

* Update the path

* Try using inputer instead

* oh provide the config

* Try that again

* Return affected user IDs

* Don't create so many forward extremities

* Add missing `Path` to name

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2022-04-28 16:02:30 +01:00
0x1a8510f2
8d69e2f0b8
Use Go 1.18 to build Docker images (#2391)
Go 1.18 has now been released for a while and the CI already tests Dendrite with Go 1.18
so there should be no issues. Go 1.18 brings some performance improvements for ARM via
the register calling convention so it makes sense to switch to it.
2022-04-27 20:19:46 +01:00
Neil Alexander
6907e42167
Use connection manager in Pinecone demos 2022-04-08 13:55:46 +01:00
Till
e5e3350ce1
Add presence module V2 (#2312)
* Syncapi presence

* Clientapi http presence handler

* Why is this here?

* Missing files

* FederationAPI presence implementation

* Add new presence stream

* Pinecone update

* Pinecone update

* Add passing tests

* Make linter happy

* Add presence producer

* Add presence config option

* Set user to unavailable after x minutes

* Only set currently_active if online
Avoid unneeded presence updates when syncing

* Tweaks

* Query devices for last_active_ts
Fixes & tweaks

* Export SharedUsers/SharedUsers

* Presence stream in MemoryStorage

* Remove status_msg_nil

* Fix sytest crashes

* Make presence types const and use stringer for it

* Change options to allow inbound/outbound presence

* Fix option & typo

* Update configs

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-06 13:11:19 +02:00
Sebastian Spaeth
1783496423
remove obsolete config entry "federation_certificates" (#2318)
* Remove all federation_certificates occurencs

This configuration value has not been used since 2019 apparently, and indeed
it is never really used in the code base. So remove all traces of it from
the various configuration files.

Also remove the unused variable FederationCertificatePaths

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>

* setup/config/config_test.go: remove federation_sender config snippet

The federation_sender: section was folded into the federation_api some
time ago, and this seems to be the only leftover in the code base.
So remove it.
2022-04-04 17:31:41 +01:00
Neil Alexander
cee12a7ab0
Enhanced calm at startup of Pinecone demos 2022-04-04 15:35:47 +01:00
Neil Alexander
c69159bda7
Update to matrix-org/pinecone@e526fa8 2022-04-04 15:16:02 +01:00
Neil Alexander
8213b2ba30
Update Pinecone P2P demo 2022-03-30 15:01:22 +01:00
S7evinK
49dc49b232
Remove eduserver (#2306)
* Move receipt sending to own JetStream producer

* Move SendToDevice to producer

* Remove most parts of the EDU server

* Fix SendToDevice & copyrights

* Move structs, cleanup EDU Server traces

* Use HeadersOnly subscription

* Missing file

* Fix linter issues

* Move consumers to own files

* Rename durable consumer; Consumer cleanup

* Docs/config cleanup
2022-03-29 14:14:35 +02:00
Neil Alexander
7972915806
User directory for nearby Pinecone peers (P2P demo) (#2311)
* User directory for nearby Pinecone peers

* Fix mux routing

* Use config to determine which server notices user to exclude
2022-03-28 16:25:26 +01:00
Neil Alexander
8099bcbc8b
P2P demo tweaks 2022-03-28 11:44:25 +01:00
Neil Alexander
34b9c8c670
Ensure Dendrite has stopped in Pinecone demo Stop() 2022-03-28 10:55:53 +01:00
Neil Alexander
1b389abbfd
Upload Docker images for releases to both Docker Hub and GitHub Container Registry (#2299)
* Upload Docker images for releases to both Docker Hub and GitHub Container Registry

* Build current images on `:main` tag

* Use Dendrite flow to trigger Docker flow for `:main`

* Tweaks

* Fix references to `env.GHCR_NAMESPACE`
2022-03-24 15:22:06 +00:00
S7evinK
01f863d248
Move CI to Github Actions (#2297)
* Initial test

* Move CI to GHA

* Naming

* Always report all linter issues

* Remove if true

* Test complement in different variations

* Try again

* Move Complement back after initial tests and readd timeout
Make linting fail further checks
Remove CodeQL

* Update and rename tests.yml to dendrite.yml

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-24 12:52:51 +01:00
Neil Alexander
c54d88aecf
Update to matrix-org/pinecone@8687b0d 2022-03-18 15:32:14 +00:00
Neil Alexander
d78e0a33e0
Remove references to userapi_devices (fixes #2259) 2022-03-09 16:38:50 +00:00
Dan
f05ce478f0
Implement Push Notifications (#1842)
* Add Pushserver component with Pushers API

Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>

* Wire Pushserver component

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add PushGatewayClient.

The full event format is required for Sytest.

* Add a pushrules module.

* Change user API account creation to use the new pushrules module's defaults.

Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.

* Add push rules query/put API in Pushserver.

This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.

Forks clientapi/producers/syncapi.go to pushserver/ for later extension.

* Add clientapi routes for push rules to Pushserver.

A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.

* Output rooms.join.unread_notifications in /sync.

This is the read-side. Pushserver will be the write-side.

* Implement pushserver/storage for notifications.

* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.

* Use one goroutine per user to avoid locking up the entire server for
  one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
  multiple devices into one push. Matches Synapse. Either we change
  Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
  the consumer.

* Implement read receipt consumers in Pushserver.

Supports m.read and m.fully_read receipts.

* Add clientapi route for /unstable/notifications.

* Rename to UpsertPusher for clarity and handle pusher update

* Fix linter errors

* Ignore body.Close() error check

* Fix push server internal http wiring

* Add 40 newly passing 61push tests to whitelist

* Add next 12 newly passing 61push tests to whitelist

* Send notification data before notifying users in EDU server consumer

* NATS JetStream

* Goodbye sarama

* Fix `NewStreamTokenFromString`

* Consume on the correct topic for the roomserver

* Don't panic, NAK instead

* Move push notifications into the User API

* Don't set null values since that apparently causes Element upsetti

* Also set omitempty on conditions

* Fix bug so that we don't override the push rules unnecessarily

* Tweak defaults

* Update defaults

* More tweaks

* Move `/notifications` onto `r0`/`v3` mux

* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly

Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-03 11:40:53 +00:00
Neil Alexander
9bd5e414c9
Missing commit from #2186 2022-02-18 11:32:45 +00:00
Neil Alexander
153bfbbea5
Merge both user API databases into one (#2186)
* Merge user API databases into one

* Remove DeviceDatabase from config

* Fix tests

* Try that again

* Clean up keyserver device keys when the devices no longer exist in the user API

* Tweak ordering

* Fix UserExists flag, device check

* Allow including empty entries so we can clean them up

* Remove logging
2022-02-18 11:31:05 +00:00
kegsay
f92b048fec
Add host mount compatible Complement image (#2187) 2022-02-15 17:40:48 +00:00
kegsay
4e75ab9930
Add postgres complement support (#2177) 2022-02-11 12:35:47 +00:00
kegsay
f800cae6d2
Point to /complement/ca not /ca (#2172) 2022-02-10 18:12:11 +00:00
tommie
c36e4546c3
Support for m.login.token (#2014)
* Add GOPATH to PATH in find-lint.sh.

The user doesn't necessarily have it in PATH.

* Refactor LoginTypePassword and Type to support m.login.token and m.login.sso.

For login token:

* m.login.token will require deleting the token after completeAuth has
  generated an access token, so a cleanup function is returned by
  Type.Login.
* Allowing different login types will require parsing the /login body
  twice: first to extract the "type" and then the type-specific parsing.
  Thus, we will have to buffer the request JSON in /login, like
  UserInteractive already does.

For SSO:

* NewUserInteractive will have to also use GetAccountByLocalpart. It
  makes more sense to just pass a (narrowed-down) accountDB interface
  to it than adding more function pointers.

Code quality:

* Passing around (and down-casting) interface{} for login request types
  has drawbacks in terms of type-safety, and no inherent benefits. We
  always decode JSON anyway. Hence renaming to Type.LoginFromJSON. Code
  that directly uses LoginTypePassword with parsed data can still use
  Login.
* Removed a TODO for SSO. This is already tracked in #1297.
* httputil.UnmarshalJSON is useful because it returns a JSONResponse.

This change is intended to have no functional changes.

* Support login tokens in User API.

This adds full lifecycle functions for login tokens: create, query, delete.

* Support m.login.token in /login.

* Fixes for PR review.

* Set @matrix-org/dendrite-core as repository code owner

* Return event NID from `StoreEvent`, match PSQL vs SQLite behaviour, tweak backfill persistence (#2071)

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-10 10:27:26 +00:00
kegsay
9130156b13
Make the Complement Dockerfile use a fresh directory for runtime (#2168) 2022-02-10 09:37:46 +00:00
Neil Alexander
bb39149ff8
Fix DendriteJS dockerfile 2022-02-08 16:18:16 +00:00