From 3e95c19911b04dbc110ccff33002e998e19cffdd Mon Sep 17 00:00:00 2001 From: Tatu Wikman Date: Thu, 27 Apr 2023 13:25:44 +0300 Subject: [PATCH] Docs: Add Nginx loadbalancing example with sticky mxid for workers (#15411) * Docs: Add Nginx loadbalancing example with sticky mxid for workers Add example nginx configuration snippet that * does load balancing for workers * respects mxid part of the token * from both url parameter and auth header * and handles since parameter Thanks to @olmari for pushing me to write this and testing the configs Signed-off-by: Tatu Wikman * Add changelog entry Signed-off-by: Tatu Wikman * Update codeblock formatter Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> * Remove indirectly related nginx-config Signed-off-by: Sami Olmari * Proper definition of action how to target username for worker Signed-off-by: Sami Olmari * Change "nginx" to general "reverse proxy" as it's concept now. Signed-off-by: Sami Olmari * Wording in better English Co-authored-by: Tatu Wikman * rename changelog entry to have correct extension --------- Signed-off-by: Tatu Wikman Signed-off-by: Sami Olmari Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: Sami Olmari Co-authored-by: Sami Olmari --- changelog.d/15411.doc | 1 + docs/workers.md | 66 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 65 insertions(+), 2 deletions(-) create mode 100644 changelog.d/15411.doc diff --git a/changelog.d/15411.doc b/changelog.d/15411.doc new file mode 100644 index 000000000..c23a8df04 --- /dev/null +++ b/changelog.d/15411.doc @@ -0,0 +1 @@ +Docs: Add Nginx loadbalancing example with sticky mxid for workers. diff --git a/docs/workers.md b/docs/workers.md index 6192a46e0..765f03c26 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -325,8 +325,7 @@ load balancing can be done in different ways. For `/sync` and `/initialSync` requests it will be more efficient if all requests from a particular user are routed to a single instance. This can -be done e.g. in nginx via IP `hash $http_x_forwarded_for;` or via -`hash $http_authorization consistent;` which contains the users access token. +be done in reverse proxy by extracting username part from the users access token. Admins may additionally wish to separate out `/sync` requests that have a `since` query parameter from those that don't (and @@ -335,6 +334,69 @@ when a user logs in on a new device and can be *very* resource intensive, so isolating these requests will stop them from interfering with other users ongoing syncs. +Example `nginx` configuration snippet that handles the cases above. This is just an +example and probably requires some changes according to your particular setup: + +```nginx +# Choose sync worker based on the existence of "since" query parameter +map $arg_since $sync { + default synapse_sync; + '' synapse_initial_sync; +} + +# Extract username from access token passed as URL parameter +map $arg_access_token $accesstoken_from_urlparam { + # Defaults to just passing back the whole accesstoken + default $arg_access_token; + # Try to extract username part from accesstoken URL parameter + "~syt_(?.*?)_.*" $username; +} + +# Extract username from access token passed as authorization header +map $http_authorization $mxid_localpart { + # Defaults to just passing back the whole accesstoken + default $http_authorization; + # Try to extract username part from accesstoken header + "~Bearer syt_(?.*?)_.*" $username; + # if no authorization-header exist, try mapper for URL parameter "access_token" + "" $accesstoken_from_urlparam; +} + +upstream synapse_initial_sync { + # Use the username mapper result for hash key + hash $mxid_localpart consistent; + server 127.0.0.1:8016; + server 127.0.0.1:8036; +} + +upstream synapse_sync { + # Use the username mapper result for hash key + hash $mxid_localpart consistent; + server 127.0.0.1:8013; + server 127.0.0.1:8037; + server 127.0.0.1:8038; + server 127.0.0.1:8039; +} + +# Sync initial/normal +location ~ ^/_matrix/client/(r0|v3)/sync$ { + proxy_pass http://$sync; +} + +# Normal sync +location ~ ^/_matrix/client/(api/v1|r0|v3)/events$ { + proxy_pass http://synapse_sync; +} + +# Initial_sync +location ~ ^/_matrix/client/(api/v1|r0|v3)/initialSync$ { + proxy_pass http://synapse_initial_sync; +} +location ~ ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$ { + proxy_pass http://synapse_initial_sync; +} +``` + Federation and client requests can be balanced via simple round robin. The inbound federation transaction request `^/_matrix/federation/v1/send/`