0
0
Fork 0
mirror of https://github.com/matrix-construct/construct synced 2025-02-16 08:40:11 +01:00

Add troubleshooting section for DNS timeouts

Jason Volk 2022-08-10 10:44:07 -07:00
parent aad9ed7596
commit 5582ab614e

@ -4,10 +4,10 @@
- [Recovering from broken configurations](#Recovering-from-broken-configurations)
- [Recovering from database corruption](#Recovering-from-database-corruption)
- [Trouble with reverse proxies and middlewares](#Trouble-with-reverse-proxies-and-middlewares)
- [Timeouts while resolving domain names](#Timeouts-while-resolving-domain-names)
***
##### Useful program options
### Useful program options
Start the daemon with one or more of the following program options to make it
easier to troubleshoot and perform maintenance:
@ -32,7 +32,7 @@ of `-nolisten -wa -console` options described below.
- `-console` convenience to immediately drop to the adminstrator console
after startup.
##### Recovering from broken configurations
### Recovering from broken configurations
If your server ever fails to start from an errant conf item: you can override
any item using an environmental variable before starting the program. To do
@ -47,7 +47,7 @@ will not be written back to the database unless they are explicitly set by
the user in the console.
##### Recovering from database corruption
### Recovering from database corruption
In very rare cases after a hard crash the journal cannot completely restore
data before the crash. Due to the design of rocksdb and the way we apply it
@ -70,7 +70,7 @@ as `-single -ro` may be advisable for some salvage techniques.
- 🔴 **recover** - Salvage mode; expert use only. Data may be lost.
- 🔴 **tolerate** - Salvage mode; expert use only.
##### Trouble with reverse proxies and middlewares
### Trouble with reverse proxies and middlewares
Construct is designed to be capable internet service software and should
perform best when directly interfacing with remote parties. Nevertheless,
@ -96,3 +96,14 @@ reverse-proxy.
3. Ensure the reverse-proxy is not rewriting the `Host:` header which is sent to Construct. The header must appear exactly as sent by remote clients.
4. Ensure the reverse-proxy is not setting `Connection: close` when communicating to Construct. The ideal middleware is configured to maintain a pool of persistent connections and pipeline requests. As a hint based on Construct's default settings at the time of this writing, the optimal connection count from the middleware is 64, and up to 128.
### Timeouts while resolving domain names
Due to the abnormal loads of Matrix, Construct implements custom DNS resolution directly over UDP. Construct does not use `127.0.0.1` or any locally provided DNS servers by default after nearly all users reported issues which required them to reconfigure or upgrade their service. To ship the least-broken solution by default, Construct is pre-configured with an array of public servers to query in a load-balanced round-robin. To view the DNS configuration in its entirety use the command: `conf ircd.net.dns`.
1. Reduce the rate-limits to slow down queries made to the servers. This can be done with `conf ircd.net.dns.resolver.send_rate` which is a millisecond value to wait between requests; higher is _slower_: `conf set ircd.net.dns.resolver.send_rate 300`
2. The `conf ircd.net.dns.resolver.send_burst` can be tweaked in conjunction with the `conf ircd.net.dns.resolver.send_rate` to more effectively shape the load as tolerated by the remote server's rate-limiting scheme. The burst is important to keep requests in flight to utilize the array while minimizing local delays for a lot of resolutions. If necessary, try setting a lower value, or `1` to never exceed the `send_rate` with any burst.
3. Add or replace the default configured array of servers. The configuration at `conf ircd.net.dns.resolver.servers` is a string of IP addresses separated by spaces. It is better to add more servers than to replace the existing, but it is worse to add a server which is configured very differently from the others. The default servers were chosen because they have reasonably high rates and are consistent among themselves; they may not be the best choice for all users, especially in Europe and Asia.
> 👉 Administrators are tempted to simply replace the array with `127.0.0.1` to use their own high-performance service: this is okay, but the `ircd.net.dns.resolver.send_rate` may need to be configured significantly faster (lower) than the default if only one server is configured rather than the default six.