Simplify adding new Redis instance

The main sticking point was `config/README.md`, which went into a lot of
detail about each instance. It was quite easy to make a copy and paste
error, as some instances had default URLs and others didn't, some had
fallback instances and others didn't, etc.

This changes a number of things:

1. `config/README.md` now has a table for the Redis instances with a
   single list of instructions that apply to all instances.
2. There are now specs to ensure that each Redis instance has a default
   URL set when no Redis config at all is present.
3. Instances that use a fallback instance for config will automatically
   use the default URL of their fallback instance. They should not
   define their own default URL.
4. The `.gitignore` file does not need to be updated for each new Redis
   instance.
5. The example config files are now gone as `request.yml.example` should
   be sufficient.
This commit is contained in:
Sean McGivern 2021-09-22 10:38:13 +00:00
parent 2f89dcdb69
commit b3c5c99600
No known key found for this signature in database
GPG key ID: DD7CC910493922AC
13 changed files with 69 additions and 405 deletions

6
.gitignore vendored
View file

@ -40,11 +40,7 @@ eslint-report.html
/config/initializers/smtp_settings.rb
/config/initializers/relative_url.rb
/config/resque.yml
/config/redis.cache.yml
/config/redis.queues.yml
/config/redis.shared_state.yml
/config/redis.trace_chunks.yml
/config/redis.rate_limiting.yml
/config/redis.*.yml
/config/unicorn.rb
/config/puma.rb
/config/secrets.yml

View file

@ -1,13 +1,13 @@
# Configuration files Documentation
Note that most configuration files (`config/*.*`) committed into
[gitlab-ce](https://gitlab.com/gitlab-org/gitlab-foss) **will not be used** for
[gitlab-foss](https://gitlab.com/gitlab-org/gitlab-foss) **will not be used** for
[omnibus-gitlab](https://gitlab.com/gitlab-org/omnibus-gitlab). Configuration
files committed into gitlab-ce are only used for development.
files committed into gitlab-foss are only used for development.
## gitlab.yml
You can find most of GitLab configuration settings here.
You can find most of the GitLab configuration settings here.
## mail_room.yml
@ -21,7 +21,7 @@ This file is called `resque.yml` for historical reasons. We are **NOT**
using Resque at the moment. It is used to specify Redis configuration
values when a single database instance of Redis is desired.
# Advanced Redis configuration files
## Advanced Redis configuration files
In more advanced configurations of Redis key-value storage, it is desirable
to separate the keys by lifecycle and intended use to ease provisioning and
@ -40,7 +40,7 @@ If desired, the routing URL provided by these settings can be used with:
2. TCP port number for each Redis instance desired
3. `database number` for each Redis instance desired
## Example URL attribute formats for GitLab Redis `.yml` configuration files
### Example URL attribute formats for GitLab Redis `.yml` configuration files
* Unix Socket, default Redis database (0)
* `url: unix:/path/to/redis.sock`
* `url: unix:/path/to/redis.sock?db=`
@ -52,170 +52,37 @@ If desired, the routing URL provided by these settings can be used with:
* TCP Socket for Redis on remote host `myserver`, port 6379, database 33
* `url: redis://:mynewpassword@myserver:6379/33`
## redis.cache.yml
## Available configuration files
If configured, `redis.cache.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for `Rails.cache` and other volatile non-persistent data which enhances
the performance of GitLab.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_CACHE_CONFIG_FILE` which provides
an alternate location for configuration settings.
The Redis instances that can be configured are described in the table below. The
order of precedence for configuration is described below, where `$NAME` and
`$FALLBACK_NAME` are the upper-cased instance names from the table, and `$name`
and `$fallback_name` are the lower-cased versions:
The order of precedence for the URL used to connect to the Redis instance
used for `cache` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
2. URL from `redis.cache.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6380`
1. The configuration file pointed to by the `GITLAB_REDIS_$NAME_CONFIG_FILE`
environment variable.
1. The configuration file `redis.$name.yml`.
1. **If a fallback instance is available**, the configuration file
`redis.$fallback_name.yml`.
1. The configuration file pointed to by the `GITLAB_REDIS_CONFIG_FILE`
environment variable.
1. The configuration file `resque.yml`.
The order of precedence for all other configuration settings for `cache`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
2. the configuration file `redis.cache.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. the configuration file `resque.yml`
An example configuration file for Redis is in this directory under the name
`resque.yml.example`.
## redis.queues.yml
| Name | Fallback instance | Purpose |
| --- | --- | --- |
| `cache` | | Volatile non-persistent data |
| `queues` | | Background job processing queues |
| `shared_state` | | Persistent application state |
| `trace_chunks` | `shared_state` | [CI trace chunks](https://docs.gitlab.com/ee/administration/job_logs.html#incremental-logging-architecture) |
| `rate_limiting` | `cache` | [Rate limiting](https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html) state |
If configured, `redis.queues.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for clients of `::Gitlab::Redis::Queues`.
These queues are intended to be the foundation
of reliable inter-process communication between modules, whether on the same
host node, or within a cluster. The primary clients of the queues are
SideKiq, Mailroom, CI Runner, Workhorse, and push services. Settings here can
be overridden by the environment variable
`GITLAB_REDIS_QUEUES_CONFIG_FILE` which provides an alternate location for
configuration settings.
If no configuration is found, or no URL is found in the configuration
file, the default URL used is:
The order of precedence for the URL used to connect to the Redis instance
used for `queues` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_QUEUES_CONFIG_FILE` environment variable
2. URL from `redis.queues.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6381`
The order of precedence for all other configuration settings for `queues`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_QUEUES_CONFIG_FILE` environment variable
2. the configuration file `redis.queues.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. the configuration file `resque.yml`
## redis.shared_state.yml
If configured, `redis.shared_state.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for clients of `::Gitlab::Redis::SharedState` such as session state,
and rate limiting.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `shared_state` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
2. URL from `redis.shared_state.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6382`
The order of precedence for all other configuration settings for `shared_state`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
2. the configuration file `redis.shared_state.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. the configuration file `resque.yml`
## redis.trace_chunks.yml
If configured, `redis.trace_chunks.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for clients of `::Gitlab::Redis::TraceChunks` which stores CI trace chunks.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_TRACE_CHUNKS_CONFIG_FILE` which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `trace_chunks` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_TRACE_CHUNKS_CONFIG_FILE` environment variable
2. URL from `redis.trace_chunks.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6383`
The order of precedence for all other configuration settings for `trace_chunks`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_TRACE_CHUNKS_CONFIG_FILE` environment variable
2. the configuration file `redis.trace_chunks.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
4. the configuration file `redis.shared_state.yml`
5. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
6. the configuration file `resque.yml`
## redis.rate_limiting.yml
If configured, `redis.rate_limiting.yml` overrides the `resque.yml` settings to
configure the Redis database instance used for clients of
`::Gitlab::Redis::RateLimiting` which stores
[rate limiting](https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html)
state.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_RATE_LIMITING_CONFIG_FILE` which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `rate_limiting` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_RATE_LIMITING_CONFIG_FILE` environment variable
2. URL from `redis.rate_limiting.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
4. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
5. URL from `resque.yml`
6. `redis://localhost:6384`
The order of precedence for all other configuration settings for `rate_limiting`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_RATE_LIMITING_CONFIG_FILE` environment variable
2. the configuration file `redis.rate_limiting.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
4. the configuration file `redis.cache.yml`
5. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
6. the configuration file `resque.yml`
1. `redis://localhost:6380` for `cache`.
1. `redis://localhost:6381` for `queues`.
1. `redis://localhost:6382` for `shared_state`.
1. The URL from the fallback instance for all other instances.

View file

@ -1,38 +0,0 @@
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/10
#
# url: redis://localhost:6380
# sentinels:
# -
# host: localhost
# port: 26380 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26380 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/10
#
# url: redis://localhost:6380
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.cache.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6380
# sentinels:
# -
# host: replica1
# port: 26380 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26380 # point to sentinel, not to redis port

View file

@ -1,38 +0,0 @@
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/11
#
# url: redis://localhost:6381
# sentinels:
# -
# host: localhost
# port: 26381 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26381 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/11
#
# url: redis://localhost:6381
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.queues.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6381
# sentinels:
# -
# host: replica1
# port: 26381 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26381 # point to sentinel, not to redis port

View file

@ -1,38 +0,0 @@
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/4
#
# url: redis://localhost:6382
# sentinels:
# -
# host: localhost
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/14
#
# url: redis://localhost:6382
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.rate_limiting.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6382
# sentinels:
# -
# host: replica1
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port

View file

@ -1,38 +0,0 @@
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/12
#
# url: redis://localhost:6382
# sentinels:
# -
# host: localhost
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/12
#
# url: redis://localhost:6382
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.shared_state.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6382
# sentinels:
# -
# host: replica1
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port

View file

@ -1,38 +0,0 @@
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/13
#
# url: redis://localhost:6382
# sentinels:
# -
# host: localhost
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/13
#
# url: redis://localhost:6382
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.trace_chunks.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6382
# sentinels:
# -
# host: replica1
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port

View file

@ -4,14 +4,6 @@ module Gitlab
module Redis
class Cache < ::Gitlab::Redis::Wrapper
CACHE_NAMESPACE = 'cache:gitlab'
private
def raw_config_hash
config = super
config[:url] = 'redis://localhost:6380' if config[:url].blank?
config
end
end
end
end

View file

@ -2,21 +2,12 @@
# We need this require for MailRoom
require_relative 'wrapper' unless defined?(::Gitlab::Redis::Wrapper)
require 'active_support/core_ext/object/blank'
module Gitlab
module Redis
class Queues < ::Gitlab::Redis::Wrapper
SIDEKIQ_NAMESPACE = 'resque:gitlab'
MAILROOM_NAMESPACE = 'mail_room:gitlab'
private
def raw_config_hash
config = super
config[:url] = 'redis://localhost:6381' if config[:url].blank?
config
end
end
end
end

View file

@ -7,14 +7,6 @@ class SharedState < ::Gitlab::Redis::Wrapper
USER_SESSIONS_NAMESPACE = 'session:user:gitlab'
USER_SESSIONS_LOOKUP_NAMESPACE = 'session:lookup:user:gitlab'
IP_SESSIONS_LOOKUP_NAMESPACE = 'session:lookup:ip:gitlab2'
private
def raw_config_hash
config = super
config[:url] = 'redis://localhost:6382' if config[:url].blank?
config
end
end
end
end

View file

@ -6,6 +6,7 @@
# Rails.
require 'active_support/core_ext/hash/keys'
require 'active_support/core_ext/module/delegation'
require 'active_support/core_ext/object/blank'
require 'active_support/core_ext/string/inflections'
# Explicitly load Redis::Store::Factory so we can read Redis configuration in
@ -150,11 +151,35 @@ def redis_store_options
def raw_config_hash
config_data = fetch_config
if config_data
config_data.is_a?(String) ? { url: config_data } : config_data.deep_symbolize_keys
else
{ url: '' }
config_hash =
if config_data
config_data.is_a?(String) ? { url: config_data } : config_data.deep_symbolize_keys
else
{ url: '' }
end
if config_hash[:url].blank?
config_hash[:url] = legacy_fallback_urls[self.class.store_name] || legacy_fallback_urls[self.class.config_fallback.store_name]
end
config_hash
end
# These URLs were defined for cache, queues, and shared_state in
# code. They are used only when no config file exists at all for a
# given instance. The configuration does not seem particularly
# useful - it uses different ports on localhost - but we cannot
# confidently delete it as we don't know if any instances rely on
# this.
#
# DO NOT ADD new instances here. All new instances should define a
# `.config_fallback`, which will then be used to look up this URL.
def legacy_fallback_urls
{
'Cache' => 'redis://localhost:6380',
'Queues' => 'redis://localhost:6381',
'SharedState' => 'redis://localhost:6382'
}
end
def fetch_config

View file

@ -38,21 +38,6 @@ sed -i 's|url:.*$|url: redis://redis:6379|g' config/cable.yml
cp config/resque.yml.example config/resque.yml
sed -i 's|url:.*$|url: redis://redis:6379|g' config/resque.yml
cp config/redis.cache.yml.example config/redis.cache.yml
sed -i 's|url:.*$|url: redis://redis:6379/10|g' config/redis.cache.yml
cp config/redis.queues.yml.example config/redis.queues.yml
sed -i 's|url:.*$|url: redis://redis:6379/11|g' config/redis.queues.yml
cp config/redis.shared_state.yml.example config/redis.shared_state.yml
sed -i 's|url:.*$|url: redis://redis:6379/12|g' config/redis.shared_state.yml
cp config/redis.trace_chunks.yml.example config/redis.trace_chunks.yml
sed -i 's|url:.*$|url: redis://redis:6379/13|g' config/redis.trace_chunks.yml
cp config/redis.rate_limiting.yml.example config/redis.rate_limiting.yml
sed -i 's|url:.*$|url: redis://redis:6379/14|g' config/redis.rate_limiting.yml
if [ "$SETUP_DB" != "false" ]; then
setup_db
elif getent hosts postgres; then

View file

@ -327,6 +327,12 @@
expect(subject.send(:fetch_config)).to eq false
end
it 'has a value for the legacy default URL' do
allow(subject).to receive(:fetch_config) { false }
expect(subject.send(:raw_config_hash)).to include(url: a_string_matching(%r{\Aredis://localhost:638[012]\Z}))
end
end
def clear_raw_config