.. | ||
disk-tuning.sh | ||
README.md | ||
sysctl.sh |
Kernel Tuning for MinIO Production Deployment on Linux Servers
![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)
Tuning Network Parameters
Following network parameter settings can help ensure optimal MinIO server performance on production workloads.
tcp_fin_timeout
: A socket left in memory takes approximately 1.5Kb of memory. It makes sense to close the unused sockets preemptively to ensure no memory leakage. This way, even if a peer doesn't close the socket due to some reason, the system itself closes it after a timeout.tcp_fin_timeout
variable defines this timeout and tells kernel how long to keep sockets in the state FIN-WAIT-2. We recommend setting it to 30. You can set it as shown below
sysctl -w net.ipv4.tcp_fin_timeout=30
tcp_keepalive_probes
: This variable defines the number of unacknowledged probes to be sent before considering a connection dead. You can set it as shown below
sysctl -w net.ipv4.tcp_keepalive_probes=5
wmem_max
: This parameter sets the max OS send buffer size for all types of connections.
sysctl -w net.core.wmem_max=540000
rmem_max
: This parameter sets the max OS receive buffer size for all types of connections.
sysctl -w net.core.rmem_max=540000
Tuning Virtual Memory
Recommended virtual memory settings are as follows.
swappiness
: This parameter controls the relative weight given to swapping out runtime memory, as opposed to dropping pages from the system page cache. It takes values from 0 to 100, both inclusive. We recommend setting it to 10.
sysctl -w vm.swappiness=1
dirty_background_ratio
: This is the percentage of system memory that can be filled withdirty
pages, i.e. memory pages that still need to be written to disk. We recommend writing the data to the disk as soon as possible. To do this, set thedirty_background_ratio
to 1.
sysctl -w vm.dirty_background_ratio=1
dirty_ratio
: This defines is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk.
sysctl -w vm.dirty_ratio=5
Transparent Hugepage Support
: This is a Linux kernel feature intended to improve performance by making more efficient use of processor’s memory-mapping hardware. But this may cause problems for non-optimized applications. As most Linux distributions set it toenabled=always
by default, we recommend changing this toenabled=madvise
. This will allow applications optimized for transparent hugepages to obtain the performance benefits, while preventing the associated problems otherwise.
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
Also, set transparent_hugepage=madvise
on your kernel command line (e.g. in /etc/default/grub) to persistently set this value.
All these system level tunings are conveniently packaged in shell script. Please review the shell script for our recommendations.
Tuning Scheduler
Proper scheduler configuration makes sure MinIO process gets adequate CPU time. Here are the recommended scheduler settings
sched_min_granularity_ns
: This parameter decides the minimum time a task will be be allowed to run on CPU before being pre-empted out. We recommend setting it to 10ms.
sysctl -w kernel.sched_min_granularity_ns=10000000
sched_wakeup_granularity_ns
: Lowering this parameter improves wake-up latency and throughput for latency critical tasks, particularly when a short duty cycle load component must compete with CPU bound components.
sysctl -w kernel.sched_wakeup_granularity_ns=15000000
Tuning Disks
The recommendations for disk tuning are conveniently packaged in a well commented shell script. Please review the shell script for our recommendations.