GIT – Note: Also running these tests in 2-5 times will give you a better idea of where you stand.  The first run if you are using any caching at all may not be primed on the first run.

the packages:
# If you are using the CentALT repo, use -stable instead
# I’m just using the fedora/centos which is an older version for these tests.
yum install nginx memcached

Not to great and the errors and time outs had me concerned.  So taking a look at my default nginx configuration I noticed a bunch of areas I could help fix the problem.

Here are a few things I did to see if I could get better results.

First thing to do is increase the ulimit size for base operating system:
# For your session (and testing)
ulimit -n 200000
# To set it permanently on reboot
vim /etc/security/limits.conf
*                                –               nofile                  200000
Next edit your /etc/nginx/nginx.conf file: (You may have to play with the number of workers and processes depending on your setup)
# This number should be, at maximum, the number of CPU cores on your system.
# (since nginx doesn’t benefit from more than one worker per CPU.)
worker_processes 24;
# Number of file descriptors used for Nginx. This is set in the OS with ‘ulimit -n 200000’
# or using /etc/security/limits.conf
worker_rlimit_nofile 200000;
# only log critical errors
error_log /var/log/nginx/error.log crit
# Determines how many clients will be served by each worker process.
# (Max clients = worker_connections * worker_processes)
# “Max clients” is also limited by the number of socket connections available on the system (~64k)
worker_connections 4000;
# essential for linux, optmized to serve many clients with each thread
use epoll;
# Accept as many connections as possible, after nginx gets notification about a new connection.
# May flood worker_connections, if that option is set too low.
multi_accept on;
# Caches information about open FDs, freqently accessed files.
# Changing this setting, in my environment, brought performance up from 560k req/sec, to 904k req/sec.
# I recommend using some varient of these options, though not the specific values listed below.
open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
# Buffer log writes to speed up IO, or them altogether
#access_log /var/log/nginx/access.log main buffer=16k;
access_log off;
# Sendfile copies data between one FD and other from within the kernel.
# More efficient than read() + write(), since the requires transferring data to and from the user space.
sendfile on;
# Tcp_nopush causes nginx to attempt to send its HTTP response head in one packet,
# instead of using partial frames. This is useful for prepending headers before calling sendfile,
# or for throughput optimization.
tcp_nopush on;
# don’t buffer data-sends (disable Nagle algorithm). Good for sending frequent small bursts of data in real time.
tcp_nodelay on;
# Timeout for keep-alive connections. will close connections after this time.
keepalive_timeout 30;
# Number of requests a client can make over the keep-alive connection. This is set high for testing.
keepalive_requests 100000;
# allow the server to close the connection after a client stops responding. Frees up socket-associated memory.
reset_timedout_connection on;
# send the client a “request timed out” if the body is not loaded by this time. Default 60.
client_body_timeout 10;
# If the client stops reading data, free up the stale client connection after this much time. Default 60.
send_timeout 2;
# Compression. Reduces the amount of data that needs to be transferred over the network
gzip on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable “MSIE [1-6]\.”;
Putting it all together on a current Nginx Installation your nginx.conf will look similar to this:
vim /etc/nginx/nginx.conf
#———————————————————————-
# Main Module – directives that cover basic functionality
#
#   http://wiki.nginx.org/NginxHttpMainModule
#
#———————————————————————-
user              nginx;
worker_processes  24;
worker_rlimit_nofile 200000;
error_log  /var/log/nginx/error.log;
#error_log  /var/log/nginx/error.log  notice;
#error_log  /var/log/nginx/error.log  info;
pid        /var/run/nginx.pid;
#———————————————————————-
# Events Module
#
#   http://wiki.nginx.org/NginxHttpEventsModule
#
#———————————————————————-
events {
    worker_connections  4000;
        use epoll;
        multi_accept on;
}
#———————————————————————-
# HTTP Core Module
#
#   http://wiki.nginx.org/NginxHttpCoreModule
#
#———————————————————————-
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  ‘$remote_addr – $remote_user [$time_local] “$request” ‘
                      ‘$status $body_bytes_sent “$http_referer” ‘
                      ‘”$http_user_agent” “$http_x_forwarded_for”‘;
    access_log  off;
    sendfile    on;
    open_file_cache max=200000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 30;
    keepalive_requests 100000;
    reset_timedout_connection on;
    client_body_timeout 10;
    send_timeout 2;
    # Load config files from the /etc/nginx/conf.d directory
    # The default server is in conf.d/default.conf
    include /etc/nginx/conf.d/*.conf;
    gzip on;
    gzip_min_length 10240;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
    gzip_disable “MSIE [1-6]\.”;
}
/etc/init.d/nginx restart
Tuning the kernels TCP stack:
vim /etc/sysctl.conf
net.ipv4.ip_local_port_range = 2000 65000
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 3240000
net.core.somaxconn = 3240000
net.ipv4.tcp_max_tw_buckets = 1440000
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_congestion_control = cubic
# To apply the changes
sysctl -p /etc/sysctl.conf

I have some other things I plan on doing to push these numbers even further, but that’s for another post when I have more time.

Print Friendly

Comments

comments

Bài viết liên quan