使用Nginx进行UDP端口的负载均衡

UDP端口的负载配置

以下安装配置方式可能已不适用于新版本的安装,若考虑安装新版本请见第二章节对高版本安装的说明!

此前一直在用HaProxy做TCP的负载均衡,但UDP上一直没有启用可行的方案,这次需要UDP,便使用Nginx作为代理服务来使用。记录下具体步骤。

编译安装Nginx并启用stream模块

以下命令均在Ubuntu 16.04中执行。
安装所需的依赖库:

apt install proc* openssl* pcre*

下载所需的版本:

wget -c https://nginx.org/download/nginx-1.14.0.tar.gz 

解压并进入相关目录,执行:

./configure  --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-threads --with-stream --with-mail --with-file-aio  --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

注意:以上编译的选项根据自己需求进行编译,也可最简易的配置。

接下来编译:

make & make install

编译完成后会输出配置的结果:

  
...
nginx path prefix: "/etc/nginx"
nginx binary file: "/usr/sbin/nginx"
nginx modules path: "/etc/nginx/modules"
nginx configuration prefix: "/etc/nginx"
nginx configuration file: "/etc/nginx/nginx.conf"
nginx pid file: "/var/run/nginx.pid"
nginx error log file: "/var/log/nginx/error.log"
nginx http access log file: "/var/log/nginx/access.log"
nginx http client request body temporary files: "/var/cache/nginx/client_temp"
nginx http proxy temporary files: "/var/cache/nginx/proxy_temp"
nginx http fastcgi temporary files: "/var/cache/nginx/fastcgi_temp"
nginx http uwsgi temporary files: "/var/cache/nginx/uwsgi_temp"
nginx http scgi temporary files: "/var/cache/nginx/scgi_temp"

然后尝试启动Nginx,是否有问题:

sudo /usr/sbin/nginx -c /etc/nginx/nginx.conf

配置负载

备份原有的配置文件

cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak
cp /dev/null /etc/nginx/nginx.conf
vi /etc/nginx/nginx.conf

填入如下内容:

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}
stream {

    upstream lvs {
       server 101.89.134.11:8989;
       server 101.89.134.12:8989;
       server 101.89.134.13:8989;
    }

    server {
        listen 8989 udp;
        proxy_responses 1;
        proxy_timeout 20s;
        proxy_bind $server_addr:$remote_port;
        proxy_pass dns;
    }
}

UDP端口段的负载(Range of Ports)

安装所需的依赖库:

apt install proc* openssl* pcre*
#或
sudo apt install zlib1g-dev
sudo apt install libpcre++-dev
sudo apt install openssl

或者编译:

wget -c https://www.openssl.org/source/openssl-1.1.1a.tar.gz
wget -c https://ftp.pcre.org/pub/pcre/pcre-8.42.tar.gz
wget -c https://www.zlib.net/zlib-1.2.11.tar.gz

下载所需的版本:

wget -c http://nginx.org/download/nginx-1.18.0.tar.gz

将上述包下载至同一目录下,并分别解压,然后进入nginx目录后执行:

./configure --prefix= --with-stream  --with-openssl=../openssl-1.1.1a --prefix=/usr/local/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf

执行完上述指令后注意观察结果,确认库及配置均配置成功,通常:

checking for zlib library ... found
creating objs/Makefile

Configuration summary
  + using system PCRE library
  + using OpenSSL library: ../openssl-1.1.1a
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/sbin/nginx"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/etc/nginx"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

然后执行编译:

make 
make install

此时最后结果会显示:

...
        || mkdir -p '/usr/local/nginx/logs'
test -d '/usr/local/nginx/html' \
        || cp -R html '/usr/local/nginx'
test -d '/usr/local/nginx/logs' \
        || mkdir -p '/usr/local/nginx/logs'
make[1]: Leaving directory '/root/nginx-1.18.0'

然后启动即可。常用命令:

nginx
nginx -s stop
nginx -s reload

若启动时报错:

[emerg] getpwnam("nginx") failed

可能是因为安装时指定了用户名或用户组,配置时去除或者创建对应的用户名或用户组即可。

配置

/etc/nginx/nginx.conf 中加入:

stream {
    proxy_connect_timeout 5s;
    include stream/*conf;
}

在同级目录创建stream文件夹,再创建新的文件:udp.conf,内容为:

upstream udp {
    server 10.0.0.1:10000;
}
server {
    listen 10000-20000 udp;
    proxy_connect_timeout 3s;
    proxy_timeout 3s;
    proxy_pass udp;
}

最后重载配置。