haproxy 2.0 DATA PLANE API动态更新haproxy.cfg

Posted by Mathew on 2019-08-09

一、目标

  使用dataplaneapi对haproxy backend server动态扩缩容,减少数据中转成本、提升数据更新时效性。

二、方案

haproxy 2.0

提供了 RESTAPI 方式 的DATA PLANE API对haproxy状态进行管理,包括frontend、backend、server等动态修改

In today’s cloud-native landscape, ephemeral services are born and die quickly, deployments happen continuously, and configuration needs to be refreshed constantly. The new Data Plane API provides a modern REST API for configuring HAProxy on the fly. You can now dynamically add and remove frontends, backends, and servers. You can create ACL rules, insert HTTP routing directives, set IP and port bindings, and much more. The API updates the configuration file as needed, reloading the HAProxy process when necessary.

测试结论:

  1. DATA PLANE API的启动参数中包含对haproxy的reload --reload-cmd,对haproxy reload(systemd ExecReload适配见下文)做适配之后可以实现seamless reload(0中断 reload),高并发场景待验证。
    image.png
  2. 如果启动dataplaneapi的时候去掉--reload-cmd参数,虽然可以启动,但是调用api修改config的操作会触发dataplaneapi这个程序去执行reload操作,找不到reload-cmd会产生报错,实际haproxy.cfg未提交至内存所以不生效。
    image.png
  3. 调用data plane api效果是直接修改haproxy.cfg文件 且 会触发启动datapalneapi时候的参数--reload-cmd操作。

  1. The new process tries to bind to all listening ports
  2. If it fails, it sends a signal to the old processes asking it to temporarily release the ports
  3. The new process tries again
  4. If it fails, it gives up and signals the old process to continue taking care of the incoming connections
  5. If it succeeds, it signals the old process it can quit once it has finished serving existing connections
    image.png

In the mean time, a new and much better SO_REUSEPORT implementation was brought to Linux kernel 3.9, allowing the load to be intelligently spread over multiple sockets. HAProxy could immediately benefit from this new improvement.

  3.9版本以后内核中的“SO_REUSEPORT”的sokect选项,即允许同一台主机上的多个socket可以binding到同一个tcp port,reload的时候,通过stats socket将当前listener的文件描述传递给新起的haproxy进程。旧的haproxy进程处理完毕当前用户连接后退出,新的haproxy进程处理新的用户连接。在这个过程中,因为新的haproxy进程也binding到了原有的listener FD上,在整个过程中不会有连接被关闭。真正实现了seamless-reload。

haproxy作者willy提到了其实这个方法在bsd上一直有,haproxy的企业版之前也一直支持这个功能,linux之前内核也有,后来消失了,在3.9内核又重新出现了更好的实现。

expose-fd listeners
This option is only usable with the stats socket. It gives your stats socket
the capability to pass listeners FD to another HAProxy process.
During a reload with the master-worker mode, the process is automatically
reexecuted adding -x and one of the stats socket with this option.

haproxy.cfg的state开启expose-fd listeners配置

1
stats socket /run/haproxy/admin.sock mode 777 level admin expose-fd listeners

seamless reload总结三点:

  1. 一个是“Ws”选项,即master-worker模式
    -W : master-worker mode. It is equivalent to the “master-worker” keyword in the “global” section of the configuration. This mode will launch a “master” which will monitor the “workers”. Using this mode, you can reload HAProxy directly by sending a SIGUSR2 signal to the master. The master-worker mode is compatible either with the foreground or daemon mode. It is recommended to use this mode with multiprocess and systemd.
  1. 即新的reload方法
    ‘-x’ option is used in the ‘ExecReload’ action of the service file and specifies the unix socket used to transfer the listening sockets from the old process. This is the same socket referenced later in the haproxy.cfg ‘expose-fd listeners’.
    就是使用-x选项,通过在haproxy.cfg中定义的stats socket,将listening socket的FD传递给新的haproxy进程
  1. haproxy stats socket加入“expose-fd listeners”
1
stats socket /run/haproxy/admin.sock mode 777 level admin expose-fd listeners

三、配置

(1)haproxy 2.0安装

1
2
3
4
tar -zxvf haproxy-2.0.0.tar.gz
cd haproxy-2.0.0
make TARGET=linux-glibc USE_OPENSSL=1 USE_SYSTEMD=1 USE_LUA=1 USE_PCRE=1 USE_ZLIB=1 && make install
make install --prefix=/usr/local/sbin/haproxy

systemd service文件创建

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[Unit]
Description=HAProxy Load Balancer
# allows us to do millisecond level restarts without triggering alert in Systemd
After=network.target

[Service]
Environment="CONFIG=/usr/local/sbin/haproxy.cfg" "PIDFILE=/var/run/haproxy.pid"
# EXTRAOPTS and RELOADOPS come from this default file
EnvironmentFile=-/usr/local/sbin/haproxyfile
ExecStartPre=/usr/local/sbin/haproxy -f $CONFIG -c -q
ExecStart=/usr/local/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS

# note that the master worker mode obsoletes the old haproxy-systemd-wrapper
#
# add RELOADOPTS to /etc/default/haproxy if you want seamless reload
ExecReload=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS $RELOADOPTS

ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
Type=notify

[Install]
WantedBy=multi-user.target

~]$ echo RELOADOPTS="-x /run/haproxy/admin.sock" >> /etc/default/haproxy

(2)data plane api安装

1
2
3
4
5
下载所需版本二进制文件
https://github.com/haproxytech/dataplaneapi/releases

chmod +x dataplaneapi
cp dataplaneapi /usr/local/bin/

dataplaneapi的使用需要开启basic authentication,因此在haproxy中添加如下配置段

1
2
userlist controller
user admin insecure-password mypassword

启动dataplaneapi(api作为一个单独的二进制程序启动,也可以托管进haproxy 2.0 program配置段跟随haproxy一起启动)

1
2
3
4
5
6
7
8
$ dataplaneapi \
--host 0.0.0.0 \
--port 5555 \
--haproxy-bin $(which haproxy) \
--config-file /etc/haproxy/haproxy.cfg \
--reload-cmd "systemctl reload haproxy" \
--reload-delay 5 \
--userlist controller

注意: 启动参数里面 --reload-cmd "systemctl reload haproxy",在后续测试时候发现,每次调用dataplaneapi的修改操作会触发reload haproxy,并且只有reload才会生效。

四、测试

待完善

技术参考:

“Truly Seamless Reloads with HAProxy – No More Hacks!”:https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/
“The SO_REUSEPORT socket option”:https://lwn.net/Articles/542629/
“Haproxy 2.0 configuration”:https://www.haproxy.org/download/2.0/doc/configuration.txt
"Using dataplane plane api":https://www.haproxy.com/documentation/hapee/1-9r1/configuration/dataplaneapi/
“Data plane api接口文档”:https://www.haproxy.com/documentation/dataplaneapi/latest/#operation/replaceBackend