I'm trying to set up a simple web load balancing using CentOS 7.5 (64bit, kernel 3.10), HAProxy 1.8.13 and systemd.
It seems that the HAProxy configuration is OK, but starting the application gives me a headache. I tried it with init.d once but was then pointed into the systemd direction, where I'm currently stuck. Been looking for the cause for almost 2 days now and I have reached my limits when it comes to linux, I also couldn't find a cause for this particular behavior anywhere. Most cases like this have some issue in the haproxy config, but that's not the case here apparently.
HAProxy config file validation output
haproxy -f /etc/haproxy/haproxy.conf -c
Configuration file is valid
When I try to start it using systemd, I get the following output in the journalctl:
journalctl -u haproxy.service
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:51 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:51 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:51 localhost.localdomain systemd[1]: start request repeated too quickly for haproxy.service
Aug 13 17:07:51 localhost.localdomain systemd[1]: Failed to start HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: Unit haproxy.service entered failed state.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service failed.
Systemctl shows the following status:
systemctl status haproxy.service
[root@localhost sbin]# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Mo 2018-08-13 17:33:46 CEST; 24min ago
Process: 1557 ExecStart=/usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid (code=exited, status=0/SUCCESS)
Process: 1556 ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q (code=exited, status=0/SUCCESS)
Main PID: 1557 (code=exited, status=0/SUCCESS)
Aug 13 17:33:46 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:33:46 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:33:46 localhost.localdomain systemd[1]: start request repeated too quickly for haproxy.service
Aug 13 17:33:46 localhost.localdomain systemd[1]: Failed to start HAProxy Load Balancer.
Aug 13 17:33:46 localhost.localdomain systemd[1]: Unit haproxy.service entered failed state.
Aug 13 17:33:46 localhost.localdomain systemd[1]: haproxy.service failed.
The systemd file for this service was mostly taken over from the sample file haproxy 1.8.13 provided. I just replaced some placeholders/variables with fixed values, since some component couldn't handle relative paths.
nano /lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network.target
[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/var/run/haproxy.pid"
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
ExecReload=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify
# The following lines leverage SystemD's sandboxing options to provide
# defense in depth protection at the expense of restricting some flexibility
# in your setup (e.g. placement of your configuration files) or possibly
# reduced performance. See systemd.service(5) and systemd.exec(5) for further
# information.
# NoNewPrivileges=true
# ProtectHome=true
# If you want to use 'ProtectSystem=strict' you should whitelist the PIDFILE,
# any state files and any other files written using 'ReadWritePaths' or
# 'RuntimeDirectory'.
# ProtectSystem=true
# ProtectKernelTunables=true
# ProtectKernelModules=true
# ProtectControlGroups=true
# If your SystemD version supports them, you can add: @reboot, @swap, @sync
# SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io
[Install]
WantedBy=multi-user.target
It appears that it starts the application, stops it again and then tries to start it again, until it gives up at some point.
When I try to start the application manually from the bash, it looks like nothing is happening:
[root@localhost sbin]# /usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
[root@localhost sbin]#
No process visible via ps or top. I tried getting some logs, but I couldn't get any log from rsyslog since it doesn't want to write into the file I specified.
Adding the haproxy config file as an appendix, but as per haproxy itself, it seems to be fine:
# Rev1
# HAProxy Global and Default definitions
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/run/haproxy.sock
stats timeout 2m
defaults
log global
option dontlognull
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 300s
timeout server 300s
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend LB_https_vs
bind *:443
mode tcp
option tcplog
default_backend SP_https_group
backend LB_https_group
mode tcp
balance source
option httpchk GET /MyPage/Logon.aspx
hash-type consistent
stick-table type ip size 200k expire 2000s
stick on src table SP_https_group
default-server inter 15000
#server srv1 192.168.1.1:443 id 1 port 443 check
server srv1 192.168.1.1:443 id 1 check
server srv2 192.168.1.2:443 id 2 check
server srv3 192.168.1.3:443 id 3 check
server srv4 192.168.1.4:443 id 4 check
server srv5 192.168.1.5:443 id 5 check
listen stats
mode http
bind :9000
stats enable
stats hide-version
stats realm HAproxy-Statistics
stats uri /haproxy_stats
stats auth admin:default
Any ideas what's going on here?
Thanks in advance
Best Answer
You need to start haproxy in a mode that runs in foreground. Furthermore, if you want to use
Type=notify
, you need haproxy to implement the systemd notify daemon.It looks like you should start it using the
-Ws
option rather than just-W
. From the --help output in the source code:Please also note that this only works if haproxy was built with systemd support. Not sure if that's the case for you, but you can check it by looking at the --help output (it doesn't seem to support
--help
directly, but it prints usage info on unknown args, so--help
should work for you.)You might want to take a look at the suggested haproxy.service template in version 1.8 and the latest one in GitHub, looks like what you're using is close to the latest one, but with some differences (such as
-W
instead of-Ws
...)