HAProxy for Docker containers

At work I am currently developing an application that provides some tools and represents aggregated data from different sources to facilitate the workflow in different departments. It basically consists of two applications, a single page application being the front-end, and a RESTful API as back-end.

The applications run in separate Docker containers on a virtual machine in the firm’s intranet, as well as a couple of other dockerized applications. In the beginning that ensemble worked quite well, because those applications were only used by specific persons or systems. So exposing different ports for different applications under the same domain was not that much of a problem. But that changed when the aforementioned SPA & API became more widely used. On one hand it isn’t that great to explain to (technically challenged) users – to use one and the same domain but different ports for their apps -, on the other hand it just ain’t pretty. And since change needed to happen I also wanted to route HTTP traffic over TLS. Because… why not?

Approach

So first a couple of subdomains, some certificates and then… what? When it came to deployment I had two options in mind:

  1. a web server on the host that routes traffic to specific containers that expose different ports
  2. a Docker container that is accessible from the intranet and acts as a reverse proxy, directing traffic to the containers

Web server on Host

At first glance that seemed the way to go. Setting up an Apache/NGINX web server and configuring it so that requests to a certain domain would point to a specific port on the host. Docker would publish the container’s port on the host, the web server take care of the TLS certificate(s) et voilà, all would be good.

All? Well, except for the fact that all those published ports would also be accessible from the network where the host resides. And I did not like that. Plus I knew that – me knowing myself – I would eventually become tired of all the different containers and all the different ports and all the whatnot. I guess there exist ways to circumvent those things, but at this point I thought there must be an easier, cleaner, lazier way.

Dockerized Reverse proxy

I already knew HAProxy, as I have an instance of it running in another place for quite some time, and that there is an official Docker image for HAProxy. So setting up the front-ends for HTTP and HTTPS was straight forward. However, the back-ends were not that straight. Because how should I point to a server?

Let’s talk about the Docker setup first. When you run a container it is connected to Docker’s default network, bridge. Three Docker networks are available from the start, bridge, host and none. A container connected to the default bridge network can communicate with others if the IP addresses are known. That bears the problem that containers do not always get the same IP address when they start/restart – unless they are explicitly defined when a docker container is run:

$ docker run --ip

I did not like the idea of assigning IPs manually to each new container. Fortunately the guys over at Docker did not like the idea either and introduced both the --hostname and the --link parameter for the run command. A container run with those parameters would be discoverable by the linked container by its hostname. However, the link parameter is deprecated, ever since Docker introduced the network feature.

Docker Network

But that actually simplifies the entire story. Because now containers are always discoverable by their container name, as long as they’re in the same user defined network.

And that solves the question from earlier. Now the back-ends configured in HAProxy simply point to the hostname:port of a container. This works if the HAProxy container is in the same network as the other containers. A container can be connected to more than just one network, which is exactly what needs to be done here. The HAProxy container is connected both to the subnet and to the bridge network, where it exposes the ports 80 (HTTP) and 443 (HTTPS).

What’s left to do tho is to configure HAProxy in a way, that it reacts to changing IP addresses. Because the hostnames are resolved only at startup. So if a container’s IP changes later on, HAProxy might point to the wrong container or none at all.

Let’s do this

Long story short

Incoming traffic is handled by HAProxy. The TLS certificate is checked and then, based on ACLs, the traffic is redirected to a back-end that points to the (host)name of a Docker container, and Docker’s internal DNS handles the hostname resolution.

Short story Long

First a new bridged network (in this case i name it hapnet) is created, along with some parameters:

  • --driver: specifies the driver for the network
  • --internal: restrict external network access
$ docker network create --driver bridge --internal hapnet

The application containers are then connect to the hapnet network with the --net parameter. That way they are not directly accessible from outside but can communicate among each other.

$ docker run -d --restart always --name frn --net hapnet frontend:latest
$ docker run -d --restart always --name bck --net hapnet backend:latest

Then the HAProxy container is started, connected to the default bridge, exposing the desired and configured ports 80 and 443.

global
	tune.ssl.default-dh-param 2048
	maxconn 100
	daemon

defaults
	mode http

	timeout connect  4s
	timeout http-request  1s
	timeout http-keep-alive  120s
	timeout client  1m
	timeout server  1m
	timeout queue  10s

	option splice-auto
	option dontlog-normal
	option tcp-smart-accept
	option tcp-smart-connect
	option forwardfor
	option http-keep-alive
	
	http-reuse safe

frontend www-http
	bind :80
	reqadd X-Forwarded-Proto:\ http
	
	# hosts
	acl host_frn hdr(host) -i front.example.com
	acl host_bck hdr(host) -i back.example.com

	# https redirect
	redirect scheme https code 301 if !{ ssl_fc } host_frn OR !{ ssl_fc } host_bck

frontend www-https
	bind :443 ssl crt /usr/local/etc/haproxy/example.com.pem
	reqadd X-Forwarded-Proto:\ https
	
	# hosts
	acl host_frn hdr(host) -i front.example.com
	acl host_bck hdr(host) -i back.example.com
	
	# backends
	use_backend front if host_frn
	use_backend front if host_bck

resolvers docker
	nameserver dns1 127.0.0.11:53

backend front
	server front frn:80 check inter 10s resolvers docker

backend back
	server back bck:80 check inter 10s resolvers docker

And here’s a little hiccup: the execution will fail, because at this point HAProxy does not see the other containers, as it is in the wrong network. Therefore the container will exit immediately and the logs will show something like:

[ALERT] 143/182147 (6) : parsing [/usr/local/etc/haproxy/haproxy.cfg:53] : 'server front' : could not resolve address 'frn'.
[ALERT] 143/182147 (6) : Failed to initialize server(s) addr.

This is why the container is started with --restart always so it will try to come up after it exited. The container could be connected even if it is exited, but I want it to restart always either way.

$ docker run -d -p 443:443 -p 80:80 --restart always --name haproxy -v /home/.../config:/usr/local/etc/haproxy:ro haproxy:1.7-alpine

Now it needs to be connected to the hapnet network, and from that point on HAProxy will see the other containers.

$ docker network connect hapnet haproxy

When the HAProxy container is run, the configuration folder is mounted into it, containing the haproxy.cfg configuration file as well as the TLS certificate(s). HAProxy resolves a hostname’s IP on start, so whenever a container’s IP changes we’ve got a problem. To avoid this, we can use a resolver to specify a DNS. Docker provides such a DNS and we can use it in HAProxy. Last but not least we need to tell the backends to use that resolver and choose an appropriate TTL.

And we’re good to go.


Special thanks to my network/HAProxy guy for helping out with the HAProxy configuration.

One Reply to “HAProxy for Docker containers”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.