HAProxy and Certbot on Docker

A while ago I wrote a post about running HAProxy on Docker, where the goal was to set up HAProxy in a Docker container so that it could provide frontends for requests and use Docker containers as backends.

The goal this time is to involve Let’s Encrypt/Certbot to generate and provide TLS certificates on the fly, so that HAProxy frontends are automatically equipped with corresponding certificates, avoiding separate server configurations for each interested container. The containers themself shall be in a Docker-internal network that is solely accessible through the HAProxy container.

Abstract

The client sends a request to the server where it is handled by an application delivery controller (ADC). The ADC interprets the request and forwards it accordingly. If it is a request with transport layer security (TLS) the ADC handles the protocol as well.

Certificates are needed to handle requests with TLS. These certificates are provided by a certificate authority (CA) and are subject to periodic validity checks and renewals. A certificate tool facilitates validity checks and renewals.

• client – the entity that creates a request (e.g. a browser, API client)
• request – a request following the hypertext transport protocol (HTTP)
• server – a computer (e.g. cloud server) that hosts applications and is accessible over a network
• application delivery controller (ADC) – an application that decides which application is provided after a request (i.e. HAProxy)
• transport layer security (TLS) – cryptographic protocol providing communication security over a network
• certificate – a document that verifies your identity
• certificate authority (CA) – widely recognized and accepted entity that issues certificates (i.e. Let’s Encrypt)
• certificate tool – obtains new certificates from the CA, checks validity of certificates and renews expired ones (i.e. Let’s Encrypt Certbot)

The plan

Basically, this project will consist and/or make use of the following Docker features:

• container: HAProxy (Docker image)
• container: Certbot (Docker image)
• network: default network (bridge)
• network: internal network
• volume: shared volume with certificates
• internal Docker DNS server

The two containers will share a volume that holds the certificates. Certbot will check the validity of the certificates according to Let’s Encrypt’s guidelines and – if necessary – update the certificates. The certificates, provided by Certbot in the form of .pem files, will then be concatenated and stored on the shared volume. HAProxy then reloads the certificate from the shared volume.
In this setup, the HAProxy and Certbot containers are the only ones publicly accessible (default bridge network). Application containers are bound to the internal network only, and accessible exclusively via the HAProxy container.

First of all, this solution needs to work with docker-compose, not only because it’s cleaner than my usual deploy.sh shell scripts, but to also support my occasional laziness. So I’m setting up a .yml file, according to the documentation, where I define the two services, the shared volume and setup the networks. Everything else shall be done by scripts.

HAProxy

Service one is HAProxy, which is based on the original image. The container exposes three ports – HTTP (:80), HTTPS (:443), and the HAProxy runtime API (:9999) – that can be mapped to any host-side ports, as long as the requests then point to the correct ones (e.g. port-forwarding on a router). This is also useful for cases in which other services are already using said ports, such as a custom DNS (e.g. PiHole). Additionally, the container must be part of two networks:

• the default bridge network, so that it is accessible from host-side
• the custom network I called “hapnet”, that was set up in the previous blog post

Furthermore it gets two mounts, one’s the configuration file straight from host (for easy-access), the other one’s the shared volume. Since HAProxy doesn’t need to modify anything in any of those files, they are read-only (:ro).

haproxy:
  image: haproxy:latest
  container_name: "haproxy"
  hostname: "haproxy"
  restart: always
  volumes:
    - ./cfg/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    - certificates:/usr/local/etc/haproxy/certificates:ro
  networks:
    - default
    - hapnet
  ports:
    - "80:80"
    - "443:443"
    - "9999"

The HAProxy configuration needs to be updated so that the runtime API is accessible (line 4) and so that HAProxy handles the requests for the acme-challenges as well. These are passed along to the Certbot container itself (line 33).

global

	# Enable HAProxy runtime API
	stats socket :9999 level admin expose-fd listeners

frontend http
	bind :80
	http-request redirect scheme https unless { ssl_fc }
	
	# ACL
	acl certbot path_beg /.well-known/acme-challenge/

	# Backends
	use_backend certbot if certbot

# HTTPS frontend
frontend https
	bind :443 ssl crt /usr/local/etc/haproxy/certificates/
	http-request add-header X-Forwarded-Proto https

	# ACL
	acl certbot path_beg /.well-known/acme-challenge/

	# Backends
	use_backend certbot if certbot

# Docker resolver
resolvers docker
	nameserver dns1 127.0.0.11:53

# Certbot backend
backend certbot
	server certbot certbot:380

Certbot

Service two is going to be a slightly customized Certbot. The only changes are the installation of socat and a bunch of additional shell scripts, that are responsible for creating/renewing certificates and generating HAProxy-friendly .pem files. The image exposes port :380 which is used in both the HAProxy configuration as well as in the create and renew scripts.

FROM certbot/certbot

# Install socat
RUN apk update && apk add socat

# Copy scripts
COPY ./scripts/ /etc/scripts/

# Expose port 380
EXPOSE 380

• create-certificates.sh – requests a new certificate and invokes the certificate concatenation script.
• renew-certificates.sh – checks whether the certificate exists or not. If it does, certbot is instructed to renew it. That’ll check if it’s still valid and generate a new certificate – in case it’s not – and concatenate the generated files to a single .pem file.
• concatenate-certificates.sh – the creation of HAProxy’s .pem file is straightforward. Certbot’s fullchain.pem and privkey.pem files are concatenated and saved to the certificate directory in the shared volume.
• update-haproxy-certificates.sh – triggers HAProxy Dynamic SSL Certificate Storage (since version 2.1).

#!/bin/bash

# Request certificates
certbot certonly --standalone \
  --non-interactive --agree-tos --email info@davole.com --http-01-port=380 \
  --cert-name davole.com \
  -d davole.com

# Concatenate certificates
. /etc/scripts/concatenate-certificates.sh

# Update certificates in HAProxy
. /etc/scripts/update-haproxy-certificates.sh

#!/bin/bash

# Certificates exist
if [ -d /etc/letsencrypt/live/davole.com ]; then
  # Check certificates and renew them
  certbot renew --http-01-port=380

  # Concatenate certificates
  . /etc/scripts/concatenate-certificates.sh

  # Update certificates in HAProxy
  . /etc/scripts/update-haproxy-certificates.sh

# Certificates don't exist
else
  #  Execute certificate creation script
  . /etc/scripts/create-certificates.sh
fi

#!/bin/bash

if [ -f /etc/letsencrypt/live/davole.com/fullchain.pem -a -f /etc/letsencrypt/live/davole.com/privkey.pem ]; then
  cat /etc/letsencrypt/live/davole.com/fullchain.pem /etc/letsencrypt/live/davole.com/privkey.pem > /etc/certificates/davole.com.pem
fi

#!/bin/bash

# Start transaction
echo -e "set ssl cert /usr/local/etc/haproxy/certificates/davole.com.pem <<\n$(cat /etc/certificates/davole.com.pem)\n" | socat tcp-connect:haproxy:9999 -

# Commit transaction
echo "commit ssl cert /usr/local/etc/haproxy/certificates/davole.com.pem" | socat tcp-connect:haproxy:9999 -

# Show certification info (not essential)
echo "show ssl cert /usr/local/etc/haproxy/certificates/davole.com.pem" | socat tcp-connect:haproxy:9999 -

Now that the Certbot image is in place too, it’s time to add the service to the .yml file. This service will build the Docker image ad-hoc when docker-compose is run with the --build option. The entrypoint (line 6) specification will run the renew-certificates.sh script every 12 hours, as specified by Let’s Encrypt

certbot:
  build: .
  image: davole/certbot:latest
  container_name: "certbot"
  hostname: "certbot"
  entrypoint: "/bin/sh -c 'trap exit TERM; while :; do . /etc/scripts/renew-certificates.sh ; sleep 12h & wait $${!}; done;'"
  restart: always
  volumes:
    - certificates:/etc/certificates
  ports:
    - "380:380"
  networks:
    - default
    - hapnet

What remains to be done is to put the two services together, and add network and volume configuration. Since I’ve already set up both the shared volume and the network, I do not need to do that again.

version: "3.7"

services:
  haproxy: [...]
  certbot: [...]

networks:
  hapnet:
    name: hapnet
    external: true

volumes:
  certificates:
    external: true

Conclusion

In this post I demonstrated how it is possible to make use of Let’s Encrypt’s certificates and Certbot in combination with HAProxy on Docker. You saw how it’s possible to configure HAProxy and Certbot to work together in a dockerized environment to serve web-applications through one frontend. All that while TLS is handled automatically without the need to update your certificates every 90 days.

Once again kudos to my HAProxy guy for the occasional nudge in the right direction.