Setting up verified TLS (transport layer security) certificates for your web services allows them to be trusted by clients connecting over HTTPS. Without a trusted certificate all web browsers will show a warning that discourages users from proceeding to a site. In this post I'm going to break down how you can set up auto renewing TLS certificates for free for any of your domain names. Note: I use the term TLS here as TLS superseded SSL and is therefore technically the correct terminology, though many people still refer to TLS as SSL.

If you want to follow along, all of the code for this project can be found at the Github repo here.

Perquisites

• A basic understanding of Docker, Docker Compose, Linux, and networking concepts
• A Linux server with Docker and Docker Compose installed
• Port 80 and 443 open on your server, with port forwarding set up for them on your router
• A domain name pointed at your server's public IP address. You can get a free subdomain from freedns.afraid.org which will work fine

Intro to Certbot & Let's Encrypt

First, let's look at what Certbot and Let's Encrypt are, and how they work together.

Certbot is a free and open source tool which runs on your server that can be used to provision a TLS certificate by using the free certification service provided by the open certificate authority Let's Encrypt. Certbot and Let's Encrypt work together to achieve this by using the ACME protocol which forms the following challenge process:

1. The Certbot instance running on the server you enabled port forwarding to will send a request to Let's Encrypt to initiate verification for a given domain name. As part of this request, Certbot will also send over a public cryptographic key (part of a pair, the private key of which is kept on the server). If you are not familiar with keys and asymmetric encryption, you can learn about it with this excellent primer: Asymmetric Encryption - Simply explained.
2. In reply Let's Encrypt will send a challenge to the domain name provided in the form of a file to be placed at a specific path on the web server.
3. If your Certbot instance is truly located at the domain provided, it will receive the file, sign it with the private key counterpart to the public key it sent to Let's Encrypt, and then place it at the web server path expected by Let's Encrypt.
4. Let's Encrypt will then make another request to the domain provided and attempt to download the signed version of the file it posted there from the expected web server path.
5. If Let's Encrypt could successfully download the file, it will use the public key provided by Certbot at the start of the process to verify through a hash that the file was correctly signed, proving that your Certbot instance is in possession of the public key's private counterpart, and it resides at the domain name provided.
6. If this is successful, Let's Encrypt will sign a 90 day TLS certificate for your domain name with its own certificate's private key and send it back to your Certbot instance, which will place it in a directory (usually /etc/letsencrypt/live/) where it can be accessed by web servers to enable HTTPS transmission.

How does this mean your certificate is trusted? In short: your certificate is now at the end of a certificate chain which can be linked back through signing/hashing to Let's Encrypt's intermediate authority certificate, which can itself be linked back to a root certificate authority's certificate (probably ISRG Root X1). This root certificate is trusted automatically by all web browsers, and can be used to verify transitively that Let's Encrypt's certificate, and therefore your own certificate can be trusted.

Provisioning a TLS Certificate

Let's look at how we can use the Certbot Docker image to request a certificate using the ACME protocol.

Create a new project in your favourite code editor, then create a directory called certbot, and within this directory create another directory called initial, and within that directory create a file called docker-compose.yml. In this file, start with the following configuration:

# ./certbot/initial/docker-compose.yml

version: 3.9

services:
  certbot-initial:
    # change to fixed version in production
    # for raspberry pi (arm) latest does not work at time of writing. Use something like arm64v8-latest
    image: certbot/certbot:latest
    container_name: certbot-initial
    volumes:
      # where certbot will place the certificates once the challenge is complete. rw allows read write.
      - /etc/letsencrypt:/etc/letsencrypt:rw
    ports:
      # bind 80 on the host, to 80 on the container. this allows certbot to receive the challenge from Let's Encrypt
      - "80:80"
    # see explanation below
    command: certonly --standalone --non-interactive --email my@email.com --agree-tos --preferred-challenges http --dry-run -d my.domain.com

If you are familiar with Docker and Docker Compose, most of this will be straightforward. For the command key, all of these arguments will be appended to the certbot command, which is the entry point of the certbot/certbot:latest image. From left to right the arguments mean the following:

• certonly is used because we only want to be issued a certificate, and we do not want TLS to be automatically configured for a web server. We will configure it manually later.
• --standalone is used because we want Certbot to run as a self contained web server instead of using Nginx or another web server as a reverse proxy, which we will do later.
• --non-interactive is used to tell certbot not to expect any user input
• --email my@email.com provides an email address to Let's Encrypt, which is required should they ever need to contact you about your certificate. Change it to your email.
• --agree-tos is required to use the Let's Encrypt service.
• --preferred-challenges http tells Certbot to request that Let's Encrypt used port 80 (the only port linked to the container) for the challenge process.
• --dry-run instructs Certbot to use Let's Encrypt's staging server. This allows us to test if the configuration defined is correct. If you omit this flag and spam Let's Encrypt's service by running a Certbot container multiple times, your server may be blocked from making verification requests for a period of time. Read more about rate limits here.
• -d my.domain.com tells Certbot the domain name which we want to get a trusted certificate for. Change it the domain name you set up and pointed at the server in the Prerequisites section.

If you have satisfied the requirements in the Perquisites section above, then copy the project onto your server if you're not already on it (I push it to a repo on GitHub and clone it down, by you could also scp it across). On your server, cd into the ./certbot/initial/ directory and run docker compose up and you should see Certbot perform the ACME challenge process with Let's Encrypt, and then report if the dry run was successful. If all goes well, remove the --dry-run flag from the certbot-initial's command key in the docker-compose.yml file, and re-run docker compose up. This will cause Certbot to use Let's Encrypt's production servers, which will return a valid TLS certificate if everything checks out. This certificate can be found at /etc/letsencrypt/live/my.domain.com/ on the host machine.

If you got this far, congratulations! You now have a verified TLS certificate for your domain which is valid for 90 days. I recommend you make a backup of this certificate. Do not whatever you do, run the certbot-initial container many times without the --dry-run flag, as you will may soon become barred from making more requests to the Let's Encrypt servers for a period of time (I did this once without backing up my certificates and was left without TLS for a week, though luckily not in production).

Using the TLS certificate with Nginx

Next let's look at how we can set up Nginx to use our newly acquired TLS certificate.

Nginx is an event loop based web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. Once we have a TLS certificate, it is very easy to set up Nginx to enable HTTPS.

In the root of the project directory on your development machine, create another directory called nginx. Within this directory create another directory called conf.d and within it, create a file called nginx-test.conf, and add the following block (this is not fit for production, it is a minimal config). Read the comments to understand the server configuration and what to change.

# ./nginx/conf.d/nginx-test.conf

server {
    # this server listens on 443
    listen 443 ssl;
    # incoming requests must have this hostname. change!
    server_name my.domain.com;
    # the location of the tls cert. change!
    ssl_certificate /etc/letsencrypt/live/my.domain.com/fullchain.pem;
    # the location of the tls cert's private key. change!
    ssl_certificate_key /etc/letsencrypt/live/wy.domain.com/privkey.pem; # change this!

    location / {
      # where nginx will serve files from. full path /usr/share/nginx/html
      root   html;
      # the file that is served if a user goes to /
      index  index.html;
    }
}

server {
    # this server listens on 80
    listen 80;
    # incoming requests must have this hostname. change!
    server_name my.domain.com;

    # the location Let's Encrypt challenge will make requests to
    location /.well-known/acme-challenge/ {
        # where certbot will create the challenge files
         root /var/www/certbot;
    }

    # redirect anything else to https
    location / {
        return 301 https://$host$request_uri;
    }

}

Make sure you change all instances of my.domain.com to match the domain name you used when provisioning for the initial certificate or your certificate will not be trusted, and the configuration may not work at all.

Having created the nginx config, go back to the root of your project and create a docker-compose.yml file, and add the following:

# ./docker-compose.yml

version: "3.9"

services:
  nginx:
    image: nginx:latest # change this in production for a fixed value
    container_name: nginx
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      # where the nginx config is located
      - ./nginx/conf.d/:/etc/nginx/conf.d/:ro
      # where certbot placed the initial tls cert, and where refreshed certs will be placed
      - /etc/letsencrypt:/etc/letsencrypt:rw
      # where certbot writes challenge files to when being challenged by a Let's Encrypt server
      - /var/www/certbot:/var/www/certbot:ro

This compose config can be used to start an nginx server which will use the config that was defined previously, as well as the TLS certificate obtained from Let's Encrypt.

Run docker compose up -d, which will pull the nginx image, and then start the server.

If you enter your domain name into your browser prefixed with https:// (i.e. https://my.domain.com), you should now land at the nginx welcome page with the browser recognising the TLS certificate of the server as trusted (shown with the padlock in the address bar).

Setting Up Certificate Auto-Renewal

We've got half the way there, and we now have a certificate that will be valid for 90 days, but after this period, clients communicating with your server will stop trusting it. Now we'll look at how we can get the certificate to automatically renew before expiry.

To get a new certificate, we could temporarily shut down the nginx container, therefore unbinding from the server's ports 80 and 443, and then run the certbot-initial container again. However this is not ideal as: our site will be unavailable for a period of time, and we have to remember when the certificate is going to expire and intervene. What we really want is a way of automatically renewing the certificate. To achieve this, we're going to make a container which will run a certbot renew command using cron (a scheduling utility) every 12 hours. This renew command will request a new certificate, with nginx serving the challenge files.

Within the ./certbot/ directory we created earlier, create a directory called renew, and within this directory create a file called cert-renew-crontab. Add the following to this file:

# ./certbot/renew/cert-renew-crontab
0 0,12 * * * certbot renew --webroot -w /var/www/certbot > /proc/1/fd/1 2>/proc/1/fd/2

Here is an explanation of the config:

• 0 0,12 * * * tells cron to run the command following at the 0th minute of the 0th hour (12am) and 12th hour (12pm), on any day of any month, on any day of the week (every day).

• certbot renew --webroot -w /var/www/certbot is the command which will be run, which tells Certbot to initiate a certificate renewal using the webroot plugin which defers challenge file serving to a web server (nginx in our case), with the challenge files being placed in /var/www/certbot.

• /proc/1/fd/1 2>/proc/1/fd/2 redirects the process's stdout and stderr to the container's stdout and stderr so it can be seen in the container logs.

  Still in the ./certbot/renew/ directory, create another file called Dockerfile and add the following to it:

FROM certbot/certbot:latest

COPY cert-renew-crontab /etc/crontabs

# load crontab file
RUN crontab /etc/crontabs/cert-renew-crontab

# run cron in foreground
ENTRYPOINT ["crond", "-f"]

This Dockerfile builds upon the Certbot image and takes advantage of the fact that it includes the cron command. When building an image based on this Dockerfile, docker will copy the cert-renew-crontab file we wrote previously into a built image's crontab directory, load the file into cron, and then run the cron daemon. When this image is run as a container, the command to renew the certificate will be run by cron at 12am and 12pm every day. You might be concerned that Let's Encrypt will start blocking renewal requests at that frequency, but Certbot will avoid doing this by checking if the certificates at /etc/letsencrypt/live/ are going to expire within 30 days, and only if they are going to expire will it actually attempt a renewal with Let's Encrypt.

To actually build and run an image based on this Dockerfile, let's go back to the project's root and modify the ./docker-compose.yml file where we defined the nginx service previously. In the file's list of services, add the following:

  cert-renew:
    container_name: certrenew
    build:
      context: .
      dockerfile: ./certbot/renew/Dockerfile
    volumes:
      # where the certs are located
      - /etc/letsencrypt:/etc/letsencrypt:rw
      # where the renew challenge will be written and then served by nginx
      - /var/www/certbot:/var/www/certbot:rw

This service will create a container with the image defined by the Dockerfile we defined above. Mounting /etc/letsencrypt allows Certbot to see the certificates that were put onto the host by the certbot-initial container. This allows Certbot to tell when they have expired, and subsequently replace them with new certificates.

Running docker compose up -d should start a new container for the cert-renew service, and as defined in the crontab file, it will renew the certificates 30 days before they expire.

Getting Nginx to Use the Renewed Certificates

There is still a problem: even if the certificate is successfully renewed by the certrenew container, the nginx container will not use them. This is because when nginx starts, it caches the certificates in memory to enable higher performance than if it were reading from the on disk files for every TLS handshake. Now we'll look at how we can get Nginx to automatically reload the certificate cache when the certificates have been renewed.

In the project's ./nginx/conf.d directory create a file called reload-cert.sh and add the following code block to it, making sure to change my.domain.com on line 8 to your certificate's domain.

#!/bin/bash

while :; do
    # format: 03/08/22 11:36:26
    now=$(date +"%d/%m/%y %T")
    logPrefix="$now [CERT RENEW]"
    # get the current certificate expiry
    expiry=$(openssl x509 -enddate -noout < /etc/letsencrypt/live/my.domian.com/fullchain.pem)
    # check if there was a problem in getting the expiry
    if [[ $? -ne 0 ]]; then
        echo "$logPrefix failed to get certificate expiry: $expiry"
    else
        # check if previous expiry not empty string, and if different from current expiry
        if [[ -n "$prevExpiry" && "$expiry" != "$prevExpiry" ]]; then
            # if different then has changed, and should reload nginx
            echo "$logPrefix certificate changed. reloading nginx"
            # Unlike restart, reload does not drop connections, but still good to keep it to minimum
            nginx -s reload
        else
            echo "$logPrefix certificate not changed"
        fi

        echo "$logPrefix current expiry: $expiry"
        # set the previous expiry to current expiry
        prevExpiry="$expiry"
    fi
    # sleep for 24 hours
    sleep 24h
# background the loop command with & so it keeps running
done &

Read the comments to understand each command in the script, but the general idea is to use openssl to check if the certificate has changed since the script was last run, and if it has changed, send a reload signal to Nginx with nginx -s reload to get it to load the new certificate into its cache. The script will then sleep for 24 hours, and then repeat.

To run this script we'll mount it into the nginx container's /docker-entrypoint.d/ directory. Any scripts in this directory will automatically be run by the container on start, and it includes the script to start the container's Nginx process. We can use this behaviour to get the container to start a second process which will run the above script, reloading Nginx when the certificate has been renewed.

To mount the script into the nginx container, change the project's ./docker-compose.yml file by adding the following to the nginx service's volume list: - ./nginx/reload-cert.sh:/docker-entrypoint.d/reload-cert.sh:ro. Then in the root of the project run docker compose restart nginx, which should restart the nginx service with the above script running. You can find log messages from the script by searching the container's logs for [CERT RENEW].

Conclusion

In this post we've seen how to:

• Generate an initial TLS certificate with Certbot in standalone mode

• Use the certificate with Nginx

• Set up automatic certificate renewal with Certbot in webroot mode

• Set up automatic certificate reload with a script running in an Nginx container

You can check out the completed project at the Github repo here.

Once you've set up this basic template, it's very easy to use it in any project whenever you need a TLS certificate. If you're building a website, the next step would be to mount the site's files into the nginx container, and configure Nginx to serve them to users visiting your domain. For a backend, the next step would be to configure Nginx to act as a TLS termination point and then reverse proxy requests to your backend services. I will probably write another post which will explain how this is done at some point.