Skip to main content

traefik proxy v3 Setup

Now that we have the foundation laid for our services by getting an LXC and VM setup with Alpine linux and docker, it is time to get our first service running and that will be traefik proxy v3. For this we will do our work on the VM as it is the manager node of our docker swarm. As mentioned in the docker section I had many issues trying to run every service in their own docker container in their own LXC, the networking portion just would not work correctly. I can't seem to find the bug report on github, if I do find it I will put the link up here.

DNS

You will need some method to do local DNS records like a Pi-hole(s), Ubiquiti Unifi, pfSense or OPNsense. As we will be playing around with CNAME records and DNS records. You will also need your own domain name, although you can just use [YOURNAME].home.arpa where home.arpa is the TLD/FQDN, there is also .test which shouldn't conflict with anything. Keep in mind I have not tested .home.arpa or .test and Let's Encrypt so use those at your own risk.

.local

Using .local could cause your network to go nuts as it is being used for mDNS.

Hint It is always DNS ha!

File Transfering

You will need some way to transfer files to your PVE host from your computer, puTTY, Termius can do this through a GUI and I am pretty sure PowerShell can but through CLI.

Manager node / system prep​

Let's create a few directories for our traefik information in the NFS share we setup. Technically you can keep everything within the traefik proxy docker container or in a docker volume if that is the path you wish to take then you will need to look for other resources that explain how to use them.

Let's begin

mkdir traefik

navigate into the new directory

info

Yes, you can stay in a root directory and do this but I prefer going in to the directory just incase I mistype something.

cd traefik

Make the directories and create the files we need

mkdir certs && touch certs/acme.json && chmod 600 certs/acme.json && mkdir logs && touch logs/traefik.log && touch logs/access.log && mkdir rules && mkdir configs && touch configs/traefik.yml

info
  • mkdir: creates a directory
  • &&: is an operator and ensures that each command is only run if the previous command is successful, if any command fails then the whole thing will fail
  • touch: creates a file
warning
  • chmod 600: makes the file so only the owner can read and write to it and no one else can access it, this needs to happen or else traefik proxy will complain alot in the traefik.log file and it will fail to get an SSL certificate for your FQDN.

We just have one more directory to create this will be in the docker directory of your NFS share.

mkdir traefik
touch traefik/docker-compose.yml
info

There are a few different schools of thought on docker compose file naming. My preference is to have a /docker directory and then add sub-directories for each service I have where the docker-compose.yml file is kept. My reasoning for this is it allows me to have multiple copies if I am trying to get something working. For example I had an issue with the service labels, rather then just delete labels and have no reference point on what things were working and what didn't. I just copy the docker-compose.yml and rename the old one as docker-compose.yml.1 and so on and so forth esentially making a poor mans file versioning. This method still would work with just leaving everything in the /docker directory and using [SERVICE-compose.yml] as your filename, it is just how you want to shift the clutter either multiple directories or multiple files in one spot.

docker-compose.yml​

Now on to creating the compose file. The file I put below is heavily commented to make navigating the settings easier.

info

There are many tutorials out there that say we should be using .env files and I do agree with that sentiment, unfortunatly docker swarm does not read them natively and it is a fair process to get it to work.

Basic Auth Password

You will need to generate a basic auth password to secure the dashboard until you get something like Authella setup.

apk add apache2-utils
echo $(htpasswd -nb "[USERNAME]" "[PASSWORD]") | sed -e s/\\$/\\$\\$/g

Make sure to copy the [PASSWORD] that was created somewhere safe as you will need it below for the docker compose file.

  • htpasswd -nb "[USERNAME]" "[PASSWORD]": The htpasswd command is used to create and update the flat-files used to store usernames and password for basic authentication of HTTP users. The -nb option tells htpasswd to create a new password and display it on the console rather than saving it to a file. [USERNAME] and [PASSWORD] should be replaced with your actual username and password.

  • $(...): This is command substitution in bash. It runs the command inside the parentheses (in this case htpasswd -nb "[USERNAME]" "[PASSWORD]"), and replaces the $(...) with the output of that command.

  • echo: This command prints its arguments to the standard output (in this case, the output of the htpasswd command).

  • |: This is a pipe. It takes the output from the command on its left (in this case, echo $(htpasswd -nb "[USERNAME]" "[PASSWORD]")) and uses it as the input to the command on its right.

  • sed -e s/\$/\$\$/g: sed is a stream editor for filtering and transforming text. The -e option allows you to pass a script to sed. The script s/\$/\$\$/g is a substitution command that replaces all occurrences (g stands for global) of \$ with \$\$. In other words, it doubles every dollar sign in the input

Make sure to change anything in [ ] while also removing the [ ]. To find your timezone Wikipedia has a list.

docker-compose.yml
services:
  traefik:
    # --  Specifies the exact image to use :latest would always pull the newest version
    image: traefik:v3.0.0
    deploy:
      placement:
        constraints:
        # --  Tells docker what node to run the service/container on to find the name use docker node ls on the manager node
          - node.hostname == [MANAGER_VM_IP_OR_HOSTNAME]
        # --  Enable a restart policy for the service/container
      restart_policy:
        # --  The service/container will always restart if it stops. If it's manually stopped, it will be restarted only when the Docker Daemon restarts or the container itself is manually restarted
        condition: any
        # --  This section will define all the labels needed for traefik to pickup the service/container
      labels:
        # --  Enables Traefik for this service/container
        - "traefik.enable=true"
        # --  Sets the host rule for routing traffic to this service/container. You can change this from traefik, but it will be used for your CNAME entries also so something easy to type and remember is my moto.
        - "traefik.http.routers.traefik.rule=Host(`traefik.[YOUR_FQDN]`)"
        # --  Specifies the internal API service for Traefik (required for the dashboard and API access)
        - "traefik.http.routers.traefik.service=api@internal"
        # --  Define the entry point for the container either web or web secure. All containers are on a perminent redirect through the traefik.yml file
        - "traefik.http.routers.traefik.entrypoints=websecure"
        # --  Enable SSL/TLS
        - "traefik.http.routers.traefik.tls=true"
        # --  Sets the middleware for basic authentication
        - "traefik.http.routers.traefik.middlewares=traefik-auth"
        # --  Defines the users for basic authentication middleware (username:password hash)
        - "traefik.http.middlewares.traefik-auth.basicauth.users=[USERNAME]:[PASSWORD]"
        # --  Specifies the port for the internal Traefik dashboard service, disable this in production!
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
    ports:
      # --  It is mapped as [HOST]:[CONTAINER]
      # --  Exposes port 80 for HTTP traffic
      - "80:80"
      # --  Exposes port 443 for HTTPS traffic
      - "443:443"
    environment:
      # --  Sets the environment variables
      # --  Cloudflare API token for DNS challenge
      - CF_DNS_API_TOKEN=[YOUR_CF_TOKEN]
      # --  Cloudflare account email
      - CF_API_EMAIL=[YOUR_CF_EMAIL]
      # --  Set the timezone for the logs, it is not needed but makes going through the logs a bit easier or else the timestamps are all UTC
      - TZ=[TZ_IDENTIFIER]
    volumes:
      # --  Mounts Docker socket to allow Traefik to communicate with the Docker daemon (read-only access). This is required even though we declair this in the traefik.yml file.
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # --  Mounts the Traefik static configuration file (read-only access)
      - /[VM_NFS_PATH]/configs/traefik.yml:/traefik.yml:ro
      # --  Mounts the directory for certificates and the ACME file (read/write access)
      - /[VM_NFS_PATH]/certs/:/certs:rw
      # --  Mounts the directory for Traefik log files (read/write access)
      - /[VM_NFS_PATH]/logs/:/logs:rw
      # --  Mounts the directory for dynamic configuration files (read/write access)
      - /[VM_NFS_PATH]/rules:/rules:rw
    networks:
      # --  Connects the service/container to the 'backend' network
      - [YOUR_CUSTOM_NETWORK_NAME]
      # --  Enable the built-in docker logging to use log rotation
    logging:
      driver: "json-file"
      options:
      # --  Specifies the maximum size of the log file before it gets rotated (in this example, 10 MB).
        max-size: "10m"
      # --  Specifies the maximum number of log files to keep (in this example, 3).
        max-file: "3"

networks:
  [YOUR_CUSTOM_NETWORK_NAME]:
    # --  Indicates that the network is created outside of this Compose file
    external: true
    # --  Uses the overlay driver for multi-host networking
    driver: overlay

Save the file to the [PATH_TO_TRAEFIK_DOCKER_FOLDER] in the NFS share.

traefik.yml​

Next we need to create our traefik.yml file.

traefik.yml
global:
  checkNewVersion: True
# --   Enable checking for new versions of Traefik.
  sendAnonymousUsage: False
# --   Disable sending anonymous usage statistics.

entryPoints:
  web:
    address: ":80"
# --     Listen on port 80 for HTTP connections.
    http:
      redirections:
        entryPoint:
          to: websecure
# --           Redirect HTTP traffic to HTTPS.
          scheme: https
# --           Use HTTPS scheme for the redirection.
  websecure:
    address: ":443"
# --     Listen on port 443 for HTTPS connections.

log:
  level: DEBUG
# --   Set the log level to DEBUG for detailed logs. You can also use TRACE for even more information in the logs.
  filePath: /logs/traefik.log
# --   Path to the log file for Traefik logs.
  format: common
# --   Log format set to 'common'.

accessLog:
  filePath: /logs/access.log
# --   Path to the log file for access logs.
  format: common
# --   Log format for access logs set to 'common'.

api:
  dashboard: True
# --   Enable the Traefik dashboard.
  debug: True
# --   Enable debug mode for the API.

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
# --     Docker socket for Docker provider.
    exposedByDefault: False
# --     Services are not exposed by default.
  swarm:
    endpoint: "unix:///var/run/docker.sock"
# --     Docker socket for Docker Swarm provider.
    exposedByDefault: False
# --     Services are not exposed by default in Swarm.
  file:
    directory: /rules
# --     Directory for file provider configuration.
    watch: True
# --     Enable watching for changes in the configuration files.

certificatesResolvers:
  cloudflare:
    acme:
      email: [YOUR_CF_EMAIL]
# --       Email address for ACME registration.
      storage: /certs/acme.json
# --       Path to store ACME certificates.
      caServer: https://acme-v02.api.letsencrypt.org/directory
# --       ACME CA server URL.
      dnsChallenge:
        provider: cloudflare
# --         DNS challenge provider.
        disablePropagationCheck: True
# --         Disable DNS propagation check.
        delayBeforeCheck: 60s
# --         Delay before checking DNS propagation.
        resolvers:
          - "1.1.1.1:53"
# --           Cloudflare DNS resolver.
          - "1.0.0.1:53"
# --           Cloudflare DNS resolver.
# --       Staging environment for testing certificates without hitting rate limits, uncomment the below section and comment out the above section while testing.
#  staging:
#    acme:
#      email: [YOUR_CF_EMAIL]
# --       Email address for ACME registration.
#      storage: /certs/acme.json
# --       Path to store ACME certificates for staging.
#      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
# --       Staging ACME CA server URL.
#      dnsChallenge:
#        provider: cloudflare
# --         DNS challenge provider.
#        resolvers:
#          - "1.1.1.1:53"
# --           DNS resolver 1.
#          - "8.8.8.8:53"
# --           DNS resolver 2.

serversTransport:
  insecureSkipVerify: True
# --   Skip verification of server certificates (insecure).

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /certs/cert.pem
# --         Path to the default TLS certificate.
        keyFile: /certs/cert-key.pem
# --         Path to the default TLS certificate key.
  options:
    default:
      minVersion: VersionTLS12
# --       Minimum TLS version to be used (TLS 1.2).

Save the file to the [PATH_TO_TRAEFIK_CONFIG_FOLDER] NFS share. This takes care of the files needed to get traefik up and running.

Starting the traefik service​

Now that we have the files we need created, let's get the service up and running. On the manager node run the following

docker stack deploy -c [PATH_TO_TRAEFIK_COMPOSE_FILE] traefik

  • docker stack deploy: because we are using docker swarm, we have to use the stack deploy even though we are not running a stack

  • -c [PATH_TO_TRAEFIK_COMPOSE_FILE]: is telling docker where the compose.yml file is located

  • traefik: this is the name of the stack

info

When using the below commands to find your service you will need to specifiy the [STACK_NAME]_[SERVICE_NAME]

docker service ps

or on the node the service is running on

docker ps

it will show up as traefik_traefik or [STACK_NAME]_[SERVICE_NAME]

If all has went well you will see no errors in the termnial. To verify that the service is running you can run

docker service ps

and it will show all the services running on the node.

Setting up DNS w/Pi-hole​

  • In your web browser navigate to your Pi-hole and login.
  • Once logged in, on the left find Local DNS, click on that
  • Now click on DNS records
  • Where it says Add a new domain/IP combination
    • Domain: traefik.[YOUR_FQDN]
    • IP Address: [IP_ADDRESS_OF_VM]
    • Click on Add

Now you should see in the list of local DNS domains the new entry. We can now navigate to traefik.[YOUR_FQDN] if everything is working properly you will be presented with a login prompt, use the [USERNAME] and [PASSWORD] that you put in the docker-compose.yml. If none of this is comming up then it is time to troubleshoot the issue. You can start by looking at the traefik.log file with

cat [PATH_TO_TRAEFIK.LOG]

scroll through the output on the screen and look for anything that says disabled or error. Alternativly you can also download the file and use your favourite text editor if that is easier for you.

Congratulations!

traefik proxy v3 should be up and running. Now any docker service you add to an LXC should automatically show up in the dashboard and from there you can add it to your Pi-hole through a CNAME record esentially you will point [SERVICE.FQDN] to [TRAEFIK.FQDN].

traefik/rules directory and manually adding services​

If you've got non-docker services in your homelab and want them to be put under your SSL certificate you will need to make a file for each one and place it in to the traefik/rules directory I prefer just to name it the service that it is. Once the yml file is placed into the directory traefik should automatically pick up the file and start routing it. The example I will use is Ubiquiti Unifi web console.

unifi.yml
http:
  routers:
    unifi-router:
      rule: "Host(`unifi.[FQDN]`)"
      service: unifi-service
      priority: 1000
      entryPoints:
        - web
        # (optional) Permanent Redirect to HTTPS
        - websecure
      tls:
        domains:
          - main: "unifi.[FQDN]"

  services:
    unifi-service:
      loadBalancer:
        servers:
          - url: "https://192.168.1.2"
  • routers: we want traefik to create a route to the [HOST] listed in the rule
  • unifi-router: this is telling traefik we want to create a router named unifi
  • rule: this is where we will specifiy what [FQDN] to use
  • service: we are telling traefik to reference a service called unifi-service (which we are establishing in the services section)
  • **entryPoints: what ports traefik will listen on when it gets unifi.[FQDN]
  • tls: that we want an SSL certificate for the [FQDN] listed below
  • services: this is being referenced by the router and where we will define WHERE traefik can find the service
  • unifi-service: this is the name of the service, and it needs to match what we put in routers--service
  • url: this is the direct IP address of the service/web page/thing you want traefik to direct traefik to
note

Make sure that the names you use in routers and services are the same Ex. unifi-service. Also double check to make sure your domain matches in the TLS and rule sections. Once you upload the file to your server make sure to add a CNAME record in Pi-hole for the service in this case unifi.[FQDN].

Force reloading of the service

If you have made changes to the traefik.yml or your docker-compose.yml file doing another

docker stack deploy -c

might not pickup the changes even though docker is saying it is updating the service. So I highly suggest to run

docker service update --force traefik_traefik
Setup complete

We should now have a functional installation of traefik proxy v3 where it will pick up any new service added to docker, docker, swarm or through files. Remember if you have issues make sure to look at your traefik.log file! If you find you are not getting good enough information from the logs then change the level to TRACE which should put more information in to the log, although keep in mind you will not want to run it at that level for production unless you setup the log rotation (like we have in our docker-compose.yml) or you could end up consuming all of your storage with massive log files from traefik proxy.