Traefik, lighttpd, and serving static sites in k8s

Published on 2020-08-06 by rkevin

Hey everyone! A quick update on my setup. This is going to be less exploratory and more me just dumping how I have set things up, but hopefully this is still interesting / useful content.

tl;dr

I have setup Traefik as my ingress controller, some defaults to make it use the HTTPS certificate we get from cert-manager in an earlier blog post, and lighttpd backends to serve my static sites. It's currently in production, so you're reading it right now! (A moment of silence to my standard Apache webserver on the first Raspberry Pi I've owned. You have served me well in the past several years.)

Setting up Traefik

I initially had a completely working setup using Traefik 1.7, which comes preinstalled with k3s. However, I eventually decided I would use some Traefik 2 features in the future, so I ripped it out and started fresh. That process was actually pretty painful, since I don't want to reinstall my k3s cluster again. I had to:

• Add --disable traefik to /etc/systemd/system/k3s.service's ExecStart on the control plane node
• Delete traefik.yaml from /var/lib/rancher/k3s/server/manifests
• Delete the HelmChart resource traefik from the kube-system namespace using kubectl (Don't bother trying to delete the pods or the job it spawns) Afterwards, I installed Traefik using the official helm chart from containous (not the one in stable) using the following values:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

ingressRoute:
  dashboard:
    enabled: false
ports:
  web:
  websecure:
  https:
    port: 8443
    expose: true
    exposedPort: 8443
    protocol: TCP
  http:
    port: 8000
    expose: true
    exposedPort: 8000
    protocol: TCP
additionalArguments:
  - --entryPoints.https.http.tls

I just didn't like the name web and websecure so I replaced it with http and https. Purely personal preference. I also added the additionalArguments section to tell Traefik to use TLS on the https endpoint. To make it use the certificate, I created a TLSStore object:

1
2
3
4
5
6
7
8
9

apiVersion: traefik.containo.us/v1alpha1
kind: TLSStore
metadata:
  name: default
  namespace: https-cert

spec:
  defaultCertificate:
    secretName: traefik-default-cert

The traefik-default-cert secret is managed by cert-manager. This sets it as the default for all TLS requests, which is useful since I don't want to write all that in my config files. That's mostly it for the Traefik setup, other than finding what port did k8s schedule the LoadBalancer service to be exposed on, and setting up the right port forwarding on my router. Of course, I skipped over a ton of debugging, most notably in the additionalArguments section when I was trying to get TLS and HTTPS redirection to work.

Static site servers

I eventually decided to use lighttpd to serve my static sites. They're small, lightweight, and still configurable enough for my needs. I would have one pod per static site, which amounts to around a dozen pods for all the virtual hosts I'm running.

I made a general purpose lighttpd image:

1
2
3
4
5
6

FROM arm32v7/alpine:3.12.0
RUN apk add --no-cache lighttpd
COPY lighttpd.conf /etc/lighttpd/lighttpd.conf
EXPOSE 8080
USER 65535:65535
ENTRYPOINT ["/usr/sbin/lighttpd", "-D", "-f", "/etc/lighttpd/lighttpd.conf"]

The corresponding lighttpd.conf is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

server.document-root = "/var/www/html/"
server.port = 8080
server.modules = (
    "mod_deflate",
    "mod_accesslog"
)
include "mime-types.conf"

server.error-handler-404 = "/404.html"
accesslog.filename = "/dev/stdout"

index-file.names = ( "index.html" )

Funnily enough, if you use server.error-handler-404 and it's a static file, lighttpd would return 200 instead. This was one of the things I would've liked, and I actually setup my original Apache server to use a 404.php which has a <?php header("HTTP/1.1 200 OK"); ?> specifically to have this behavior. This is to throw anyone dirbusting the website into a loop and confuse the heck out of them, and it's actually pretty funny when looking at the logs. I thought I would have to pick some sort of webserver that allows me to rewrite HTTP response codes and is still efficient and only serves static sites, but apparantly lighttpd just does it. Awesome. If you don't want this behavior though (like on some of my virtual hosts that needs 404 to work for API purposes), you can use server.errorfile-prefix = "/var/www/html/" instead, and it will use 404.html in /var/www/html/ for the 404 page.

I built this image, tagged it as registry.rkevin.dev/static-site-server:0.2, and pushed it onto my private registry. I also built a couple of other images for some sites that have specific needs, but most static sites (including this one) can just use this general purpose image and mount something in /var/www/html.

k8s deployment/ingress

I eventually used an Ingress rather than Traefik's preferred IngressRoute for a) simplicity reasons, and b) I got this to (sort of) work with Traefik 1.7 before I switched to 2.2, so I was too lazy to change everything. Anyway, here's the complete yaml that declares the lighttpd deployment, k8s service, and the ingress to let Traefik forward stuff to the pod:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-blog
  namespace: static-sites
spec:
  replicas: 1
  selector:
    matchLabels:
      static-site: blog
  template:
    metadata:
      labels:
        static-site: blog
    spec:
      securityContext:
        runAsNonRoot: true
      containers:
        - name: lighttpd
          image: registry.rkevin.dev/static-site-server:0.2
          volumeMounts:
            - name: html
              mountPath: /var/www/html
          ports:
            - name: http
              containerPort: 8080
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
          securityContext:
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            runAsUser: 65535
      volumes:
        - name: html
          nfs:
            server: wavelength.lan
            path: /srv/synchrotron/static-sites/blog
            readOnly: true
---
apiVersion: v1
kind: Service
metadata:
  name: service-blog
  namespace: static-sites
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 8080
      targetPort: http
  selector:
    static-site: blog

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-blog
  namespace: static-sites
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: https-cert-httpsredir@kubernetescrd
spec:
  rules:
    - host: rkevin.dev
      http:
        paths:
          - backend:
              serviceName: service-blog
              servicePort: http

It is sort of a lot of yaml, but I can just copy this yaml, do a sed s/blog/newsite/, change the hostname in the Ingress object, and call it a day. I actually think this way of managing things is slightly better than the 30+ Apache configuration files I had before, mostly cuz I had to edit multiple files to get HTTP->HTTPS redirection working for each virtual host.

The deployment has a selector to tell the service what blog it is serving, and it also has some liveness and readiness probes to tell k8s when the service is considered healthy. There's also a NFS volume mount, so I can put files into the server without rebuilding the image. The same NFS server is used to provision PersistentVolumeClaims, but I eventually decided against using PVCs and just mounted a static path so I can mount the same predictable path on my laptop and copy files directly.

The Service object is pretty predictable, a standard k8s service that selects all pods (well, only one, but I guess I can scale it if I really cared about HA) that match the labels. The Ingress tells Traefik to route what hosts to what k8s service.

By the way, I also added a middleware using a Middleware object as follows:

1
2
3
4
5
6
7
8
9

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: httpsredir
  namespace: https-cert
spec:
  redirectScheme:
    scheme: https
    permanent: true

I can then use the traefik.ingress.kubernetes.io/router.middlewares annotation to set it on virtual hosts that need SSL redirection. I experimented with setting it globally as command line arguments to Traefik, but some of my virtual hosts require HTTP to work and not HTTPS, and I couldn't get it to work.

On a side note, Traefik's documentation sucks when it comes to examples of arguments. It took me a very long time of searching to find the resource@provider syntax and it's at a non-obvious location (I would think this would be under the providers section, but nope). There are also no examples on what the provider name is (I know it's the Kubernetes CRD provider, but what is it called internally? @crd? @kubernetes? @kubernetes-crd? we may never know), or how to reference namespaced k8s objects as the resource (there is a note about the reverse, please don't include the k8s namespace when referencing objects out of k8s, but what if I want to reference an object in the namespace? Do I do namespace/object@kubernetescrd? Or object@namespace@kubernetescrd?). Eventually I found the solution by literally digging through Traefik's source code on Github. Either way, it works now.

Of course, this isn't the only configuration I have (for example, for the virtual hosts that must only use HTTP and not HTTPS I had to use the annotations traefik.ingress.kubernetes.io/router.entrypoints: http and traefik.ingress.kubernetes.io/router.tls: "false", and I used a different lighttpd image to serve a site that must have some reverse proxied stuff to an external server (lighttpd supports that!), but I won't explain how I set those up in detail.

Now that I have a proper ingress, I can add more services fairly easily. For example, the private registry I'm running is literally just the stable/docker-registry helm chart, and it has an option to add an ingress automatically, which exposes it automatically with zero fuss about reverse proxying or manual Traefik configuration. Cool!

With that in place, I'll move on to setting other stuff up. Candidates include a VPN server that's behind port 443 and routes using SNI (Traefik supports this) for maximum sneak (well, slightly less than maximum since I mentioned it on this blog on the same port that I intend to serve it from, but it should bypass the filtering that I care about), or porting my entire ECS189M setup to k8s (photon was down for quite some time due to a failing power supply, and it would definitely be interesting to try and port everything to be k8s native). Until then, take care.