Traefik, lighttpd, and serving static sites in k8s
Hey everyone! A quick update on my setup. This is going to be less exploratory and more me just dumping how I have set things up, but hopefully this is still interesting / useful content.
I have setup Traefik as my ingress controller, some defaults to make it use the HTTPS certificate we get from
cert-manager in an earlier blog post, and lighttpd backends to serve my static sites. It's currently in production, so you're reading it right now! (A moment of silence to my standard Apache webserver on the first Raspberry Pi I've owned. You have served me well in the past several years.)
Setting up Traefik
I initially had a completely working setup using Traefik 1.7, which comes preinstalled with k3s. However, I eventually decided I would use some Traefik 2 features in the future, so I ripped it out and started fresh. That process was actually pretty painful, since I don't want to reinstall my k3s cluster again. I had to:
ExecStarton the control plane node
- Delete the
kubectl(Don't bother trying to delete the pods or the job it spawns) Afterwards, I installed Traefik using the official helm chart from containous (not the one in stable) using the following values:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
I just didn't like the name
websecure so I replaced it with
https. Purely personal preference. I also added the
additionalArguments section to tell Traefik to use TLS on the https endpoint. To make it use the certificate, I created a
1 2 3 4 5 6 7 8 9
traefik-default-cert secret is managed by cert-manager. This sets it as the default for all TLS requests, which is useful since I don't want to write all that in my config files. That's mostly it for the Traefik setup, other than finding what port did k8s schedule the
LoadBalancer service to be exposed on, and setting up the right port forwarding on my router. Of course, I skipped over a ton of debugging, most notably in the
additionalArguments section when I was trying to get TLS and HTTPS redirection to work.
Static site servers
I eventually decided to use
lighttpd to serve my static sites. They're small, lightweight, and still configurable enough for my needs. I would have one pod per static site, which amounts to around a dozen pods for all the virtual hosts I'm running.
I made a general purpose
1 2 3 4 5 6
lighttpd.conf is as follows:
1 2 3 4 5 6 7 8 9 10 11 12
server.document-root = "/var/www/html/"
Funnily enough, if you use
server.error-handler-404 and it's a static file,
lighttpd would return 200 instead. This was one of the things I would've liked, and I actually setup my original Apache server to use a
404.php which has a
<?php header("HTTP/1.1 200 OK"); ?> specifically to have this behavior. This is to throw anyone dirbusting the website into a loop and confuse the heck out of them, and it's actually pretty funny when looking at the logs. I thought I would have to pick some sort of webserver that allows me to rewrite HTTP response codes and is still efficient and only serves static sites, but apparantly
lighttpd just does it. Awesome. If you don't want this behavior though (like on some of my virtual hosts that needs 404 to work for API purposes), you can use
server.errorfile-prefix = "/var/www/html/" instead, and it will use
/var/www/html/ for the 404 page.
I built this image, tagged it as
registry.rkevin.dev/static-site-server:0.2, and pushed it onto my private registry. I also built a couple of other images for some sites that have specific needs, but most static sites (including this one) can just use this general purpose image and mount something in
I eventually used an
Ingress rather than Traefik's preferred
IngressRoute for a) simplicity reasons, and b) I got this to (sort of) work with Traefik 1.7 before I switched to 2.2, so I was too lazy to change everything. Anyway, here's the complete yaml that declares the lighttpd deployment, k8s service, and the ingress to let Traefik forward stuff to the pod:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
It is sort of a lot of yaml, but I can just copy this yaml, do a
sed s/blog/newsite/, change the hostname in the Ingress object, and call it a day. I actually think this way of managing things is slightly better than the 30+ Apache configuration files I had before, mostly cuz I had to edit multiple files to get HTTP->HTTPS redirection working for each virtual host.
The deployment has a selector to tell the service what blog it is serving, and it also has some liveness and readiness probes to tell k8s when the service is considered healthy. There's also a NFS volume mount, so I can put files into the server without rebuilding the image. The same NFS server is used to provision PersistentVolumeClaims, but I eventually decided against using PVCs and just mounted a static path so I can mount the same predictable path on my laptop and copy files directly.
The Service object is pretty predictable, a standard k8s service that selects all pods (well, only one, but I guess I can scale it if I really cared about HA) that match the labels. The Ingress tells Traefik to route what hosts to what k8s service.
By the way, I also added a middleware using a
Middleware object as follows:
1 2 3 4 5 6 7 8 9
I can then use the
traefik.ingress.kubernetes.io/router.middlewares annotation to set it on virtual hosts that need SSL redirection. I experimented with setting it globally as command line arguments to Traefik, but some of my virtual hosts require HTTP to work and not HTTPS, and I couldn't get it to work.
On a side note, Traefik's documentation sucks when it comes to examples of arguments. It took me a very long time of searching to find the
resource@provider syntax and it's at a non-obvious location (I would think this would be under the providers section, but nope). There are also no examples on what the provider name is (I know it's the Kubernetes CRD provider, but what is it called internally?
@kubernetes-crd? we may never know), or how to reference namespaced k8s objects as the resource (there is a note about the reverse, please don't include the k8s namespace when referencing objects out of k8s, but what if I want to reference an object in the namespace? Do I do
object@namespace@kubernetescrd?). Eventually I found the solution by literally digging through Traefik's source code on Github. Either way, it works now.
Of course, this isn't the only configuration I have (for example, for the virtual hosts that must only use HTTP and not HTTPS I had to use the annotations
traefik.ingress.kubernetes.io/router.entrypoints: http and
traefik.ingress.kubernetes.io/router.tls: "false", and I used a different lighttpd image to serve a site that must have some reverse proxied stuff to an external server (lighttpd supports that!), but I won't explain how I set those up in detail.
Now that I have a proper ingress, I can add more services fairly easily. For example, the private registry I'm running is literally just the
stable/docker-registry helm chart, and it has an option to add an ingress automatically, which exposes it automatically with zero fuss about reverse proxying or manual Traefik configuration. Cool!
With that in place, I'll move on to setting other stuff up. Candidates include a VPN server that's behind port 443 and routes using SNI (Traefik supports this) for maximum sneak (well, slightly less than maximum since I mentioned it on this blog on the same port that I intend to serve it from, but it should bypass the filtering that I care about), or porting my entire ECS189M setup to k8s (photon was down for quite some time due to a failing power supply, and it would definitely be interesting to try and port everything to be k8s native). Until then, take care.