Howdy!

I’ve had a bit of spare time and decided to spin up a simple site, using Hugo. I don’t have a ton of detail to talk about today, but I wanted to get some kind of writing ground up so I have somewhere to dump my senseless ramblings in the future.

Some of this will end up being a repeat of the ‘About’ page in the nav of this page. Worst-case, this will end up being a documentation of my own thoughts when setting something up, so I can figure out what the hell I was thinking at the time.

So I can have some content here, let’s talk about what I had to do to get this site rolled out. As a developer with some interest in systems, I love to hear about other setups and how people design their operations, so maybe someone out there will be interested to see how I do it.

I’ve wanted to have a location for write-ups and ramblings for a while and kept putting off rolling out a service for it. I didn’t want the nightmare of managing something like WordPress or equivalent, since their footprint is morbidly obese in terms of what I actually want to do here.

Enter Hugo, a very simple static site generator written in Go. Now, I don’t have a particular attachment to Go like everyone else seems to, but I can appreciate fast and lean tech which requires little maintenance. There’s a lot of benefits here: for one, it’s static pages. You really can’t beat a static page when you want to throw words on a screen. I’ve got bigger fish to fry and I can’t be bothered to update WordPress every week because some plugin had yet another CVE disclosed.

This isn’t really intended to be a how-to on Hugo, but it might help you get on the road.

First, I had no idea where to start with Hugo. Fortunately, starting a new site is very easy (once you’ve installed the application).

hugo new site

and suddenly my current directory was filled with random files that meant little to me.

I took the configuration file (config.toml) that was recommended by my choice of theme (Hello Friend, see the footer of this page). That made the whole process significantly easier, since most of the work was done for me. Your theme will give you appropriate instruction for how to utilize it, but I added my theme as a git submodule based on their directions. If you want to do something similar to the “About” button above, you’ll need to add it to your config. However, it directs you right to a Markdown file in the ~/content directory, so you should be ready to start writing up something at that point.

By now, I wanted to see the site before I committed. While hugo is a static page generator, it does have a bundled web server.

hugo server

There’s some options you can use with this command, but I defined everything in my config.

At this point, I wanted to start writing something since it wouldn’t take long for me to forget to do so. Hugo does prepend a little header to each Markdown file which populates some of the metadata for the post (read: title, date). You can honestly write this yourself if you really wanted, but you could also do

hugo new posts/<title>.md

and it’ll create an empty markdown file with the header prepended for you. One nice feature of the local server is that it will refresh when you change the file structure at all. Creating, deleting, or updating content will immediately reflect the changes, which is handy to see everything grow in front of you.

Next, I built my pipeline for this. This site is hosted on my own servers, through a reverse proxy instance. I use Jenkins, so I first built my Jenkinsfile for a multibranch pipeline. Sometimes, I will package the image and the deployment separately, but I’m rolling this into one pipeline as it’s simpler. This is practically a template that I use for all of my other pipelines, so there’s not that much special here.

pipeline {
    agent any

    options {
        disableConcurrentBuilds()
    }

    environment {
        PROJECT_NAME = "blog_lildirt_com"
        STACK_NAME = "blog"
        GIT_COMMIT_SHORT = sh(script: "git rev-parse --short ${GIT_COMMIT}", returnStdout: true).trim()
        VERSION_STRING = "${GIT_COMMIT_SHORT}-${BRANCH_NAME}"

        VERSIONED_IMAGE_NAME = "${DOCKER_REGISTRY}/${PROJECT_NAME.toLowerCase()}:${VERSION_STRING}"
        BRANCH_IMAGE_NAME = "${DOCKER_REGISTRY}/${PROJECT_NAME.toLowerCase()}:${BRANCH_NAME}"
    }

    stages {
        stage('Prepare') {
            steps {
                checkout scm
            }
        }
        stage('Obtain External Resources') {
            steps {
                sh "echo 'Nothing to do.'"
            }
        }
        stage('Update Build Version') {
            steps {
                buildName "${VERSION_STRING}"
            }
        }
        stage('Build Image') {
            steps {
                sh "docker build --no-cache --pull -t ${VERSIONED_IMAGE_NAME} --file=Dockerfile ."
            }
        }
        stage('Push to Registry') {
            steps {
                sh "docker image tag ${VERSIONED_IMAGE_NAME} ${BRANCH_IMAGE_NAME}"
                sh "docker push ${VERSIONED_IMAGE_NAME}"
                sh "docker push ${BRANCH_IMAGE_NAME}"
            }
        }
        stage('Deploy') {
            when {
                branch 'master'
            }
            steps {
                sh "docker stack deploy ${STACK_NAME} --compose-file=docker-stack.yml"
            }
        }

    }
}

The idea behind this template is to make copy-pasting it around much easier. Copy the file from elsewhere, redefine some variables, check it in, and you’re off to the races. I’m using Docker Swarm since my scale isn’t sufficient for Kubernetes. This template is probably going to change soon, but it’s what operates this site at the moment of writing.

Basically what you’re seeing above is the process of building the image, pushing it to my private registry, then deploying the Docker Stack. The Stack and Dockerfile aren’t anything special (just a webserver with the generated Hugo files merged into it), so I won’t be including them.

So, during the steps of this pipeline, we will:

  • go grab the source from git
  • build the Dockerfile in this repository (which is using the public klakegg/hugo image based on Alpine and just copying the files into it)
  • push that completed image to the registry
  • re-deploy the stack and updating the image.

You might notice that I use a versioned image tag and a branch image tag. Effectively, this is so that I can use master instead of latest so I know I will always be using the stable version of something. Is this a great policy? Probably not. I still can’t determine what version of the image I’m running, but that’s less of a problem for a deployment like this.

There’s a couple of other things I have to do that you won’t see. Most notably, creating the Jenkins job, registering a token for Gitea to use as an on-push webhook, and registering that webhook URL in Gitea itself. I would have included images here, but honestly I didn’t feel like blurring out the contents since it has secrets involved.

Voila! The pipeline is ready to go and you’re looking at the finished product.

Just kidding, there’s more. I have a reverse proxy between clients and the actual backing site, for a few different reasons. Furthermore, we haven’t even talked about TLS. Do we actually need TLS for this site? Maybe not. However, I’d prefer we make use of it.

Let’s start by configuring the reverse proxy. I use Ansible to manage my reverse proxy server, so we’ll need to refer to my Ansible configuration for that server.

Image: Ansible directory listing

By now, this is a fairly established process for me (thanks to Ansible). There’s a role I’ve created in this playbook named apache which is devoted to managing, well, Apache httpd. The sites directory contains a bunch of site configurations for virtual hosts, so let’s add a new one and save it to that directory. All of these files look practically identical, barring some site-specific exceptions here and there.

Image: New VirtualHost config

I’ve omitted the IP and port here, so if you’re copying this word-for-word, be aware of that. I can touch on a few things I’m doing here, though I won’t go to in-depth of the basics of virtual hosts and Apache config.

SSLProxyEngine and the related Proxy* directives are enabling the reverse proxy aspect of this virtual host. As I said, you’re running through a proxy and not connecting to this site directly. Additionally, I am using TLS from both the client to the proxy to the backend server, as all of this traffic is going over the Internet.

There’s some other SSLProxy* directives which are effectively saying “ignore bad certificates on the backend.” This is a bit problematic (and is a problem I am researching a solution for), since we’re blindly trusting whatever the backend server has. I’m confident I’ll keep ownership of the backend network, so I’m not worried about correcting this for the meantime. Ultimately, this is happening because we are using self-signed certificates. I don’t have a means to get verified certificates refreshing on the backend yet and the pressure hasn’t been too high to fix it. Eventually, I’d like to fix this but it’s a significant infrastructure investment to get a clear route to an ACME endpoint.

The next directive block, SSLEngine, SSLCertificate*, is simply defining the TLS configuration for this server that you’re on now. This is your standard Let’s Encrypt ACME setup, with auto-renewing certificates and all the usual things. This is pretty easy to maintain, since I’m generating certificates on the server that they will be used on.

Finally, RequestHeader directives are just including some HTTP headers for the downstream applications. This is mostly used for keeping an eye on things for my own sake.

OK, now that all that’s out of the way, let’s talk about rolling this out. I’ve made my changes, did a test deployment in a VM, and am ready to roll them out. There’s a couple small tasks I need to tackle, first being updating the DNS entry. No big deal; I’ll be doing this through my DNS provider (Namecheap).

Image: DNS configuration

Let’s just check that I can resolve it, then move forward.

Image: DNS ping test

Great! Now, the other task is to update my Let’s Encrypt certificate. It’s not every day that I add a new domain, but I need to add a common name to this certificate for the new domain (blog.lildirt.com). This is always a road bump for me, since I have to do it somewhat infrequently that I end up forgetting how to handle it. Probably something I need to write in documentation soon, or have some automated means for handling it.

Anyway, I have to effectively request a new certificate from Let’s Encrypt for something like this.

certbot certonly -d "proxy.lildirt.com, ... ,blog.lildirt.com, ..."

Image: Let&rsquo;s Encrypt update

Fantastic. The certificate is updated. I went and verified this through my browser, so I’m good to go. I can install a virtual host here and carry on with rolling this out. Let’s run that playbook I was talking about before.

Image: Ansible output

The virtual host is now live! So, up to this point we have:

  • Created a Hugo site.
  • Made a Docker image to run it in.
  • Written a Stack to deploy it with.
  • Added a Jenkins pipeline to process the Dockerfile into an image and deploy the stack.
  • Created a DNS record for the new blog.lildirt.com domain.
  • Added that domain to my existing reverse proxy setup.
  • Added that domain to my existing Let’s Encrypt certificate common names.

That’s still not everything though. We’ve got the service running and we have the edge service configured, but we haven’t actually made a route from the proxy server to the backend server yet, nor have we configured TLS for the backend server. The backend server is firewalled off, so we’ll need to punch a hole through so it can be accessed, too.

Getting TLS working is fairly straightforward. We’ll need to provision some certificates and feed them to the Hugo container. We’ll include the certificate and key file as secrets in the deployment and mount them appropriately. Ultimately, we’re just running Apache on the backend here to serve the static files, so dropping in the certificates and static files is all we need to do.

Finally, let’s talk about that firewall. This really is the last step involved, then we’re done. I configured the virtual host to use a specific port, so we’re going to create a NAT rule for that specific port.

Image: Firewall rule

Hooray! The backend server should now be reachable. It’s only accessible by this server, so I don’t have to worry about anyone else trying to reach the backend server. Perhaps at some point, I’ll go over my strife with container networking and getting applications routed from the Internet, through a firewall, to the container. In my case, the networking involved is a bit complex, but that’s out-of-scope for this (already lengthy) article.

Summary

Well, that’s it. That’s the full lifecycle of introducing a deployment to my home cluster. It’s not too complicated once you understand all the moving pieces in it. Overall, I’m relatively happy with my ingress for self-hosted sites.

If you’ve actually read all of this, that’s great! I hope you found it helpful or at least somewhat enlightening. I hope it can improve your setup, if you’re struggling with something I talked about here. This is just how I handle ingress from the Internet, and there’s more complexity behind the scenes, but this is what I have to go through to expose something to the Internet.

Most of the resources I’ve mentioned here aren’t hosted in a public repo, so you’ll have to manage them on your own. None of it is terribly exotic though.

I’m not sure how much I’ll end up posting here, though it shouldn’t go down anytime soon since it’s now deployed. I’ll mostly use this site for rambling about technical topics or games. I’d like to use this as a thinking outlet, so I can summarize what’s going through my head with whatever technical task I’m working on at the moment. Stay tuned, hopefully.