Bringing all of my projects together under one umbrella

post, Jul 1, 2023 on Mitja Felicijan's blog

What is the issue anyway?

Over the years, I have accumulated a bunch of virtual servers on my -DigitalOcean account for small experimental -projects I dabble in. And this has resulted in quite a bill. I mean, I wouldn't -care if these projects were actually being used. But there were just being there -unused and wasting resources. Which makes this an unnecessary burden for me.

Most of them are just small HTML pages that have an endpoint or two to read data -from or to, and for that reason I wrote servers left and right. To be honest, -all of those things could have been done with CGI -scripts and that would -have been more than enough.

Recently, I decided to stop language hopping and focus on a simpler stack which -includes C, Go and Lua. And I can accomplish all the things I am interested in.

Finding a web server replacement

Usually I had Nginx in front of these small web servers -and I had to manage SSL certificates and all that jazz. I am bored with these -things. I don't want to manage any of this bullshit anymore.

So the logical move forward was to find a solid alternative for this. I have -ended up on Caddy server. I've used it in the past -but kind of forgotten about it. What I really like about it is an ease of use -and a bunch of out of the box functionalities that come with it.

These are the pitch points from their website:

  • Secure by Default: Caddy is the only web server that uses HTTPS by -default. A hardened TLS stack with modern protocols preserves privacy and -exposes MITM attacks.
  • Config API: As its primary mode of configuration, Caddy's REST API makes -it easy to automate and integrate with your apps.
  • No Dependencies: Because Caddy is written in Go, its binaries are entirely -self-contained and run on every platform, including containers without libc.
  • Modular Stack: Take back control over your compute edge. Caddy can be -extended with everything you need using plugins.

I had just a few requirements:

  • Automatic SSL
  • Static file server
  • Basic authentication
  • CGI script support

And the vanilla version does all of it, but CGI scripts. But that can easily be -fixed with their modular approach. You can do this on their website and build a -custom version of the server, or do it with Docker.

This is a Dockerfile I used to build a custom server.

FROM caddy:builder AS builder
-
-RUN xcaddy build \
-  --with github.com/aksdb/caddy-cgi
-
-FROM caddy:latest
-RUN apk add --no-cache nano
-
-COPY --from=builder /usr/bin/caddy /usr/bin/caddy
-

Getting rid of all the unnecessary virtual machines

The next step was to get a handle on the number of virtual servers I have all -over the place.

I decided to move all the projects and services into two main VMs:

  • personal server (still Nginx)
    • git server
    • static file server
    • personal blog
  • projects server (Caddy server)
    • personal experiments
    • other projects

I will focus on projects' server in this post since it's more interesting.

Testing CGI scripts

The first thing I tested was how CGI scripts work under Caddy. This is -particularly import to me because almost all of my experiments and mini projects -need this to work.

To configure Caddy server, you must provide the server with a configuration -file. By default, it's called Caaddyfile.

{
-  order cgi before respond
-}
-
-examples.mitjafelicijan.com {
-  cgi /bash-test /opt/projects/examples/bash-test.sh
-  cgi /tcl-test /opt/projects/examples/tcl-test.tcl
-  cgi /lua-test /opt/projects/examples/lua-test.lua
-  cgi /python-test /opt/projects/examples/python-test.py
-
-  root * /opt/projects/examples
-  file_server
-}
-
  • The order is very important. Make sure that order cgi before respond is at -the top of the configuration file.
  • Also, when you run with Caddy v2, make sure you provide adapter argument -like this /usr/bin/caddy run --watch --environ --config /etc/caddy/Caddyfile --adapter caddyfile. Otherwise, Caddy will try to use a different format for -config file.

I did a small batch of tests with Bash, -Tcl, Lua and -Python. Here is a cheat sheet if you need it.

Let's get Bash out of the way first.

#!/usr/bin/bash
-
-printf "Content-type: text/plain\n\n"
-
-printf "Hello from Bash\n\n"
-printf "PATH_INFO     [%s]\n" $PATH_INFO
-printf "QUERY_STRING  [%s]\n" $QUERY_STRING
-printf "\n"
-
-for i in {0..9..1}; do
-  printf "> %s\n" $i
-done
-
-exit 0
-

This one is for Tcl script.

#!/usr/bin/tclsh
-
-puts "Content-type: text/plain\n"
-
-puts "Hello from Tcl\n"
-puts "PATH_INFO     \[$env(PATH_INFO)\]"
-puts "QUERY_STRING  \[$env(QUERY_STRING)\]"
-puts ""
-
-for {set i 0} {$i < 10} {incr i} {
-  puts "> $i"
-}
-

And for all you Python enjoyers.

#!/usr/bin/python3
-
-import os
-
-print("Content-type: text/plain\n")
-
-print("Hello from Python\n")
-print("PATH_INFO     [{}]".format(os.environ['PATH_INFO']))
-print("QUERY_STRING  [{}]".format(os.environ['QUERY_STRING']))
-print("")
-
-for i in range(10):
-  print("> {}".format(i))
-

And for the final example, Lua.

#!/usr/bin/lua
-
-print("Content-type: text/plain\n")
-
-print("Hello from Lua\n")
-print(string.format("PATH_INFO     [%s]", os.getenv("PATH_INFO")))
-print(string.format("QUERY_STRING  [%s]", os.getenv("QUERY_STRING")))
-print()
-
-for i = 0, 9 do
-  print(string.format("> %d", i))
-end
-

Basic authentication

One thing was also to have an option for some sort of authentication, and -something like Basic access -authentication would -be more than enough.

Thankfully, Caddy supports this out of the box already. Below is an updated -example.

{
-    order cgi before respond
-}
-
-examples.mitjafelicijan.com {
-  cgi /bash-test /opt/projects/examples/bash-test.sh
-  cgi /tcl-test /opt/projects/examples/tcl-test.tcl
-  cgi /lua-test /opt/projects/examples/lua-test.lua
-  cgi /python-test /opt/projects/examples/python-test.py
-
-  root * /opt/projects/examples
-  file_server
-
-  basicauth * {
-    bob $2a$14$/wCgaf9oMnmQa20txB76u.nI1AldGMBT/1J7fXCfgOiRShwz/JOkK
-  }
-}
-

basicauth * matches everything under this domain/sub-domain and protects it -with Basic Authentication.

  • bob is the username
  • hash is the password

To generate these passwords, execute caddy hash-password and this will prompt -you to insert a password twice and spit out a hashed password that you can put -in your configuration file.

Restart the server and you are ready to go.

Making Caddy a service with systemd

After the tests were successful, I copied caddy to /usr/bin/caddy and copied -Caddyfile to /etc/caddy/Caddyfile.

Now off to the systemd. Each systemd service requires you to create a service -file.

  • I created a /etc/systemd/system/caddy.service and put the following content -in the file.
[Unit]
-Description=Caddy
-Documentation=https://caddyserver.com/docs/
-After=network.target network-online.target
-Requires=network-online.target
-
-[Service]
-Type=notify
-User=root
-Group=root
-ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile --adapter caddyfile
-ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force --adapter caddyfile
-TimeoutStopSec=5s
-LimitNOFILE=1048576
-LimitNPROC=512
-PrivateTmp=true
-ProtectSystem=full
-AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
-
-[Install]
-WantedBy=multi-user.target
-
  • You might need to reload systemd with systemctl daemon-reload.
  • Then I enabled the service with systemctl enable caddy.service.
  • And then I started the service with systemctl start caddy.service.

This was about all that I needed to do to get it running. Now I can easily add -new subdomains and domains to the main configuration file and be done with -it. No manual Let's Encrypt shenanigans needed.