One Base Image, Many Servers

Share
One Base Image, Many Servers
One Base Image, Many ServersOne Base Image, Many Servers

I first heard about NixOS from a colleague at a recent conference. What pulled me in was the idea that the whole system is configuration. Packages, services, users, firewall rules, even the bootloader all live in a file you can read, version in Git, and rebuild from scratch. Change the file, run one command, and the machine becomes what the file says.

The more I read about that, the more interesting it was. If a server is just a config file, could I build one clean base, then stamp out a web server, a database, or a Git server by writing only the few lines that make each one different?

This project is me finding out. The goal: Build a NixOS base image on my Proxmox server, and a repeatable way to turn copies of it into whatever server I need next.

The code is all on GitHub at mwgolden/nixos.

The project started the old-fashioned way. I wrote the first pieces by hand: the initial Nix configuration, the build container, the early Proxmox playbooks. Progress was slow. A Sunday morning would go to one stubborn error, and at that pace the whole thing was looking like months of Sunday mornings. So I tried using claude to speed up development and completed in one morning what would have taken a few months.

I still make it a point to understand what's happening in each playbook and guide what the system should do, run every playbook against my own hardware, and ask why until I understand the answers.

The Idea

Adding a new server isn't a new project. It's a new folder. The whole repo hangs on what every server shares, and what makes each one different.

Shared lives in modules/common.nix: the GRUB bootloader, OpenSSH, an admin user that logs in with my deployment key and gets passwordless sudo, and a couple of packages like Python and Git so Ansible has something to work with. Every machine gets exactly this.

Unique configurations live under hosts/. Each host folder has its own configuration.nix that imports the common module and then adds only what that server is for. The webserver config is basically "turn on nginx, serve /var/www, open port 80." The Postgres config turns on PostgreSQL and mounts a separate data disk.

The glue between them is a tiny template, modules/network.nix.j2. Ansible fills in each host's name and static IP from its vars.yaml, so the network setup is written once and stamped out per machine.

There are two stages, and they map cleanly onto that shared/unique split.

Stage one is ./bootstrap.sh. It builds a custom NixOS installer ISO inside a Docker container so I don't need Nix on my workstation, uploads it to Proxmox through the API, and boots a VM into the installer. Ansible then partitions the disk, drops in common.nix, and runs nixos-install. Before converting the VM into a template, the script logs in as admin. Only the installed system has that user, so a successful login proves the VM booted from its own disk and not the ISO.

Stage two is one playbook per server. Each one clones the template, boots the clone, copies in the shared module plus that host's configuration.nix, and runs nixos-rebuild. Building first means any mistake shows up before the machine changes. On switch, the clone moves off the template's IP to its own address.

What I've Built so Far

  1. A webserver was the proof of concept to see if I could clone the template and get a page served.
  2. Postgres: Database files live on a separate disk, so the OS and the data are independent. The mount uses nofail, which keeps the VM bootable if the disk goes missing, and PostgreSQL is told to require that mount before it starts. That way it refuses to run instead of quietly creating a fresh, empty database on the system disk. Growing the disk is a one-line change in vars.yaml and a re-run. Database users and their per-schema access are defined in YAML too. It can also load the AdventureWorks sample database, which gives me realistic data to play with right away.

Read more