Introduction

Documentation for my homelab.

NixOS Server Configs

The nixos/ directory contains the system definitions for all of my NixOS servers.

To reload the config, see here.

Directory Layout

NixOS is configured using the den framework.

Testing

Automated tests can be ran via nix flake check.

To spin up an interactive VM for a check, use nix run .#checks.x86_64-linux.<CHECK_NAME>.driverInteractive.

Check Architecture

Checks should be located close to the implementation, in the same file if possible.

Checks for hosts should be in their host file (modules/hosts/<HOST_NAME>.nix).

Networking

Hickory DNS

The DNS server is Hickory; see nixos/services/hickory-dns.nix for details.

Legacy Networking

Networking config is defined at the host level (i.e. in hosts/<HOSTNAME>) Eventually, this code duplication could be moved into base/.

Importantly, Both hostXYZ entries under networking are changed per-host. Additionally, the link name and IP address in matchConfig.Name and networkConfig.Address respectively also need to be changed per-host.

{...}: {
  imports = [
    ../../base/configuration.nix
    ./disks.nix
  ];

  networking = {
    hostName = "ophiuchus";
    hostId = "e7ea22a6"; # `head -c4 /dev/urandom | od -A none -t x4`
  };

  systemd.network = {
    enable = true;
    netdevs = {
      "10-vlan9" = {
        netdevConfig = {
          Name = "vlan9";
          Kind = "vlan";
        };
        vlanConfig.Id = 9;
      };
    };
    networks = {
      "10-enp0s13f0u1" = {
        matchConfig.Name = "enp0s13f0u1";
        vlan = ["vlan9"];
        networkConfig.LinkLocalAddressing = "no";
        linkConfig.RequiredForOnline = "carrier";
      };

      "10-vlan9" = {
        matchConfig.Name = "vlan9";
        gateway = ["10.0.9.1"];
        networkConfig = {
          Address = "10.0.9.104/24";
        };
      };
    };
  };
}

GitOps

The machine configs are synced every minute using comin.

To see the status:

• On machine, use comin status
• Metrics are exported on <HOST_IP>:4343/metrics

Rebuilding

Wait at most a minute for comin to pull and start a rebuild, or:

sudo nixos-rebuild switch --flake /var/lib/comin/repository/nixos#(hostname)

Note: comin can be paused using systemctl stop comin.

Testing

To test on one machine, ensure branches.testing.name is unset and push changes to a branch named testing-<HOSTNAME>.

To test changes on all machines, set branches.tesing.name to the name of the testing branch.

{...}: {
  services.comin = {
    enable = true;
    flakeSubdirectory = "nixos";
    exporter = {
      openFirewall = true;
      port = 4243;
    };
    remotes = [
      {
        name = "origin";
        url = "https://github.com/KGB33/homelab.git";
        branches.main.name = "main";
        branches.testing.name = "nixos-is-the-new-proxmox";
      }
    ];
  };
}

Secrets

Nix Secrets are managed by sops-nix.

Create a secret in-repo using sops host/<HOSTNAME>/<SERVICE_NAME>Secret.[env/yaml/etc].

Import it into the config via:

{...}: {
sops = {
    secrets = {
      "SERVICE_NAME" = {
        sopsFile = ./SERVICE_NAME_Secrets.env;
        format = "dotenv";
      };
    };
  };
};

The private key must also be on the machine.

scp ~/.config/sops/age/keys.txt $HOSTNAME:~/.config/sops/age/keys.txt

Uptime Kuma

uptime.kgb33.dev

An off-site uptime monitoring solution hosted on Fly.io.

Scripts to deploy to both AWS and Fly.io exist in the repo; However, due to cost, Uptime Kuma is only deployed to Fly.io. AWS documentation and Scrips are kept to demonstrate AWS experience on a resume.

Cloudflare Rules

Cloudflare (occasionally) tries to block this bot. To prevent this, add a new “Configuration Rule” with a custom filter expression where the IP source matches the Fly.io IPv4 or IPv6 address assigned to the machine. This rule turns off the Browser integrity check, and sets the Security Level to “Essentially Off”.

Fly.io Deployment

From flyio/uptime_kuma, just run the following, It’ll deploy Uptime Kuma to Fly.io, validate the DNS challenge for SSL certificates, and add A/AAAA records. If you use down instead of up, it’ll do the reverse. Don’t worry about running the commands multiple times, they’re both idempotent.

dagger call \
    --fly-api-token=FLY_API_TOKEN \
    --fly-toml=fly.toml \
    --pulumi-access-token=PULUMI_ACCESS_TOKEN \
    --cloudflare-token=CLOUDFLARE_API_TOKEN \
    up

[!note] For simple updates (no DNS certificate) you can just bump the image tag and run flyctl deploy.

AWS Deployment (Depreciated)

This has been removed, please see ff7a87d8 for the historical setup.

Secrets required:

• Cloudflare token (with write access to kgb33.dev) as CLOUDFLARE_API_TOKEN
• Allow Pulumi access to AWS (See here

AWS Permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "acm:DeleteCertificate",
                "acm:DescribeCertificate",
                "acm:ListTagsForCertificate",
                "acm:RequestCertificate",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:CreateTags",
                "ec2:DeleteSecurityGroup",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "iam:AttachRolePolicy",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:DetachRolePolicy",
                "iam:GetRole",
                "iam:ListInstanceProfilesForRole",
                "iam:ListRolePolicies",
                "logs:DeleteLogGroup",
                "logs:ListTagsLogGroup"
            ],
            "Resource": "*"
        }
    ]
}

Then just pulumi up and navigate to uptime.kgb33.dev