Carl Andersson

Kubernetes deployment

Deployed Kubernetes on-premise using RKE2 on Ubuntu with Ansible to consolidate with previously deployed AWS infrastructure. Kubernetes replaced a custom SaltStack based "scheduler" that installed systemd units. Collaborated with application developers enabling them to write control-plane software (Use Kubernetes jobs to run heavy processing jobs)

• VMware deployment: Deployed vSphere + vCloud + NSX-T + vSAN stack to replace a unreliable Hyper-V S2D cluster. Customers got self-service via vCloud portal
• Datacenter consolidation: Migrated Dialect conglomerate hosting from InterXion to Skövde datacenter. Live-migrated VMs over stretched L2, then physically relocated and remounted servers to reuse hardware
• Network modernisation: Continued L2 removal for remaining edge cases, deployed EVPN on Cumulus Linux (Mellanox + Broadcom hardware) for appropriate L2 stretching, initiated IPv6 rollout
• Backup infrastructure: Deployed Cohesity for VMware and legacy Hyper-V workloads

• Network architecture redesign: Eliminated chronic STP looping issues by migrating customer connectivity from L2 tunneling to L3 routing, requiring subnet migrations across the customer base
• O365 crisis response: When parent company went bankrupt, wrote PowerShell + Puppeteer automation to create admin users and accept new CSP partnerships across 1000+ Microsoft 365 tenants
• Internal tooling: Built PBX queue monitoring with notifications, custom TeamViewer auto-registration installer (before native support existed), HTTP API for physical phone PBX group login/logout
• MikroTik and networking instructor for all Dialect branch offices

Problem: Container images have inefficient layer sharing, fragile build caching, and no intrinsic SBOM. Nix solves these at the package level but needed Kubernetes integration.

How it works:

1. DaemonSet deployment—no node modifications required. Works on managed clusters (EKS, GKE, AKS).
2. Creates a node-local Nix store at /var/lib/nix-csi.
3. For each CSI volume, creates a hardlink view of requested store paths.
4. Mount modes: read-only (direct bind mount, shared inodes) or read-write (overlayfs layer).

Key benefit: Hardlinked files share page cache across pods. When one pod loads a shared library, other pods get cached pages. Container layers can't do this—each container gets separate inodes even with the same base image.

Comparison: Similar concept to nix-snapshotter, but doesn't require containerd modifications. Inspired Flox's "imageless Kubernetes" approach.

Written in Python. In production use in hetzkube.

Why: Helm templates are stringly-typed and hard to debug. Kustomize patches are limited—strategic merge works until it doesn't, JSON patches are verbose. The NixOS module system already solves configuration composition well.

Features:

• Deep merging from multiple sources
• Type checking with clear errors
• Override at any level with mkForce/mkOverride
• Conditional configuration with dependency tracking

Helm compatibility: Renders Helm charts and imports results into the module system. Override rendered resources like any other. Covers most use cases, though some hooks/lifecycle features don't translate.

nix-csi integration: Reference Nix packages directly in manifests—nix-csi automatically identifies and pulls required store paths.

In production use managing all hetzkube deployments: cert-manager, Cilium, kube-prometheus-stack, Keycloak, and applications.

Stack:

• Control plane: ClusterAPI managed CX22 node(s).
• Workers: NixOS nodes rebuilt from scratch on every initialization.
• Networking: Cilium for CNI, ingress, Gateway API and network policies. Full dual-stack (IPv4+IPv6) enforced via Kyverno.
• Custom IPAM: Reuses node IP addresses as LoadBalancer addresses to avoid Hetzner floating IP/LB costs. Each node's /64 IPv6 block is carved up for services. Kyverno enforces IP sharing annotations to maximize IPv4 utilization.
• Storage: hetzner-csi for traveling volumes, local-path-provisioner for node-bound volumes.
• Observability: kube-prometheus-stack (Prometheus, Grafana, Alertmanager).
• Auth: Keycloak OIDC for all! (Kube, Grafana, Headlamp, pgAdmin, ...)

Deployed components:

• cert-manager
• Cilium
• CloudnativePG
• CoreDNS
• external-dns
• external-secrets
• Gateway API
• Headlamp
• Keycloak
• kube-prometheus-stack
• Kyverno
• metrics-server
• pgAdmin
• Vertical Pod Autoscaler

  All manifests generated via easykubenix. Application delivery via nix-csi.
</details>

Problem: WoW factions (Alliance/Horde) had imbalanced PvP populations. On private servers, 80/20 splits meant 30+ minute queues for one faction, instant for the other.

Implementation (C++):

• Reused arena faction field to temporarily assign players to opposite faction during battlegrounds
• Invalidated Player info cache to all BG players
• Faked faction responses in player info cache packets
• Modified scoreboard packets so client UI displayed teams correctly
• Queue modes: simple (first-come-first-served) and item-level balanced (with gear-swap prevention)

Impact: Published publicly—first implementation any server could use. Spread rapidly across the private server community during the 3.3.5a era when faction imbalance was severe. Still in use today.

Blizzard later added crossfaction instances to retail WoW (dungeons/raids first, then battlegrounds). The concept was proven on private servers years earlier.

One of my first real programming projects—modifying a large C++ codebase to solve a problem affecting thousands of players.