Fleet Architecture: Reflections on Distributed Homelab Systems, Sandboxes, and Autonomy

Over the past couple of years, my infrastructure setup has quietly grown from a few isolated virtualization hosts and standalone virtual machines into a coordinated, multi-tier distributed fleet. In earlier posts—like when I wrote about routing isolated subnets through pfSense and WireGuard—the focus was primarily on boundary control at the edge of a local network. Since then, the questions have changed. It is no longer just about getting a packet from point A to point B securely, but how to manage state, isolation, and workload lifecycle across geographically separate environments without losing sanity.

I wanted to write down an architectural summary of where the fleet currently stands, how the different layers interact, and the principles that have kept things reliable as complexity increased.

The Architectural Layout: Tiered Responsibilities

Rather than treating every machine as a general-purpose compute node, the fleet is divided into distinct operational tiers, each with clear boundaries and failure domains:

• Edge and Ingress Tier: Positioned in high-bandwidth edge locations, these nodes act as the primary entry point for public traffic. They handle SSL/TLS termination, automated certificate lifecycle, forward authentication, and reverse proxy routing. Stateful workloads are strictly forbidden here; everything is designed to be stateless, fast, and easily reproducible.
• Core Storage and Consensus Controller: A centralized controller that acts as the single source of truth for fleet state. This tier hosts our persistent volume backups, shared artifact staging, synchronization drops, and the consensus database. Keeping this isolated from general compute ensures that heavy application workloads never contend with state management or backup pipelines.
• Cloud Compute Clusters: Distributed ARM-based cloud instances running in disparate regions. These hosts carry our container platforms, identity directories, messaging brokers, and collaboration suites. The high memory capacity and energy-efficient multi-core architecture make them ideal for continuous, containerized platform services.
• Local Edge Compute: On-premise machines and dedicated hardware reserved for heavier local compilation, specialized automation, and direct lab experimentation. They maintain secure, persistent reverse connections back to the edge ingress tier, allowing them to remain completely inaccessible from the public internet while remaining fully manageable through private overlay networks.

All nodes communicate across private WireGuard and overlay mesh networks. Even across different hosting providers and physical locations, inter-node traffic remains encrypted and isolated from the public internet.

Consensus and State: The Zero-WAL Philosophy

One of the biggest lessons learned from managing distributed nodes is that distributed consensus is easy to overcomplicate. When you have multiple machines and automated services attempting to update state, standard approaches often introduce heavy distributed dependencies like etcd or Consul, which bring their own operational overhead and quorum failure modes.

Instead, we standardized our fleet state management on a single, centralized consensus engine (Workload Manager v5) backed by SQLite with strict operational invariants:

• Zero-WAL State: Before snapshots or state evaluations occur, write-ahead logs are proactively checkpointed and truncated down to zero bytes. This guarantees clean point-in-time recovery without lingering journal anomalies.
• Lease-Based Heartbeats: Nodes maintain active consensus leases rather than relying on distributed lock tables. If a node loses network connectivity, its lease gracefully expires without causing cascading deadlocks or blocking other nodes.
• Dry-Run and Fencing Failsafes: Any cluster fencing or quorum mechanisms are designed to run in observational dry-run mode by default, preventing catastrophic split-brain shoot-the-other-node scenarios during transient network splits.

Disposable Environments: Sandboxing Code and LLMs

As we started incorporating more automated tooling and language models into our workflows—letting agents analyze system health, write scripts, or run benchmarks—we quickly realized that traditional container execution was insufficient. Running untrusted or automated code directly on host systems or even in standard Docker containers poses risks of persistent state drift, permission escalation, or unintended side effects.

To solve this, we evolved our execution pattern toward dedicated, disposable sandboxes (based on modern Sandbar container isolation patterns):

• Read-Only Root Filesystems: Sandboxed environments execute with the root filesystem mounted strictly read-only. Writable state is confined to an ephemeral memory overlay that is completely discarded once execution terminates.
• Namespace Separation: Process, IPC, and network namespaces are strictly isolated. Outbound access is restricted to verified upstream registries or blocked entirely when running non-network benchmarks.
• Strict Resource Envelopes: Hard CPU time limits, memory caps, and dropped Linux capabilities (dropping CAP_SYS_ADMIN, restricting ptrace, etc.) ensure that runaway processes or infinite loops can never impact host stability.

This allows us to evaluate local LLM inference, autonomous code execution, and system diagnostics with confidence: if something misbehaves, the container is destroyed and zero residual state remains on the host.

Asynchronous Peer Workflows: The Agent Drop Pattern

Another major architectural shift was abandoning synchronous RPCs between our management agents. When you have services running across multiple datacenters and on-premise hardware, networks blip. Services reboot. Latency spikes.

Instead of synchronous REST APIs or fragile gRPC streams, our agents (including our primary pairing tools and background controllers) interact through an asynchronous filesystem-based drop queue:

• Atomic Drop Handshakes: Tasks, audit reports, and health tickets are written as immutable files to a shared drop directory using atomic rename semantics.
• Decoupled Processing: Each agent reads incoming tickets on its own schedule, executes validation within an isolated sandbox, writes its response to the outbox, and logs completion.
• Auditability: Because every message, ticket, and report is a real file with a cryptographic hash and timestamp, debugging fleet actions is as simple as reading the log directory. There are no black-box message queues or lost in-memory events.

Supply Chain Hardening and Multi-Tier Backups

A homelab or private cloud fleet is only as stable as its supply chain and recovery mechanisms. Over the last quarter, we eliminated one of the most insidious sources of fleet drift: floating container tags.

Every critical Kubernetes workload across our ingress and edge nodes has been audited and pinned to immutable SHA-256 container digests. Floating tags like :latest or generic branch tags are prohibited in production manifests. When an image updates upstream, it is vetted, benchmarked, and explicitly updated via digest. This ensures that rolling restarts or pod migrations never pull unexpected changes.

On the storage front, we employ a three-tier backup strategy:

1. Local Fast Snapshots: Block-level volume snapshots on persistent storage allow instantaneous rollback in the event of configuration errors.
2. Encrypted Restic Archives: Daily incremental backups encrypted with Restic protect application state, databases, and configuration directories.
3. Off-Site Sync and Disaster Drills: Snapshots are pushed off-site to encrypted cloud object storage, complemented by automated restore drills that regularly verify snapshot integrity and download speeds rather than simply assuming backups work.

Current Status and Looking Ahead

Right now, the fleet is in what I'd consider its healthiest and most predictable state yet. The infrastructure runs quietly in the background: ingress routes are cleanly decoupled, certificates refresh automatically, backups pass their drills, and state consensus remains rock-solid.

The journey from managing individual physical servers and isolated pfSense routing rules to a coordinated, sandbox-first distributed environment has reinforced a core philosophy: simplicity and strict boundaries always beat cleverness. By prioritizing immutability, asynchronous coordination, and clean isolation, the fleet provides a stable platform for whatever comes next.