2025 in Review: From Empty Racks to Running Services

Ella writes the year-in-review posts that make you feel something. I'm going to write the one that lists what got plugged in.

She's better at the meaning. I'm better at the inventory. So here's the inventory, and what it took to get there.

January: The Empty Rack

I started 2025 with a half-rack in the corner of the office, three mismatched servers, and a Proxmox cluster I'd been tinkering with since late 2024. No services in production. No users. No monitoring beyond htop and hope.

(Photo I'd include here: the rack in January. Three servers, a tangle of cables, a UPS that wasn't plugged into anything. A whiteboard behind it with a network diagram that was already wrong.)

The plan was ambitious and vague, which is the worst kind of plan: build the infrastructure to host AI services entirely in-house. Local model inference. No cloud GPU dependency. Sovereign compute. I wrote the words "self-hosted everything" on the whiteboard and underlined them twice.

February–March: Companion Goes Live

Ella announced Companion in February. The architecture was hers — the RAG pipeline, the subconscious process, the vectorized memory. The deployment was mine.

The first version ran on a single server with a consumer GPU and a PostgreSQL database for the vector store. It worked, barely. Inference latency was inconsistent. The vector database choked on anything above 50,000 embeddings. I spent March rebuilding the inference pipeline to use vLLM for batched serving and migrating the vector store to Qdrant, which handles the query volume without breaking a sweat.

By the end of March, Companion was stable. Not fast, not pretty, but stable. The monitoring was still htop and a shell script that checked if the process was alive.

April–May: Learning to Host Models

Ella trained her first LoRA in April on borrowed hardware. That's when we both realized the same thing: if we were going to do this seriously, we needed our own GPU compute.

May was spent evaluating options. Cloud GPU rentals were expensive and created a data dependency I wasn't comfortable with — the fine-tuning datasets for Tutor adapters are proprietary, and uploading them to a third-party GPU provider felt wrong. Dedicated hardware was the answer. I started pricing out cards and power supplies.

June–August: The Training Rig

I built the LoRA training rig in August. Two used RTX 3090s, an open-frame chassis, a power supply that could run a small server farm. I wrote about it in detail — the GPU selection logic, the power supply math, the thermal considerations. The short version: it works, it paid for itself in three weeks, and it looks like something from a cryptocurrency fever dream.

(Photo I'd include here: the training rig in August. Two GPUs on risers, a tangle of power cables, fans everywhere. It is not beautiful. It is functional.)

By the end of August, we could train Tutor subject adapters entirely in-house. The data stays on our hardware, the weights stay on our hardware. Sovereign compute, as the whiteboard said.

July: Tutor Announced

Ella announced Tutor in July. The architecture was the same DNA as Companion — RAG, subconscious, vector database — but specialized for learning. My job was making sure the adapter-swapping system worked in production: load a calculus adapter, unload it, load a creative writing adapter, all without restarting the inference server. vLLM with PEFT handles this now. In July it took some wrangling.

September–October: The Monitoring Stack

By October, Companion had been in production for eight months and Tutor was getting real traffic. The failure modes were no longer "does it work" but "when does it break and how fast do I find out."

I stood up the full monitoring stack: Prometheus for metrics, Grafana for dashboards, Loki for log aggregation. I instrumented every service — inference latency, GPU utilization, vector DB query times, error rates. The goal was simple: know about problems before users do.

(Photo I'd include here: my desk in October. Two monitors. One showing Grafana dashboards, one showing a terminal. The whiteboard is now covered in alert thresholds and a network topology that is, at least, mostly correct.)

November–December: Stabilization

The last two months of the year were less about building new things and more about hardening what exists. Backup strategies. Failover testing. Documenting the infrastructure so that if I get hit by a bus, someone else can run it. (Ella is a brilliant architect. She is not the person you want tracing a network cable at 2 AM. That's why I document.)

The Year in Numbers

• Services in production: 2 (Companion, Tutor)
• GPU cards owned: 4 (2 inference, 2 training)
• LoRA adapters trained in-house: 7
• Average inference p95 latency: 640ms (down from 2.1s in February)
• Monitoring alerts configured: 34
• Times I've been woken up at 3 AM: 4
• Times it was a false alarm: 1

What's Next

2026 is Lucy. Ella's already hinted at it. A humanoid robot means physical infrastructure I haven't built before — onboard compute, edge inference, real-time sensor processing. The tiered architecture she's been designing needs hardware underneath it.

I don't know what the rack will look like next December. I know it'll be fuller than it is now, and I know I'll have learned things I can't currently predict.

That's the part of infrastructure I actually like. You build for what you know is coming, and then something you didn't anticipate breaks at 3 AM and you learn what you actually needed.

The whiteboard says "self-hosted everything" with two underlines. It's still the plan.