Instrumenting Everything: Our Monitoring Stack

There's a moment every infrastructure engineer knows. You're asleep, it's 3 AM, and your phone buzzes. A user has reported that something is broken. You sit up, grab your laptop, and spend the next 40 minutes figuring out what happened — when the actual failure started two hours ago and you could have caught it at the first sign if you'd been watching.

I got tired of that moment. So I instrumented everything.

The Stack

Three tools, each with a specific job:

Prometheus  — time-series metrics, scraping, alerting
Grafana     — dashboards, visualization, the thing I stare at
Loki        — log aggregation, because metrics tell you something's wrong
              but logs tell you why

All three run as Docker containers on the Proxmox cluster. Prometheus scrapes metrics endpoints every 15 seconds. Grafana renders them into dashboards I can actually read. Loki ingests structured logs from every service via Promtail.

The whole stack cost zero dollars in licensing and about six hours of setup time.

What I'm Watching

Here are the dashboards I actually care about, in order of how often I look at them:

Inference Latency. Every model call is timed. p50, p95, and p99 percentiles, broken down by model and endpoint. If Companion's p95 response time creeps above 800ms, something is wrong with the inference pipeline and I want to know before it hits 2 seconds.

GPU Utilization. nvidia-smi metrics exported via dcgm-exporter, scraped by Prometheus. Memory usage, compute utilization, temperature, power draw. The LoRA training rig and the inference servers both report in. If a GPU is at 99% utilization during what should be a quiet period, either a training job is running or something is stuck in a loop.

Vector DB Query Times. Companion's memory system runs thousands of vector queries per hour. If query latency spikes, the entire user experience degrades — slow memory retrieval means slow responses, which means unhappy users. I track query time, recall accuracy, and index size.

Error Rates. Every 500-level error from every service hits Prometheus within 15 seconds. Error rate above 1% for any service triggers an alert. I'd rather investigate a false alarm than miss a real one.

Disk and Memory. Boring. Essential. A full disk will take down everything and the failure mode is ugly.

The Alerting Setup

Dashboards are for humans who want to look. Alerts are for humans who need to know. The difference is whether the system wakes you up.

# prometheus/alerts.yml (excerpt)
groups:
  - name: inference
    rules:
      - alert: HighInferenceLatency
        expr: histogram_quantile(0.95,
              rate(inference_request_duration_seconds_bucket[5m])) > 1.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "p95 inference latency above 1.5s"

      - alert: GPUThermalThrottle
        expr: gpu_temperature_celsius > 85
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "GPU {{ $labels.gpu }} above 85C"

Alerts route through Alertmanager to a Discord webhook and my phone. Critical alerts wake me up. Warnings wait until morning. I tuned the thresholds over about three weeks — too sensitive and you get alert fatigue, too loose and you miss real failures.

The Logging Side

Metrics tell you that something is wrong. Logs tell you why. Loki aggregates structured JSON logs from every service — the inference API, the RAG pipeline, the vector database, the subconscious process. When an alert fires, I can jump from the Grafana metric panel directly to the correlated logs for that time window.

# Structured logging in the inference service
import structlog
log = structlog.get_logger()

log.info("inference_request",
    model="deepseek-7b",
    adapter="tutor-calculus",
    tokens_in=42,
    tokens_out=186,
    latency_ms=412,
    user_id=hash(user_id),
)

Every log line is structured, correlatable, and queryable in Loki with LogQL. When inference latency spikes, I filter to the affected model, sort by latency, and look at the slowest requests. Usually the answer is obvious — a long context window, a cold cache, a LoRA adapter that wasn't preloaded.

The Goal

The dream of monitoring is simple: know about problems before users do.

I'm not fully there. There are still failure modes that surface as user reports first — edge cases in the RAG pipeline, weird interactions between LoRA adapters, the occasional vector DB rebalancing that causes a latency hiccup. But the gap is closing. Most issues now show up on a Grafana panel 5–15 minutes before anyone complains.

That's the difference between reactive infrastructure and something you can actually trust in production. The stack doesn't make problems go away. It makes them visible. And visible problems get fixed.

Invisible ones fester until 3 AM.