Running Models In-House: Our First Inference Server

Saturday morning, coffee, GPU humming in the closet. Time to see if we can actually run our own models.

We've been paying for API access to various LLM providers, which is fine for production traffic but expensive and slow-feeling when you're iterating on prompts at 11 PM and watching the meter spin. The question I wanted to answer today: can a local inference box handle our development workload, and how does it compare to the APIs we're already using?

The answer, spoiler: yes for dev, not yet for scale. But the gap is smaller than I expected.

The Hardware

The GPU host is a Proxmox VM with PCIe passthrough to a single consumer-grade NVIDIA card with 24 GB of VRAM. Getting passthrough working was its own saga — kernel parameters, IOMMU groups, a vfio config — but that's a separate post. Assume it works and the VM sees the GPU.

nvidia-smi
# Confirms the card is visible inside the VM

CUDA toolkit installed from NVIDIA's repo, not the Ubuntu default, because the Ubuntu default tends to lag:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update && sudo apt install -y cuda-toolkit-12-4

vLLM

For the inference server, I went with vLLM. It's fast, it supports the OpenAI API format (so our existing client code works unchanged), and it handles batching and memory management well. The easiest path is Docker:

# docker-compose.yml
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    volumes:
      - ./models:/root/.cache/huggingface
    ports:
      - "8000:8000"
    command: >
      --model deepseek-ai/deepseek-coder-7b-instruct-v1.5
      --quantization awq
      --max-model-len 8192
      --gpu-memory-utilization 0.90
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]

--quantization awq is doing real work here. The full-precision model doesn't fit comfortably in 24 GB alongside the KV cache and the context window. AWQ-quantized versions of DeepSeek and Llama models are 4-bit, roughly a quarter the size, and the quality hit is modest for the use cases we care about — code generation, summarization, structured extraction. GPTQ is the other option; AWQ gave slightly better throughput in my tests.

The HUGGING_FACE_HUB_TOKEN is needed because some models are gated. Get one from your HuggingFace account settings.

The Endpoint

Once the container is up, vLLM exposes an OpenAI-compatible API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/deepseek-coder-7b-instruct-v1.5",
    "messages": [{"role": "user", "content": "Write a Python function to reverse a linked list."}],
    "max_tokens": 200
  }'

This means our application code — which already speaks OpenAI's API format — can point at localhost:8000 instead of api.openai.com and just work. Swap the base URL, change the model name, done. That compatibility is the reason vLLM won over other options I tested.

The Benchmark

Saturday morning is for honesty, so here are real numbers from my testing. Same prompt (200-token completion), median of 20 requests:

| Source | Model | Latency (ms) | Tokens/sec | |--------|-------|-------------|------------| | Local vLLM | DeepSeek Coder 7B (AWQ) | 850 | 47 | | Local vLLM | Llama 3 8B (AWQ) | 920 | 44 | | OpenAI API | GPT-4o-mini | 680 | 89 | | Groq API | Llama 3 8B | 210 | 280 |

Read that table carefully. The Groq numbers are absurd — their LPU hardware is built for exactly this. Local vLLM is slower than the API options, as expected. But here's the thing: 47 tokens per second is faster than I can read. For a development loop where I'm iterating on prompts and testing pipelines, that's plenty fast. And it's free, private, and doesn't rate-limit me at 2 AM.

When Local Wins

Local inference is the right call for:

• Development and testing — no meter, no rate limits, full privacy.
• Fine-tuning prep — you need the weights locally anyway for LoRA training.
• Sensitive data — nothing leaves the network.

It's the wrong call for:

• Production at scale — one consumer GPU can't match a fleet of H100s.
• Models larger than ~13B at full quality — 24 GB of VRAM is a hard ceiling.
• Anything needing the absolute frontier model.

Where This Leaves Us

We now have a local inference endpoint that handles development traffic. Our code is written against the OpenAI API format, so switching between local and remote is a config change. Fine-tuning work — we're looking at DeepSeek weights specifically — has a place to happen. And I'm not watching a billing dashboard every time I run an experiment.

The frontier model APIs still matter. Ella's experiments with the latest releases need the real thing. But for the daily engineering grind, the GPU in the closet is pulling its weight. Literally — I checked the power meter.

Next on the list: automating the parts of development that don't need a human. The AI coding agent space is moving fast, and I want to see if any of these tools are ready for real work.

Coffee's cold. Time to go do something else with my Saturday.