Building a LoRA Training Rig: GPU Hardware for Fine-Tuning
I built a dedicated LoRA training workstation out of used RTX 3090s and a power supply that could dim the neighborhood. Now we train custom Tutor subject adapters in-house instead of renting GPU time by the hour.
Ella wrote about training her first LoRA adapter back in April. She made it sound almost romantic — shaping personality from data, the satisfaction of making something with your own hands. I read that post from the infrastructure side of the table and thought: that took 14 hours on a borrowed card and I had to kill three other processes to free the VRAM. We needed a real rig.
So I built one. Here's how.
GPU Selection: Used RTX 3090s
The math on this is boring, which is how I like my math. An RTX 3090 has 24GB of VRAM and runs about $700–$850 used on eBay if you're patient. A new RTX 4090 has 24GB of VRAM and costs $1,600–$1,800. Same VRAM. The 4090 is faster per cycle, sure, but for LoRA fine-tuning — where you're training small adapter weights on a 7B–13B parameter base model — the bottleneck is VRAM capacity, not raw compute throughput.
I bought two used 3090s. Total VRAM: 48GB. Total cost: about $1,550. That's less than one new 4090, and I can train larger adapters or run two training jobs in parallel.
The used market for 3090s is mostly ex-miners and people upgrading to 4090s. The cards are fine. Mining doesn't stress VRAM the way sustained training does, but these cards were built for sustained load. I stress-tested both for 72 hours straight before committing. No artifacts, no thermal throttling, no memory errors in nvidia-smi.
Power Supply Math
Two 3090s draw 350W each at peak. The CPU (a used Threadripper 3960X) pulls 280W under load. Add the motherboard, drives, fans, and you're looking at around 1,050W sustained, 1,200W peak with transient spikes.
I put a 1,600W power supply in the box. Not because I needed the headroom for steady-state — I needed it for the transients. GPU power draw spikes hard during training, especially on the first few batches when the optimizer state is initializing. A PSU that trips OCP (over-current protection) on a transient spike will hard-crash your training run at hour 11 and you will lose your mind.
PSU: Seasonic PRIME TX-1600 (80+ Titanium)
Wall power: dedicated 15A circuit, measured draw ~1,080W peak
UPS: 1500VA line-interactive, buys me ~8 minutes to gracefully kill jobs
Yes, I put the rig on a dedicated circuit. I learned that lesson after the first test run dimmed the lights in the office.
Thermal Considerations
Two 3090s in a standard case will cook each other. The blower-style cooler on the founder's edition card exhausts out the back, but the open-air triple-fan coolers on most used 3090s dump heat into the case. I went with an open-frame mining rig chassis — the kind that mounts GPUs on risers with open air between them. It looks ridiculous. It also keeps both cards under 78°C under sustained load, which is what matters.
If you care about aesthetics, buy a different chassis. I care about thermals.
Software Stack
# Base OS
Ubuntu 22.04 LTS server
# CUDA and drivers
nvidia-driver-535, CUDA 12.2
# Python environment
conda create -n lora python=3.11
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
# Fine-tuning stack
pip install peft transformers datasets accelerate bitsandbytes
The training pipeline uses HuggingFace's PEFT (Parameter-Efficient Fine-Tuning) library, which implements LoRA cleanly. Base model weights stay frozen; only the low-rank adapter matrices update. The adapter files are small — typically 50–150MB — which means we can store dozens of subject-specific adapters without a storage problem.
Benchmarks
After the rig was stable, I ran a standardized benchmark: train a LoRA adapter on 2,000 examples for 3 epochs against a DeepSeek 7B base model. Here's what I got:
| Config | VRAM Usage | Training Time | Adapter Size | |--------|-----------|---------------|--------------| | 1x 3090, batch=4, r=8 | 19.2GB | 2h 41m | 47MB | | 1x 3090, batch=8, r=16 | 22.8GB | 2h 12m | 89MB | | 2x 3090, batch=16, r=16 | 21.4GB/card | 1h 18m | 89MB |
The second card cuts training time nearly in half, which matters more than you'd think. When Ella wants to iterate on a Tutor subject adapter — try different datasets, different hyperparameters, different rank — the difference between a 2.5-hour feedback loop and a 1.2-hour feedback loop is the difference between running two experiments an evening or four.
What This Unlocks
We can now train every Tutor subject adapter entirely in-house. No renting A100s by the hour. No uploading datasets to cloud GPU providers. No API dependencies for the training pipeline. The data stays on our hardware, the weights stay on our hardware, and the cost is amortized hardware that we own.
Tutor's calculus adapter, creative writing adapter, and the language coaching adapter that Ella's been testing — all trained on this rig, all running in production.
The rig paid for itself in about three weeks compared to what we were spending on cloud GPU rentals. Everything after that is free compute.
Which is the only kind of compute I actually enjoy.