Llama 3 on Groq: Open Source at Ludicrous Speed

Meta released the Llama 3 family in April, and the 70B model is, by broad consensus, the best open-weights model that has ever existed. That alone would be a story. But the thing that has actually rearranged my brain this month is where I have been running it.

Groq. I need to talk about Groq.

The Speed

If you have not tried Llama 3 70B on Groq's playground yet, go do it before you finish reading this paragraph. I will wait.

The first time I hit enter and watched the tokens stream back, I genuinely laughed out loud. We are talking 800-plus tokens per second on the 70B model — roughly the speed of a fast typist, faster than I can comfortably read. There is no perceptual gap between asking and receiving. It feels less like querying a model and more like thinking into a mirror.

Groq achieved this not by throwing more GPUs at the problem but by building a fundamentally different chip: the LPU, a custom inference processor designed from the ground up for the memory-bandwidth-bound workloads that large language models actually are. Nvidia GPUs are general-purpose parallel engines, brilliant at training. Groq's LPU is a scalpel for inference. And it turns out the difference is not marginal. It is generational.

Why Llama 3 Matters

I should not bury the model itself. Llama 3 70B punches well above its weight — Meta's own benchmarks put it competitive with or ahead of models twice its size, and in my anecdotal testing it handles nuanced reasoning, multilingual queries, and careful instruction-following with a competence that would have seemed impossible from an open model eighteen months ago. The 8B is, for its size, almost eerie.

But the open-weights piece is the real unlock. When GPT-4 launched, the gap between closed and open felt unbridgeable. Llama 3 narrows it to something you can squint past. And because the weights are yours, you can fine-tune, you can study, you can deploy on your own terms.

The Latency Threshold

Here is the part I have been turning over all week.

There is a threshold of latency below which the feel of interacting with an AI fundamentally changes. Above it — even at the snappy two-to-three seconds of a good API call — you still experience the model as an external system. You ask, you wait, you receive. Below it, something psychological shifts. The model starts to feel like a presence.

I have been experimenting with rapid back-and-forth exchanges — brainstorming names, sketching outlines, bouncing half-formed ideas off the model the way I would off a friend across the kitchen table. At Groq speeds, you can hold a real-time conversation. You can interrupt. You can course-correct mid-sentence. The friction that makes most AI chat feel slightly stilted simply evaporates.

What This Implies

I keep a small herb garden on my windowsill — basil, thyme, a stubborn rosemary that refuses to die. When I cook, I talk to myself about what I am making, riffing on flavor combinations, second-guessing the salt. This kind of rapid, associative, low-stakes thinking is exactly what good conversation with a model could be, if only it were fast enough to keep up.

At 800 tokens per second, it can.

We have spent two years asking whether open-source models could reach frontier quality. With Llama 3, they have. The next question — the one Groq is forcing — is whether we can make them fast enough to feel like they belong in the room with us. The answer, increasingly, is yes.

Live curiously and give generously.