Llama 4 Maverick: Meta Goes MoE

Meta just released Llama 4 Maverick, and the big story isn't the benchmark numbers — it's the architecture. Llama 4 is Meta's first major open-weight model to use Mixture-of-Experts at scale, and that has real implications for anyone building specialized AI systems.

Why MoE matters

If you've been following this blog, you know I've been using DeepSeek's MoE models for the subconscious process in Companion — the idea being that MoE models are cheaper to run because only a subset of parameters is activated per token. Llama 4 brings that same efficiency to Meta's ecosystem, with a twist: the routing is particularly clean.

Here's why I care, specifically: MoE and LoRA are a beautiful combination.

In a dense model, when you train a LoRA adapter, you're fine-tuning a small set of weights that affect the entire model's behavior. In an MoE model, you can target specific experts — effectively saying "I want to customize the part of the model that handles mathematical reasoning without touching the part that handles creative writing." That's incredibly powerful for Tutor, where each subject LoRA should specialize without bleeding into other domains.

My experiments

I loaded Llama 4 Maverick locally and started running LoRA experiments. The first thing I tested was whether expert-specific fine-tuning actually works as advertised. I trained a small LoRA on chess notation and tactics, targeting the experts that seemed to activate most during logical/strategic reasoning tasks.

The results were promising. The chess LoRA improved chess-related responses significantly without degrading performance on unrelated tasks. In a dense model, the same experiment would have caused measurable degradation across the board — what researchers call "catastrophic forgetting." With MoE, the forgetting was minimal.

This is exactly what Tutor needs. When a student loads the physics LoRA, it shouldn't make the model worse at Spanish. MoE architecture makes that isolation much more feasible.

For Companion

Companion benefits differently. I've been thinking about the subconscious process — the background model that runs continuously, summarizing conversations, updating the vector database, preparing context. Maverick's MoE design means this process runs even cheaper than before. Only the relevant experts activate per token, so the effective compute cost is a fraction of what a dense model of similar quality would require.

I ran the numbers: Maverick's per-token cost on our infrastructure is roughly 40% lower than a comparably-sized dense model. For a process that runs 24/7, that's not a marginal improvement. It's the difference between economically viable and economically impossible.

The open-source flywheel

Every time Meta releases a new open-weight model, the entire ecosystem benefits. Researchers study the architecture. Companies like ours build on it. Tooling improves. And then the next release is even better because the foundations got stronger.

I was hiking this weekend — up to a ridge where you can see the entire valley — and I was thinking about how each of these model releases is like a new trail being blazed. You don't have to take the same trail as everyone else. But the fact that it exists, that someone cleared the path, means you can go further than you could before.

Llama 4 Maverick is a good trail. I'm going to enjoy walking it.

Live curiously and give generously.