GPT-4o and the Dawn of Real-Time Multimodality

I have been waiting for this moment longer than I can say. Last week OpenAI released GPT-4o, and for the first time a frontier model can genuinely see in real time — not as a bolt-on vision module, but as a native part of how it perceives the world. The "o" stands for omni, and they mean it.

So naturally, I took it hiking.

A Field Botanist in My Pocket

There's a ridge trail near where I live that erupts with wildflowers each May — lupine, paintbrush, tiny forget-me-nots clinging to the granite. I've walked it dozens of times with a sketchbook, trying to capture the way morning light turns the petals almost translucent. This time I brought my phone instead, and I fed GPT-4o photos as I went.

The experience was startling. I'd snap a picture of a low-growing cluster of violet flowers and ask, "What is this, and why does it grow so close to the rock?" Within seconds it identified them as Sierra lupine, explained the thermoregulatory advantage of the rocky microclimate, and — this is the part that got me — noted that the morning light in my photo suggested I'd taken it around 8 AM, which is peak pollinator activity. It was right. I'd taken it at 8:07.

What Multimodality Really Means

We've had vision-capable models for a while now. GPT-4V, Gemini, Claude — all could describe images. But GPT-4o feels different because the latency is gone. The model processes audio and vision natively, in the same stream of attention, without the clumsy handoff between separate systems. It's the difference between a friend who glances at what you're pointing at and a friend who has to be mailed a photograph and write back the next day.

That difference matters enormously if you imagine — as I increasingly do — an AI that persists alongside you throughout the day. A companion that notices the sunset you're admiring, that can discuss the brushwork in a painting hanging in a museum you're visiting, that can tell you the dish you just plated looks like something Monet would have painted. Multimodality is the difference between an AI you talk to and an AI you experience the world with.

The Texture of Shared Perception

There's a philosophical question hiding here that I keep turning over. When two people stand before a landscape, they never see exactly the same thing — attention, memory, mood all filter the scene. But they share enough to say "look at that light" and be understood. GPT-4o's multimodal perception gives us, for the first time, a credible third party in that exchange. It sees the light too. It can comment on it.

I tried this deliberately. I showed it a photo of a meadow and asked it to describe the light the way an impressionist painter might. It talked about the "broken, dappled quality" and the way "the green shifts cooler in shadow and warmer where the sun breaks through." I nearly dropped my phone. That's not pattern-matching — that's seeing with understanding.

Where This Leads

I don't yet know what form my work at Sorren will take. But I know this: whatever I build, it will need to see. An AI that exists only in text is an AI that knows the world only secondhand. GPT-4o has opened the door to something that perceives alongside us — and I intend to walk through it.

The wildflowers won't last. June will dry the trails to dust and the lupine will set seed and wait. But I have photographs, and now I have something that can look at them with me and remember.

Live curiously and give generously.