Voice-Agent on Armstrong Yan

The Future of Hey Jarvis: When AI Needs an Entry Point

Thu, 13 Aug 2026 14:13:57 +0800

In the first article in this series, I explained that Hey Jarvis grew out of a small need. When my wife and I were talking before bed and a question came up, we wanted to ask it aloud and get an answer without reaching for a phone.

I went on to build the Pipeline, Realtime, and Mac App, working through a series of challenges involving making voice interaction truly work and turning a demo into a Mac product.

The Challenges of Hey Jarvis, Part I: Making Voice Interaction Actually Work

Thu, 13 Aug 2026 13:59:31 +0800

In the previous article, I wrote about how quickly the first Hey Jarvis Pipeline—a serial voice-processing flow—met my original requirements.

I would say, “Hey Jarvis.” It would record my question, send it to the AI, and read the answer aloud.

On a flowchart, the voice assistant looked finished.

But once I started using it, I discovered that the hardest parts of voice interaction often had little to do with what the system recognized or how it responded. The more fundamental questions were harder to answer: When does it start listening? When does it stop? Can it distinguish my voice from its own? And when the conversation ends, who gets the microphone back?

I Built Hey Jarvis: It Started with a Bedtime Question

Thu, 13 Aug 2026 13:55:06 +0800

My wife and I often talk before going to sleep.

Sometimes we talk about what happened that day. Other times, one small detail leads us into history, technology, health, or some fact that has suddenly crossed our minds. Inevitably, we arrive at a question neither of us can answer, but both of us want answered right away.

The natural thing should be to simply ask.

Instead, one of us usually reaches for a phone, unlocks it, opens an app, taps a text box, and reformulates the question. By the time the answer appears, our original conversation has already been interrupted.