September 2026 update: Meta has announced Muse Charm, giving personal AI an always-available voice hardware interface that people can carry with them. It reinforces the central argument of this article: AI needs an independent entrance that is low-friction and continuously available. It is natural for vendors to begin by building proprietary hardware for their own products. I believe vendors will eventually emerge whose hardware can connect to AI products and services from multiple providers.
In the first article in this series, I explained that Hey Jarvis grew out of a very simple need. When my wife and I were talking before bed and a question came up, we wanted to ask it aloud and get an answer without reaching for a phone.
I later built Pipeline, Realtime, and the Mac App, working through a series of challenges involving making voice interaction truly work and turning a demo into a Mac product.
The further I got, however, the more clearly I saw a fundamental limitation:
AI can become ever more intelligent, but without a natural, reliable, and trustworthy entrance, it remains just another app that someone has to open deliberately.
At first, I was tempted to reduce the problem to a more emotional claim: Siri was holding back the development of AI assistants.
I now think that was only half right.
Siri Is Both an Assistant and a System Interface Link to heading
From the user’s perspective, Siri’s most distinctive capability may not be the intelligence of its answers, but the fact that it already lives inside the system.
It has a wake word, lock-screen access, microphone permissions, a system UI, and an identity that follows the user across devices. There is no app to find first and no background process to understand. You speak, and the entrance is already there.
Even with a better model, a third-party AI app has little chance of occupying exactly the same position.
Apple has not shut third parties out completely. Through App Intents, apps can expose their actions and content to Siri, Apple Intelligence, Spotlight, and Shortcuts in a structured form. The 2026 updates added support for long-running and cancellable background tasks, entities shared across devices, and confirmation for sensitive operations. Apple has also announced its next generation of Siri AI, emphasizing personal context, on-screen awareness, knowledge from the web, and actions across apps.
But there is an important distinction:
Exposing an app’s capabilities to the system assistant
≠
Allowing the app itself to become the system assistant
A third-party app can give Siri an action to invoke. That does not give the app its own system-level wake word, let it remain ready in every device state, or allow it to replace the default voice interface without friction.
One concrete example is Apple’s current support for launching a third-party conversational app with the iPhone side button. However, the official assistant activation capability is available only in Japan, requires a dedicated Side Button Access entitlement, and is subject to regional and device restrictions.
This shows that Apple understands that third-party voice assistants need a hardware entry point. That entry point matters so much that the platform must define, authorize, and constrain it separately.
Those Restrictions Are Not All Bad Link to heading
If any app could register a permanent wake word, keep the microphone active from the lock screen, read what is on the screen, and take actions across apps, AI assistants might become more convenient—but the system would quickly become impossible to trust.
Users need to know:
- when the microphone is active;
- which audio is processed only on the device and which is uploaded;
- which app receives the content;
- who can access their personal data;
- which actions can run automatically;
- whether additional confirmation is required before making payments, sending messages, or deleting content;
- how to shut down a misbehaving background assistant immediately.
Apple’s restrictions on permissions, background execution, hardware entry points, and app distribution establish entirely reasonable boundaries for security, privacy, battery life, and accountability.
My own development experience has also helped me understand why those boundaries exist. A leftover page can cause audio to play twice. A background thread may reopen the microphone while shutting down. After the Mac wakes from sleep, the interface may incorrectly claim that it is still listening. Diagnostic logs can accidentally become conversation histories.
So this cannot be reduced to “openness is progress, restrictions are backward.”
The real tension is that the platform must protect its users. Yet when the most natural system-level voice interface belongs primarily to the platform’s own assistant, the space available for third-party AI innovation is also determined by the platform’s product roadmap.
What Hey Jarvis Really Lacks Is Not Another Model Link to heading
If a new model appeared today that was smarter, cheaper, or better at Chinese, Hey Jarvis could theoretically switch to it.
But changing the model would not change the fact that Hey Jarvis still runs on a Mac.
A Mac gets closed, goes to sleep, travels to the office, and may be nowhere near the bed. It is first and foremost a personal computer, not a voice device permanently stationed in a room. To keep Hey Jarvis available after the screen locks, I have had to manage power assertions, keep media tracks alive, restore WKWebView, and work within system permissions. That engineering can improve the experience, but it cannot change what the device was designed to be.
This gradually led me to a conclusion:
The scarce resource in voice AI is not a model that can generate answers. It is a low-friction, persistent entrance that users can trust.
Such an entrance needs at least four properties:
- It can be awakened naturally at any time.
- It clearly indicates when it is listening and when it is uploading data.
- It has sufficiently reliable input, output, and network connectivity.
- Its permissions belong to the user rather than being permanently tied to a single model.
The next step for Hey Jarvis should not simply be to support more models or add a few more tools. Adding models and tools does not solve the entrance problem.
Near Term: Becoming a More Complete Mac Agent Link to heading
Independent hardware is a longer-term goal. Before then, the Mac version still has substantial room to evolve.
The first step is proper distribution. A publicly downloadable v0.1.0-internal GitHub Release already exists, but it still lacks Developer ID signing, notarization, and automatic updates. To reach ordinary users, signing, notarization, a stable bundle identity, upgrades, and rollbacks all need to become part of a formal release process.
The second step is reducing startup friction. Launch at login, a clear indication of background status, and reliable recovery after an operating system update or crash would make Hey Jarvis feel more like a persistent assistant and less like an app that must be opened manually from time to time.
The third step is using the system interfaces Apple already provides. Hey Jarvis can expose its actions through App Intents and Shortcuts so that the system knows what it can do. This will not give it a new system-level wake word, but it can make Hey Jarvis less isolated from other Mac workflows.
The fourth step is turning tools from demo features into personal workflows. Weather, time, exchange rates, and stock prices have helped demonstrate the boundaries of tool calling. A genuinely useful assistant, however, should be able—with explicit authorization—to connect to calendars, notes, reminders, and the services I use every day.
At that point, the security model must evolve as well. Queries can be answered directly. Operations that modify something should clearly identify the object they affect. High-risk actions should be auditable, while reversible actions should support undo.
Further Ahead: A Hardware Entrance for Personal AI Link to heading
If I take Hey Jarvis beyond the Mac, I do not want to build yet another closed smart speaker.
I would rather think of it as a personal AI peripheral: something that gives personal AI ears, a mouth, and a physically trustworthy boundary without dictating which model or service the user must adopt.
The device’s core capabilities are not mysterious:
- a microphone array designed for room-scale use, with echo cancellation;
- on-device wake-word detection;
- a short audio buffer that is not uploaded before activation;
- status indicators physically wired to the microphone and network circuitry;
- a physical mute switch that truly disconnects the microphone;
- a reliable speaker and support for full-duplex conversation;
- device identity, encrypted communication, secure boot, and signed updates;
- a control plane on a phone or Mac for configuring permissions, checking status, and revoking access.
It would not necessarily need to run the most powerful model locally.
The local device could handle activation, audio processing, privacy decisions, and emergency controls. Depending on the user’s preference, complex reasoning could run on a Mac, a home server, or a cloud model. The model would be a replaceable computing resource; the hardware would provide a stable sensory interface and trust boundary.
Hardware in the room
→ Local activation and physical privacy controls
→ Personal Agent control plane
→ Replaceable models
→ Authorized tools and personal data
This architecture would not make the future device an entirely different product from Hey Jarvis as it exists today.
The current project already keeps local activation, realtime sessions, tools, language behavior, and privacy logs behind relatively independent boundaries. What would actually need to be replaced is the audio interface that depends on the Mac, WKWebView, and the desktop sleep lifecycle.
Hardware Does Not Create Trust Automatically Link to heading
Owning the hardware interface would also mean taking on more responsibility than a Mac App requires.
An always-on microphone in a bedroom cannot earn trust merely by claiming that “we care about privacy.” Its physical state must make clear whether the microphone is truly disconnected, whether audio is leaving the device, and which Agent is responding.
It must also account for the consent of everyone in the household. The device owner’s authority to configure it does not mean that everyone else in the room has agreed to be recorded or have their audio uploaded. Children, guests, shared spaces, and sensitive conversations all require stricter boundaries than a personal computer does.
As tool capabilities grow, so does the risk. Reading a calendar or turning on a light should not require the same level of authorization as sending a message, purchasing an item, or controlling a door lock.
The hardware I envision, then, is not an all-powerful Agent that remains permanently online. It should be a system with limited capabilities by default, permissions that expand layer by layer, and the ability to be physically shut off at any time.
Security is not about putting the brakes on AI. It is what allows users to feel safe leaving the entrance open over the long term.
Ultimately, the Question Is Not How Smart Siri Is Link to heading
If Siri AI eventually becomes intelligent enough to understand personal context and take action across apps, the system may directly absorb some of Hey Jarvis’s original product value.
That would not make this project meaningless.
On the contrary, it has helped me break down a vague dissatisfaction into several distinct problems: model capability, voice experience, system access, personal data, execution permissions, and hardware trust are not the same thing.
Siri may become a better AI assistant. Apple may also continue to open more capabilities to third parties. But as long as users cannot freely choose who occupies the most natural interface, who manages their context, and who acts on their behalf, “personal AI” will still largely mean AI provided to us by a platform.
The future I would rather see is one where:
- the entrance belongs to the user;
- models can be replaced;
- memories can be migrated;
- tool permissions can be inspected and revoked;
- high-risk actions require confirmation;
- every action leaves a comprehensible record;
- the hardware makes it unmistakably clear when it is listening.
The first version of Hey Jarvis was simply my attempt to find an entrance for AI within the Mac.
Perhaps one day it will no longer live in the Mac at all, but will instead have a pair of ears that truly belong to the individual.
Building Hey Jarvis series: