Dan Baruch · · 9 min read
Adding Live Voice to HCM Intelligence
HCM Intelligence has a live mode. Open Ask on your phone, start a call, and ask how ADP's stock did this month. Ask reads the answer aloud while the closing-price chart and sources appear on screen. You can interrupt it, ask a follow-up, or tap a source while it's talking.
The voice agent doesn't answer anything itself. It sends each question back to the open page, which already knows who is signed in, which conversation they're in, and how to show a cited answer. A spoken follow-up uses the same history, research tools and source checks as a typed one.
That keeps the voice side small. Most of the care goes into a pause, a phone speaker, or someone saying "What'd you say?"
How the pieces fit together
The HCM app runs on Node.js and Express, with SQLite holding company records, news, filings, stock closes and Ask history. Live mode adds a WebRTC room hosted by LiveKit Cloud, on its free Build plan, and a small voice agent built with the LiveKit Agents SDK. The agent uses Deepgram's Nova-3 for speech recognition and Aura-2 with the Athena voice to speak. Company names like Paycom, Paychex and Paylocity are in the recognition vocabulary, because a transcript can be one company name off and still look like a perfectly reasonable research question.
The whole thing is deliberately light: one server and one SQLite file. At thousands or millions of concurrent callers, those are the parts that would change.

When you start a call, the HCM server checks your remaining voice time, creates a unique room name, and signs a token that can join that room for the next ten minutes. The token also carries your remaining seconds and tells LiveKit which agent to dispatch. The ten minutes are the window for joining. The agent reads the separate allowance to decide how long the call can run.
Once you're connected, your microphone audio travels through LiveKit and Deepgram transcribes it. When the agent decides you've finished speaking, it calls a method named hcm.ask on your browser:
const me = this.room.localParticipant;
return me.performRpc({
destinationIdentity: identity,
method: 'hcm.ask',
payload: JSON.stringify({
question, id, replaces,
}),
responseTimeout: 120_000,
});The browser passes the question to the same onAsk handler the text box uses. The full response goes to the page, and a short spoken version goes back to the agent. The agent never needs a database connection, a research tool or a model key. The Stop button uses that room connection too, in the other direction, so you can silence Ask without having to get a word in.

Deciding when you've finished talking
Try saying "Compare Paycom and Paylocity," taking a breath, and adding "on revenue growth." The first half sounds like a complete request. It's also broad enough to send Ask off doing research you didn't want.
LiveKit's turn detector listens for whether you sound finished. It waits between half a second and three seconds, with more room for a pause when you sound mid-sentence. Preemptive generation is off, because each submitted question uses quota:
turnHandling: {
turnDetection:
new inference.TurnDetector(),
preemptiveGeneration: {
enabled: false,
},
endpointing: {
minDelay: 500,
maxDelay: 3000,
},
interruption: {
mode: 'vad',
minDuration: 500,
minWords: 2,
},
},If the detector does split that example into two turns and the first answer hasn't started playing, the agent withdraws the first question and sends the combined version. That's what replaces is doing in the call above.
Interrupting an answer works on different settings. The agent yields after half a second of speech and two words, and a plain "stop" or "hold on" also counts. You can take over mid-answer, and a stray sound doesn't cost you the rest of one.
A phone on speaker hears Ask too. Ask says something, the phone mic picks it up, and Ask could end up with a question from itself. The browser's echo cancellation and noise suppression handle most of it. For the rest, the agent remembers what it's saying, strips those words from the front of an incoming turn, and drops a turn made entirely of echo. That protection stays on for six seconds after the agent finishes.
One answer for the screen and one for the ear
When you ask about a stock, the chart, date, ticker and citation all earn their space on screen. Read the same answer word for word and you get to listen to parentheses.

The page makes the spoken version with ordinary code. It takes up to three points and 400 characters, keeps whole sentences, removes parenthetical asides and ticker symbols, and points you to the screen for the rest. Because it selects from the finished answer, there's no second generation step that can change a price or introduce a new claim.
The ear has no chart, so every stock line names its company. Two closes read aloud come out as "Okta last closed at $182.87 on November 13, 2026. Also, Workday last closed at $196.45 on November 13, 2026." You never have to look down to work out which number you just heard.
The wait before the answer matters too. Across the first 67 voice questions answered in production, Ask took 1.1 to 12.2 seconds to finish an answer, with a median of 4.9, and most of that is research. If the answer isn't back after two and a half seconds, the agent says a short line like "One moment." A fast answer skips it.
Where the answer comes from
Once the page submits the question, the research path is the same one Ask uses for text. There are two model calls with different jobs, and the research call can take several rounds.

First, the researcher gets the question and 11 read-only tools. It can look up a company, retrieve stock closes, find a filing or read a relevant article. It gets up to six rounds and can request up to four tools at once. The server runs those tools and keeps track of exactly which evidence came back for this question. The researcher finishes by selecting source IDs.
The writer gets a fresh context containing the question and those selected sources. It returns structured findings. Each finding has the sentence Ask wants to show and a reference to the evidence behind it. For a filing, that might be @statement:4, the fourth numbered statement in the catalog the writer was given, paired with the source it came from. The server resolves the reference against the material it actually supplied. If a reference or quote doesn't match, the writer gets the reason and tries again, up to three attempts in all.
The server then renders the citations, dates and charts. That distinction matters when you look at the evals. Code can verify that statement four exists and that the writer quoted it accurately. Whether the statement supports the conclusion is a different question.
Both model calls go through one server-side client for the standard chat-completions API, set by a base URL, a model name and a key. The model provider (DeepSeek, Google, Anthropic, OpenAI or OpenRouter) can change without touching LiveKit, Deepgram or the page.
How I evaluate a change
Every change goes through the same four steps. A bad answer gets caught, it becomes a test case, the new build and the live build answer the same questions, and a gate decides whether the change ships. One real question shows all four.

A caller asked for Workday's latest stock price, and Ask gave the last stored close, $196.45. A question later, they said, "What'd you say?" Ask went back to research, and its answer described its own tool output: "The only data returned is a company directory." That's not what you want to hear after asking someone to repeat themselves.
First, the bad answer gets caught. Explorer, a private admin app, flags weak answers for review. This one was flagged for having no sources. Reading it showed the real problem. A repeat request doesn't need new research. It needs the previous answer.

Second, it becomes a test case. The case holds the conversation, ending with the Workday answer, and then the question "What'd you say?" A good reply has to say Workday and 196.45, stay short, and avoid internal phrases like "company directory." It doesn't need a citation, because the caller is asking Ask to repeat itself.
Third, both builds answer the same questions. The new build and the live build each answer all 73 real questions in the suite three times, on the same frozen data and settings. Web search is off, so a newer filing or market close can't creep into one side.
On this case, the live build repeated $196.45 all three times. The new build got it right once. Twice it looked the price up again and gave a newer close, $197.40. Both prices were in the data. Only one answered the question.

Fourth, the gate decides. That build passed the speed and cost checks and still failed, because the gate compares question by question. Improvements elsewhere can't average away a follow-up that used to work. One rewritten instruction line fixed it, and the next build repeated $196.45 all three times.
What else the gate checks
The gate is there so a change can't quietly make Ask worse. In plain words: a question the live build gets right can't start failing, a question both builds fail needs an owner and an expiry date, and speed and cost have to stay close to the live build.
| Check | Rule |
|---|---|
| New failures | A question the live build passes can't fail two of its three trials. |
| Known failures | Each needs a reason, an owner and an expiry date. An expired one blocks the release. |
| Completion | At most 4% of answers stop early. Any failed request fails the gate. |
| Speed | Median and 90th-percentile latency within 1.3 times the live build. |
| Cost | Model calls within 1.1 times the live build, prompt tokens within 1.15 times. |
Unit tests cover what doesn't need a real model, and a few fictional-company cases are scored by hand. Failed runs are kept, never rerolled. Holdout questions stay unread. Once one is opened to fix something, it moves to the development set.
Some checks only report. Code checks that every number in an answer appears in the evidence it cites, but nearly everything it flags is a real number cited to the wrong sentence, so it logs instead of blocking. A second model that flags unsupported claims reports too, without a vote.
What happens when you hang up
A browser tab can disappear and a phone can lock, so the server settles call time from signed LiveKit events. It uses the event for the caller leaving the room. The room itself hangs around for roughly another 40 seconds, which is a long time to charge someone for a conversation they've already finished. If the end event never arrives, the fallback charge is capped at three minutes.
The agent also handles silence. After 45 seconds it asks if you're still there, then hangs up 20 seconds later if you don't answer. Its session logs record interruptions, ignored echoes, merged questions and time limits without writing the spoken words. That's enough to investigate a call that kept cutting someone off without needing its audio.
Tools and reading
- LiveKit Agents: the SDK the voice agent is built on. LiveKit Cloud hosts the room and the agent.
- LiveKit turn detector: decides when you've finished speaking.
- Turn-taking tuning: the endpointing and interruption settings.
- Deepgram models and languages: Nova-3, for speech to text.
- Deepgram voices: Aura-2 and the Athena voice.
- Demystifying evals for AI agents: Anthropic on tasks, trials, graders and reading transcripts.
- Evaluation best practices: OpenAI's guide to eval-driven development, with human feedback to calibrate automated scoring.
Try it
Live mode is a super cool way to interact with an agent. With the tools, libraries and platforms available today, building one and taking it through evals is totally approachable. Try it for yourself at hcmintelligence.com/ask.