The voice agent doesn't answer anything itself. It sends each question back to the open page, which already knows who is signed in, which conversation they're in, and how to show a cited answer. A spoken follow-up uses the same history, research tools and source checks as a typed one.

That keeps the voice side small. Most of the care goes into a pause, a phone speaker, or someone saying "What'd you say?"

Live voice in Ask: two spoken questions and an interruption. The panel on the right shows each call as it happens.

How the pieces fit together

The HCM app runs on Node.js and Express, with SQLite holding company records, news, filings, stock closes and Ask history. Live mode adds a WebRTC room hosted by LiveKit Cloud, on its free Build plan, and a small voice agent built with the LiveKit Agents SDK. The agent uses Deepgram's Nova-3 for speech recognition and Aura-2 with the Athena voice to speak. Company names like Paycom, Paychex and Paylocity are in the recognition vocabulary, because a transcript can be one company name off and still look like a perfectly reasonable research question.

The whole thing is deliberately light: one server and one SQLite file. At thousands or millions of concurrent callers, those are the parts that would change.

Architecture diagram. On the left the user's browser, which exchanges audio with LiveKit Cloud, and below it Deepgram, which the voice agent calls for speech. In the middle one DigitalOcean droplet holding Caddy, the Node.js and Express app, SQLite and the ingestion jobs. On the right the outside services the server calls: the model API, which also provides the capped web search tool, Brave Search for news ingestion only, and SEC, news feeds and YouTube.
The browser never calls the model. Only the server calls out.

When you start a call, the HCM server checks your remaining voice time, creates a unique room name, and signs a token that can join that room for the next ten minutes. The token also carries your remaining seconds and tells LiveKit which agent to dispatch. The ten minutes are the window for joining. The agent reads the separate allowance to decide how long the call can run.

Once you're connected, your microphone audio travels through LiveKit and Deepgram transcribes it. When the agent decides you've finished speaking, it calls a method named hcm.ask on your browser:

const me = this.room.localParticipant;
return me.performRpc({
  destinationIdentity: identity,
  method: 'hcm.ask',
  payload: JSON.stringify({
    question, id, replaces,
  }),
  responseTimeout: 120_000,
});

The browser passes the question to the same onAsk handler the text box uses. The full response goes to the page, and a short spoken version goes back to the agent. The agent never needs a database connection, a research tool or a model key. The Stop button uses that room connection too, in the other direction, so you can silence Ask without having to get a word in.

Sequence diagram of live voice across five lanes: user, browser, LiveKit room, voice agent and app server. The browser gets a short-lived token, joins the room, the agent greets, the user speaks, the agent asks the browser to run the question, the browser sends a normal Ask question to the server, the full answer goes on screen and a short spoken version is played.
The agent hears and speaks. The page asks Ask as the signed-in user, exactly as if the question were typed.

Deciding when you've finished talking

Try saying "Compare Paycom and Paylocity," taking a breath, and adding "on revenue growth." The first half sounds like a complete request. It's also broad enough to send Ask off doing research you didn't want.

LiveKit's turn detector listens for whether you sound finished. It waits between half a second and three seconds, with more room for a pause when you sound mid-sentence. Preemptive generation is off, because each submitted question uses quota:

turnHandling: {
  turnDetection:
    new inference.TurnDetector(),
  preemptiveGeneration: {
    enabled: false,
  },
  endpointing: {
    minDelay: 500,
    maxDelay: 3000,
  },
  interruption: {
    mode: 'vad',
    minDuration: 500,
    minWords: 2,
  },
},

If the detector does split that example into two turns and the first answer hasn't started playing, the agent withdraws the first question and sends the combined version. That's what replaces is doing in the call above.

Interrupting an answer works on different settings. The agent yields after half a second of speech and two words, and a plain "stop" or "hold on" also counts. You can take over mid-answer, and a stray sound doesn't cost you the rest of one.

A phone on speaker hears Ask too. Ask says something, the phone mic picks it up, and Ask could end up with a question from itself. The browser's echo cancellation and noise suppression handle most of it. For the rest, the agent remembers what it's saying, strips those words from the front of an incoming turn, and drops a turn made entirely of echo. That protection stays on for six seconds after the agent finishes.

One answer for the screen and one for the ear

When you ask about a stock, the chart, date, ticker and citation all earn their space on screen. Read the same answer word for word and you get to listen to parentheses.

Live voice answering How is ADP stock doing this month. The full written answer with a citation and a closing price chart is on screen, and the voice bar below shows Speaking, the question, and a caption of the spoken line ADP last closed at 307.72 dollars on November.
The screen carries the full answer, citation and chart. The voice says a shorter version, names the company without the ticker, and captions it.

The page makes the spoken version with ordinary code. It takes up to three points and 400 characters, keeps whole sentences, removes parenthetical asides and ticker symbols, and points you to the screen for the rest. Because it selects from the finished answer, there's no second generation step that can change a price or introduce a new claim.

The ear has no chart, so every stock line names its company. Two closes read aloud come out as "Okta last closed at $182.87 on November 13, 2026. Also, Workday last closed at $196.45 on November 13, 2026." You never have to look down to work out which number you just heard.

The wait before the answer matters too. Across the first 67 voice questions answered in production, Ask took 1.1 to 12.2 seconds to finish an answer, with a median of 4.9, and most of that is research. If the answer isn't back after two and a half seconds, the agent says a short line like "One moment." A fast answer skips it.

Where the answer comes from

Once the page submits the question, the research path is the same one Ask uses for text. There are two model calls with different jobs, and the research call can take several rounds.

Pipeline diagram in six steps. One, a gate before any model call. Two, a research loop of up to six rounds where the model decides what to look up and the server runs up to four read-only tools in parallel. Three, the model selects source IDs. Four, a writer in a clean context produces findings with exact quotes. Five, the server checks quotes against the evidence and rejects mismatches. Six, the server renders citations, sources, dates and charts and streams the answer.
The model picks evidence and writes sentences. The server does everything that has to be exact.

First, the researcher gets the question and 11 read-only tools. It can look up a company, retrieve stock closes, find a filing or read a relevant article. It gets up to six rounds and can request up to four tools at once. The server runs those tools and keeps track of exactly which evidence came back for this question. The researcher finishes by selecting source IDs.

The writer gets a fresh context containing the question and those selected sources. It returns structured findings. Each finding has the sentence Ask wants to show and a reference to the evidence behind it. For a filing, that might be @statement:4, the fourth numbered statement in the catalog the writer was given, paired with the source it came from. The server resolves the reference against the material it actually supplied. If a reference or quote doesn't match, the writer gets the reason and tries again, up to three attempts in all.

The server then renders the citations, dates and charts. That distinction matters when you look at the evals. Code can verify that statement four exists and that the writer quoted it accurately. Whether the statement supports the conclusion is a different question.

Both model calls go through one server-side client for the standard chat-completions API, set by a base URL, a model name and a key. The model provider (DeepSeek, Google, Anthropic, OpenAI or OpenRouter) can change without touching LiveKit, Deepgram or the page.

How I evaluate a change

Every change goes through the same four steps. A bad answer gets caught, it becomes a test case, the new build and the live build answer the same questions, and a gate decides whether the change ships. One real question shows all four.

Diagram of the eval loop. A weak real answer becomes a case with frozen evidence. A change is made. The candidate runs every case three times on the same snapshot as a recorded run of the live build. Each answer gets mechanical checks, the two builds are compared question by question at a gate, and a pass ships while a fail goes back to the change.
Real failures become cases, and every change runs against the live build on the same frozen evidence.

A caller asked for Workday's latest stock price, and Ask gave the last stored close, $196.45. A question later, they said, "What'd you say?" Ask went back to research, and its answer described its own tool output: "The only data returned is a company directory." That's not what you want to hear after asking someone to repeat themselves.

First, the bad answer gets caught. Explorer, a private admin app, flags weak answers for review. This one was flagged for having no sources. Reading it showed the real problem. A repeat request doesn't need new research. It needs the previous answer.

Explorer's review queue showing one production voice answer flagged No sources. The question is What'd you say, the answer says the only data returned is a company directory and Salesforce does not appear in it, and a note for the eval owner says it should have repeated the last answer in plain words. A Request eval case button sits under the note.
The real turn. The question just before was about Salesforce, which the site didn't track yet, so Ask researched again and repeated that answer, internal wording included.

Second, it becomes a test case. The case holds the conversation, ending with the Workday answer, and then the question "What'd you say?" A good reply has to say Workday and 196.45, stay short, and avoid internal phrases like "company directory." It doesn't need a citation, because the caller is asking Ask to repeat itself.

Third, both builds answer the same questions. The new build and the live build each answer all 73 real questions in the suite three times, on the same frozen data and settings. Web search is off, so a newer filing or market close can't creep into one side.

On this case, the live build repeated $196.45 all three times. The new build got it right once. Twice it looked the price up again and gave a newer close, $197.40. Both prices were in the data. Only one answered the question.

An eval run in Explorer for the question What'd you say. The candidate row shows run 1 passing and runs 2 and 3 failing the mustMention check; all three baseline runs pass. Below, the full report shows run 1 repeating the 196.45 close and runs 2 and 3 giving a newer close of 197.40.
Runs 2 and 3 fail must-mention: they quote a fresh close of 197.40 instead of repeating 196.45.

Fourth, the gate decides. That build passed the speed and cost checks and still failed, because the gate compares question by question. Improvements elsewhere can't average away a follow-up that used to work. One rewritten instruction line fixed it, and the next build repeated $196.45 all three times.

What else the gate checks

The gate is there so a change can't quietly make Ask worse. In plain words: a question the live build gets right can't start failing, a question both builds fail needs an owner and an expiry date, and speed and cost have to stay close to the live build.

CheckRule
New failuresA question the live build passes can't fail two of its three trials.
Known failuresEach needs a reason, an owner and an expiry date. An expired one blocks the release.
CompletionAt most 4% of answers stop early. Any failed request fails the gate.
SpeedMedian and 90th-percentile latency within 1.3 times the live build.
CostModel calls within 1.1 times the live build, prompt tokens within 1.15 times.

Unit tests cover what doesn't need a real model, and a few fictional-company cases are scored by hand. Failed runs are kept, never rerolled. Holdout questions stay unread. Once one is opened to fix something, it moves to the development set.

Some checks only report. Code checks that every number in an answer appears in the evidence it cites, but nearly everything it flags is a real number cited to the wrong sentence, so it logs instead of blocking. A second model that flags unsupported claims reports too, without a vote.

What happens when you hang up

A browser tab can disappear and a phone can lock, so the server settles call time from signed LiveKit events. It uses the event for the caller leaving the room. The room itself hangs around for roughly another 40 seconds, which is a long time to charge someone for a conversation they've already finished. If the end event never arrives, the fallback charge is capped at three minutes.

The agent also handles silence. After 45 seconds it asks if you're still there, then hangs up 20 seconds later if you don't answer. Its session logs record interruptions, ignored echoes, merged questions and time limits without writing the spoken words. That's enough to investigate a call that kept cutting someone off without needing its audio.

Tools and reading

Try it

Live mode is a super cool way to interact with an agent. With the tools, libraries and platforms available today, building one and taking it through evals is totally approachable. Try it for yourself at hcmintelligence.com/ask.