blog · · 8 min read
Cue: a small on-device model for turn-taking in voice mode
Orvena ↗ has a voice mode that keeps listening while it speaks. That only works if the assistant can tell what a short piece of speech means. While it is answering, “mm-hm” should not stop it, “wait, stop” should, and “ok got it, now check Lisbon” should stop it and become the next request. When it is idle, “can you hear me?” or “thanks” does not need the main language model at all.
Cue-0.4B is the model I trained to make both decisions from the speech recognizer’s transcript and the assistant’s own words. It is half of Qwen3-0.6B-Base with three small heads, covers 30 languages, runs in about 10 ms on the iPhone’s Neural Engine while the main model uses the GPU, and is released with open weights and a technical report. This post goes through how it works.
01 / 10
Hearing speech is not the same as being interrupted
Echo-cancelled input lets the microphone stay open while Orvena talks, so the recognizer hears everything: backchannels, reactions, someone else in the room, the television. When speech arrives, Orvena ducks its voice and records how much of the current sentence had been played. When the user pauses, Cue reads the sentence with that cut point and the user’s words, and returns one of three actions: keep going, stop or request.
For a request it also points at the word where the request starts, so from “ok got it, now check Lisbon” only “now check Lisbon” becomes the next turn.
three interruptions of the same sentence · phrases from the report, probabilities recorded from Cue-0.4B 02 / 10
Some new turns need no language model
The second decision is made when a user turn starts, before the main model is called. “Can you hear me?”, “thanks”, “hold on” and “say that again” have short, predictable answers, and the app handles nine of these quick intents itself. “Say that again” replays the last answer exactly, which a model can’t promise.
The hard part is that the same words also start real requests. “Thanks, now add milk to the shopping list” is a look-alike and goes to the main model, and so does “okay” right after the assistant asked “Should I add it to your calendar?”. That is why Cue also sees the end of what the assistant last said.
new-turn classes and replies from the report 03 / 10
It runs beside the main model, so it runs on the Neural Engine
Both decisions have to take a few tens of milliseconds at most, and they come while the main model is decoding on the GPU. I measured one forward pass at 128 tokens on an iPhone 17 Pro Max. A 28-layer version took 42 ms on the GPU when the phone was idle and 890 ms while a decode loop ran there, and 104 and 523 ms on the CPU. On the Neural Engine it took 18.6 and 23.4 ms.
The Neural Engine is the only unit that stays fast while the GPU is busy, and it sets the rules for the model: fixed input shapes, float16 arithmetic and compact weights. Keeping half the layers brought a pass to 8.2 ms idle and 10.5 ms under load.
Table 10 of the report · measured before training, with untrained heads 04 / 10
Half of a small language model, with three heads
Cue keeps the embedding table, the first 14 of the 28 decoder layers of Qwen3-0.6B-Base and the final norm. A rank-16 LoRA adapter on all seven linear projections of every layer is trained and then merged, so the released model keeps the original layer shapes. That is 376.4 million parameters, 5.6 million of them trained.
Three heads read the hidden state of the last prompt token: an action head with three outputs, an intent head with ten, and a pointer over the user’s words. Interruption prompts end with “Decision:” and new-turn prompts with “Reply:”, so one forward pass serves both questions and the prompt says which one is being asked.
parameter counts from the saved checkpoints (Table 3) 05 / 10
One forward pass, recorded
This is the released model running on the prompt for “ok got it, now check Lisbon”, with every value recorded from that run. The prompt is 33 tokens. The window is always 128, so 95 pad tokens follow, and a one-hot mask marks the last prompt token. The app gathers the 33 int8 embedding rows itself and passes them in as float16.
The bars show how much the last token attends to each token in each of the 14 layers, averaged over the 16 heads. Through the middle layers it mostly attends to the prompt’s own structure: the colons, the line breaks and “Decision”. In the last layers it attends to the user’s words. Token 0 is left out; from layer 4 on it takes 20 to 66% of the attention, an attention sink, as in many language models.
Then the heads read the last token’s state. The action logits divided by 3.80 give request 0.997 after the softmax, so p(keep going) is far below 0.7, and the pointer puts 1.00 on “now”. The same pass gives keep going 0.987 for “mm-hm” and stop 0.9995 for “wait, stop”.
values recorded from Cue-0.4B on these prompts · the int8 Core ML package gives the same probabilities to within 0.0005 06 / 10
The pointer marks where the request starts
The pointer projects the last token’s state and each user token to 256 dimensions and scores them against each other. Only word starts in the user’s text can be chosen; every other token gets −10⁴.
Its most likely word sometimes lands too late. Losing a “now” is harmless, losing the verb is not, and keeping an extra “okay” costs the main model nothing. So the app takes the earliest word start with a probability of at least 0.15.
The figure is a request from the Finnish test set: “meneekö sinne bussi myös sunnuntaisin”, does a bus go there on Sundays too. The most likely start is “myös” at 0.64, which would send only “myös sunnuntaisin”, also on Sundays. “Meneekö” has 0.36, so the app sends the whole request. On the 340 test requests the model decided as requests, the rule moved the exact start from 92.4% to 92.6% and the requests that lost words from 6.2% to 5.0%.
a request from the Finnish test set · probabilities from the int8 Core ML model · rates on the independent test (section 7.4) 07 / 10
Training data from phrase banks in 30 languages
There is no corpus of people interrupting an assistant in 30 languages, so language-model agents wrote phrase banks for each one: sentences an assistant says, and what users say in each category, from backchannels and side talk to stops and requests. Every language has the same 302 interruption entries, plus about 430 phrases for new turns.
Examples are composed from the banks. A category is drawn by weight, an assistant sentence is cut at a random word, and the user’s words are built from pieces such as an acknowledgement, a connector and a request; 40% lose their punctuation and case, as recognizer output does. Each bank is split by item before composing, so no test phrase appears in training. That gave 62,996 training examples.
The numbers I report come from test sets written separately: 900 interruptions and 1,800 new turns, by agents told not to read the banks.
bank sizes and splits from Tables 4 and 5 · the composed example is illustrative 08 / 10
Calibrated probabilities, and thresholds that weigh errors differently
A model fine-tuned on clean synthetic labels is overconfident. One temperature per head, fitted on the validation split, fixes most of that: 3.80 for actions, which brings the expected calibration error on the synthetic test from 0.051 to 0.011, and on the independent test from 0.089 to 0.039.
The errors don’t cost the same. Stopping on a backchannel cuts the answer short; talking through a stop ignores the user. Cue keeps going only when p(keep going) is at least 0.7, and otherwise takes the more likely of stop and request. For new turns a quick reply to a real request is the worst error, so a quick reply needs 0.9, and a goodbye, which closes voice mode, needs 0.97. At 0.9, 1 of 990 requests got a quick reply, and 10.4% of quick turns went to the main model instead.
Table 7 (development set) and Table 8 (quick test), 14-layer Core ML model 09 / 10
Fitting half of Qwen3 onto the Neural Engine
The PyTorch model is traced at one fixed shape and converted with coremltools: always 128 tokens, right-padded, with a one-hot mask that selects the last prompt token by multiplication instead of indexing. The 151,936 × 1,024 embedding table would be the largest tensor in the package, so it stays outside as int8 rows with one scale each, memory-mapped by the app, which gathers the prompt’s rows and passes float16 embeddings in.
RMSNorm needed a trick. Computing it directly squares the activations, and in float16 a square overflows once a value passes 256. The LayerNorm of [x, −x] has zero mean, so its variance is the mean square of x and the first half of its output is RMSNorm(x). Linear weights are int8 per output channel, at a cosine similarity of 0.9996 to float32; 4-bit palettization ran at the same speed but drifted to 0.956.
conversion choices from section 8.1 · activation values illustrative 10 / 10
How it does, and where it fails
On the independent tests, with the shipped thresholds, Cue makes the right interruption decision in 90.4% of 900 cases (95% interval 88.3 to 92.2) and the right new-turn decision in 95.2% of 1,800. A 28-layer model trained the same way scores 91.3% and 96.0%, which is not a significant difference at this test size (McNemar p = 0.37 and 0.08), so the app ships the half that is 2.3 times faster.
Most false interrupts are side talk or background speech phrased like a request: ordering a coffee, telling a child to put the tablet away, a quiz host reading a question. From text alone it can’t be told who is being addressed. By language, Estonian interruptions are the weakest at 70%, though with 30 examples per language most differences between languages are noise.
per-language results (Figure 5) and errors (Table 15) of the 14-layer model