blog · · 13 min read
Orvena 4B: training a 4B model to do phone tasks without thinking
Orvena ↗ is an iPhone assistant I’ve been building, and its main model was Qwen3.5-4B running locally on the phone. Qwen3.5-4B gets more tasks right when it thinks first, and thinking means writing several hundred tokens of reasoning before each action. The phone generates about 21 tokens a second, so a typical task with thinking took more than half a minute. Turning thinking off brought that down to about 16 seconds and left too many tasks wrong, so for a while the best results in the app came from Apple’s Private Cloud Compute, which needs the internet.
I wanted the local model to be good enough without thinking, so I fine-tuned Qwen3.5-4B on scenarios built around the app’s own tools and released the result as Orvena 4B. It passes 440 of the 547 held-out scenarios on a bench that runs the app’s agent loop, and the stock model passes 345. It also runs at about the same speed as the stock model with thinking off.
01 / 10
Thinking costs seconds on a phone
The timings in this section come from a phone test I ran at the end, with 14 tasks run twice each on an iPhone 17 Pro Max, and they show the problem I started with. The app’s default at the time was thinking auto, and in that mode the base model wrote 757 tokens per turn and needed a median of 37.0 s per task. Generating 757 tokens at 20.6 tokens a second takes 36.7 s, so nearly all of that time is the model writing. The same model with thinking off wrote 213 tokens and needed 16.3 s.
Requests that nothing on the phone can do took a median of 78.3 s, because the base model with thinking kept trying tools and made 27 calls across six of them before it said it couldn’t help.
I looked for a way around this before training anything. I couldn’t find a newer small model that was better at calling tools in 30 languages. Moving the model to the Neural Engine that Cue runs on wouldn’t have helped either, because generating text with a 4B model is limited by how fast the weights can be read from memory, and the only Core ML port I found was 5.7 GB.
Turning thinking off was always possible, so the real question was why the model got tasks wrong without it. It answered without a tool and made up the facts, asked for things the phone already knew, got relative dates wrong, set a reminder when you asked for an alarm, or stopped after the first step of a longer task. These looked more like habits than missing knowledge, and fine-tuning can change habits.
phone test from section 7.7 of the report · decode time is tokens ÷ 20.6 per second 02 / 10
A bench that runs the app’s own agent loop
I needed a way to tell whether a model was better at what people ask Orvena to do before I trained anything, and public benchmarks couldn’t tell me that because they don’t use Orvena’s tools, prompts or phone data. I got the app’s Swift packages building on macOS so the bench could run the same agent loop, MLX backend and MCP client that ship in the app, with the system prompt and tool catalog exported from a phone.
Everything around the agent is simulated, with a simulated iPhone that answers the 38 native tools from each scenario’s calendar, contacts, places, routes and files in the same format as the real tools, and fake MCP servers that stand in for 12 connected services such as Notion, Linear and Todoist with their real tool lists. The clock is always set to Wednesday 4 November 2026 at 10:30.
Language-model agents wrote the scenarios in 30 languages. Each scenario has a request, the data on the phone and a set of checks for which tools should be called with which arguments, what must not be called, and what the answer has to say and in which language. The figure replays one of them at the speed it was recorded, where the user asks for an alarm to leave for a job interview by tube.
The first runs also found three bugs in the app itself. The prompt-injection filter was removing connector arguments named
titleanddescription, calendar search only matched whole words, and the model saw an empty tool list right after loading a skill.The base model passed 60.1% of the held-out scenarios with thinking off at a median of 7.2 s per task on the Mac, and 64.0% with thinking auto at 15.6 s. A second run with a different seed changed the result for 13 to 18% of the scenarios, so I compared every model over more than one seed from then on.
en-chain-interview-leave-alarm from the published bench results, seed 7 03 / 10
Only runs that pass become training data
The training data comes from the same bench, where agents wrote 2,616 more scenarios in six task families that don’t overlap with the held-out ones. A run on one of them only becomes training data if it passes every check, whichever model produced it. I used Qwen3.6-35B-A3B at 4 bits with thinking off as the teacher for the first runs, and it passed 1,533 of them.
The teacher was weaker than the base model on travel and files even though it is a much bigger model, and it read the Bulgarian word “вдругиден” as tomorrow when it means the day after tomorrow. That is why the checks decide what goes into the training data, and the size of the model that wrote a run doesn’t.
I also learned to run one model job at a time here. Running the 35B teacher, a paused training run, Xcode and four simulators at once made my laptop so unresponsive that I had to force quit the terminal.
The figure shows the data as it ended up after the fixes in the next sections. The teacher got a second try with thinking on the scenarios it had failed, and its reasoning was left out of the training text. The base model’s own passing runs went in as well, along with extra sets for no-tool answers, connections, places and restraint. The final set has 11,161 rounds from 3,397 scenarios, where a round is one model call.
counts from section 3 and Figure 2 of the report 04 / 10
Training on a Mac
I started training on my M3 Max with mlx-lm and didn’t get very far. Qwen3.5 has 24 Gated DeltaNet layers that use a kind of linear attention, and mlx-lm trained them with a loop that goes through the sequence one token at a time. Training stopped at 8,192 tokens with
Resource limit (499000) exceeded, so I replaced that loop with one that processes 128 tokens at a time and gives the same results to float32 precision.The bigger saving came from how the agent turns are laid out for training. A single turn is several model calls, and each call’s prompt is the previous prompt plus the model’s last tool call and the tool’s result, so training each call as its own example means going through the same prompt over and over. I joined the rounds that extend each other into one sequence and took the loss only on the model’s own outputs, so each output still sees exactly the context it had when it was recorded. That cut the tokens by 49% and brought an epoch on the Mac down to about 13 hours. Each output’s loss is also averaged over its own tokens, so a 15-token tool call counts as much as a 65-token answer.
the recorded turn from the previous figure, tokenized with Qwen3.5’s chat template · weights from equation 1 of the report 05 / 10
The first model picked up the teacher’s habits
The first model was an adapter trained only on the teacher’s passing runs. I compared it with the base model over three seeds on the 407 scenarios the bench had at that point. It passed 63.4% and the base model 59.5%, which just missed the 4-point gain I had set as the bar.
The average also hid that it got much better at multi-step tasks, calendar questions and follow-ups and much worse at requests that need no tool. The no-tool scenarios in its data were all general-knowledge questions, so it had never seen a calculation it should just do itself, and it called
get_current_timebefore splitting a bill or working out VAT and then got the sum wrong. It also learned from the teacher to stop too early with connected services, because the teacher’s runs without thinking only passed 56% of the MCP scenarios.I changed the data for the second model to fix that. The base model’s own passing runs are used for no-tool and MCP scenarios wherever there is one, because they keep what it already did well. The teacher got a second try with thinking on the MCP and multi-step scenarios it had failed, and I added 450 new no-tool scenarios with 15 per language.
Table 4 of the report, mean of three seeds on the first 407 held-out scenarios 06 / 10
Trying to make it faster
I also wanted to see whether the model could be faster than the base model with thinking off. Turning it into a mixture of experts was out, because converting a dense model takes billions of tokens, I had about 12 million, and the result would need more memory rather than less.
Removing layers looked more realistic, with one constraint from the architecture. Every fourth layer in Qwen3.5 is full attention and the app’s loader depends on that pattern, so layers can only be removed in groups of four. I measured how much the loss went up with each group removed, and layers 8 to 15 cost the least. Removing only 8 to 11 without retraining dropped the model from 12 of 24 test scenarios to 1, because it stopped calling tools. Training the remaining 24 layers after removing all eight got it to 50.4%, and the base model was at 60.2%. Those eight layers are a quarter of the model, and they held general abilities that 12 million tokens of Orvena data couldn’t put back, so I shelved the idea.
The other option was the multi-token prediction head in Qwen3.5, usually called the MTP head. It is an extra layer that guesses the token after the next one, so a single pass produces two tokens when the guess is right. The community 4-bit conversion the app used had left it out, so I put it back and retrained it on Orvena 4B’s own outputs, and generation on the phone went from 20.6 to 23.0 tokens a second.
Orvena 4B only writes about 170 tokens a turn, so the faster generation only takes it from 8.2 s to 7.7 s. The median task took 14.0 s with the head and 13.5 s without it, and that difference is within the noise between runs. The head would also add 67.9 MB to the download, so the app ships without it and the bf16 release keeps it for servers that run speculative decoding.
draft hits computed with the released 4-bit model and the retrained head on Orvena 4B’s recorded answer · acceptance and phone numbers from section 5 of the report 07 / 10
The second model passed the bench and failed on my phone
The second model passed 70.1% and roughly doubled the base model on multi-step tasks, so I put it on my phone, where it did worse than those numbers suggested. It searched my Notion and Airtable in 3 of 8 attempts when I asked for opening hours, the weather or a transit route. It also listed cafés around me in Copenhagen all four times I asked for cafés near a square in another city, as if they were near that square.
The Notion and Airtable searches came from the training data, where connections were installed in only 12% of the everyday scenarios and in every MCP one, so the model learned that an installed connection meant the task was about it. The café results came from the bench, because the simulated phone returned places in whatever city the request named and the app’s
search_nearby_placeshad no location parameter at all, so on a real phone it could only search around me.I fixed both in the app and in the training data. The tool got a
nearparameter, and the training data got 600 copies of everyday scenarios with two to six connections installed and new scenarios for places, music and transit. I also added 140 scenarios to the bench, with 40 that copy the failures from my phone and 100 everyday tasks with connections installed.Table 5 of the report: 100 held-out everyday copies with 2 to 6 connections installed 08 / 10
The third model got worse at MCP and at holding back
The third model no longer made unneeded connector calls, and its MCP score dropped to 31.7 of 62 while the base model had 45, because its MCP examples had been recorded against an older version of the app’s tool list. A short extra round of training on new MCP examples changed the whole model and made it worse almost everywhere. I also got curious whether editing the adapter by hand could keep the gains and get MCP back, so I tried scaling some of its layers and removing some directions from it, and none of the edits beat the unedited adapter.
I ended up recording all the MCP examples again on the current tool list and moving training to a rented H100. The run took 71 minutes and cost about $4, and the model passed 437 of 547. The same training took about a day on the Mac.
The public benchmarks showed a problem my bench hadn’t caught. BFCL has a live irrelevance test where none of the offered tools fits the request, and the base model correctly called nothing in 78.1% of those cases and this model in only 34.2%. IFEval checks whether answers follow instructions, and the model’s score there fell from 82.1 to 71.2.
Orvena 4B is that model with two more sets of data. The first is 370 passing runs on requests that need no tool or that nothing can do in all 30 languages, and the second is 600 instruction-following answers from the base model that passed IFEval’s checker. They brought irrelevance back up to 50.1% and IFEval to 78.6, and the bench score stayed about the same at 440.
Table 6 of the report, 60 held-out restraint scenarios, seed 7 · BFCL and IFEval from Table 11, bf16 on vLLM 09 / 10
Results
Orvena 4B passes 80.4% of the 547 held-out scenarios, the base model 63.1% and Apple’s on-device model 19.0%. Apple’s model supports 13 of the 30 languages and the app only sends it requests in those, so the fairer comparison is on those 13 languages, where the three models pass 84.6%, 67.3% and 29.6%.
The biggest gain is on multi-step tasks, where Orvena 4B passes 43 of 70, the base model 17 and Apple’s model 1. One of them is set in Copenhagen, where the user says “My notes from the flat viewing have the plumber’s number in them somewhere, remind me tomorrow at 10 to call him.” Orvena 4B searched the files, found that the plumber is Henrik and set a reminder for 10:00 to call him. The base model searched the existing reminders instead of the files and set one to “Call plumber about flat viewing notes”, and Apple’s model scheduled a task.
It is still a little behind the base model in three small categories. It passes 16 of 23 no-tool requests where the base model passes 19, 9 of 13 requests nothing can do where the base model passes 10, and 5 of 6 contact tasks where the base model passes all 6. It never answered in the wrong language, which the base model did 4 times and Apple’s model 13 times. Its median on the phone is 15.1 s per task with thinking off, which is about the same as the 16.3 s of the base model with thinking off and well under the 37.0 s with thinking auto.
I compared the candidate models on this bench before picking one, so it isn’t a fully independent test of the result.
Orvena bench, seed 7, 4-bit MLX in the app’s agent loop with thinking off · Table 7 of the report 10 / 10
Where it still guesses
Orvena 4B matches the base model on public benchmarks where a tool call is clearly needed, and both score 81.5 on BFCL’s non-live AST tests even though those use different tools and conventions from Orvena’s. It tends to call a tool anyway when a detail or a fitting tool is missing. It asked the user in 8 of 30 held-out requests that leave out a detail the phone can’t supply, and the base model asked in 18. It set a 60-minute timer when asked to set one for the lasagna, and When2Call and ToolSandbox show the same pattern.
I trained a version with examples of asking, and it asked in 26 of the 30. That version also got worse on connected services, with 41 of 62 where Orvena 4B has 51, and it got stuck in loops on BFCL’s multi-turn tasks, so I didn’t release it. Asking instead of guessing is the main thing I want to fix in the next version.
recorded turns from the published ask set, seed 7 · public benchmarks from Table 11 of the report, bf16 on vLLM, thinking off