Mehran Toosi · Principal Software Engineer · Copenhagen

blog · · 5 min read · updated

Running an agent with a 4B model on an iPhone

Orvena ↗ is an iPhone assistant where the whole agent runs on the phone: the model, the tool loop, speech recognition and speech output. It uses Qwen 3.5 4B through MLX and needs an iPhone with 8 GB of memory. I had built agent harnesses before and expected the agent to be the hard part. It was the model. A 4B model at 4 bits has very little room for error, especially when it calls tools.

This post goes through the parts that decided whether it works: memory, the prompt cache, thinking, how tools are offered to the model, how its calls are repaired, a runtime update that broke it, and the tests that run on the phone.

  1. 01 / 09

    Memory decides the context window

    iOS ends the foreground app at about 6.4 GB on an 8 GB iPhone, and there is no swap to fall back on. The 4-bit weights, including the vision encoder, take about 3 GB. What is left has to hold the KV cache, the speech models, the runtime’s buffers and the app itself.

    So the context window is a memory budget, not a model setting. Orvena caps the local window at 16,384 tokens, which is what the memory left over can afford.

    an 8 GB iPhone · weights from the model files, window from the app, KV at fp16
  2. 02 / 09

    Only 8 of the 32 layers keep a KV cache

    Qwen 3.5 uses a hybrid layout. Every fourth layer uses full attention and keeps keys and values for every token. The other 24 use linear attention and keep a fixed-size state, about 25 MB in total, however long the conversation gets.

    With 4 KV heads of 256 dimensions at fp16, a full-attention layer costs 4 KB per token, so the eight of them cost 32 KB, and a 16k window costs about 512 MB. If all 32 layers kept a cache it would be 128 KB per token and 2 GB, and the window would have to be a quarter of the size.

    layer layout and head sizes from the model config · sizes at fp16
  3. 03 / 09

    The prompt renders the same bytes every turn

    On a phone, prefill is the slowest part of a turn. The system prompt and the skill summaries are a few thousand tokens, and computing them again on every turn costs seconds. MLX can reuse the KV cache of a prefix it has already processed, so each turn is planned as the cached prefix plus whatever is new. The system prompt is prefilled while the app is still opening, so the first message doesn’t wait for it.

    That only works if the prompt renders to exactly the same tokens every time. A timestamp in the system prompt, or anything else injected per turn, changes the first tokens and throws the whole cache away.

    segment lengths illustrative
  4. 04 / 09

    A language tag that keeps the cache

    I live in Copenhagen and my calendar is mostly Danish, so tool results are often Danish too. At some point the model started answering my English questions in Danish. A rule in the system prompt didn’t help: all the Danish in the context outweighed the instruction.

    What works is tagging each user message with its language, detected on the phone with Apple’s language recognizer and written as [message language: English] right after the message text. The tag depends only on the message, so the message renders the same in every later turn and the cached prefix still matches. Telling the model the user’s language once per turn would have been simpler, and it would have broken the cache.

    calendar events from the on-device test scenario · answers illustrative
  5. 05 / 09

    Thinking is set by the phone’s memory

    Qwen 3.5 can reason before it answers. That helps with multi-step tool use, but it makes a turn three to five times slower. Deciding per request whether thinking is worth it would need another model in memory, and there is no room for one.

    So the budget comes from the device: no thinking under 6 GiB of physical memory, 288 tokens under 8 GiB, 576 from 8 GiB, and half of that in Low Power Mode. Users can change the budget or turn thinking off.

    budget rule from the app · the model’s 0.75 multiplier is already applied
  6. 06 / 09

    38 tools, loaded as 16 skills

    Sending all 38 tool schemas on every turn costs prefill time and makes the choice harder for a small model. The tools are grouped into 16 skills. The model starts with a one-line summary of each skill and a load_skills tool, and the schemas of a loaded skill join the same request.

    At first the skills were listed in the system prompt as name and description lines, and the model kept calling them as if they were tools, whatever the prompt said. Moving the same list into load_skills itself, as the allowed values of its skills argument, took hallucinated skill calls to zero. The content didn’t change, only where it lived.

    skills and tool counts from the app
  7. 07 / 09

    Calls are repaired before they run

    A 4B model sends imperfect arguments. Before a tool runs, the call is resolved against the turn’s catalog, its arguments are coerced to the schema, and the result is validated. A number sent as "9" becomes 9, and an object or array sent as a JSON string is parsed.

    A call that still doesn’t fit doesn’t run. The model gets the reason with the exact path, such as $.minutes must be at most 1440., and one more try. A skill name used as a tool loads that skill and tells the model which tools it now has, and any other unknown name is told to use a declared one.

    messages, schema and approval text from the app · the request is illustrative
  8. 08 / 09

    A runtime update that broke tool calls

    I updated the inference runtime for speed. Everything got faster, the tests were green, the prompts were byte-identical, and the model started calling tools that don’t exist. The update had changed how the prompt is split for prefill: from chunks of exactly 512 tokens to the fewest equal chunks.

    Qwen 3.5 carries its linear-attention state from chunk to chunk, so other boundaries round differently and the logits move a little. A 4B model has no headroom, and that was enough to pick a tool name that isn’t there. A temperature-0 A/B with the old chunking reproduced the old output verbatim. The old boundaries are now pinned in code and in a test, and every runtime update gets the same A/B.

    chunking rules from the runtime and the app · prompt length and the invented name illustrative
  9. 09 / 09

    Every change runs on the phone first

    None of this showed up in the simulator or in public benchmarks. Every prompt or harness change runs an XCTest suite on a physical iPhone with the real model: nine scenarios, such as moving a calendar event, setting an alarm from a travel time, or retrying after a rejected argument. Tool results are canned, and each scenario asserts which tools were called and what the answer must and must not contain.

    Single runs swing, so decisions are made on repeated runs. About half of the changes I was sure would help failed, and nothing ships without a clean run.

    scenario names from the test suite · pass and fail pattern illustrative

Orvena runs on iPhones with 8 GB of memory, the iPhone 15 Pro and newer. It won’t beat a large cloud model on intelligence or speed, but it does real work without anything leaving the phone. If you prefer a cloud model, you can connect your own OpenRouter or OpenAI key, and the app asks before anything is sent to that provider.