Mehran Toosi · Principal Software Engineer · Copenhagen

blog · · 5 min read

Routing an agent with four small models

ecoai is the internal AI platform for Slack and desktop that I built at Visma e-conomic. Before its main model can work on a message, three things have to be decided: which model setup to use, which of its roughly 160 tools to load, and what to do first. For a long time the model worked these out itself, by trial.

This post is about the four small models that now make those decisions before the first model call. They are TF-IDF features and logistic regression, they run in about a tenth of a millisecond, and most of the work went into making them stay quiet when they are not sure.

  1. 01 / 09

    Three decisions come before the work

    A message can need the cheap, fast setup or a reasoning model with a stronger fallback. It can need tools for the data warehouse, the code, tickets or monitoring. And it needs a first step.

    The tools are the expensive part. Each one comes with a schema and guidance, and with about 160 of them the definitions alone would take a large share of every request. So a turn starts with a small set of core tools, and the model asks for more when it realises it needs them. Each of those requests, and each wrong first step, is another model call: a few seconds, with the whole context sent again.

    the tool clusters follow the real catalog’s group sizes · the call sequence is illustrative
  2. 02 / 09

    Small trained models are faster than asking another model

    There are three ways to make these decisions. The model can find out by trial, which is the detours above. Another LLM call can decide before every turn, which ecoai once did for the model tier; that adds a round trip to every message and is hard to test exactly. Or small models trained on our own threads can predict the answers.

    I went with the third. All four models together score a message in about 0.1 ms, so the first real model call already has the right setup and tools.

    block lengths illustrative · scoring time measured on the deployed code
  3. 03 / 09

    The agent gets hints, not orders

    Take a question like how many invoices were paid this month, and how many last month? The tools model is confident this is a data warehouse question, so the bi tools are loaded before the first call. A route turns that category into a first step, search_knowledge. The workflow model recognises an analytics query and adds a short step-by-step guide.

    The tier model leans towards tier2 but doesn’t clear its threshold, so it says nothing and the agent uses its default setup. Every hint is optional. When a model stays silent, that part of the turn runs as it did before.

    question and scores illustrative · thresholds from the shipped policy
  4. 04 / 09

    Text becomes a sparse vector with TF-IDF

    A model needs numbers, so the text becomes a vector first. Each word and word pair gets its count, damped with a logarithm, so a word used twice counts 1.69 rather than 2. That is multiplied by the inverse document frequency, so terms that are rare in the training data, like “invoices”, weigh more than common ones like “how”.

    The tier model reads the message three ways: words and word pairs, character chunks of three to five letters, and the first 32 words on their own. Each view is scaled to length one, so a long message doesn’t score higher just by having more words. Together that is a vector of about 96,000 slots, and a typical question fills about 150 of them.

    formulas as the Worker computes them · IDF values illustrative
  5. 05 / 09

    Logistic regression turns the vector into probabilities

    Each possible answer has one learned weight per slot. The model multiplies the message’s values by those weights, adds them up with a bias, and gets a raw score for each answer.

    For the model tier only one answer can be right, so softmax turns the three scores into probabilities that add up to one: exponentiate each score and divide by the total. Tool categories are different, because one message can need several groups of tools, so each category gets its own yes-or-no probability from a sigmoid instead.

    scores illustrative · threshold from the shipped policy
  6. 06 / 09

    A hint is given only when the model is sure

    A probability isn’t a decision. A hint is given only if the top answer is above its threshold and leads the runner-up by a margin. Otherwise the model stays silent.

    The costs are lopsided, which is why the bars are high. A missing hint costs a little time, while a wrong one sends the turn in the wrong direction with the wrong tools or setup. The shipped thresholds are 0.96 for the tier, 0.882 for tool categories, 0.696 for the first step with a margin of 0.137, and 0.676 for the workflow with a margin of 0.003.

    There is one more rule. When a tool category is confident and has a known first tool, that route sets the first step, even if the first-step model on its own is unsure.

    thresholds and margins from the shipped policy · scores illustrative
  7. 07 / 09

    The labels say what should have happened

    The models learn from reviewed ecoai threads. Each one is labelled against a written rubric with what should have happened, not with what the agent did at the time, so the models don’t learn the old mistakes.

    Near-duplicate threads are grouped so the same question can’t land on both sides of the split, and the split is by time: older threads train, newer ones check. That is how the models are used in production, always on messages newer than anything they saw.

    Training minimises cross-entropy, minus the log of the probability given to the right answer. A confident wrong answer costs far more than an unsure one, and the gradient is simply the prediction minus the truth, times the input.

    thread positions illustrative
  8. 08 / 09

    A genetic algorithm sets the thresholds

    The thresholds and margins are tuned by a genetic algorithm on the validation threads. Each policy is scored by replaying the hints it would have given: one point for every right hint and minus five for every wrong one, so a policy can’t win by guessing more.

    Each generation keeps the best 8 of 32 policies and breeds the other 24 from them, taking each number from one of two parents and sometimes nudging it. The animation is the real run, replayed from seed 42. Over 25 generations the best score went from −120 to −24, and the winner is the policy that shipped.

    the recorded run, replayed from seed 42 · each point is one policy
  9. 09 / 09

    Python trains, the Worker scores, and both have to agree

    Training happens in Python with scikit-learn, but the models run in a TypeScript Worker on Cloudflare. The Python bundle is too large for a Worker, so the vocabularies, IDF values and weights are exported, and the TypeScript code reimplements the feature extraction down to how Python splits text on whitespace.

    Before every deploy, every held-out example is scored both ways and the probabilities have to match. That check has already caught one tokeniser bug. In production, when the first-tool hint is given, the first tool is right 93% of the time; when the models aren’t sure, they stay silent and the turn runs as it always did.

    example points illustrative · 93% is the measured precision of first-tool hints when given

None of the parts are new. TF-IDF and logistic regression are decades old, and so are genetic algorithms. What made them work here was treating every hint as optional, putting a price on a wrong one, and letting the reviewed threads decide how sure each model has to be before it speaks.