Local decision model as memory search reranker.

2026-10-12 · by Dennis Gubsky · research notes · ~6 min read

If you have ever used a search engine, you know the order of the results matters. If the thing you need is third, you scroll. If it is thirteenth, you might never see it.

Behind many modern search systems is a two-step process. The first step is fast and rough: it fetches a pool of maybe 20 candidate passages that could plausibly answer the question. The second step is slower and smarter: it reorders those 20 to put the best ones at the top. That second step is called a reranker. It is usually the most expensive part of the whole pipeline.

The question we had to find the answer on was: can we make the reranker cheaper without losing quality?

What we tried

We tested five ways to rerank search results on a public benchmark called ConditionalQA. For each of 800 questions, we fetched 20 candidate passages from the search index and let one of five methods reorder them. Every model in this test runs locally through Ollama. Nothing in these numbers calls a cloud API. The whole pipeline runs on our own hardware.

1. No reorder at all. Take the pool as-is from the search index. This is the floor.
2. The one we currently ship. A general-purpose language model (27 billion parameters, called qwen3.8) that reads all 20 candidates together and writes out a new order in prose.
3. A decision model, one call per question. A smaller model (9 billion parameters, called nimble) answers a multiple-choice question: "Which one of these 20 passages best answers the question?" It picks one and returns probabilities for all 20.
4. The same decision model, 20 calls per question. This time one call per candidate: "Does THIS passage answer the question? Yes or no." Then sort by the probability of yes.
5. An even smaller decision model. A 4 billion-parameter model (tev1) doing the same one-at-a-time thing.

The idea behind a decision model is that it does not write prose. It answers a typed question with probabilities over a fixed set. It cannot forget to answer. It cannot invent a passage number that was not in the input. It cannot wrap commentary around the answer. The output is a vector of numbers, period.

What we got

We measured a few things. The two that matter most for a reader of search results are "did we put a good answer in the top 5?" (recall@5) and "did we put the best answer first?" (recall@1).

Here is the summary across 800 questions.

The 9B model with one call (method 3) matched the 27B model (method 2) at recall@5. Both put a good answer in the top 5 about 94% of the time. The 9B model did it about three times faster on the same hardware.

The 20-calls-per-question version (method 4) also matched the 27B model at recall@5, but was no faster, because 20 calls instead of one erased the per-call speed advantage.

The 4B model (method 5) did not match. It barely beat not reranking at all.

So the headline is: a decision model with no prose output, no reply to parse, no risk of hallucinating a passage number, can replace a 27B general model and save about two-thirds of the reranker's compute, with no measurable loss in whether the right answer is in the top handful.

Where the smaller model loses

At recall@1 (the "put the best answer first" question), the 9B model did worse. 78% vs 81% for the 27B. On 800 questions that difference is real.

Why? When you ask a model "which ONE of these 20 passages is best?" and it returns a probability for each, the probabilities form a softmax. One number is large and the rest are small. The model is good at picking a winner from the field. But below that winner, the small numbers do not separate second place from third place very well.

A general-purpose language model, by contrast, writes out the whole ordering as text. It is in some sense thinking about the ordering as a whole.

For an agent that reads the top three or top five results and considers them, this difference does not matter. The right answer is in that handful at the same rate. If your product needs the single best result and acts on it right away, the smaller model is the wrong choice.

Decision model does not detect abstention

We also hoped the model could tell us when NONE of the 20 candidates actually answered the question. If none of them do, we would like the system to say "I do not know" instead of returning a bad answer. The model gives a probability with every pick. We thought that number could be the signal: if it is too low, abstain.

It did not work. We tested it directly. For each question we built two pools: one with the correct passage in it, one without. We checked whether the pool-with-correct-passage got a higher score than the pool-without. It did, but only 75% of the time. We needed over 85%.

The reason is that the model's confidence is not calibrated to reality. When it said it was 97% sure, it was actually right 42% of the time. When it said it was between 50% and 90% sure, it was right less than 15% of the time. The numbers are fine for sorting candidates against each other, but they are not meaningful on their own. A score of 0.9 from this model does not mean a 90% chance of being right.

The benchmark marks one specific passage per question as "the correct one", but sometimes another passage in the pool also answers. Some of what looks like overconfidence is the model noticing a correct passage that the benchmark did not label. The real number is probably better than 42%. It is still not good enough to use as a gate.

For now, if we want a system that abstains when nothing fits, these probabilities are not the signal. We have to find another one.

Follow-on: two newer models, same conclusion

A few days after this test, two newer decision models landed from the same model family: clef (27B, multimodal) and clef-flash (9.1B). We ran a smaller test on a different benchmark (LoCoMo memory reranking, 300 questions) to see if either beat nimble.

Neither did. On recall@5 nimble scored 0.750. clef scored 0.739, at more than three times the latency. clef-flash scored 0.711. The newer and bigger models were not better here. Their calibration was different (underconfident, rather than nimble's overconfident), but still not good enough to use as a gate.

For now nimble is the model we recommend for reranking. The results on one 800-question corpus do not generalize automatically to every corpus, but they are consistent with what we see on others.

What this means in practice

If you are building a system that reranks search results and the first handful is what matters, a small typed-answer model can replace a bigger general model and save roughly two-thirds of the compute per rerank, with no measurable loss in whether the right answer is in the top five.

If you need the single best answer at position one and you cannot afford to be wrong, keep the bigger general model, or combine the two.

If you want to use these models' probabilities as a signal for "I do not know, nothing in this pool answers your question", they are not calibrated well enough for that today. The underlying rank order is good; the probability numbers are not meaningful in an absolute sense.