Jevstral: a decision model on Ministral 3 8B
Benchmark: Decision Index leaderboard · full Jevstral run
I first heard about decision models through the hype around Jev. LLMs don't solve every problem, and this looked like a good alternative for a whole class of them.
A lot of work in AI companies is routing and making decisions based on text, and engineers often just call the OpenAI API directly. Jev is fast, but what caught my attention was calibration: a model that gives you an answer and tells you how confident it is. That felt close to what teams need, and to where these models are going.
The best way I know to understand a model is to build one. So I built Jevstral, a decision model on Ministral 3 8B Base, inspired by Kev.[1] I picked it for three reasons: the Apache 2.0 license, Mistral's models are very well engineered for their size, and I live in France, so I preferred a French model to a Chinese open-source one.
A decision model reads a document and a set of questions, and gives a probability for each option of each question. It doesn't write any text, and it does all its work in one forward pass. Most sections below have a small playground you can play with. The code, weights and benchmark results are all public.[2]
Jevstral is a personal learning project. It is not affiliated with Mistral AI, with TypeSafe (the company that makes Jev) or with the author of Kev.
Why decision models?
I've used chat models since 2023, and I've seen them hallucinate even on small tasks. Ask the same yes/no question twice and the answer could flip. The usual fix was to overengineer it: query the model several times and take a majority vote, or add a second LLM call to check the first one's answer. That made things slower, on top of latency that was already a problem. And when a chat model says "I'm 90 % sure", that number is just more text, not a calibrated probability.
Classifiers are the old answer. A logistic regression or a fine-tuned BERT can route support tickets well, and it's fast and cheap. But it only knows the labels it saw in training: add a new team and you're labeling data and retraining. Want to also ask "is this urgent?"? That's a second classifier.
A decision model tries to keep the good part of each. Like a chat model, it takes the question and the options with the request, so it works on options it has never seen (zero-shot). Like a classifier, it gives one probability per option, so you can set a threshold: if the model is above 0.9, act automatically; if not, send the case to a person.
It's also fast. A chat model works in two steps: it reads the whole prompt in one parallel pass (the prefill), then writes the answer one token at a time, one pass per token (the decode). A decision model does only the prefill, and a small head reads the hidden states and scores each option.
So a decision takes about as long as the backbone needs to produce its first token, and the answer is always one of the given options.
What's inside
Jevstral has three parts: the Ministral 3 8B text decoder (frozen), small LoRA adapters inside its 34 layers, and a pointer head on top. Only 46.7 million weights are trained, about 0.6 % of the decoder. Scroll through the diagram to follow one request from text to probabilities.
The text is split into tokens, and each token gets an id. Five special tokens mark the parts of the row: <state>, <q>, <opt>, </opt> and <decide>.
The input
In the diagram, this is steps 1 and 2. The input is a state (the document) and one or more questions, each with its options. Each question becomes one row of tokens:
<s> <state> document <q> question <opt> option 1 </opt> <opt> option 2 </opt> ... <decide>
Five reserved tokens of the Mistral tokenizer mark the parts. I call them delimiters. A request with three questions gives three rows, each with the full state, so one question can't see another.
These five tokens confused me at first. Their embeddings were all exactly zero, and I'd expected random values. Here's what I found. Control tokens never come out of encoding raw text; the tokenizer only emits them when you build a chat or instruct template explicitly. Pretraining data is plain text, so these rows never got a gradient. They're in the Base vocabulary only because Mistral ships one tokenizer for Base and Instruct. The exact zeros look like a deliberate choice before release: a random row is noise with an arbitrary norm, and if you feed it in by accident it puts junk into the residual stream. Zero is neutral.
So Jevstral has to learn these embeddings, and they're the only part of the embedding table that changes.[3]
Attention and the KV cache
In the diagram, this is the Attention box in step 3. The row goes through 34 decoder layers. Attention is causal: each token reads only itself and the tokens before it. The row order is built around that. Each </opt> comes right after its option's text, so its hidden state holds that option's meaning. <decide> comes last, so it can see the state, the question and every option. Click a row to see what each token reads.
| <s> | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| <state> | |||||||||||||||
| charged | |||||||||||||||
| twice | |||||||||||||||
| refund | |||||||||||||||
| <q> | |||||||||||||||
| Which | |||||||||||||||
| team? | |||||||||||||||
| <opt> | |||||||||||||||
| billing | |||||||||||||||
| ▸ </opt> | |||||||||||||||
| <opt> | |||||||||||||||
| shipping | |||||||||||||||
| ▸ </opt> | |||||||||||||||
| ▸ <decide> |
<decide> can read 15 of 15 tokens. It comes last, so it reads the state, the question and every option. That is why the head uses it as the query.
Because the model is autoregressive, an early option can't read the options after it, so order can affect the result. I shuffle the options during training so the model doesn't learn to depend on position.
The same rule gives a free speed-up. The state comes first in every row and can't see the questions after it, so its keys and values are identical in every row. When a request has several questions, Jevstral runs the document once, caches it (the KV cache), and runs only each question on top. For a single question there's nothing to share and the extra call makes it slower, so the cache is used only when there's more than one question.[4]
LoRA
In the diagram, LoRA is the blue "+ LoRA" tags in step 3. Training all 8 billion weights needs too much GPU memory, and the model can forget what it already knows. So the decoder stays frozen, and LoRA adds a small trained path next to each of the 7 projections in every layer (q, k, v, o, gate, up, down): two thin matrices B and A, with rank 16 and α = 32. B starts at zero, so at step one the model behaves exactly like the base. After training, the path is merged into W, so inference costs nothing extra.
The pointer head
In the diagram, this is steps 4 and 5. For the head, I skipped the usual classifier layer with one output per option slot. It caps the number of options, and the last slots barely get trained. Jev uses a pointer instead, and so does Jevstral. Two linear layers, q and k, project each vector to 256 numbers, and each option's score is a dot product:
It's the calculation of one attention head: <decide> is the query, and each option is a key. The same k scores every option, so there's no limit on the number of options, and the score comes from what the option says, not where it sits.
A softmax turns the scores into probabilities. In training, the loss is −log of the correct option's probability. In use, you compare the top probability with your threshold. Try both:
Move T. The bars become flatter or sharper, but the blue option never changes: division by a positive number keeps the order of the scores.
Training
Training has four stages. Stage 1 teaches the format and the basic decision skill. Stage 2 adds date arithmetic and what to do when the document can't decide. Stage 3 is long, real documents. Stage 4 covers long policies, trade-offs, multi-hop reasoning, judging and developer tools. Stages 2 to 4 also replay records from stage 1 so the model keeps its earlier skills.
The model has to learn what each option means, not its position or its exact words. So training shuffles the options, sometimes adds a "none of the above" (with or without the correct option), and sometimes throws in something unrelated like "A recipe for pancakes".[5]
The four stages took about 3 hours and 25 minutes on one H100, with no failed runs.[6] I ran everything on Modal, which turned out much nicer than RunPod: no YAML files or container setup, just Python functions that run on a GPU.
Stage 4 is where the hard skills come in. Between stage 2 and stage 4 (I didn't measure these sets after stage 3), hard-v1 goes from 0.517 to 0.810 and devtools-v1 from 0.563 to 0.710. With replay, decision-v7 stays at 0.88 the whole way. I didn't test training without replay.
Calibration
When I looked at the outputs after training, the model was too sure of itself. A model is calibrated if, of all the times it says 0.8, it's right about 80 % of the time. Jevstral wasn't: on data it hadn't trained on, it was wrong more often but stayed just as confident.
To measure this I used the expected calibration error (ECE). Group the predictions by confidence (0.9 to 1.0, 0.8 to 0.9, …), and in each group compare the average confidence with how often the model was actually right. ECE is the average gap, so 0 is perfect, and an ECE of 0.14 means the confidence is off by about 14 points on average.
The fix is one number, the temperature T, used in step 6 of the diagram. Divide every score by T before the softmax, and a T above 1 makes the probabilities less extreme. With the scores 4, 1 and 0:
| billing | shipping | returns | |
|---|---|---|---|
| T = 1 | 0.94 | 0.05 | 0.02 |
| T = 2.04 | 0.73 | 0.17 | 0.10 |
Dividing by a positive number keeps the scores in the same order, so the top answer never changes and neither does accuracy. You can check this with the T slider in the playground above.
I picked T on 648 questions from datasets no training stage used, choosing the value with the lowest log loss: T = 2.04.[7] ECE went from 0.141 to 0.043, and accuracy stayed at 0.679. Two caveats: I measured ECE on the same questions I used to pick T, so 0.043 is optimistic, and those questions come from other datasets, which is why accuracy is lower than on the dev sets (0.71 to 0.89).
Results
The Decision Index 0.2.1 is a public benchmark for decision models: 38 benchmarks in five areas, each chance-corrected so 0 is random guessing and 100 is perfect. I ran the full suite, 150,317 requests, on one NVIDIA RTX PRO 6000.[8]
Jevstral scored 34.3, 36th of 74 on the 1 October leaderboard, with a median latency of 28.3 ms.[9] It's now listed on Decision Index 0.4.1, which uses a new formula and has 248 models: 32.2, 164th. For $30 of Modal credits I'm happy with that. The goal was to build the core loop end to end, not to top a benchmark.
Smaller models do better: Decider 4B (40.7 at 12.6 ms) and Hopper 1.2 4B (40.8 at 23.3 ms) are both faster and more accurate. By area, against Kev and Jev:
| Area (weight) | Jevstral 8B | Kev 4B | Kev 9B | Jev |
|---|---|---|---|---|
| Knowledge and reasoning (25.8 %) | 24.5 | 22.9 | 26.2 | 51.4 |
| Language understanding (25.8 %) | 34.6 | 35.3 | 41.7 | 62.0 |
| Retrieval and classification (20 %) | 39.3 | 41.0 | 43.7 | 55.4 |
| Tools and automation (18.3 %) | 52.9 | 52.6 | 54.5 | 75.1 |
| Arts and human taste (10 %) | 14.8 | 17.9 | 22.4 | 37.7 |
| Decision Index | 34.3 | 34.6 | 38.5 | 57.9 |
The gap to Jev (57.9) is large in every area. Kev 9B scores only 3.9 points more than Kev 4B, so I think most of the gap comes from training data, not size. Jevstral fits that: trained on Kev's data, its 8 B backbone lands at Kev 4B's level.
The best models under 10 B tell the same story. PurpleMIST uses almost the same design (a base model, LoRA and a pointer head) and scores 47.5 with 10× more data. The models above 55 mostly start from an instruct model and train on 1 to 6 million examples. Some also read short inputs twice, so every token can see the whole request, which fixes the option-order problem above without any training.
With more time, I'd try that, then build a specialized decision model for one task and test it on a real-world use case.
Acknowledgements
The method, the training recipe and the training data come from Kev. The design follows the analysis in Jev's Architecture Unmasked. The benchmark and its harness come from the Decision Index. The base model comes from Mistral AI, under Apache 2.0.
Notes
- Mistral AI publishes the model as mistralai/Ministral-3-8B-Base-2512, under Apache 2.0. Jevstral uses only its text decoder. ↩
- Code: github.com/AmirBraham/jevstral. Weights: AmirBraham/jevstral-8b. Full benchmark run: AmirBraham/jevstral-decision-index. All under Apache 2.0. ↩
- Five zero rows are all the same, so the model can't tell the five delimiters apart. So stage 1 starts each row as a random sample from the distribution of the real embeddings, with their mean and covariance. The other 131,067 rows stay frozen. ↩
- On a sample of 100 benchmark requests, the cache halved the p95 latency. ↩
- "None" with the correct option removed: 10 % of questions. "None" with the correct option kept: 12 %. Unrelated option: 15 %. Stage 1 also adds minimal pairs: two copies of one question that differ in one option. One copy has the correct option, and the other does not. ↩
- Stage 1: 70 minutes. Stage 2: 10. Stage 3: 45. Stage 4: 80. Calibration: 5. Settings: AdamW with weight decay 0.01, a one-cycle learning rate with 10 % warm-up, gradient clipping at 1.0, fp32 weights with bf16 computation. All GPU work ran on Modal. Training and checks cost approximately $29, and the benchmark run approximately $12. ↩
- I tried T from 0.50 to 5.00 in steps of 0.01. Log loss is a proper scoring rule: only honest probabilities give the lowest value. The fit uses held-out data because, on training data, the model is correct most of the time, so high confidence costs nothing. A fit there gives a T that is too small. ↩
- The run took 3 hours 21 minutes with the public harness. Two overlaps: BANKING77 is in the benchmark and in the training data, and 200 MMLU-Pro questions were in the calibration set. ↩
- p95 latency: 185.1 ms. Mean: 79.3 ms. One request at a time. Jev is a hosted API, so its latency includes the network. ↩