cerno: JEV's System One primitives on a local model

A week ago TypeSafe AI came out of stealth with Jev (opens in a new window) , a “System One” model that doesn’t write answers at all. You hand it a piece of state and a list of typed questions, and it hands back typed decisions: a probability, one option out of a closed set, or a level on a rubric. Nothing to parse, and no way for it to answer outside the schema you gave it.

I read the announcement and had three reactions in quick succession. First, this is a genuinely good interface for the boring 80% of what people actually use LLMs for: routing tickets, flagging messages, grading things. Second, I wanted to know how it works, because I like to understand my AI tools before I trust them. And third, I was fairly sure the core of it doesn’t need a hosted model. That gave me an excuse to write a lot more Rust than my day job asks of me.

The result is cerno: the same three primitives, served over HTTP, answered by whatever model you already run in Ollama, vLLM, llama.cpp or LM Studio. Nothing leaves your machine unless you point it at a remote endpoint. The name comes from Latin cernere: to sift, to distinguish, to decide.

Source is at github.com/cebor/cerno (opens in a new window) . Version 0.1.0 is on crates.io, PyPI and npm.

The three primitives

JEV’s API and cerno’s both expose three question types at POST /v1/systemone:

A request carries one piece of state and any number of questions about it:

curl localhost:3000/v1/systemone -H 'content-type: application/json' -d '{
  "state": "Ticket: Serverraum-Klima ausgefallen, 31 Grad und steigend.",
  "questions": [
    {"id": "urgent", "noul": "Ist das dringend?"},
    {"id": "team", "choice": {
      "question": "Welches Team?",
      "options": ["IT", "Facility", "HR"]}},
    {"id": "sev", "score": {
      "question": "Wie schwer?",
      "levels": ["unkritisch", "gering", "mittel", "hoch", "kritisch"]}}
  ]}'

{"answers": {
   "urgent": {"type": "noul",   "noul": 0.9834, "truncated": false},
   "team":   {"type": "choice", "choice": "Facility", "index": 1,
              "confidence": 0.978, "truncated": false},
   "sev":    {"type": "score",  "score": 5, "expected_score": 4.80,
              "legend": "kritisch", "confidence": 0.692, "truncated": false}},
 "model": "gemma4:e2b-it-qat",
 "usage": {"input_tokens": 297, "questions": 3},
 "timing_ms": {"total": 96}}

Three questions in 96 ms, on a 4 GB model on my own GPU. (The response is trimmed. Every answer also carries its raw_logprobs, and a choice or score its full probabilities.)

The trick: one token, read as a distribution

A generative model answers by writing words, and then something has to parse those words back into a decision. That parsing step is where free-text LLM classification goes wrong: “Facility (though IT might also be involved)” is not an enum.

cerno never lets the model get that far. Each question becomes a short prompt in which the options are lettered A, B, C, and so on. The model is asked for exactly one token. cerno ignores the token it picks and reads the probability distribution over that one position instead, via the runtime’s top_logprobs. P(A), P(B) and P(C) are all there after a single forward pass, however many options there are.

question ──> options ──> labels A,B,C ──> prompt ──> model
                                                       │
                                          distribution over one token
                                                       │
typed answer <── calibrate <── fold variants, floor <──┘

This is where both the speed and the probabilities come from. The speed comes from generating one token instead of a sentence. The probabilities are the model’s own, not a number it was asked to make up, and asking a model “how confident are you, from 0 to 100?” produces a number that is mostly vibes.

Under the hood a noul is just a two-option choice with “Yes” first, and a score is a choice whose options are the rubric’s legend. That reduction lives in one function, and it is the only place the three primitives differ:

/// Reduce a question to its options.
/// This is the only place the three primitives differ.
fn ballot_for(kind: &QuestionKind) -> Ballot<'_> {
    match kind {
        // "Yes" first, so the noul probability is always option 0.
        QuestionKind::Noul(question) => Ballot {
            question: Some(question),
            options: vec!["Yes".to_string(), "No".to_string()],
        },
        QuestionKind::Choice(spec) => Ballot {
            question: spec.question.as_deref(),
            options: spec.options.clone(),
        },
        QuestionKind::Score(spec) => Ballot {
            question: spec.question.as_deref(),
            options: spec.levels.legend(),
        },
    }
}

Everything after it — labelling, prompting, the single host call, folding, calibrating, normalising — is shared. A fourth primitive would be one more match arm and a response shape, not a second code path.

What the measurements forced

That’s the concept, and it fits on a napkin. What took the time was that real models, behind real runtimes, disagree with the napkin in six ways. Every one of them would have produced answers that looked plausible and were quietly wrong.

The first token wasn’t the answer

On gemma4, the first generated token was <|channel|>, a chat-template control token that opens the thinking channel. The answer letter came after it, so the distribution I was reading was the distribution over “should I start thinking now”.

So every request sends think: false. Models without a thinking mode reject that field, so the host retries without it once and remembers the refusal per model. That costs one round trip per model, not one per question. On the OpenAI-compatible path the equivalent switch is reasoning_effort: "none". Behind Ollama’s /v1 endpoint, without it, <|channel> came first at a logprob of -0.03 with the actual answer at -3.5.

Sampling runs before logprobs are reported

This one scared me the most, because nothing looked broken. Ollama applies its sampling transform before it reports logprobs. With the default top_k: 40, the distribution is truncated and whatever survives is renormalised. Against gemma4:26b that turned a four-option question into A = 0.0 (probability 1), with every rival at -17, and B, C and D missing from the list entirely. The answer was still sensible. Only the probabilities were meaningless.

The fix is a set of options that are pinned in cerno-host and deliberately unreachable from outside the crate:

top_k: 0, top_p: 1, min_p: 0, temperature: 1, num_predict: 1

These are correctness conditions, not tuning knobs. Calibration happens later, in cerno-core, where it is explicit and can be reversed.

“Nein” is not a token

My first version used word labels. Measured on gemma4:26b, a German yes/no question produced Ja, JA, Ja and ja as four separate entries, each holding part of the probability mass. “Nein” wasn’t in the list at all, because it isn’t one token. It arrives as Ne + in, so a single-token distribution can never contain it.

Single capital letters avoid both problems. Some tokenisers still hand back " A" or "a", though, so every token that spells a label counts toward that label. The variants are summed in probability space with a numerically stable logsumexp, rather than picking one and dropping the rest:

pub fn logsumexp(logprobs: &[f64]) -> f64 {
    if logprobs.is_empty() {
        return f64::NEG_INFINITY;
    }

    let max = logprobs.iter().copied().fold(f64::NEG_INFINITY, f64::max);
    if !max.is_finite() {
        return max;
    }

    max + logprobs.iter().map(|lp| (lp - max).exp()).sum::<f64>().ln()
}

A missing label is not a zero

Runtimes report at most 20 ranked tokens. When a model is sure of itself, the losing label drops out of that window. For the 26B reference model this happened in 39% of cases. Treating the missing label as probability zero would be a lie. What we actually know is that it is below the weakest token that was reported.

So cerno substitutes that floor, which is a strict upper bound, and sets truncated: true on the answer, naming the affected labels in truncated_labels. A caller who acts on the winner can ignore this. A caller who acts on the probability can tell an observation from a bound.

The same ceiling explains the 20-option limit. A 21st label could never be observed, so a larger choice is refused with a 422 instead of degrading quietly.

Big models are overconfident

gemma4:26b answers clear cases at P = 1.0000. That’s the correct answer with a useless probability, because a threshold like “escalate above 0.9” can’t tell a clear case from a borderline one when both read 1.0. cerno applies temperature scaling to the logits, set per request or per model in the config, and the benchmark (below) fits the best temperature for each model.

A small model is enough

Which leads to the finding that shaped the defaults: a 26B model answered in 87 ms and a 4 GB one in 36 ms, with no loss of accuracy on this task. For picking a letter, you don’t need the big model.

Confidence

A choice and a score carry a confidence in 0..=1, defined as one minus the normalised entropy of the distribution:

confidence = 1 - H(p) / log(n)

Dividing by log(n) puts a two-option choice and a ten-level score on the same scale. It is a scale, not an invariant, though: at a fixed top probability of 0.9, ten options score 0.76 and two options score 0.53. That’s intended, since narrowing ten candidates down to one is the stronger statement. But it means confidences are only directly comparable between questions of the same shape, and a test pins both numbers so nobody “fixes” it by accident.

A noul has no confidence, because JEV’s doesn’t either. The probability already is the whole answer, and its distance from 0.5 is the certainty. The SDKs treat asking for one as a type error, not a silent None.

Choosing a model

cerno-bench runs candidate models over a labelled dataset of 36 cases (14 noul, 12 choice, 10 score) and writes the result to docs/model-selection.md:

The reference model is a yardstick, not a candidate.

Fidelity is a gate, not a score. It measures the share of cases answered with one of the offered letters at all. A model that sometimes answers in prose is unusable here however accurate it is, which is why granite4:3b is out despite being the fastest.

The interesting row is the second one. The 4 GB gemma4:e2b matches the 26B model’s accuracy at a quarter of the footprint and several times the speed, and it is better calibrated. The large model needs its logits flattened by a factor of 3.2 before its confidences mean anything. The small one comes out at 0.8, meaning it is very slightly under-confident. It’s the default. cerno still ships a calibration temperature of 1.0, because 36 cases is thin evidence for baking an adjustment in, and callers can always pass their own.

Hosts: the seam is one question

The part that makes “local” cheap is the ModelHost trait. It knows nothing about nouls, choices or scores. It answers one question: what is the distribution over the next token?

#[async_trait]
pub trait ModelHost: Send + Sync {
    fn capabilities(&self) -> HostCapabilities;

    /// A human-readable name for the host, used in logs and diagnostics.
    fn name(&self) -> &str;

    async fn first_token(
        &self,
        req: FirstTokenRequest,
    ) -> Result<FirstTokenDistribution, HostError>;
}

There are two implementations. One speaks Ollama’s native API. The other speaks /v1/chat/completions, and a small Flavour enum decides which runtime-specific fields go out with it, because each runtime needs its own way of saying “don’t sample before you report”:

The generic openai flavour sends nothing non-standard, because OpenAI answers 400 to fields it doesn’t know. So it is only as good as the server’s own defaults. A host that doesn’t report top_logprobs at all fails loudly with a 502 that says so, instead of making something up.

To be honest about coverage: Ollama is measured in both its native and /v1 form, and on gemma4:26b the generic body returns exactly the same distribution as the native path. vLLM, llama.cpp and LM Studio are implemented against their documentation, not yet run live. Whether LM Studio reports top_logprobs at all is still an open question.

The Rust side

Part of the point was to get better at Rust, so here is what the workspace looks like:

crates/
  cerno-types    wire types — serde always, utoipa behind a feature
  cerno-host     ModelHost trait, the Ollama and OpenAI-compatible adapters
  cerno-core     labels, prompt, logprob maths, engine
  cerno-server   axum + utoipa
  cerno-sdk      Rust client
  cerno-tui      terminal front end, on cerno-sdk
  cerno-bench    model benchmark
sdks/python      uv package `cerno-sdk`, imported as `cerno`
sdks/typescript  npm package `cerno-sdk`
spec/            openapi.json + conformance cases

A few things I’d do the same way again:

• The contract is generated, and drift fails a test. The server is axum with utoipa annotations, and spec/openapi.json is produced from the code by a small cerno-openapi binary. A test regenerates it and compares, so the document can’t silently fall behind. http://localhost:3000/docs serves Swagger UI off the same spec.
• Validation happens before a model is loaded. The handler validates the whole request before any host call: option counts, rubric sizes, unknown fields, a question naming two primitives, more than 32 questions. A bad request costs nothing, and a test pins that with a mock host that expects zero calls.
• Wire types live in their own crate, with serde always on and utoipa behind a feature. The Rust client depends on them without pulling in the server’s OpenAPI machinery.
• Nothing in the test suites needs a model. Every host is mocked with mockito. The live checks, a smoke example and the benchmark, are separate commands you run on purpose. Live models also wobble in the third decimal between identical requests (GPU reduction order), so exact equality would be the wrong assertion anyway.

The terminal front end

cerno-tui is a ratatui front end built on the Rust client, and it exists for one reason: curl tells you the winner, the terminal shows you how close the runner-up was.

┌ State (1) ───────────────────────┐┌ Answers ──────────────────────────────┐
│Ticket: Serverraum-Klima          ││sev      score → 5 "kritisch" conf 0.69│
│ausgefallen, 31 Grad und steigend.││          expected 4.80                │
└──────────────────────────────────┘│  1 unkritisch░░░░░░░░░░░░░░░░░░  0.9% │
┌ Questions (2) ───────────────────┐│  4 hoch      ██░░░░░░░░░░░░░░░░  8.2% │
│  urgent    noul   Ist das dring… ││  5 kritisch  █████████████████░ 87.6% │
│  team      choice IT | Facility  ││                                       │
│> sev       score  unkritisch | … ││team     choice → Facility   conf 0.965│
│  + add question  (a)             ││  IT          ░░░░░░░░░░░░░░░░░░  0.2% │
│                                  ││  Facility    ██████████████████ 99.4% │
└──────────────────────────────────┘└───────────────────────────────────────┘
 localhost:3000 ● · default model · ^S send · ? help

That’s the difference between “Facility” and “Facility, but it was nearly IT”. t and T step the calibration temperature, so you can watch the bars flatten, and m cycles through the models the service offers.

The TUI deliberately validates nothing. It builds the request through cerno-sdk and lets the service decide. After cerno-core, the OpenAPI document and the SDKs, a fourth copy of the rules would be the one that drifts.

Three SDKs, one set of cases

The Rust, Python and TypeScript clients share one builder shape:

let answers = client.systemone(text)
    .noul("urgent", "Is this urgent?")
    .choice("team", "Which team?", ["IT", "Facility"])
    .send().await?;

answers = (client.systemone(text)
    .noul("urgent", "Is this urgent?")
    .choice("team", "Which team?", ["IT", "Facility"])
    .send())

const answers = await client.systemone(text)
  .noul("urgent", "Is this urgent?")
  .choice("team", "Which team?", ["IT", "Facility"])
  .send();

All three are written by hand against spec/openapi.json rather than generated. Generated clients are correct and unpleasant to use. What keeps the hand-written ones honest is spec/conformance/cases.json, a shared file of requests, responses and expected errors that each SDK’s test suite runs. If one client starts accepting something the others refuse, its tests fail.

Limits

• 20 options. That’s the top_logprobs ceiling, as above. Splitting a large set across two questions works today. Doing it automatically doesn’t.
• Position bias. Options are lettered in request order, and models have some preference for A. Shuffling and averaging would fix it at the cost of a second pass, so for now it’s not done.
• No authentication. The service listens on 127.0.0.1:3000 by default. Only bind it to 0.0.0.0 on a network you trust.

Try it

You need a model and the service:

ollama pull gemma4:e2b-it-qat          # once, 4.3 GB
cargo install cerno-server cerno-tui

cerno-server                           # terminal 1, listens on 127.0.0.1:3000
cerno-tui                              # terminal 2

The first request takes 5–10 seconds while Ollama loads the model into VRAM. After that it’s around 150 ms per request. To use another runtime, set CERNO_HOST:

CERNO_HOST=vllm cerno-server
CERNO_HOST=openai \
CERNO_HOST_URL=http://gpu-box:9000/v1 \
CERNO_HOST_API_KEY=… cerno-server

The clients:

cargo add cerno-sdk         # Rust
pip install cerno-sdk       # Python, imported as `cerno`
npm install cerno-sdk       # TypeScript

Everything is on crates.io (opens in a new window) , PyPI (opens in a new window) and npm (opens in a new window) , all at the same version. The source, the benchmark and the full changelog are at github.com/cebor/cerno (opens in a new window) .

JEV is the polished, hosted version of this idea, and if you need its accuracy on hard cases, use it. But the idea itself, reading a decision straight out of one token’s distribution, turns out to fit comfortably on a laptop GPU. It has been a very good way to learn where the probabilities an LLM reports actually come from.