Jev Calibration — do you know what you don't know?
Answer 16 true/false statements, then watch Jev answer the same ones with a calibrated probability behind each. The twist: keep only Jev's confident answers and its accuracy climbs toward 100%. That's calibration — it knows what it doesn't know. Do you?
Goldfish have a memory of only about three seconds.
True or false?
The same demo is one Jev call
- Take each true/false statement
- Ask Jev with a calibrated yes/no (
noul) - Jev returns a probability, not just an answer
- Keep only its high-confidence answers → accuracy climbs
const res = await fetch("https://jevtypesafeai.com/api/v1/decide", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.JEV_API_KEY}` },
body: JSON.stringify({
state: statement, // the claim to judge
questions: {
verdict: { type: "noul",
instructions: "Is this statement true? Return your calibrated probability." },
},
}),
});
const { answers } = await res.json();
const p = answers.verdict.noul; // P(true); confidence = max(p, 1-p)See a real decision NO · 6% · 191ms
The Great Wall of China is visible from space with the naked eye.
What calibration is, and why it matters
A prediction is calibrated when its confidence matches how often it's right: of all the times a model says it's 80% sure, it should be correct about 80% of the time. Calibration is a different thing from raw accuracy. A model can be accurate on average yet wildly overconfident — sure of itself exactly where it's wrong — and that's the failure mode that quietly breaks products. If you can't trust the number attached to a decision, you can't route the easy cases automatically and escalate only the hard ones.
Humans are famously badly calibrated on general knowledge. We're most certain precisely where the common misconceptions live — that bulls hate red, that we use 10% of our brains, that the Sahara is the largest desert — so our confidence is highest exactly where it should be lowest. This game makes that visible: you'll likely be very sure about several answers that are simply wrong.
Jev is built the other way around. Every decision comes with a calibrated probability, so its confidence is a usable signal. In this game we prove it live: sort Jev's answers by confidence, throw away the least-confident ones, and its accuracy on what remains climbs toward 100%. It knows what it doesn't know.
What is Jev?
Jev is TypeSafe AI's first System One model. Instead of writing prose like a chatbot, it reads a situation and returns a typed, calibrated decision — a choice between options, a score on a scale, or a yes/no probability (a noul) — in tens to hundreds of milliseconds, with no hallucinations. Here each statement becomes one noul call: "Is this true? Answer with your calibrated probability that it is TRUE." The number that comes back, and how well it's calibrated, is the whole point.
The same model powers ticket routing, content moderation, lead scoring, agent tool-call guardrails and model routing in real products — anywhere a small, fast, bounded decision needs a probability you can actually act on.
How the game works
- You answer 16 true/false statements — a deliberate blend of plain facts and popular misconceptions where people are confidently wrong.
- Hit Reveal, and Jev answers every statement itself: 16 real API calls run five at a time, each returning its calibrated probability that the statement is true.
- Jev's boolean answer is P(true) ≥ 50%; its confidence is how far that probability sits from a coin flip — max(P, 1−P).
- The Calibration Explorer lets you slide a confidence threshold. Over just the answers Jev was at least that sure about, you see how many it kept and how many it got right — and the accuracy climbs as the bar rises.
FAQ
What does it mean that Jev is 'calibrated'?
Calibrated means its stated confidence matches reality: across all the answers it's 80% sure of, roughly 80% are correct. That makes the probability a usable signal — you can auto-accept the confident decisions and escalate only the uncertain ones. The game demonstrates it: keep only Jev's high-confidence answers and its accuracy rises toward 100%.
Why do people score worse than Jev on this?
The statements are seeded with common misconceptions — things most people are sure about but that are false. Human confidence isn't calibrated: we're often most certain exactly where we're wrong, so the easy-feeling questions are the traps.
Is Jev actually answering, or is it scripted?
Jev answers each statement for real. On Reveal, the page fires one live API call per statement (five at a time) asking for a noul — a calibrated yes/no probability. Nothing is precomputed; you can watch the latency and running cost.
Is this free, and what does it cost to run?
It's free to play — no key, no signup, just rate-limited calls to the live Jev API. Each statement is a single noul decision that returns in tens to hundreds of milliseconds and costs a fraction of a cent; the page shows the total.
More: Moral Dilemmas — you vs Jev · Jev Wikiracer · All decision games · Jev API docs