What a judge is here
A rule set is one to four questions: a choice over 2–12 criteria labels, a noul over true and false criteria, or a score over 2–7 ordered levels. One request answers all of them for one record and costs one credit, whether it asks one question or four. Each answer returns under the id you sent, typed as you asked it, and an answer outside your criteria is refused rather than handed back to you.
That is the difference from asking a chat model to grade a response. There is no generated rationale to read and no free-form verdict to parse; what you store is the answer to a question you wrote. It also means the judge is only as good as your criteria: vague instructions produce a confident answer about nothing in particular.
{
"text": "Refund issued twice for INV-2291; the customer asked us to stop the second transfer.",
"questions": [
{"id": "outcome", "type": "choice", "instructions": "Which outcome does this note record?", "criteria": {"resolved": "The issue is closed.", "in_progress": "Someone is still working on it.", "blocked": "Waiting on a third party."}},
{"id": "is_negative", "type": "noul", "instructions": "Does the note report a fault in our handling?", "criteria": {"true": "It reports a mistake in our process.", "false": "It reports progress or a normal handover."}}
],
"threshold": 0.9
}Write the rules, or generate a draft and inspect it
You can write the questions yourself, or describe the decision in a short prompt and let the generator propose them: POST /api/judges from a signed-in session, a prompt of 5–800 characters, and a locale for the wording. The generator is a chat completion over one fixed model, and its output is validated into the same typed questions the API accepts before you see it.
A generation costs one credit and a failed one is refunded. Nothing runs automatically: a generated rule set is a draft you edit and then run yourself. Check that the criteria are mutually exclusive, that the levels are ordered, and that no question asks for personal data. Treat the prompt as data — the generator writes rules, it does not decide anything.
curl https://jevapi.pro/api/judges \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: judge-support-triage-v1' \
-b 'jev_session=YOUR_SESSION' \
-d '{"prompt": "Decide whether a support note is resolved, in progress or blocked, and whether it reports a fault in our process.", "locale": "en"}'\n\n{
"judge": {
"name": "Support note review",
"questions": [{"id": "outcome", "type": "choice", "instructions": "Which outcome does this note record?", "criteria": {"resolved": "…", "in_progress": "…", "blocked": "…"}}]
},
"model": "openai/gpt-4.1-mini",
"id": "6f2c1a4e-9d7b-4c11-8f3a-2b5e9c0d1a77",
"latencyMs": 2140,
"credits": 19
}Batch classification and exports · OpenRouter decisions API reference
Check the judge against people, not against its own confidence
Compare the answers with labels a knowledgeable person wrote, question by question, and count the errors that are costly rather than overall agreement. A probability is the model’s own output; it is not a measured accuracy, and a high value can still be wrong on your data.
Re-run the comparison whenever instructions, criteria or the model change, and keep the request id and model name with the results so a later run means something. Route uncertain records to review instead of treating a judge as a final decision, and publish accuracy figures only from your own held-out set.
Evaluate classification before automating work · Human-in-the-loop decisions · OpenRouter: TypeSafe SDK and Jev