What LLM-as-a-judge means, and what the research measured
LLM-as-a-judge means asking a strong language model to evaluate an output instead of, or before, a person does. The term was popularised by Zheng et al. in “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023), which described three variants: pairwise comparison, where the judge picks the better of two answers or declares a tie; single-answer grading, where it scores one answer; and reference-guided grading, where it compares an answer with a reference solution.
Their headline result is often quoted without its conditions. On MT-Bench, counting only non-tie votes, GPT-4 as a pairwise judge agreed with expert human voters 85% of the time, while the experts agreed with each other 81% of the time. That is agreement on open-ended chat answers from six 2023 models, measured by the authors. It says nothing about your labels, your language or today’s models, which is why the rest of this page is about measuring your own judge.
What a judge is here
A rule set is one to four questions: a choice over 2–12 criteria labels, a noul over true and false criteria, or a score over 2–7 ordered levels. One request answers all of them for one record and costs one credit, whether it asks one question or four. Each answer returns under the id you sent, typed as you asked it, and an answer outside your criteria is refused rather than handed back to you.
That is the difference from asking a chat model to grade a response. There is no generated rationale to read and no free-form verdict to parse; what you store is the answer to a question you wrote. It also means the judge is only as good as your criteria: vague instructions produce a confident answer about nothing in particular.
{
"text": "Refund issued twice for INV-2291; the customer asked us to stop the second transfer.",
"questions": [
{"id": "outcome", "type": "choice", "instructions": "Which outcome does this note record?", "criteria": {"resolved": "The issue is closed.", "in_progress": "Someone is still working on it.", "blocked": "Waiting on a third party."}},
{"id": "is_negative", "type": "noul", "instructions": "Does the note report a fault in our handling?", "criteria": {"true": "It reports a mistake in our process.", "false": "It reports progress or a normal handover."}}
],
"threshold": 0.9
}Write the rules, or generate a draft and inspect it
You can write the questions yourself, or describe the decision in a short prompt and let the generator propose them: POST /api/judges from a signed-in session, a prompt of 5–800 characters, and a locale for the wording. The generator is a chat completion over one fixed model, and its output is validated into the same typed questions the API accepts before you see it.
A generation costs one credit and a failed one is refunded. Nothing runs automatically: a generated rule set is a draft you edit and then run yourself. Check that the criteria are mutually exclusive, that the levels are ordered, and that no question asks for personal data. Treat the prompt as data — the generator writes rules, it does not decide anything.
curl https://jevapi.pro/api/judges \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: judge-support-triage-v1' \
-b 'jev_session=YOUR_SESSION' \
-d '{"prompt": "Decide whether a support note is resolved, in progress or blocked, and whether it reports a fault in our process.", "locale": "en"}'\n\n{
"judge": {
"name": "Support note review",
"questions": [{"id": "outcome", "type": "choice", "instructions": "Which outcome does this note record?", "criteria": {"resolved": "…", "in_progress": "…", "blocked": "…"}}]
},
"model": "openai/gpt-4.1-mini",
"id": "6f2c1a4e-9d7b-4c11-8f3a-2b5e9c0d1a77",
"latencyMs": 2140,
"credits": 19
}Known biases, and the fixes that were tested
The same paper measured three biases. Position bias: with the default prompt, GPT-4 gave the same verdict after the two answers were swapped in only 65% of pairs, and weaker judges less often. Verbosity bias: when a list was padded with rephrased items that added nothing, GPT-3.5 and Claude-v1 preferred the longer answer in 91.3% of 23 cases and GPT-4 in 8.7%. Self-enhancement bias, favouring answers from the judge’s own model, appeared in some judges, but the authors could not confirm it with their data. Judges also mis-graded maths questions they could solve on their own.
Mitigations were tested as well. Judge each pair twice in both orders and count a win only when both verdicts agree. Add a reference answer, which cut GPT-4’s grading failures on ten maths questions from 14 of 20 to 3 of 20. Add few-shot examples, which raised position consistency from 65% to 77.5% at roughly four times the prompt cost.
A typed question about one record avoids pairwise position effects by construction, and there is no free-text verdict for length to inflate. It does not remove wording effects. Test those directly: reorder the criteria, rephrase an instruction without changing its meaning, and check that the answers on your held-out set stay the same.
Agent-as-a-judge: when the thing being judged is a trajectory
Agent-as-a-judge extends the idea to agentic systems. Zhuge et al. (2024) gave the judge tools to map the workspace an agent produced, locate the relevant files, read them and search its trajectory, and applied it to DevAI, a benchmark of 55 AI development tasks with 365 hierarchical requirements. In their experiments the agent judge agreed with the consensus of three human evaluators on about 90% of requirements, against about 70% for an LLM judge, at roughly 2.3% of the cost and time of the human evaluation.
This workspace judges one record at a time; it does not browse files or replay an agent run. If you evaluate agents, split what can be judged into records, one requirement or one step per request, and keep a person or a dedicated agent judge for checks that need to open files or run code.
Zhuge et al., Agent-as-a-Judge (arXiv 2410.10934) · Agent workspace
Check the judge against people, not against its own confidence
Compare the answers with labels a knowledgeable person wrote, question by question, and count the errors that are costly rather than overall agreement. A probability is the model’s own output; it is not a measured accuracy, and a high value can still be wrong on your data.
Re-run the comparison whenever instructions, criteria or the model change, and keep the request id and model name with the results so a later run means something. Route uncertain records to review instead of treating a judge as a final decision, and publish accuracy figures only from your own held-out set.
Evaluate classification before automating work · Human-in-the-loop decisions · OpenRouter: TypeSafe SDK and Jev
