Can This LLM Judge Be Trusted?
Order consistency caught what agreement alone missed
Built with
llm-judge-kit tells you whether an LLM judge can be trusted before you let it grade anything. Using an LLM as a judge is the default way to evaluate generated text, but the judge is still a model with its own biases. It can prefer whichever answer it sees first, and it can disagree with the people it's standing in for. I use LLM judges in my own work, and my internship research tested one, so I wanted a way to know when to trust one.
How it checks a judge
It uses the public MT-Bench human judgments, where expert annotators compared answers from six models on 80 multi-turn questions. Every pair is judged twice, once in each order, and if the verdict flips that counts as an order inconsistency. It also measures agreement with the annotators next to the human ceiling, which is how often two annotators agree with each other on the same pairs. A judge can't really be expected to beat that.
I fixed the pass bar in the first commit, before running any judge. It needs at least 90% order consistency, and agreement of at least 80% or within 10 points of the human ceiling. Until a judge clears that it stays advisory, and after that it's trusted. Every number has a bootstrap confidence interval that resamples pairs rather than votes.
What it found
I ran it on 100 MT-Bench pairs in October 2026. Claude Sonnet and Claude Haiku both cleared the bar, and Sonnet sits about as close to the annotators as they are to each other. The GPT-4 verdicts released with the dataset agreed with the annotators most often on the decisive pairs, but flipped on about one pair in five when the answers were swapped. On the first 60 of those pairs, a local Qwen3.6 35B model run through Ollama did the same more strongly, mostly picking whichever answer came first. That's why order consistency is a separate gate.
Running it
It runs through the Claude Code CLI or any local model through Ollama, and caches every call so an interrupted run loses nothing. It has 26 tests and an MIT licence.
Worth knowing
- A hundred pairs is enough to tell a good judge from a bad one, though not to separate two good ones.
- MT-Bench answers come from 2023 models, so today's judges may find them easier to tell apart than answers from current models.