LLM-as-judge means asking one language model to evaluate the output of another, scoring it against a set of criteria you define. Instead of a person reading every answer, the judge model rates each one for accuracy, tone or completeness, so you can measure quality across thousands of responses.
Think of a cooking competition. You cannot have the head chef taste every plate during a busy service, so a trusted sous-chef tastes each dish first and flags the ones that are off. The head chef then only checks the flagged plates. The judge model plays that sous-chef role, which makes it a practical tool inside evals and broader model evaluation.
In practice the judge gets a prompt that spells out what good looks like, the original question, and the answer to grade. You can ask for a simple pass or fail, a score from one to five, or a short reason for the rating, which is handy when you want to spot patterns in what tends to fail. The same setup checks whether a RAG answer actually used the source it was given. A neat trick is asking the judge for its reason first and the score second, because writing out the reasoning tends to make the final score more consistent.
It is fast and cheap, but it is not infallible. A judge model can favor wordy answers, lean toward its own style, or score the same answer differently on a second pass. The fix is a clear rubric and a habit of comparing the judge’s scores against a small batch graded by real people.
At TopDevs we use LLM-as-judge to keep an eye on the quality of AI features after launch, so problems surface early instead of reaching a customer first.
