A rubric is supposed to make grading consistent. In practice, most hiring rubrics don't — they're a list of qualities ("strong communicator," "good problem solver") with no definition of what counts as evidence for any of them. Two graders using the same rubric end up scoring the same answer differently, which defeats the point of having one. Writing a rubric that actually holds up under repeated use is a specific skill, and it's worth doing well before you write a single question — here's the research case for why structure beats gut-feel judgment in the first place.
Start from the evidence, not the trait
The failure mode is writing criteria as traits ("shows leadership") instead of observable evidence ("names a specific decision they made under ambiguity, and explains the trade-off they weighed"). A trait is a judgment call that varies by grader. Evidence is something you can point to in the answer and agree on. If you can't write the sentence "a strong answer would include ___," the criterion isn't ready to grade against yet.
Four things a gradable criterion needs
- A specific, observable signal — not a trait. "Explains why they chose X over Y" is gradable; "thinks critically" is not.
- A defined bar for partial credit. Real answers are rarely all-or-nothing, so decide up front what a partial match looks like, not mid-grading.
- Independence from the other criteria. If two criteria are really the same thing worded twice, a strong answer double-counts and a weak one gets double-penalized.
- A weight that reflects the role, not the order you thought of it in. Not every criterion should count equally — decide which ones are pass/fail-level before you see a single answer.
Write it before you see any answers
The single biggest source of rubric drift is writing or adjusting criteria after reading a few candidates' answers — it's natural to start grading toward what you've already seen, which quietly moves the bar for everyone who answers later. Freeze the rubric before the first real answer comes in, and treat any mid-run change as a signal the rubric needed more work, not a normal adjustment.
This is exactly what "frozen rubric" means on Gradogram
This isn't just good practice — it's how grading works end to end on the platform. The rubric you write is locked in for the hiring run before any candidate answers, the AI grades every answer against it criterion by criterion with the evidence shown, and a person on your team confirms the result before it counts. Writing a gradable rubric up front is what makes that whole flow work.
A short example
Weak criterion: "Demonstrates strong system design skills." Nothing here tells a grader what to look for, so two graders will disagree. Gradable version: "Identifies at least one specific bottleneck in the proposed design and proposes a concrete mitigation (not just 'add caching' with no reasoning)." The second version tells any grader — a person or an AI working from the same rubric — exactly what evidence counts as a pass, a partial, or a miss.
Where this breaks down
A rubric this specific takes real effort to write, and it's tempting to reuse a generic one across very different roles to save time. Don't — a rubric written for a senior role rarely transfers to a junior one without rewriting the evidence bar, and a rubric that's vague enough to fit every role is vague enough to be useless for all of them. Budget the time to write one rubric per question that actually matters, not one rubric per job title.
Once the rubric is written, the grading itself — consistent, evidenced, with a person confirming the result — is the part covered in how rubric-based AI grading works. This post is the upstream half: get the rubric right, and the grading takes care of itself.