Every candidate taking a coding test today has an AI assistant open in another tab, whether your process allows it or not. That's not a hypothetical — it's just what writing code looks like now. Which leaves a traditional coding test asking a question it can no longer actually answer: is this candidate's work theirs?
Banning it doesn't work. Ignoring it doesn't either.
Recruiters we've talked to are stuck between two bad options. Ban AI, and you're grading a task that has nothing to do with how the candidate would actually work on your team — and you can't really enforce it anyway. Ignore it, and every score drifts toward the ceiling: the assistant is good enough that the test stops telling anyone apart. Either way, the thing you actually wanted to know — can this person do the work, and how do they use the tools available to them — never gets measured.
What we built instead
AI Debugging Challenge is a new question type: the recruiter writes a piece of code with a real, intentional bug and a rubric for what a correct fix looks like. The candidate opens it in a code editor with the platform's AI assistant sitting right next to it, and works the problem the way they actually would on the job — asking questions, testing suggestions, pushing back when something looks wrong.
There's no run button. The candidate isn't graded on whether code executes in a sandbox; they're graded on the fix and the reasoning behind it, against the rubric the recruiter wrote. Every message they send the AI, and every paste event, becomes part of a transcript that's frozen the moment they submit.
The same AI, for everyone
The part that makes this fair is easy to miss: every candidate gets the identical assistant. Nobody can bring their own ChatGPT account, a better model, or a friend on a second screen. It's not a proctoring rule you have to trust someone to follow — access to the tool is just built the same way for everyone taking the assessment.
That's the piece that turns “did they use AI” from an unanswerable, unenforceable question into a controlled one: everyone had the same help available. What's left to measure is what they did with it.
A category, never a score
Here's what we didn't build, on purpose: a 0–100 “AI collaboration score.” It was the obvious thing to ship, and we cut it.
A number like that implies a precision the underlying signal hasn't earned yet — nobody has proven that “73” versus “81” on an AI-collaboration score means anything real about hiring outcomes. Worse, a number invites exactly the kind of automatic cutoff we don't think belongs in a hiring decision: reject everyone under 60, no human required. So instead of a score, the platform reads the transcript and returns a category — did the candidate validate what the AI gave them, accept it without checking, or mostly work it out themselves — and attaches the specific chat turns that led to that read. It's a smaller claim than a score, and it's one we can actually stand behind.
This is the same instinct behind everything we've shipped: a black-box number is a worse product than an answer you can click into and check yourself.
Evidence you can click into
The report a recruiter sees looks like this: pass/fail on the code, sitting next to the collaboration category. Click the category, and you're looking at the exact moment in the transcript it came from — not a black box telling you to trust it. Every one of those output events is written to the same audit log as the rest of a candidate's assessment, so the read is reviewable after the fact, not just at grading time.
A read, not a ruling
None of it decides anything on its own. The report is evidence for a person to weigh — never a rule that files a candidate out automatically. That's not a compliance footnote we bolted on; it's why the categorical output exists in the first place instead of a score that would have made an automatic cutoff tempting.
What this deliberately isn't, yet
In the spirit of not overselling this: there's no code execution sandbox behind this challenge — candidates write the fix, they don't run it, and grading works from the code and the rubric, not a passing test suite. And we're not claiming a validated “AI literacy score” exists, because it doesn't yet — we'd rather ship a category we can defend than a number we can't. If the categorical signal proves useful over the next few months of real hiring decisions, a more quantitative version is something we'd revisit — not something we're claiming today.
We're onboarding our first hiring teams into AI Debugging Challenge now. If you've got a role where “can this person work with AI without babysitting it” actually matters — which, increasingly, is most technical roles — bring us a real bug from your codebase and we'll show you the candidate side and the report together.