The HackerRank Open-Source ATS Didn't Have a Bug. It Had a Measurement Error.

I came across a post by Dan Kinsky that I haven't been able to stop thinking about. He ran HackerRank's open-source resume screener on the same resume a hundred times. The scores ranged from 66 to 99. If the company's cutoff sat at 85, his identical resume failed 65% of the time. He switched to a stronger model, which tightened the spread but didn't close it: at a cutoff of 60, the same resume still failed 28% of the time.

It stuck with me because I spend my days thinking about measurement, and what Kinsky documented is not really a story about AI hiring tools and bad luck, though that framing is fair enough. The deeper story is older than LLMs, and assessment science has a precise vocabulary for what went wrong. This isn't a bug. It's measurement error, and it's the exact failure mode psychometrics was built to catch.

What the numbers actually show

Every measurement decomposes into two parts: a true score and error. Observed score = true score + error. When you score one fixed resume a hundred times, the true score is constant by construction. There is nothing left to vary. So every point of spread between 66 and 99 is an error. The instrument is reporting noise and labeling it signal.

In testing terms, the author ran an accidental reliability study. Reliability is the share of score variance that reflects real differences between candidates rather than chance. Here, on a fixed input, that share is zero. A score that swings 33 points on identical input has, in the only sense that matters for a hiring decision, no reliability at all.

Why some categories held and others didn't

Two words worth defining first. Validity is whether an assessment measures what it’s meant to: a data-analysis test that evaluates real analysis skill, not just tool familiarity. Reliability is whether it gives consistent results across times, forms, and conditions, so the same candidate scores about the same on a retake. You need both. This piece is a story about a tool that had neither.

The category breakdown is the most useful part of the post, because it shows the failure isn't uniform.

Technical skills scored 8/10 in 98 of 100 runs. That category is a checklist: you list React or you don't. It's an objective, low-inference judgment, and machines make those consistently. Projects, by contrast, were all over the map, because "is this project architecturally complex" is a high-inference holistic judgment with no fixed answer. The model is acting as a rater, and on a subjective construct it's an untrained, uncalibrated, single rater. That is the textbook recipe for low inter-rater and intra-rater reliability.

The experience category is the sharpest lesson. It scored 25/25 on every run, for a principal engineer and an intern alike. Perfectly consistent, perfectly useless. A measure that returns the same value for everyone has zero variance, which means zero ability to discriminate, which means zero information. Consistency is necessary for a good measure but nowhere near sufficient. You also need the score to track the thing you care about. That's validity, and a two-line prompt with no anchors has none.

So the tool fails in both directions at once. Its judgmental category (projects) is valid in theory, but too noisy to be reliable. Its anchored-but-thin category (experience) is consistent but invalid. Either way the decision is uninformative.

How an assessment scientist would fix it

The fixes are not exotic. They're standard practice, and something my team has spent a lot of time thoughtfully putting into place at Workera. None of it is secret. It seems to have simply been skipped here.

1. Quantify reliability before you trust a single score. The author's hundred-run experiment is exactly the study you run first, not a gotcha that you discover after. If a measure isn't stable on repeated administration, you stop and fix the measure. You don't ship it.
2. Build rubrics with behavioral anchors and exemplars, not adjectives. "Give extra points for founder roles" is not a rubric. A real one defines what a 15 looks like versus a 22, with anchor cases for each band. Anchored rubrics are how decades of structured interviewing and essay scoring turned subjective judgment into something repeatable. Note the caveat the data makes obvious: rubric detail alone isn't enough. The projects category had a detailed rubric and was still the noisiest. Some constructs are genuinely hard to judge, which is why the next two steps matter.
3. Calibrate the rater against human judgment. Train the scorer on data that experts have scored and labeled, then evaluate it on a holdout sample the model never saw. If it can't reproduce expert scores on held-out cases, it isn't ready to rate that construct.
4. Aggregate multiple independent ratings. A single LLM call is a single rater having a single opinion. Averaging several independent scorings shrinks the standard error of measurement (roughly by the square root of the number of ratings). It's the cheapest reliability gain available and the tool uses exactly one.
5. Report bands, not points, and respect them in decisions. A score of 83 with a standard error of 9 is not meaningfully different from an 85. Hard cutoffs inside the error band manufacture false precision and turn noise into rejections. Decision consistency, the odds two administrations put a candidate on the same side of the line, is the number that should govern any pass/fail threshold.
6. Match the construct to the job, then validate it. Weighting open source and personal projects at 65% is a content-validity choice, and a questionable one, given that plenty of strong engineers ship nothing public. The real test is criterion validity: does the score predict on-the-job performance? If no one has checked, the weights are a guess wearing a number.

The takeaway

LLMs are not the problem here. They're excellent at the parsing and checklist tasks in this pipeline. The problem is using one as an unreliable holistic rater with no rubric anchors, no calibration, no aggregation, and no validation, and then drawing a hard line through the output. A tool that can't tell candidates apart isn't filtering for quality. It's filtering. The discipline that prevents this already exists. 

‍

Source: Dan Kinsky, "HackerRank open sourced its ATS...", Dan Unparsed, June 28, 2026.

‍