Why Technical Interviews Don't (Always) Predict Job Performance

Most companies treat the technical interview as gospel. Whiteboard a binary tree, solve a dynamic programming puzzle under time pressure, and you're in. Fail, and you're out. It feels rigorous. It feels like engineering. But the research says it's barely better than flipping a coin.

The Meta-Analysis That Should Have Changed Everything

In 1998, Frank Schmidt and John Hunter published what remains the most comprehensive meta-analysis of hiring methods ever conducted. They examined 85 years of research across hundreds of studies and thousands of hires. Their central finding:

Unstructured interviews (the kind most companies run) predict job performance at only r=0.38.

To put that in context: a perfect predictor would score r=1.0. A coin flip scores r=0.0. Most engineering interviews land closer to the coin flip than to certainty. And yet entire hiring pipelines (months of recruiter screens, phone calls, on-sites) are built on this foundation.

Even more damning: years of experience predicts job performance at just r=0.18. The thing most job postings filter on first is one of the weakest signals available.

Google Figured This Out the Hard Way

In 2015, Laszlo Bock, then SVP of People Operations at Google, published Work Rules! detailing what Google learned from analyzing thousands of interviews against actual on-the-job performance data. The results were humbling:

Brainteaser questions had zero predictive validity. Google banned them entirely.

Their data showed that interview scores from individual interviewers were essentially random noise. One interviewer's strong hire was another's reject, with no correlation to who actually performed well on the job. The signal-to-noise ratio was so poor that Google concluded most of their interview data was worthless.

This wasn't a small company running casual interviews. This was Google, with all its data science capability, with thousands of data points, with every incentive to make interviews work. They couldn't. They had to rebuild their entire process from scratch.

Whiteboard Interviews Measure Anxiety, Not Ability

In 2020, researchers at North Carolina State University (Behroozi et al.) ran a controlled study that should have ended the whiteboard interview forever. They gave qualified developers the same coding problems under two conditions:

  1. Solving the problem on a whiteboard while being observed
  2. Solving the problem privately on their own computer

The results: half of the developers who solved the problem privately failed when observed. The performance gap wasn't about ability. It was about anxiety. The same engineer, the same problem, completely different outcomes based solely on whether someone was watching.

The researchers concluded that whiteboard interviews are "chiefly measuring whether a candidate is comfortable coding in front of an audience," a skill that has almost nothing to do with building production software. Worse, this anxiety effect disproportionately impacts women, underrepresented minorities, and anyone who doesn't fit the stereotypical "confident tech bro" mold. The interview format isn't just inaccurate. It's systematically biased.

What Software Engineering Researchers Found Next

The Behroozi team's 2020 study wasn't isolated. It was part of a sustained research program at North Carolina State studying technical hiring from an engineering perspective.

Their earlier work (2019, presented at VL/HCC) analyzed thousands of developer discussions on Hacker News about interview experiences. The qualitative findings were consistent: developers described interviews as lacking real-world relevance, biased toward younger candidates, and testing memorized algorithms rather than engineering judgment. The gap between what interviews measure and what engineering work requires was a recurring theme across hundreds of responses.

In 2022, the same team published a follow-up study at FSE that won a Distinguished Paper Award. They proposed asynchronous technical interviews, where candidates submit recorded think-alouds instead of coding under live observation. Removing the supervision significantly improved clarity, informativeness, and reduced stress. The improvement was especially pronounced for candidates who identify as women, directly addressing the bias their earlier studies documented.

The Metrics Problem

Even when companies move beyond traditional interviews, they often replace one flawed signal with another. GitHub activity graphs, commit frequency, lines of code produced: these are the metrics that show up in "data-driven" hiring.

In 2021, researchers from GitHub and Microsoft Research (Forsgren, Storey, Maddila, Zimmermann, and others) published the SPACE framework, identifying five dimensions of developer productivity: Satisfaction, Performance, Activity, Communication, and Efficiency. Their central finding: no single metric captures developer productivity. Activity metrics alone (commits, PRs, lines of code) should never be used to evaluate developers.

Noda, Storey, Forsgren, and Greiler extended this work in 2023 with the DevEx framework, showing that developer experience (encompassing flow state, feedback loops, and cognitive load) drives real productivity. Output-volume signals that companies use as screening proxies are shown to be poor measures of actual productive contribution.

The pattern across this research is clear: whether the signal is interview performance, resume keywords, or GitHub commit graphs, the metrics companies use to evaluate engineers are consistently poor proxies for engineering ability.

The Resume Problem

Resumes compound the issue. Schmidt and Hunter's data showed that biographical information and reference checks add minimal predictive value. Yet resumes remain the primary filter for most engineering hiring.

The result is a system optimized for pattern matching, not ability detection. Recruiters scan for brand-name companies, prestigious universities, and keyword density. Engineers who took non-traditional paths (self-taught developers, career changers, people from underrepresented backgrounds) get filtered out before anyone evaluates their actual capability.

A 2019 study by the National Bureau of Economic Research found that callback rates for identical resumes varied by 50% based solely on the name at the top. The resume screen isn't just weak. It actively introduces bias that subsequent interview stages can't correct.

The Feedback Loop Nobody Talks About

Here's what makes broken hiring self-reinforcing: companies only see the performance of people they hire. They never see the performance of people they reject.

If your interview process systematically filters out anxious but brilliant engineers, you'll never know. Your "successful hires" will all be people who are good at interviews, and you'll conclude your process works. Meanwhile, the engineers you rejected (who might have outperformed everyone on your team) go somewhere else. Or leave the industry entirely.

This survivorship bias means most companies have no idea how much talent their process discards. Google only discovered the problem because they had enough data and analytical rigor to study it. Most companies don't, and they never question the process.

What This Means

The research paints a clear picture: the standard engineering hiring pipeline (resume screen, recruiter call, phone screen, whiteboard on-site) is a system designed to feel rigorous while producing mediocre results.

The methods that actually work (work sample tests, structured evaluation, cognitive assessment, long-term observation) require more investment. But they exist, and the evidence for them is overwhelming. A 2015 systematic literature review by Lenberg, Feldt, and Wallgren defined "Behavioral Software Engineering" as a field, mapping where psychology and organizational behavior concepts have (and have not) been applied to software development. Their conclusion: the knowledge exists. The gap is adoption.

The question isn't whether better methods are available. It's whether companies are willing to use them.

References