Why your best-interviewing AI candidate may not be your best AI hire | Daily Reports Online
Demand for AI fluency has risen nearly sevenfold in two years, faster than demand for any other skill, according to the McKinsey Global Institute.
My conversations with technology leaders echo this: nearly every one I speak with is hiring for AI skills.
The problem, however, is that fewer can tell me what AI fluency actually looks like on the job at their organizations.
Latest Videos FromTechRadar
CEO and Co-founder, TestGorilla.
McKinsey found nearly 90% of companies have invested in AI. Fewer than 40%, however, report measurable gains.
Most AI post-mortems take a hard look at models, data and workflows.
Few look at the people hired to champion AI transformation or how and why they were chosen.
The risks of mistaking confidence for competence
A side effect of AI advancement is that it has made the ability to speak about AI use much easier. A candidate might start by naming every model, then walking you through an architecture they read about last week. In a short conversation, fluent language is almost impossible to separate from fluent practice.
Those who win over the hiring manager are those who sound the most at ease when speaking to AI. Whether they can actually use AI to do the job is a separate question, and most hiring processes never ask it.
Think of an interview as a demo on the candidate’s chosen grounds with controlled setup and their narration. On-the-job AI use is where the trouble starts: without proper AI fluency, it’s easy to lose control of messy data, missing edge cases, systems that refuse to talk to each other, or a model that hallucinates at the worst possible moment.
Just because a candidate dazzles in the demo doesn’t mean they won’t stall in a production environment.
On the surface, this looks like a people problem. In reality, it’s a hiring process problem.
What does a bad AI hire really cost?
The true cost of bad AI hires typically takes time to manifest. It’s rarely on day one, and it can often take several forms. For example, a new AI model goes into production and starts hallucinating in front of a client.
Digging further into the root of this risk reveals that most companies have no shared definition of what “good” looks like for working with AI. One manager may test it one way, another may test it completely differently, and a third goes on gut feel. When the fluency bar changes with every interviewer, organizations are limited to collecting surface-level impressions rather than assessing skills.
Without that shared definition, the damage eventually reveals itself. A project stalls because the person who talked a great game can’t get a model to produce anything reliable. Or worse: they can, but only for themselves. They 10x their output, then leave light documentation no one else can follow and route around security and compliance to do it.
Therein lies more risk: productivity that works for one person and breaks the system around them has wide-reaching ramifications.
Eventually, the work has to be redone and the role reopens. Project deliverables slip another quarter and the company is forced to burn more budget.
Bad hiring is not the only reason AI investment underdelivers, but it is a bigger one than most leaders admit.
Test for competence, not vocabulary
Simply adding another requirement to job descriptions isn’t going to solve this issue. Instead, it’s deciding the scope of AI use required by the role, then building a process that makes job candidates demonstrate this level of use rather than describe it.
Too many companies set the standard around whether a candidate knows the tools exist. Going deeper to set the bar at independent, verified use makes the fluency gap immediately visible.
There are several changes teams can make to the hiring process to shift the measurement of AI fluency:
1. Test the work before the conversation
Put candidates in front of a scenario that mirrors the real job and score it against a fixed set of capabilities agreed in advance. Say you’re hiring for a product designer, then drop them into a live brief with a real product constraint and ask “why” at every step.
The candidate limited to fluent talking hands you forty polished directions. The one who can actually do the job tells you which one survives the edge cases, what accessibility rules are needed, and the engineer who has to build it. That’s the test most interviews never run.
2. Change the interview questions
Stop asking which tools people use. Ask about the last time AI gave them a wrong answer and how they caught it, or to show a prompt that failed and how they fixed it.
Go deep on one real piece of work: what changed, what broke, what they checked, and who the output affected. None of that survives someone who has not done the work.
3. Pivot the script
Give the candidate a realistic AI task, then change it halfway through by taking a tool away, adding a constraint, or moving the goal. Strong candidates reframe and carry on, while performers stall or describe what they would theoretically do.
Ask to see the workings, not the output. Let people use AI in the assessment on the condition that they show you their workflows, automation, prompts and the reasoning behind them. Anyone can produce a polished output now. How they got there is more important.
4. Score it together
Give every interviewer the same dimensions to judge, set before the conversation, then compare evidence in a shared debrief session. The shared evidence review becomes the critical lever for the final hiring decision and fuels the definition of AI fluency for that candidate.
Today’s AI hiring continues to fall into the trap of selecting the candidate who interviews best, only to find a quarter later that they can’t deliver the actual work.
A vague hiring process may feel efficient because it asks less of the people running it. However, the business impact is real as it moves the cost downstream into deployment, where it becomes much harder to trace and far more expensive to fix.
Next time you make an AI hire, skip the questions about which tools they use. Instead, focus on the last time one of those tools failed them, and what they did about it.
We’ve featured the best HR software.
This article was produced as part of TechRadar Pro Perspectives, our channel to feature the best and brightest minds in the technology industry today.
The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here:
