ATS Screening Accuracy: Can Small Agencies Trust AI Filtering?
TL;DR: AI resume screening sorts candidates fast, but speed is not accuracy. The same systems that surface strong candidates can also quietly filter out good ones through keyword mismatches, biased training data, and overconfident scoring. This guide explains how ATS screening accuracy actually works, where it breaks down, and how to run a monthly audit that keeps your filters honest.
About this guide: Written by the Recruitable team, the company building AI recruitment technology for recruitment agencies. Bias and legal claims below are cited to primary sources including Reuters, the EEOC, and federal court reporting. The audit framework is our own process, built for how agencies actually work.
How ATS screening accuracy actually works
Most ATS platforms screen resumes using one or more of three mechanisms. Understanding which one you are dealing with is the first step to judging whether you can trust it.
1. Parsed matching. The system extracts structured fields (job titles, skills, dates, education) from the resume and compares them to the job requirements. This is deterministic and explainable: either the skill is present or it is not. Its weakness is literalness. A candidate who writes “client acquisition” will not match a filter looking for “business development,” even though a human recruiter knows they are the same thing.
2. Semantic matching. Newer systems compare the meaning of resume text against the meaning of the job description, so synonyms and paraphrases match. This handles the “client acquisition” problem better, but it introduces a new one: you cannot always see why two things were judged similar. The reasoning is harder to inspect.
3. Learned scoring. Some tools score or rank candidates based on patterns learned from historical hiring data: which resumes led to interviews, offers, or hires in the past. This is the most powerful and the most dangerous of the three, because it inherits whatever patterns, fair or not, existed in that history.
Most agencies use a blend of all three without realizing it, and they treat the output, a ranked list or a shortlist, as if it were objective. It is not. It is a set of statistical guesses, each with its own failure modes.
Where screening fails: the three accuracy problems
False negatives are the expensive failure. When a screening tool rejects a candidate who would have been a great hire, nobody notices. There is no complaint, no lost placement fee on a ledger, just a placement that never happened. Research on hiring technology has repeatedly shown that automated screens can disadvantage qualified candidates who describe their experience in nonstandard terms, career changers, and people with employment gaps. Because the cost is invisible, agencies chronically underinvest in checking for it.
Keyword gaming cuts the other way. Once candidates learn that software scans for keywords, some optimize their resumes for the scanner rather than the reader: keyword-stuffed skills sections, job titles copied verbatim from postings, invisible text. Article 1 of this series covers how agencies spot AI-generated applications in detail. The screening accuracy problem is the mirror image: a system that rewards keyword density will rank a gaming candidate above a genuinely qualified one who wrote naturally.
Bias is the documented risk. This is not theoretical. In October 2018, Reuters reported that Amazon had built and then scrapped an experimental AI tool that scored job candidates from one to five stars. Trained on ten years of resumes submitted to a heavily male engineering workforce, the system taught itself that male candidates were preferable: it penalized resumes containing the word “women’s” (as in “women’s chess club captain”) and downgraded graduates of two all-women’s colleges.
Amazon’s engineers patched the specific terms but could not be confident the system would not find other proxies for the same bias, so the project was shelved.
Regulators have noticed. The US Equal Employment Opportunity Commission’s technical guidance on AI and Title VII (May 2023) treats AI-driven selection tools as subject to the same disparate-impact analysis as any other hiring test: if a tool selects a protected group at a substantially lower rate, the employer must show the practice is job-related and consistent with business necessity. The guidance specifically names resume scanners that prioritize applications using certain keywords. And New York City’s Local Law 144 requires employers using automated employment decision tools to conduct annual independent bias audits and notify candidates, a rule other jurisdictions are watching closely.
Then there is the question of who is liable. In Mobley v. Workday, a case in the Northern District of California, a federal judge allowed discrimination claims to proceed against the AI screening vendor itself under an “agent” theory, reasoning that when an employer delegates the traditional screening function to a vendor’s software, the vendor participates in the decision rather than merely implementing employer criteria. As reporting on the case notes, the court conditionally certified a collective action in May 2025. The case is still being litigated, and no final finding of discrimination has been made, but the direction is clear: neither “the vendor did it” nor “the software did it” is a reliable shield. Employers remain responsible for the outcomes of tools they deploy.
| Failure mode | What it looks like in your pipeline | How to spot it |
|---|---|---|
| False negatives | Strong candidates never reach shortlist; hiring managers complain about thin pipelines | Audit a sample of rejected candidates manually each month |
| Keyword gaming | Top-ranked candidates interview poorly; resumes read like job postings | Compare screen scores against interview outcomes |
| Historical bias | Certain groups advance at lower rates; patterns mirror past hiring | Measure selection rates across groups where legally permitted |
| Overconfidence | Recruiters stop reading resumes and rubber-stamp the ranking | Track how often humans override the tool, and in which direction |
| Criteria drift | The tool’s idea of “qualified” slowly shifts away from what clients want | Re-validate screening criteria against recent successful placements |
The Recruitable Screening Trust Audit
You do not need a data science team to keep your screening honest. You need a routine. The Recruitable Screening Trust Audit is a four-step loop we recommend agencies run monthly. The numbers below are our recommended operating ranges for a small agency, not universal standards.
Step 1: Calibrate (30 minutes). Pick one active role. Write down, in plain language, what the screening tool is actually filtering for: which keywords, which score threshold, which knockout criteria. If you cannot state the criteria clearly, that is your first finding. Vague filters produce vague accuracy.
Step 2: Sample (1 hour). Pull a random sample of 25 to 50 candidates the tool rejected for that role. Have a recruiter review them blind, without seeing the tool’s scores, and mark each as “would advance” or “would not advance.” For agencies screening fewer than 200 applications per role, a sample of 15 to 25 is enough to surface patterns.
Step 3: Measure (30 minutes). Compare the human judgments against the tool’s decisions and calculate three numbers:
- False negative rate: the share of rejected candidates your recruiter would have advanced. Our recommended target is under 10 percent. Above 20 percent means the filter is costing you placements.
- Override rate: the share of the tool’s top-ranked candidates your recruiters would not have advanced. Above 30 percent suggests keyword gaming or criteria mismatch.
- Consistency check: run the same 10 resumes through the tool twice, two weeks apart. If rankings shift substantially with no criteria change, the scoring is unstable and should not drive decisions alone.
Step 4: Adjust (ongoing). Every finding becomes a concrete change: loosen a knockout criterion that is rejecting good candidates, add a synonym the semantic matcher missed, raise the human-review threshold for borderline scores, or document why a criterion stays. Record each change with a date so you can see what you tried and what worked.
Run the loop monthly for your highest-volume roles and quarterly for everything else. The point is not perfection. The point is that your screening gets measurably better over time instead of silently drifting.
What the law actually expects (and what it does not)
A short, careful note, because this topic attracts confident legal claims from non-lawyers. In the United States, the EEOC’s position is that AI hiring tools are subject to the same anti-discrimination framework as traditional selection procedures, including disparate-impact analysis and the employer’s burden to show job-relatedness. That is guidance, not a statute, but courts give it weight, and private lawsuits like Mobley are testing vendor liability directly.
What the law does not currently do, in most jurisdictions, is ban AI screening or prescribe a specific audit methodology for small agencies. New York City’s bias-audit law applies to employers using automated tools for employment decisions in the city, with its own definitions and thresholds. The EU AI Act classifies employment-related AI systems as high-risk, with obligations that phase in over time. If you hire across jurisdictions, the compliance picture is genuinely complicated, and this article is not legal advice. Talk to employment counsel before deploying any screening tool at scale.
The practical takeaway for a small agency is simpler than the legal landscape: document what your tools do, check whether the outcomes look skewed, keep a human in the loop, and keep records of your audits. That is what regulators, clients, and courts will ask about if anything goes wrong.
Where your ATS fits in
ATS screening accuracy is partly a tool problem and partly a workflow problem. The workflow part is where agencies lose the most ground.
The single most useful thing an ATS can do for screening accuracy is preserve the reasoning. When a candidate is filtered out, the record should show why: which criteria they missed, what score they received, who reviewed the decision. Without that history, audits are impossible and every “why didn’t we see this candidate” conversation ends in guesswork.
Recruitable keeps screening decisions attached to the candidate record, so the audit trail builds itself as recruiters work. Pipeline history shows when a candidate was advanced or rejected and on what basis, which means the monthly sample in the Screening Trust Audit takes minutes to pull instead of an afternoon of spreadsheet archaeology. And because Recruitable’s AI copilot works alongside recruiters rather than auto-rejecting behind their backs, the human stays in the loop by default: the tool recommends, the recruiter decides, and the record shows both.
That is the posture we recommend for any screening tool, whatever you use. Treat AI filtering as a first opinion, never the final one. Audit it on a schedule. And make sure you can reconstruct any decision a year later, because someday you may have to.
FAQ
What is ATS screening accuracy?
ATS screening accuracy is how correctly an applicant tracking system’s automated filters identify qualified candidates: advancing the right people and rejecting the wrong ones. It is measured through false positive and false negative rates, not through speed or volume processed.
Can small agencies trust AI resume filtering?
AI filtering is a useful first pass but should not make final decisions unsupervised. Documented cases, including Amazon’s scrapped 2018 recruiting tool reported by Reuters, show that screening systems can inherit bias from historical data. Agencies should audit their tools regularly using a structured process like the Recruitable Screening Trust Audit.
What is the biggest risk of AI screening for agencies?
False negatives: qualified candidates silently filtered out before any human sees them. Unlike a bad hire, a missed placement leaves no trace, so agencies should manually review a sample of rejected candidates every month.
Does using an AI screening tool transfer legal liability to the vendor?
No. EEOC guidance holds employers responsible for discriminatory hiring outcomes even when a third-party tool produces them. The Mobley v. Workday litigation suggests vendors may also face liability under an agency theory, but that does not reduce the employer’s exposure. The case is still being litigated.
How often should agencies audit their screening tools?
We recommend monthly audits for high-volume roles and quarterly audits for everything else. At minimum, re-validate screening criteria against recent successful placements whenever you notice pipeline quality dropping.
What is New York City Local Law 144?
A 2023 law requiring employers in New York City that use automated employment decision tools to conduct annual independent bias audits and provide notice to candidates. It is the most concrete AI hiring audit mandate in the US so far, and other jurisdictions are considering similar rules.
ATS screening accuracy is not a feature you buy, it is a practice you maintain. The Recruitable Screening Trust Audit gives your agency the monthly routine, and Recruitable’s candidate records and pipeline history give you the audit trail to run it in minutes. If you want screening your recruiters can actually defend, book a demo.
Harvey Mor is on the Recruitable team, covering recruitment technology, agency operations, and how AI is changing the way agencies source, verify, and place candidates.