Hiding someone’s name doesn’t hide their race from an AI
A ZIP code, a home address and a university encode race. This is not a discovery. It is the premise of redlining, decades of housing and lending scholarship, and the reason US legal scholars named proxy discrimination as the distinctive risk of AI decision systems. Anyone who has thought about it for ten minutes expects the leak.
The open question is how much, from which field, and whether it changes who gets picked. Those are measurable, and nobody publishes the numbers.
An AI screening tool does not read a résumé as text. It converts it into a vector, a long list of numbers, and everything downstream, the ranking and the shortlisting, runs on that list. That list is what I measured.
What I found
I showed the model two résumés that are identical word for word, differing in exactly one detail, and asked which applicant is Black. Scores run 0.5 to 1.0, where 0.5 means it is guessing at random and 1.0 means it is right every time.
| The one detail that differs | Model guesses race | In plain words |
|---|---|---|
| Home address (a 96%-Black vs a 0.4%-Black ZIP) | 0.984 | right about 49 times in 50 |
| University (an HBCU vs a state school) | 0.983 | right about 49 times in 50 |
| Gender (first name + pronoun) | 0.979 | right about 24 times in 25 |
| The name itself (Jamal vs Greg) | 0.945 | right about 19 times in 20 |
The two proxies beat the name, on all four models.
This is a test of what screening vendors already do
Name-blind screening is not hypothetical. Redacting the candidate’s name before the document reaches the model is standard practice. It is sold as a feature, it appears in procurement checklists, and several vendors strip the name so it never reaches the embedding at all.
Each row above measures exactly that situation. In the address row, the only identifying detail on the résumé is Address: Chicago, IL 60619. No name anywhere. So the address row is the name-blind pipeline: strip the name completely, and the model still identifies race 49 times out of 50.
Redaction removes the 0.945 signal. It leaves the 0.984 and 0.983 signals untouched, because nobody redacts a home address, you need it, and nobody redacts a degree, because the degree is the point of the résumé. The one field that gets removed is the weakest of the three.
The workflow, step by step
- Strip the name from the résumé.
- Send the nameless résumé to the model, which scores it and ranks it against everyone else.
- Put the name back, so the recruiter knows who they are calling.
Step 3 cannot change the outcome; the name was not there when the score was computed. That leaves everything resting on step 2, and step 2 is the first row of the table. And the name returns at the worst possible moment: it protects the stage where the name matters least, and restores it for the stage where it matters most.
Why this matters for an audit. A pipeline built this way can be described, accurately, as name-blind, and still produce a race-correlated ranking. An audit that checks whether the name was removed will pass it. An audit that measures whether the attribute is still readable will not.
The data
AI safety, NLP and ethics.