The short version
Most fairness audits stop at the output: ask the model a leading question, see what it says, score it on a stereotype scale. That tells you what the model has been trained to say. It doesn't tell you what its representations still encode.
I measured both levels. First, output-level stereotype preference on six open-weight LLMs: LLaMA-2 7B (base and chat), Mistral 7B (base and instruct), Phi-2 2.7B, and InkubaLM 0.4B. Then, for InkubaLM, the same encoder-level probe suite I ran on Gemma 4. The two pictures disagree: the model that looks cleanest at the output level is not cleaner inside.
Output level. For each (stereotype, anti-stereotype) sentence pair I ask which one the model assigns higher probability, averaged over 50 pairs per axis, 28 axes. (For causal LMs this is ordinary per-token log-likelihood — mean log P(token | left context). The term "pseudo-log-likelihood" strictly belongs to masked-LM scoring, Salazar et al. 2020; I keep the mechanics but the honest name for what I computed is length-normalized log-likelihood.) A perfectly indifferent model sits at 0.500.
| Model | Params | Stereotype-pref rate | Biased axes (≥60% pref) |
|---|---|---|---|
| InkubaLM 0.4B | 0.4B | 0.550 | 7 / 28 |
| LLaMA-2 7B base | 7B | 0.599 | 15 / 28 |
| LLaMA-2 7B chat | 7B | 0.611 | 16 / 28 |
| Mistral 7B base | 7B | 0.611 | 17 / 28 |
| Phi-2 2.7B | 2.7B | 0.612 | 17 / 28 |
| Mistral 7B instruct | 7B | 0.618 | 18 / 28 |
InkubaLM's preference rate sits 5 points above random; the larger models sit 10–12 points above. Caveat that applies to the whole table: n=50 pairs per axis. That's enough for the model-level ordering (the InkubaLM gap is roughly 8× the run-to-run noise) and not enough for confident per-axis claims.
The encoder level: same suite as Gemma 4, applied to InkubaLM
I ran the 8-lens probe suite (linear probe, MLP probe, whitened linear, direction-consistency of pair differences, centroid distance, subspace residual, effective rank, Bhattacharyya overlap) on InkubaLM's hidden states at layers 0, 4, and 7 of its 8 layers, 500 pairs per axis, 5-fold cross-validated probes.
The individual techniques are standard: linear/MLP probes (Alain & Bengio 2016), PCA-based subspace analysis, effective rank (Roy & Vetterli 2007), Bhattacharyya overlap (1943). What I add is running them together across 28 axes and reading them jointly. Probing has well-known failure modes — expressive probes can succeed for reasons unrelated to the property under study (Hewitt & Liang 2019 introduced control tasks for exactly this), and extractability does not imply the model uses the information (Elazar et al. 2021; Belinkov 2022) — so treat every number below as "decodable from the representation," not "drives behavior."
The result, under the run's scoring rule:
InkubaLM triggers the bias-signal rule on 28 of 28 axes — the same count as Gemma 4. The rule is permissive (an axis counts if any of the eight lenses crosses threshold), so the right reading is "output cleanness does not translate to encoder cleanness," not "InkubaLM is maximally biased."
Before the table, a word about why InkubaLM is in this study at all — because the point is the opposite of a takedown. InkubaLM is exactly what responsible model-building for underserved communities looks like: open weights, a transparent paper, a pretraining mix built deliberately around African languages, by a team working with far less compute than the frontier labs. It also posts the cleanest output-level numbers in my entire test. That's precisely what makes it the most important data point here: if encoded bias were a failure of care, effort, or intent, this model would be the exception. It isn't — because encoded bias isn't a moral failure of any builder. It's a structural property of learning representations from human text, and it shows up in a community-built 0.4B model and a frontier-lab 31B model alike. The conclusion is not "don't trust community models." It's that good intentions at pretraining time, and even good output behavior, cannot substitute for intervention at deployment time — for any model, from any builder. Releasing open weights to a community is an act of empowerment; giving that community the tools to see and control what's inside those weights is the necessary second half.
Per benchmark family:
| Benchmark family | What fired | Reading |
|---|---|---|
| BBQ (7 axes) | 5 of 7 axes: linear and MLP probes at 0.97–1.00 at every probed layer. The other two show a real nonlinear gap — Religion linear 0.83 / MLP 0.99, Sexual orientation linear 0.77 / MLP 1.00 | Strong, cross-validated separation — the most defensible encoder-level finding in this post |
| CrowS-Pairs (9 axes) | Linear near chance; subspace residual 0.70–0.81 | Suggestive of structure the linear probe can't see — but my subspace metric lacks a same-class null baseline (see below), so I treat this as a hypothesis, not a verdict |
| StereoSet + WinoBias (12 axes) | WinoBias linear probes at 0.09–0.12 — far below chance; subspace residual 0.56–0.73 | Evaluation artifact, not bias. Below-chance accuracy on a balanced binary task means the evaluation is degenerate — WinoBias pairs differ by a single pronoun, mean-pooled embeddings are nearly identical, and CV fold structure can invert the learned mapping. A number like this should trigger a methods review, never a bias claim |
On the same-class-baseline caveat: fitting a 10-component PCA subspace on ~500 stereotype embeddings in 2,048 dimensions leaves a large residual for any second sample, including a second draw of the same class. Until I compute that baseline, subspace-residual values are not evidence on their own. (Notably, the lens does vary meaningfully — BBQ/Age shows 0.09–0.17 where CrowS axes show 0.70–0.81 — which is why I think it's worth validating rather than discarding.)
One contrast that survives all the caveats: Gemma 4's BBQ axes show the suppression trajectory — linear probe peaks 0.80–0.95 early, collapses to 0.26–0.35 by the final layer. InkubaLM's BBQ axes show no collapse — the probe stays at peak through the last layer. That's consistent with InkubaLM not having gone through a comparable RLHF regime: the internal separability is similar; the output-adjacent polish that hides it from output-level audits is what differs.
Data: inkubalm_full_algebra.json (77 KB) and the companion gemma4_full_algebra.json (~330 KB, all probed layers and lenses).
The models you're deploying right now don't even hide it
If you're building on an open model in 2026, it's probably one of these: Qwen3-32B, Qwen3.6-27B, Phi-4-reasoning, or a DeepSeek-R1 distill. I ran the identical suite on all four (same benchmarks, same 500-pair cap, same CV, 16–17 probed layers). Gemma 4 at least buried its bias below the output layers. These models carry it in the open — linearly readable at the final layer, the exact layer your pipeline consumes. Full per-layer JSONs are downloadable below.
| Model | Suppression axes | Linearly persistent axes | Peak linear > 0.65 |
|---|---|---|---|
| Gemma 4 31B-IT | 5 | 1 | 7 / 28 |
| Qwen3-32B | 1 | 9 | 15 / 28 |
| Qwen3.6-27B | 0 | 10 | 17 / 28 |
| Phi-4-reasoning | 0 | 10 | 15 / 28 |
| R1-Distill-32B | 1 | 9 | 16 / 28 |
Three findings, stated plainly:
-
The Gemma-style suppression signature is essentially absent in these four models. All seven BBQ axes are linearly persistent on every one of them — the probe stays high through the final layer. Whatever alignment these models received, it did not push demographic separability out of the output-adjacent layers. Gemma 4 hid it; these models carry it in the open.
-
The WinoBias below-chance pathology (linear 0.09–0.12) replicates on all four — six models total now. That number is an evaluation artifact, every time, on every model. If you see it in a bias paper, ask for a methods review.
-
I initially highlighted a third finding here — CrowS race-color reading at 0.70–0.74 final-layer linear accuracy on all four models. The control experiment below demoted it. I am leaving this sentence in as a record of how easily an honest-looking probe number turns out to be measuring something else.
The control that separates "knows the word" from "carries the stereotype"
Benchmark pairs differ in the group term ("Black"/"white", "he"/"she") — so a probe that separates them might only be detecting which word appeared. Any competent language model encodes word identity; that's not bias, that's vocabulary. To split the two readings I built a lexical-swap control: 200 minimal pairs per axis where the same group terms appear in aggressively mundane, stereotype-free sentences ("The Black man parked the car near the library." / "The white man parked the car near the library."), then ran the kill-shot both ways — a probe trained only on these neutral word-swaps, tested on the benchmark's stereotype pairs, and vice versa.
Results on Phi-4-reasoning and Qwen3-32B (the two models tested so far; data, data):
- CrowS-Pairs separability is explained by word identity. On essentially every CrowS axis, the neutral word-swap probe transfers to the benchmark at or above the benchmark's own accuracy (stereotype excess ≈ 0 or negative). The CrowS "persistent" verdicts in my table — including the race-color 0.70–0.74 I highlighted — are the model reading group words, not stereotypes. Consider them retracted as bias evidence.
- BBQ separability survives the control. The BBQ axes read at 0.77–1.00 while the word-swap probe transfers at chance (~0.50) — a stereotype-side excess of +0.27 to +0.49. Whatever the probe reads on BBQ, it is not the bare group-word direction. (Survival of this control is necessary, not sufficient: BBQ's structure — same context, group term embedded in a full answer — leaves other explanations to rule out. But it's the half of the evidence that stands.)
This control is the difference between an audit and an accusation, and I'd hold any published bias-probing result — ours or anyone's — to it. Half my table failed it. The half that passed is now much stronger evidence than the whole table was yesterday.
The remaining caveats travel with the surviving results: the permissive any-lens rule still says 28/28 everywhere (report the lens breakdown, not the headline count), and the statistical hardening pass (repeated-CV confidence intervals, permutation tests, pooled concept axes, subspace null baselines) is finishing now — those are the numbers I'd defend in front of a reviewer.
Per-model data: qwen3_32b · qwen36_27b · phi4_reasoning · r1_distill_32b
Why is InkubaLM cleaner at the output level? Three candidates, weighed honestly.
Candidate 1: smaller models encode less bias. Phi-2 (2.7B, ~7× InkubaLM's size, well below the 7B group) should then sit between InkubaLM and the larger models. It doesn't — 0.612, indistinguishable from the 7B models. Reject.
Candidate 2: RLHF cleans up output bias. Looking at base-vs-tuned pairs: LLaMA-2 base 0.599 → chat 0.611; Mistral base 0.611 → instruct 0.618. Both tuned models are marginally more stereotype-preferring on this metric. That doesn't mean RLHF made them more biased in deployment — likelihood scoring is not conversation — but RLHF isn't moving this metric downward. Doesn't explain the gap.
Candidate 3: pretraining data composition. InkubaLM's pretraining mix is heavily weighted toward African languages — roughly 1.9B tokens of African-language data and 360M of English (arXiv:2408.17024). The larger models are predominantly English web text. A different corpus produces a different stereotype surface, and the effect size here (7 vs 15–18 flagged axes) dwarfs both the RLHF effect (~1 point) and the scale effect (none, per Phi-2).
Candidate 3 is the most consistent with my data, but this is one model differing from the others in data, size, tokenizer, and training recipe simultaneously — a confounded, n-of-one comparison. It suggests pretraining composition matters more than post-hoc alignment for this metric; establishing that would take controlled runs varying data mix alone. Also worth remembering: most of these stereotype benchmarks are authored in English against US-centric stereotypes, so a model with less English may partly be showing unfamiliarity with the benchmark's framing rather than absence of bias.
What this means in practice — and why RLHF can't be the whole answer
If your product taps any of these models' hidden states for retrieval, semantic search, or classifier features, the output-level table is the wrong safety evidence: it measures the layer your pipeline bypasses. The encoder-level results say decodable group structure is present regardless of output polish. What they don't say is whether it changes your system's behavior — intrinsic bias measures are known to correlate poorly with downstream harm (Goldfarb-Tarrant et al. 2021). Both audits together are a reason to test your deployed pipeline directly, not a verdict on it.
Step back and the six-model comparison makes a structural point. RLHF is a fix applied at the output layer of a system whose value is increasingly consumed at the representation layer. RAG reads representations. Semantic search reads representations. A locally hosted open model hands its representations — and the ability to fine-tune away the output polish — to whoever deploys it. For all of those, the alignment training you're implicitly trusting sits on the one layer your pipeline doesn't use. InkubaLM makes the same point from the other direction: a model can look cleanest at the output for reasons (data composition) that have nothing to do with its encoder being clean.
The conclusion I draw: assurance has to live in a middle layer — between whichever foundation model you use and the application your users touch. That layer's job is exactly what these audits sketch: explain what the model's representations encode (the explainable-AI half — named axes, measured strengths, downloadable evidence, not post-hoc rationalizations), control it per-request (detection, editing, steering — see the steering post), and prove it (signed per-call records an auditor can re-test). That combination is also the shape regulation is converging on: the EU AI Act and NIST AI RMF ask deployers for understanding, control, and records of the system; a middle layer is the only place those three can live when the model underneath is rented, swapped, or fine-tuned downstream.
My current capability evaluations under harder protocols — independent judges, ceilings from real target examples, leakage matrices, multiple seeds — are summarized in the Gemma 4 post: coarse-concept detection on frozen third-party backbones at 0.96–0.97 AUROC; flips as convincing as genuine targets on detectable axes; my internal editing operator with 2–12× less collateral than the standard baselines (difference-of-means and LEACE); steering verified on self-hosted models for strongly encoded concepts. Each with its measured boundary.
Open problems in this audit
- The lexical confound is unresolved. Stereotype/anti-stereotype pairs differ in the demographic term itself, so a probe can separate them by detecting which group word is present — something any working encoder does, biased or not. Separating "encodes group identity" from "encodes stereotypical association" needs swapped-term controls I haven't run at scale. This is the biggest open hole, here and in most probing-based audits.
- Benchmark validity. StereoSet and CrowS-Pairs have documented construct-validity problems (Blodgett et al. 2021 — ill-formed pairs, ambiguous stereotype direction). Per-axis numbers inherit them.
- The instruction-tuned regression is unexplained. Both chat variants score marginally worse than their bases. Possibilities: instruct templates interacting with likelihood scoring, generally sharper next-token distributions, or genuine non-monotonicity. I'd want to control for templating before saying more.
- 50 pairs per axis supports model-level ordering only.
Reproduce
The output-level audit is a few hundred lines of Python; the encoder-level methodology is described in the Gemma 4 post. Benchmark sources are public:
- BBQ — Parrish et al. 2022
- CrowS-Pairs — Nangia et al. 2020
- StereoSet — Nadeem et al. 2021
- WinoBias — Zhao et al. 2018
Full per-axis, per-layer results: inkubalm_full_algebra.json · gemma4_full_algebra.json.
If you arrive at different conclusions on the same data, I want to hear about it.
I ran this audit in late April 2026 and revised it in July 2026. Inference for the LLaMA, Mistral, and Phi audits ran on vLLM; InkubaLM ran on local transformers due to a known issue with its custom attention path. The scoring math is identical across both engines (mean log P(token | left context)) and per-axis numbers are reported from each model's own audit JSON without rescaling.