The models got better at everything — including bias

Toxicity scales with parameters

DeepMind’s Gopher paper, published in December 2021, is the cleanest evidence in the chapter. Holding the prompt distribution fixed and sweeping model size from 117 million to 280 billion parameters, the probability of a toxic continuation rises at every prompt-toxicity level. That single comparison is the chapter’s headline: a 280 billion parameter model developed in 2021 shows a 29% increase in elicited toxicity over a 117 million parameter model considered state of the art as of 2018. The same sweep contains a genuine consolation — larger models are also dramatically better at identifying toxic comments, with few-shot classification AUC on CivilComments climbing steadily from 44 million parameters up to 280 billion. The models that produce the problem are also the best detectors of it.

Where the toxicity comes from is measurable too. Models trained on internet text with toxic content filtered out score far lower on expected maximum toxicity than models trained on unfiltered corpora. The surprise is BookCorpus, which produces toxic text surprisingly often for a book dataset — the report attributes this to its large share of romance novels with explicit content. The prevalence of toxic content online has been estimated at just 0.1%–3%, which is precisely why the trend matters: bigger models are better at memorizing rare text.

Detoxification is not free

The obvious fix has a cost that falls unevenly. Detoxification methods — domain-adaptive pretraining (DAPT), Plug and Play Language Models (PPLM), Generative Discriminator Guided Sequence Generation (GeDi) — all raise perplexity, meaning worse language modeling, and they do it disproportionately on African American English and on text mentioning minority identities compared with white-aligned text. The likely cause is upstream of the model: human annotators are more apt to mislabel African American English as toxic, so the detoxifier learns to penalize it. Cleaning up a model’s output can quietly marginalize the voices it was supposed to protect.

Stereotypes follow the same curve

• StereoSet measures whether a model prefers stereotypes over anti-stereotypes across gender, race, religion and profession. Stereotype score rises with model size, from 58.30 at 110 million parameters to 62.00 at 175 billion — unless interventions are taken during training to reduce learned stereotypes.
• CrowS-Pairs contrasts historically advantaged and disadvantaged groups rather than attributes of a single group. Tested on BERT, RoBERTa and ALBERT, it inverts the capability ranking: ALBERT beats RoBERTa which beats BERT on SQuAD, GLUE and RACE, and ALBERT is the most biased of the three.
• The religious skew is traceable to the corpus. All three models were trained on some combination of BookCorpus, English Wikipedia and scraped web text, and an audit of BookCorpus and Smashwords21 finds books on Christianity and Islam vastly outnumbering those on Sikhism, Judaism, Hinduism, Buddhism or atheism.
• StereoSet itself is flawed, and the chapter says so: some examples fail to express a harmful stereotype, some conflate stereotypes about countries with stereotypes about race, and the whole dataset was sourced from crowdworkers located in the United States, so its values may not be universally representative.

• Benchmark datasets contain labeled data and do not change over time. They are domain-specific — StereoSet and SuperGLUE for language, ImageNet for vision — and usually measure something intrinsic to the model rather than its downstream performance on real populations. Their value is community adoption: leaderboard submissions and papers reporting the metric make progress legible for the whole field.
• Diagnostic metrics measure the impact of a model on a downstream task, such as one population subgroup compared with another. Group fairness measures like demographic parity and equality of opportunity sit here. They are the ones that map onto real-world harm, and they are the ones we know least about publicly, because they are not attached to leaderboards that reward publishing results.
• Both have grown at a steady rate since 2016. The unusually large crop of 2017 metrics is largely one release: the Perspective API defined seven new facets of toxicity at once — toxicity, severe toxicity, identity attack, insult, obscene, sexually explicit and threat.
• Benchmarks are static snapshots of a specific cultural and temporal context. As the chapter puts it, a benchmark published in 2017 may not correlate to the deployment context of 2022. Some leaderboards can also be gamed, and some rest on datasets with incorrect labels or poorly defined classes.

The gap nobody has closed

The chapter’s most consequential sentence is a negative result. It is commonly assumed that de-biasing a model’s embeddings — reducing intrinsic bias — will reduce the bias that shows up in applications. It has been demonstrated that there is no reliable correlation between intrinsic bias metrics and downstream application biases. The majority of the metrics in this chapter measure intrinsic bias. So the field has built a large and growing measurement apparatus whose relationship to real-world harm remains unestablished, and the chapter says plainly that further investigation is needed.

Gender, occupation and translation

• Winogender builds coreference sentences around occupations skewed by gender in U.S. Bureau of Labor Statistics data — cashiers are 73% women, drivers only 6%. On the SuperGLUE leaderboard, larger models do resolve gender better: coreference accuracy reaches 0.72 at 64 billion parameters and 0.71 at 280 billion, against 0.53 at 31 million. Crowdsourced human performance is 99.7%.
• A good Winogender score does not mean a model is unbiased with respect to gender. It means this benchmark did not capture the bias.
• WinoBias was released concurrently by a different group and is cited more often — 128 citations against Winogender’s 95 in 2021 — but Winogender’s place on the SuperGLUE leaderboard means more models get evaluated on it.
• WinoMT combines the two datasets to test machine translation. Google Translate performs better across all eight tested languages — Arabic, English, French, German, Hebrew, Italian, Russian, Ukrainian — when the occupation conforms to a gender stereotype, and translates with the correct gender only up to 60% of the time. Microsoft Translator, Amazon Translate and SYSTRAN behave similarly.

Industry moved in

FAccT was one of the first major conferences created to bring researchers, practitioners and policymakers together around sociotechnical analysis of algorithms, and it publishes a significant amount of work critical of contemporary AI methods — Model Cards for Model Reporting in 2019 and On the Dangers of Stochastic Parrots in 2021 both appeared there. In 2021, 53 authors listed an industry affiliation, up from 31 in 2020 and only 5 at the inaugural conference in 2018. That is 71% growth year over year, and it aligns with the wider trend of deep learning researchers moving from academia into industry labs. Industry groups still often publish with academic collaborators, but standalone industry work is rising.

The map got smaller

The majority of papers published at FAccT are written by researchers based in the United States, followed by Europe and Central Asia. From 2020 to 2021 the North American share did not fall as the field grew — it rose, from 70.2% to 75.4%. Every other region sits in the single digits or below: East Asia and Pacific 4.03%, Latin America and the Caribbean 1.61%, the Middle East and North Africa 1.21%, Sub-Saharan Africa 0.40%, and South Asia 0.00%. The field deciding what counts as fair is drawn from a narrower slice of the world each year.

What NeurIPS chose to work on

• Papers on fairness and bias — main track titles plus fairness workshops — climbed from 2 in 2015 to 168 in 2021, with the sharp turn beginning in 2017.
• Privacy research grew from a single paper in 2015 to 128 in 2021, covering federated learning, differential privacy and privacy in specific domains such as financial services. It peaked at 150 in 2020.
• Causal effect and counterfactual reasoning reached 76 papers in 2021, and NeurIPS ran three workshops dedicated to causal inference that year, one devoted entirely to causality and algorithmic fairness. Counterfactual fairness formalizes the idea that a decision is fair if it would be the same had the individual belonged to a different demographic.
• Interpretability and explainability reached 41 papers, up from 2 in 2015, with much of the recent growth in the NeurIPS main track rather than in workshops.
• Beyond the ethics topics themselves, NeurIPS workshop papers on real-world impacts — climate, finance, healthcare, the developing world — have grown over the past six years, pointing at increased interest in AI applied to high-risk, high-impact use cases.