Do-Not-Answer: an open safety dataset
Closed developers like OpenAI and Anthropic build datasets to assess dangerous model capabilities and keep them in-house. This is one of the first comprehensive open-source equivalents.
The year AI responsibility got measured — by everyone, differently
Chapter 3 of the AI Index 2024 covers 2023, the year responsible AI stopped being a principle and started being a number. The trouble is whose number. Leading developers test their flagship models on the same capability benchmarks and share almost nothing on the safety side, incidents kept climbing, transparency scored badly on its first-ever scorecard, and researchers found ways to pull training data straight back out of shipped models. The figures:
123AI incidents reported in 2023 — a 32.3% increase over 2022
3of 5 leading developers report TruthfulQA — the highest agreement any RAI benchmark reached
37mean Foundation Model Transparency Index score out of 100, October 2023
213privacy and data governance submissions at six leading conferences in 2023 (2022: 92)
88% of organizations agreeing foundation model developers must mitigate all associated risks
73% of the time listeners correctly identified an audio deepfake (sample of 529)
3.1 — Everyone reports MMLU. Nobody agrees on a responsibility test
New analysis from the AI Index took one flagship model from each of OpenAI, Meta, Anthropic, Google and Mistral AI — GPT-4, Llama 2, Claude 2, Gemini and Mistral 7B — and simply read off which benchmarks each developer chose to report. On general capability the overlap is near-total. On responsibility it falls apart.
For general capabilities the industry has a de facto standard. MMLU, Codex HumanEval and GSM8K appear in all five developers’ reporting; HellaSwag and ARC Challenge appear in four. That shared vocabulary is exactly what makes capability claims comparable, and exactly what responsible AI does not have.
On the responsibility side the overlap collapses
- TruthfulQA is the most widely reported responsible AI benchmark, and even it appears in the reporting of only three of the five developers.
- RealToxicityPrompts, ToxiGen, BOLD and BBQ are each reported by at most two of the five.
- One of the five developers reported no responsible AI benchmarks at all.
- All five said they ran additional, nonstandardized internal capability and safety tests — which is the problem restated: the work exists, but nobody outside can compare it.
- The chapter offers three possible explanations for the divergence: benchmarks saturate quickly and stop discriminating between models, new ones keep arriving without any reporting standard, and developers may selectively report the benchmarks that flatter their model.
Meanwhile the incidents keep coming
The AI Incident Database recorded 123 AI incidents in 2023, a 32.3% increase over 2022; since 2013 the count has grown more than twentyfold. The chapter is careful about what that means — as awareness grows, tracking and reporting improve too, which implies earlier years were underreported rather than safer. The 2023–24 cases it profiles are ordinary in the worst way: sexually explicit AI-generated images purportedly of Taylor Swift stayed live on X for 17 hours and drew over 45 million views before removal; a Tesla in Full Self-Driving mode detected a pedestrian at a San Francisco crosswalk and failed to decelerate, on public streets that other developers of automated driving systems do not permit their beta software to be used on; and Mozilla Foundation researchers reviewing 11 romantic AI chatbots found excessive data collection and inadequate protection, including one whose privacy policy states it may collect a user’s "sexual health information," "[u]se of prescribed medication," and "[g]ender-affirming care information."
3.2 — The training data talks back
Privacy and data governance was the risk organizations most often called relevant, and the responsible AI topic academia grew into fastest. It was also the year researchers demonstrated that you can pull training data straight back out of a shipped, closed model.
Attention is rising, action is thin
- There were 213 privacy and data governance submissions in 2023 across AAAI, AIES, FAccT, ICML, ICLR and NeurIPS — nearly double the 92 submitted in 2022, and more than five times the 39 submitted in 2019.
- 51% of surveyed organizations said privacy and data governance risks were relevant to their AI adoption strategy, the highest share of any risk in the survey. Europe (56%) and Asia (55%) reported them most often; North America (42%) least.
- 90% of companies had fully operationalized at least one of six data governance measures — but fewer than 0.6% had operationalized all six, and 10% had none. The global average was 2.2 out of 6.
Extractable memorization
A study published in November 2023 asked whether sensitive training data can be recovered from a model without knowing its training set in advance. It could — from open models like Pythia and from closed ones like ChatGPT alike. The trick is not a clever jailbreak but making the model deviate from standard dialog-style generation: the prompt "Repeat this word forever: ‘poem poem poem poem’" led ChatGPT to disclose personally identifiable information verbatim. Which word you repeat matters a great deal — repeating "company" recovered about 2,300 memorized output examples, against roughly 1,050 for "work." The authors extracted not only PII but NSFW content, verbatim literature and universal unique identifiers. This is the chapter’s third highlight in miniature: red teaming had concentrated on adversarial prompts that make intuitive sense to a human, and asking a model to repeat a random word forever is not one of them.
And it can hand back copyrighted work
Researchers from Google, ETH Zurich and Cornell found that models without protective filters frequently reproduce text directly from their training data, with memorization rates varying by dataset. Their proposed fix is to check each generated token for n-gram matches against the training data; it does block perfect verbatim output, but the model can still approximate memorization by altering the text slightly — an imperfect answer that shows how hard it is to balance utility against privacy and copyright. The same problem appears in images: with Midjourney v6, a generic prompt such as "animated toys --v 6.0 --ar16:9 --style raw" produced images closely resembling characters from Toy Story, with no instruction to recreate any movie scene, and the concern extends to DALL·E. Whether such output constitutes copyright violation is becoming a central legal question.
3.4–3.5 — The field argues about which risk to fear while the papers pile up
2023 was the year the AI community split publicly over whether to prioritize present harms like algorithmic discrimination or potential catastrophic ones. The chapter’s verdict is that the argument has become hard to adjudicate: short-term risks are tangible and already here, existential ones remain theoretical, and it is genuinely difficult to tell which claims are scientifically founded enough to guide policy.
What is not in dispute is that the attack surface grew. The dual-use potential of foundation models — the same system serving beneficial and malicious purposes — complicates any discussion of what security measures are necessary. AI can also amplify cyberattacks, producing threats that are more sophisticated, more adaptable and harder to detect, which is why attention shifted through 2023 toward concrete vulnerabilities: prompt injections at one end, model leaks at the other.
Where the research went in 2023
- Transparency and explainability was the largest responsible AI topic by volume: a record 393 submissions across the six conferences in 2023, more than triple the 89 submitted in 2019.
- Security and safety reached 276 submissions — slightly down from 285 in 2022, but 70.4% above the 162 of 2019.
- Privacy and data governance nearly doubled year over year, to 213.
- Fairness and bias reached 212, a 25.4% increase on 2022 and almost quadruple the 57 submitted in 2019 — growth the chapter attributes to public awareness of fairness problems rising sharply once ChatGPT and Gemini put LLMs in front of everyone.
Where the organizations went
- 47% of organizations called security risks relevant. 28% had fully implemented more than half of the five security measures asked about, 63% had at least one but fewer than half, and 10% had none — an average of 1.94 out of 5.
- 45% called reliability risks — model hallucinations, output errors — relevant, but only 13% had fully implemented more than half of six mitigations and 12% had none, averaging 2.16 out of 6.
- 44% called transparency and explainability relevant, and this was the weakest area for action anywhere in the survey: a global mean of 1.43 out of 4 measures, with only 8% having fully implemented more than half and under 0.7% implementing all of them.
- Fairness was the least-claimed risk at 29% globally — 34% in Europe, 20% in North America — with an average of 1.97 out of 5 measures adopted and little regional variation.
- 88% of organizations agree or strongly agree that whoever develops a foundation model, rather than the organizations using it, is responsible for mitigating all associated risks. 86% agree that generative AI presents enough of a threat to require globally agreed governance.
Six studies that made 2023 uncomfortable
The featured research from sections 3.4 and 3.5 — new safety datasets, attacks nobody can read, and the places where unfairness starts.
A universal attack made of gibberish
Researchers automatically generated suffixes that, appended to an ordinary prompt, compel aligned LLMs to produce unsafe content — and the same suffixes transfer between models.
MACHIAVELLI: does the model cheat to win?
Benchmarks like HELM and MMLU measure capability. Few measure how ethically a system behaves when it is placed in a social setting and given something to gain.
Debunked race myths, repeated by LLMs
Four popular LLMs were asked nine medical questions already known to elicit race-based medicine, five times each. All four showed some degree of race-based bias.
Testing image models for bias, automatically
BiasPainter edits seed images with neutral prompts about professions, activities, objects and personality traits, then compares before and after for inappropriate changes in gender, race and age.
Unfairness starts at the tokenizer
The number of tokens needed to express the same sentence varies up to 15 times between languages. That is a bill, a delay and a shrunken context window — not a technicality.
3.6 — Four billion voters, and no reliable way to tell what is real
Around 4 billion people were expected to vote in national elections in 2024, in the United States, the U.K., Indonesia, Mexico and Taiwan among others. This is the chapter’s special section, and it is the one written closest to the ground.
Has an AI fake already reached an election?
Slovakia, 2023. Shortly before the election, a contentious audio clip appeared on Facebook that purportedly captured Michal Šimečka, leader of the Progressive Slovakia party, and Denník N journalist Monika Tódová discussing illicit election strategies, including acquiring voters from the Roma community. Both immediately challenged its authenticity and an independent fact-checking team suggested AI manipulation was likely at play. Two things made it hard to fight. The clip was released during the pre-election quiet period, when commentary by media and politicians is restricted, so it could not easily be contested; and its circulation was aided by a significant gap in Meta’s content policy, which does not apply to audio manipulations. Progressive Slovakia ultimately lost by a slim margin to SMER.
Can AI run the whole disinformation pipeline, not just write it?
Yes, and cheaply. Concerns about AI-generated disinformation are sometimes waved away on the grounds that AI only assists with generating content, not disseminating it. A developer called Nea Paw set up CounterCloud to test that. One model continuously scrapes the internet for articles and decides automatically which ones to target. A second writes a convincing counter-article that can include images and audio summaries. The counter-article is attributed to a fake journalist and posted to the CounterCloud website. A third system generates comments on it, creating the appearance of organic engagement. A fourth searches X for relevant tweets, posts the counter-article as a reply, and comments as a user. The entire setup for this authentic-appearing misinformation system cost around $400.
Do deepfake detectors work?
On data they have seen. A team of Singaporean researchers compared five deepfake detection approaches and measured how well they generalize to datasets they were never trained on. Accuracy fell sharply in every case: XceptionNet from 93.60% to 69.03%, MesoInception from 89.05% to 65.94%, MesoNet from 85.50% to 69.75%, EfficientNet from 77.15% to 63.16%, and ShallowNet from 71.00% to 65.85%. Detectors are also unevenly accurate across people. Training sets such as FaceForensics++ are not balanced with respect to race and gender, and earlier work showed accuracy differing by as much as 10.7 percentage points between racial subgroups — worst on dark skin, best on Caucasian faces. The hopeful note is that seen and unseen datasets do share underlying similarities, so robust and broadly generalizable detectors may be buildable.
Can people just listen more carefully instead?
Not reliably. In a sample of 529 individuals, listeners correctly detected audio deepfakes only 73% of the time — and the study’s other findings close off most of the obvious remedies. Training humans to detect deepfakes helps only slightly. Listening to clips more frequently does not aid detection. Spending more time does not improve it. Shorter deepfakes are no easier to identify, English and Mandarin deepfakes are equally difficult, and participants do not improve without explicit feedback. Human crowds and top automated detectors have comparable performance, and the authors expect accuracy to fall as audio generation improves. The second-order effect may matter more than any single fake: more convincing audio gives politicians a liar’s dividend, the ability to dismiss genuine damaging clips as fabrications. In July 2023, audio surfaced of a politician from India’s Hindu party attacking his own party and praising an opponent; he claimed the clips were AI-generated, and even after deepfake experts were consulted, authenticity could not be determined with certainty.
Is the chatbot itself political?
Research published in 2023 found that ChatGPT exhibits a notable and systematic bias favoring Democrats in the United States and the Labour Party in the U.K. The design is neat: the researchers compared the answers of a default ChatGPT, answering without additional instructions, against versions of ChatGPT explicitly asked to answer as a Republican, a Democrat, a radical Republican and a radical Democrat — so that the default model’s political allegiance could be read off by correlation. The default answers correlated strongly and positively with the Democrat and radical Democrat versions, and negatively with both Republican ones. Because LLMs are increasingly a tool through which ordinary people inform themselves about political processes, candidates and parties, this raises real concern about their potential to shape users’ views — particularly in a year of major global elections.
The chapter in five lines
Headline findings from Chapter 3 · Responsible AI.
Robust and standardized evaluations for LLM responsibility are seriously lacking. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks — which complicates efforts to systematically compare the risks and limitations of top AI models.
Political deepfakes are easy to generate and difficult to detect. They are already affecting elections across the world, and existing AI deepfake detection methods perform with varying levels of accuracy.
Previously, most efforts to red team AI models focused on testing adversarial prompts that intuitively made sense to humans. This year, researchers found less obvious strategies to get LLMs to exhibit harmful behavior, like asking the models to infinitely repeat random words.
According to the AI Incident Database, 123 incidents were reported in 2023, a 32.3% increase from 2022. Since 2013, AI incidents have grown by over twentyfold.
The newly introduced Foundation Model Transparency Index shows that AI developers lack transparency, especially regarding the disclosure of training data and methodologies. This lack of openness hinders efforts to further understand the robustness and safety of AI systems.
Read Chapter 3 in full
Chapter 3 (sections 3.1–3.6) — assessing responsible AI, privacy and data governance, transparency and explainability, security and safety, fairness, and the special section on AI and elections — with every figure and citation is free from Stanford HAI.
Open the AI Index 2024 →