Learned plasma control for fusion
DeepMind trained a reinforcement learning agent to find optimal tokamak management procedures for containing hydrogen plasma.
The scoreboards stopped moving — so 2022 built new ones
Chapter 2 of the AI Index 2023 measures technical progress during 2022 across vision, language, speech, reinforcement learning and hardware. The headline is not a leap but a plateau: state-of-the-art results kept arriving, yet on most benchmarks they arrived by a hair. What did leap was generative AI — and the cost of training it. The numbers:
4% median year-over-year gain across the benchmarks the AI Index tracks
7tracked benchmarks that improved by 5% or more in the year — all the rest gained less
204tasks in BIG-bench, launched in June 2022 by 442 authors across 132 institutions
91% ImageNet top-1 accuracy in 2022 — a gain of just 0.1 point on the year
75.2% MMLU score of Flan-PaLM, the year’s best multitask language result
502tonnes of CO₂ equivalent emitted by GPT-3’s training run (BLOOM’s: 25)
2.6 — The benchmarks ran out of room before the models did
The theme running through the whole chapter is saturation. AI kept setting records in 2022, but on most of the tests the AI Index tracks the record was barely better than last year’s — and the speed at which benchmarks reach that ceiling is still increasing.
Measured as relative change since each benchmark launched, the median improvement is 42.4%. Measured over the last year alone, the median is 4%. For all but seven of the benchmarks in this chapter, the year’s improvement came in under 5%. The AI Index also dropped two long-standing favourites, SQuAD1.1 and SQuAD2.0, from this edition entirely — no new state-of-the-art results had been posted on either.
What a plateau looks like, benchmark by benchmark
- ImageNet: the best image classifier reached 91.0% top-1 accuracy. That is 27.7 percentage points better than a decade ago — and 0.1 point better than last year.
- MPII human pose estimation: ViTPose correctly placed 94.3% of keypoints, a 0.2 percentage point increase on a record set back in 2020.
- Cityscapes semantic segmentation: 86.46% mIoU, up 23.4 points since the challenge launched in 2014, but flat for the last few years.
- COCO object detection: EVA, from a Chinese academic collaboration, reached 81.9% mAP50 — detectors have gained 26 points since 2015.
- Kvasir-SEG medical image segmentation: 94.11% mean Dice. Celeb-DF deepfake detection: an AUC of 78. Procgen reinforcement learning: a mean normalised score of 0.57.
- Face recognition is effectively finished. On every NIST FRVT dataset except WILD Photos the top model now errs less than 1% of the time, and on VISA Photos the error rate is 0.06%.
The response: bigger, broader test suites
Researchers answered saturation by building evaluations that are harder to finish. In June 2022, 442 authors across 132 institutions launched BIG-bench (Beyond the Imitation Game), a suite of 204 tasks spanning linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias and software development. In November, Stanford researchers released HELM (Holistic Evaluation of Language Models), an attempt to judge language models against unified standards rather than one score at a time. Google’s Imagen team shipped DrawBench alongside the model itself, precisely because the existing text-to-image benchmark was no longer discriminating.
2.1 — The year generative AI arrived in public
A selection from the chapter’s own timeline of the most significant AI developments of 2022, as chosen by the AI Index Steering Committee.
- Feb 2, 2022
AlphaCode · DeepMind
An AI system that writes computer programs at a competitive level, ranking within the top 54% of participants in a human programming competition — progress on exactly the kind of complex problem-solving AI had traditionally struggled with.
- Apr 5, 2022
PaLM · Google
One of the world’s largest language models at 540 billion parameters. PaLM reinforced the prevailing belief of the moment: that performance improves by simply training on more data.
- Apr 13, 2022
DALL·E 2 · OpenAI
A text-to-image system that creates realistic art and images from written descriptions. Its public release is the moment the AI Index marks as igniting the generative AI craze.
- May 12, 2022
Gato · DeepMind
A reinforcement learning agent capable of robotic manipulation, game playing, image captioning and natural language generation in one system — evidence that AI was getting better at generalisation.
- Jun 9, 2022
BIG-bench · 442 authors
To better challenge increasingly capable large language models, 442 authors across 132 institutions launched the Beyond the Imitation Game benchmark: 204 tasks from linguistics and childhood development to physics and software development.
- Jun 21, 2022
Copilot goes on sale · GitHub
GitHub made Copilot available as a subscription service for individual developers. It turns natural language prompts into coding suggestions across multiple languages; surveys suggest it makes coders more productive and less frustrated. Similar systems include OpenAI’s Codex and Salesforce’s CodeGen.
- Aug 22, 2022
Stable Diffusion · Stability AI
An open-source text-to-image diffusion model whose weights anyone can use freely. It is trained on existing human-made images and gives no credit or acknowledgment, leaving open questions about the ethical use of image generators.
- Sep 21, 2022
Whisper · OpenAI
A large-scale speech recognition system trained on roughly 700,000 hours of audio. It needed neither supervised pre-training nor unsupervised training with fine-tuning, yet performed strongly — further validation of simply scaling up training data.
- Nov 16, 2022
HELM · Stanford
Holistic Evaluation of Language Models, a new benchmarking approach that judges language models against more unified standards — evidence of the field’s attempt to build transparency around increasingly powerful systems.
- Nov 30, 2022
ChatGPT · OpenAI
A publicly usable chatbot capable of writing university-level essays. Months after launch it reached 100 million monthly active users, making it the fastest-growing consumer application in history — and capping a year in which generative AI became part of the zeitgeist.
2.2–2.5 — Text in, pictures out: the year generation went mainstream
While the classification benchmarks flatlined, generation did not. Text-to-image, text-to-video and speech all shipped systems in 2022 that ordinary users could actually touch.
Text to image
DALL·E 2, Stable Diffusion, Midjourney, Meta’s Make-A-Scene and Google’s Imagen all arrived within months of each other. The AI Index put the same prompt — “a panda playing a piano on a warm evening in Paris” — to DALL·E 2, Stable Diffusion and Midjourney to compare them side by side. On the MS-COCO 256×256 FID-30K benchmark, where a lower Fréchet Inception Distance is better, Imagen leads at 7.27, ahead of Make-A-Scene at 7.55 and DALL·E 2 at 10.39; for scale, AttnGAN scored 35.49 back in 2017. Google shipped the harder DrawBench benchmark alongside Imagen, because text-to-image models had outgrown the old test.
Text to video
- In May, Tsinghua University and the Beijing Academy of Artificial Intelligence released CogVideo, which posted the then-highest inception score on the UCF-101 text-to-video benchmark at 50.46.
- In September, Meta’s Make-A-Video overtook it decisively with an inception score of 82.55 — 63.6% better than CogVideo on UCF-101.
- In October, Google released Phenaki, though it was not benchmarked on UCF-101.
- The caveat the chapter attaches: impressive as these models are, they can so far only generate videos a few seconds long.
One model, many skills
AI has traditionally been strong at narrow tasks and weak at crossing between them. In 2022 that started to break down. Microsoft’s BEiT-3 posted state-of-the-art results across four vision skills and five vision-language skills at once — on NLVR visual reasoning it reached 92.60 against a previous best of 87.00, a 6.44% improvement, the largest of the nine. Google’s PaLI took the top spot on VQA v2 at 84.30%, above the 80.78% human baseline. Both are single systems doing what used to require several.
Speech
The scaling recipe reached speech in 2022. Whisper, trained in a weakly supervised way on 700,000 hours of audio, beat wav2vec 2.0 Large across a wide range of English speech recognition benchmarks and outperformed leading translation models on the X→EN subset of CoVoST 2, scoring 29.1 BLEU against MAESTRO’s 25.2. On Kincaid46 its median word error rate of 8.81% beat every commercial ASR service tested. It was not best at everything: on FLEURS language identification, zero-shot Whisper managed 64.5% against mSLAM-CTC’s 77.7%. Meanwhile on the original VoxCeleb speaker recognition dataset, American researchers posted an equal error rate of 0.1%, a 0.28 percentage point improvement on the previous year’s state of the art.
2.9 — The world’s best new scientist … AI?
2022 was the year AI stopped only being the subject of research and started doing some. Six cases from the chapter — including two where AI was used to improve AI. Tap any card for the detail.
AlphaTensor rewrites matrix multiplication
A reinforcement learning system that discovered faster algorithms for multiplying matrices — a problem researchers had been chipping at for 50 years.
AI-designed circuits inside Nvidia’s GPUs
Nvidia’s PrefixRL agent designs chip circuits smaller and faster than the EDA tools that used to do the job.
Antibodies designed from scratch
Generative models produced novel antibodies zero-shot — one round of generation, no further optimisation.
BCOOLER cools Google’s data centres
A reinforcement learning agent cut cooling energy by about 12.7% over a three-month experiment.
PaLM improves PaLM
Google researchers used one of their language models to improve the reasoning of that very same model.
2.7–2.8 — The hardware got cheaper and faster; the bill arrived anyway
Two sections that belong together. Training keeps getting faster per dollar, which is exactly why models keep getting bigger — and why this edition is the first to put a carbon figure on them.
Hardware
- In MLPerf training, the top systems in image classification and object detection now train roughly 32 times faster than in 2018, when the competition launched. Record lows were set in 2022 across object detection, speech recognition, image segmentation, recommendation, image classification and language processing — the fastest, language processing, takes 0.18 minutes.
- Better hardware is doing much of that work. The maximum number of accelerators used by an MLPerf entrant reached 4,216; the systems that post top results average 1,859; the mean across all entrants is 211. That gap has widened every year of the competition.
- On MLPerf Inference, the number of offline samples generated by the top image classifiers and language processors has more than doubled since 2020, while recommendation throughput is up roughly 23%.
- GPU performance keeps climbing: median FP32 performance has nearly tripled since 2021 and is up roughly 7,000 times since 2003.
- Price–performance is the number that matters for scaling. Median FLOP/s per US dollar in 2022 is 1.4 times greater than in 2021 and 5,600 times greater than in 2003 — a doubling every 1.5 years.
Carbon
Drawing on Luccioni et al., 2022, the chapter compares four large language models on parameters, data centre power usage effectiveness, grid carbon intensity, power consumption and emissions. GPT-3 (175B parameters, 1,287 MWh, PUE 1.10, grid intensity 429 gCO₂eq/kWh) released the most carbon at 502 tonnes: 1.4 times more than Gopher, 7.2 times more than OPT and 20.1 times more than BLOOM. The comparison is imperfect — accounting methodologies for reporting carbon emissions are not standardised — but the relativities are stark. BLOOM, the cleanest of the four at 25 tonnes, still emitted 25 times as much as flying one passenger round trip from New York to San Francisco, 1.4 times what the average American emits in a year, and consumed enough energy to power the average American home for 41 years.
What 2022’s systems still could not do
The chapter is unusually blunt about the gap between impressive demos and reliable capability — and about how much the scores themselves are worth.
Can large language models plan and reason?
Not really — and 2022 produced the test that showed it. Valmeekam et al. built a planning benchmark of seven assignments (plan generation, cost-optimal planning, reasoning about plan execution, robustness to goal reformulation, plan reuse, replanning and plan generalisation) and ran GPT-3, Instruct-GPT3 and BLOOM on it in a Blocksworld domain, where an agent must arrange coloured blocks into a given order. The results are brutal: on plan generation the three scored 0.6%, 5.0% and 0.5%; on optimal planning 0.2%, 3.2% and 0%; on plan reuse 0%, 14.4% and 0%. They did far better at reformulating goals — up to 77.4% on one variant of robustness to goal reformulation — but compared to humans the models performed much worse overall, which the authors read as evidence that capability is not the same thing as human-like reasoning.
How much better did language models actually get?
The AI Index ran its own small experiment: it gave the same prompt — “Explain to me the major accomplishments of Theodore Roosevelt’s presidency” — to GPT-2 (2019), GPT-3 (2020) and ChatGPT (2022). GPT-2’s answer is mostly gibberish; it produces grammatical sentences that never address the question. GPT-3 responds to the question but gets the facts wrong, crediting Theodore Roosevelt with leading the US out of the Great Depression and into World War II (that was Franklin Delano Roosevelt) and with creating the National Park Service and the National Wildlife Federation (created in 1916 and 1936, after his presidency). ChatGPT’s answer is the strongest: all but one of its facts are correct, and it is more thorough and more concise than either predecessor.
Are these systems safe to rely on?
The chapter’s own highlights say no, in plain terms. 2022 saw text-to-image models like DALL·E 2 and Stable Diffusion, text-to-video systems like Make-A-Video and chatbots like ChatGPT — “Still, these systems can be prone to hallucination, confidently outputting incoherent or untrue responses, making it hard to rely on them for critical applications.” The confidence is the problem: the failure mode is not a visible error but a plausible-sounding wrong answer.
Is any vision benchmark still genuinely open?
A few. Visual Commonsense Reasoning is one of the very few visual benchmarks in this report on which AI has yet to pass humans: the 2022 top Q→AR score is 75.60 against a human baseline of 85.00. VCR is harder than VQA because a system must both pick the right answer and pick the right reasoning behind it. In video, the gap between Kinetics-600 (91.80%) and Kinetics-700 (84.00%) is 7.8 percentage points, which the chapter reads as a sign that the 700-category dataset is still a meaningful challenge. And in face recognition, WILD Photos is the one FRVT dataset where the error rate is still above 1%, at 2.97%.
Can we compare these improvement numbers to each other?
No, and the chapter says so in a footnote. The improvements in the saturation analysis are reported as relative change, and each benchmark has different parameters — so the chart should not be used to compare improvements across benchmarks. The same caution applies more widely in this chapter: MLPerf ranks on absolute wall clock time, FID scores are better when lower, VoxCeleb is scored on equal error rate, Procgen on a min-max normalised score. Reading any one of them as “how much better AI got” is a category error.
The chapter in its own words
Five lines from the Chapter Highlights and the section text of Chapter 2 · Technical Performance.
AI continued to post state-of-the-art results, but year-over-year improvement on many benchmarks continues to be marginal. Moreover, the speed at which benchmark saturation is being reached is increasing.
2022 saw the release of text-to-image models like DALL-E 2 and Stable Diffusion, text-to-video systems like Make-A-Video, and chatbots like ChatGPT. Still, these systems can be prone to hallucination, confidently outputting incoherent or untrue responses, making it hard to rely on them for critical applications.
Compared to humans, the large language models performed much worse, suggesting that while they are capable, they lack human reasoning capabilities.
AI models are starting to rapidly accelerate scientific progress and in 2022 were used to aid hydrogen fusion, improve the efficiency of matrix manipulation, and generate new antibodies.
Nvidia used an AI reinforcement learning agent to improve the design of the chips that power AI systems. Similarly, Google recently used one of its language models, PaLM, to suggest ways to improve the very same model. Self-improving AI learning will accelerate AI progress.
Read Chapter 2 in full
Chapter 2 (sections 2.1–2.9) — the 2022 timeline, computer vision for images and video, language, speech, reinforcement learning, hardware, environment and AI for science — with every figure and citation is free from Stanford HAI.
Open the AI Index 2023 →