The year AI became an instrument scientists actually pick up

Chapter 5 is new in the 2024 edition — the AI Index added it in recognition of AI’s growing role in scientific and medical discovery. Its subject is 2023: sorting routines discovered by reinforcement learning and merged into the C++ standard library, 2.2 million new crystal structures, 71 million genetic variants scored in a single pass, and a language model that cleared 90% on a medical board-exam benchmark. Meanwhile the FDA’s approval list kept lengthening — while still containing no generative AI at all.

139AI-related medical devices approved by the FDA in 2022 (2021: 124)

90.2% — GPT-4 Medprompt on MedQA, up 22.6 points on 2022’s best score

2.2million new crystal structures uncovered by GNoME

71million missense variants scored by AlphaMissense (89% classified)

50% of observed SARS-CoV-2 mutations predicted by EVEscape (prior model: 24%)

10day global weather forecasts from GraphCast, produced in under a minute

5.1 — Six results that changed what a lab can do in a week

In 2022 AI began to advance scientific discovery. In 2023 it started shipping: algorithms into production libraries, materials into databases, forecasts into 80-plus countries. These are the six milestones picked by the AI Index Steering Committee.

AlphaDev — faster sorting, in the standard library

A reinforcement learning system wrote sorting routines with fewer instructions than the human benchmarks — and they were merged into the LLVM standard C++ sort library.

algorithms

FlexiCubes — 3D meshes that match the ground truth

Nvidia’s gradient-based approach to isosurface extraction cut mesh reconstruction error to 34.87% on the IN>5° metric, against 48.66% for the best DMTet variant.

graphics

Synbot — a robotic chemist that closes the loop

An AI planning layer, a robot control layer and a physical lab, wired into a feedback loop: Synbot hit 100% conversion yield on a target molecule where the human reference sat at 85%.

chemistryrobotics

GraphCast — 10-day forecasts in under a minute

A graph neural network that beats the industry state of the art, HRES, on root mean squared error for ten-day forecast skill.

weather

GNoME — 2.2 million new crystal structures

Google’s graph network beat the Materials Project, a leading materials-discovery method, on stable crystal count — and surfaced structures human researchers had overlooked.

materials

Flood forecasting — five days of warning, in 80+ countries

Google researchers built hydrological models that work on ungauged basins, matching or beating GloFAS on both precision and recall.

climate

5.2 — Medicine got five new instruments, and none of them is a chatbot

The systems the AI Index Steering Committee picked out of 2023 all do the same kind of job: they take a measurement that was previously too noisy, too slow or too expensive to make, and make it routine.

Rescuing data that already exists

SynthSR converts ordinary clinical brain scans into high-resolution T1-weighted images, the format with the contrast and structural clarity that advanced research needs. That matters because scan quality varies enormously, and variability has kept a large share of existing clinical scans out of research entirely. Correlations between SynthSR output and ground-truth volumes ran from 0.89 to 0.99 at subject level (n=41) across white matter, cortical and subcortical gray matter, ventricles, hippocampus and amygdala; at scan level (n=435) they were lower and more uneven, from 0.60 to 0.99.

Measuring what mass spectrometry cannot

Diagnosing Parkinson’s and Alzheimer’s depends on identifying biomarkers fast and precisely. Mass spectrometry and ELISA can quantify protein levels but cannot discern changes in structural state — which is where the disease signal often sits. The 2023 answer was ImmunoSEIRA: AI-coupled plasmonic infrared sensors using Surface-Enhanced Infrared Absorption spectroscopy together with an immunoassay technique. Across test samples spanning fibril concentrations from 0% to 100%, the deep neural network’s predicted percentages tracked the actual ones closely.

Predicting a virus before it mutates

Traditional methods for predicting viral escape rely on real-time strain and antibody data — exactly what is missing in the early stages of a pandemic, when the prediction is worth the most. EVEscape is trained instead on historical sequences plus biophysical and structural information, so it evaluates escape independently of current strain data. Working from prepandemic information only, it predicted 50.0% of observed SARS-CoV-2 mutations, against 46.2% and 32.3% for experimental scans that had pandemic antibody and sera data available, and 24% for a previous model.

Two jobs humans had barely started

AlphaMissense, from Google DeepMind, predicted the pathogenicity of 71 million missense variants — the genetic alterations that impair protein function and can lead to diseases including cancer. It classified 89% of them: 57% likely benign, 32% likely pathogenic, 11% uncertain. For comparison, human annotators have confirmed the nature of just 0.1% of all missense mutations.
The Human Pangenome Reference Consortium — 119 scientists from 60 institutions — used AI to build a more representative map of the human genome. The first draft was released in 2000 and updated in 2022, but the update omitted variations such as blood type and mapped diverse ancestry groups poorly, which makes disease detection unreliable in some populations. The new version annotated a median of 99.07% of protein-coding genes, 99.42% of protein-coding transcripts, 98.16% of noncoding genes and 98.96% of noncoding transcripts.

Share of observed SARS-CoV-2 mutations predicted, by method

EVEscape used only prepandemic sequence, biophysical and structural data, and still beat experimental scans that had pandemic antibody and sera data to work from. The previous model managed 24%.

Clinical knowledge — 90% on the board exams, without fine-tuning

MedQA was supposed to be hard: over 60,000 questions taken from professional medical board exams and designed to challenge doctors. In 2023 a model crossed 90% on it — and the method that got it there was prompting, not training.

Introduced in 2020, MedQA is the standard test of AI clinical knowledge. Performance on it has improved remarkably: the leading system of 2023, GPT-4 Medprompt, reached 90.2% accuracy, an increase of 22.6 percentage points over the top score in 2022. Since the benchmark’s inception, AI capability on MedQA has nearly tripled.

The assumption that broke

It was commonly assumed that a general-purpose LLM needs significant fine-tuning on domain-specific data before it can handle specialized medical questions. Research from Microsoft in late 2023 overturned that. GPT-4 Medprompt used prompt engineering alone to steer GPT-4 across MultiMedQA, a suite of four demanding medical benchmarks, and exceeded the top 2022 model Flan-PaLM 540B on the multiple-choice sections by 3.0 points on PubMedQA, 21.5 on MedMCQA and 16.2 on MMLU. It also beat the then state-of-the-art Med-PaLM 2, and it was the first system to pass 90% on MedQA. Prompt engineering, the chapter concludes, is a promising alternative to domain fine-tuning.

The open-source gap is 20 points

GPT-4 Medprompt is closed: its weights are not available to the public. The strongest open medical model of 2023, MediTron-70B, scored 70.2% on MedQA — the highest yet achieved by an open-source model, a significant improvement over the previous open state of the art, and ahead of Llama 2 at 63.8%, but 20 points behind Medprompt and 16 behind Med-PaLM 2. The chapter’s point is not that the gap is embarrassing; it is that medical AI only reaches its potential if its capabilities are widely accessible, and right now the best clinical knowledge sits behind closed weights.

MedQA accuracy for selected models, %

Five models, two release years, one visible split: the three closed models sit at the top, the open ones at the bottom. MediTron-70B and GPT-4 Medprompt were both released in November 2023.

Diagnosis and paperwork — where the gains are actually measured

Away from the benchmarks, 2023’s clinical research asked a narrower question: not whether a model knows medicine, but whether adding it to an existing workflow changes an outcome a hospital can count.

CoDoC — knowing when to defer

AI imaging systems miss diagnoses clinicians catch, and clinicians miss diagnoses AI catches. CoDoC (Complementarity-Driven Deferral to Clinical Workflow) is built around that asymmetry: it decides when to rely on the AI and when to defer to the traditional clinical path. Across four medical datasets its sensitivity beat clinicians by an average of 4.5 percentage points and a standalone AI model by 6.5; its specificity beat clinicians by 2.7 points and the standalone model by 5.7. It also reduced clinical workflow by 66%.

PANDA — screening for a cancer that hides

Pancreatic ductal adenocarcinoma is usually found too late for surgery, and screening asymptomatic people is hard because prevalence is low and false positives are costly. A Chinese research team built PANDA to detect and classify pancreatic lesions. In multicenter validation on 6,239 patients it beat the mean radiologist by 34.1% in sensitivity and 6.3% in specificity. In a real-world, multi-scenario test covering 20,530 patients it reached 92.9% sensitivity and 99.9% specificity.

Two more results from the same year

A meta-review of mammography-image-based AI algorithms found that predicting future breast cancer risk from mammography images alone achieves accuracy comparable to or better than traditional risk assessment tools.
X-Raydar and X-Raydar-NLP, two open-source neural networks for classifying chest X-rays from images and free-text reports, performed at levels comparable to human experts and stayed robust when applied to external datasets.

MedAlign — the benchmark that measures the boring part

Knowledge benchmarks like MedQA and the USMLE do not capture what clinicians actually spend their hours on: information-intensive administrative work over electronic health records. MedAlign, introduced in 2023, is the first extensive EHR-focused benchmark — 983 questions and instructions with 303 clinician responses, drawn from seven medical specialties. Tested on it, the best system was a GPT-4 variant using multistep refinement, and its correctness rate was 65.0%. That is the honest number for administrative competence in 2023, and it sits far below the same family’s 90.2% on MedQA.

AI-related medical devices approved by the FDA, by year

139 devices in 2022, a 12.1% increase on 2021 and more than a 45-fold increase since 2012. The AI Index stops at 2022 because the FDA last updated its list in October 2023, leaving that year incomplete.

Six questions this chapter answers

The findings that are easiest to over-read, with the numbers attached.

Does a 90.2% on MedQA mean AI can practise medicine?

No, and the chapter supplies its own counterweight. MedQA is over 60,000 multiple-choice questions from professional medical board exams — real medical knowledge, but knowledge in the format of an exam. MedAlign was built precisely because benchmarks like MedQA and the USMLE do not capture the diverse, information-intensive tasks clinicians actually perform over electronic health records. On MedAlign the best system, a GPT-4 variant with multistep refinement, was correct 65.0% of the time.

Is any FDA-approved medical device running a large language model?

No. As of October 2023, when the FDA last updated its list, it had not approved any device that uses generative AI or is powered by an LLM. What the 139 approvals of 2022 mostly are is imaging: 87.1% of them were radiology devices, and the next largest specialty, cardiovascular, accounted for 7.2%. The vertical growth curve and the generative AI boom are two separate stories that had not yet met.

Was 2023 really different from 2022 for AI in science?

The chapter’s framing is that 2022 was when AI began to advance scientific discovery, and 2023 was when significantly more consequential applications launched. The tell is deployment rather than publication: AlphaDev’s routines were merged into the LLVM standard C++ sort library — the first update to that part of the library in over 10 years — and the flood forecasting model is open source and already predicting floods in over 80 countries. GNoME’s 2.2 million crystal structures are a database other researchers can search.

How far behind are open-source medical models?

About 20 percentage points on MedQA at the end of 2023. MediTron-70B scored 70.2%, the highest yet by an open model, against 90.2% for GPT-4 Medprompt and 86.2% for Med-PaLM 2, both closed. Llama 2 was at 63.8%. Notably, Medprompt and MediTron-70B were released in the same month, November 2023, so this is a like-for-like snapshot rather than a lag.

Do these diagnostic systems replace radiologists?

The two flagship results point the other way. CoDoC exists specifically to decide when not to use the AI, and its gains come from combining the two: 4.5 points of sensitivity over clinicians and 6.5 over a standalone AI model, with clinical workflow cut by 66%. PANDA does beat the mean radiologist outright — 34.1% higher sensitivity in validation on 6,239 patients — but its role is screening a cancer that is currently not screened for at all, which is added coverage rather than substitution.

What is AI doing for basic biology, as opposed to the clinic?

Closing coverage gaps that manual work could never close. AlphaMissense scored 71 million missense variants and classified 89% of them, against the 0.1% of all missense mutations that human annotators have managed to confirm. The Human Pangenome Reference Consortium — 119 scientists across 60 institutions — rebuilt the genome map so that groups underrepresented in the 2000 draft and its 2022 update are covered, annotating a median 99.07% of protein-coding genes and 98.16% of noncoding genes.