Microsoft's diagnosis AI scored 83.2% on hard NEJM cases; generalist doctors scored 36.1% on a subset. A simulation, not the clinic

Microsoft's AI diagnostic system, MAI-DxO, got the right diagnosis on 83.2% of hard New England Journal of Medicine cases. Generalist doctors with search and reference tools got 36.1% on 56 of those cases, which had been held back from development. The paper doesn't say whether the 83.2% came from those same 56 cases or from all 304. The figures come from a peer-reviewed paper in Science, "Sequential diagnosis with language models," published in print on 22 October 2026. It was a simulation on written case records, not live patients, and the paper says so: "These comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice." All 15 authors work for Microsoft and hold equity in it. Microsoft paid for the study, and eight of the authors are named inventors on a Microsoft patent application for the evaluation method.

What was measured: the paper's benchmark, SDBench (Sequential Diagnosis Benchmark), turns 304 hard teaching cases published by NEJM, known as clinicopathological conference cases, into step-by-step encounters. A doctor or the AI starts from a short case summary and has to ask for history, examination findings and tests. A separate AI "gatekeeper" answers from the case record. An AI "judge" scores the final diagnosis against the published one, using a rubric written by physicians. A diagnosis counts as correct if management would be unchanged. Every test ordered adds an estimated cost, priced from the Kaiser Permanente Washington hospital chargemaster (its master price list), and questions to the patient count as a fixed doctor-visit fee. So the benchmark scores how you got to the answer, not just the answer. A second set adds 325 emergency department admissions from the MIMIC-IV-ED research database.

MAI-DxO isn't a new model. It's a software layer that directs an existing commercial language model, GPT-5 for the headline figures. Microsoft describes it as operating "like a panel of virtual physicians: each proposing diagnoses, selecting useful questions or tests, and challenging the emerging view." Averaged across the models tested, the paper says, that layer added 7.9 percentage points of accuracy on the NEJM cases and 4.4 on the emergency cases. TechCrunch reports GPT-5 alone scored 78.9% on the NEJM cases, so for GPT-5 the panel added about four points.

The doctors were practicing generalists, in primary care and hospitals, each scored individually on part of the 56 held-back cases (an average of 36.4 cases each closed book and 34.8 open book). With no outside help, the "closed book" round, they got 22.3% right. With search and reference tools, the "open book" round, they got 36.1%. Looking things up helped but cost them: average time per case rose from 11.9 to 18.1 minutes, and diagnostic spend from $3,049 to $4,725 a case. The best single doctor reached 51.8% and was still beaten on both accuracy and cost. The comparison didn't include specialists matched to each case, or doctors working as a team.

A 2025 preprint of the same work, run on OpenAI's o3, reported 80% against 20% for physicians who were told not to use search engines or language models. The peer-reviewed figures replace those.

For AIVI the value here is accuracy and cost per case, not fewer doctors. Without a budget cap, MAI-DxO averaged $5,431 in diagnostic costs per NEJM case for its 83.2%, more than the $4,725 the doctors with references spent. With a budget cap it got 80.9% right at an average $2,241 a case, under half the doctors' figure, and 79.6% at $1,303. The doctors' costs come from the 56 held-out cases. As with accuracy, the paper doesn't say which cases the AI's costs cover, so both comparisons may be across different sets of cases. On the emergency cases it reached 80.3% at a mean $4,979 a case, against 46.8% at $3,400 for physicians with reference tools, and 73.2% at $2,634 with a budget cap. The authors frame the system as support, saying its structured reasoning "could augment human clinical judgment." The study didn't test doctors working with the AI.

The paper lists its own limits: NEJM cases "may overrepresent diagnostically challenging presentations," and neither data set includes healthy or benign visits. Doctors and AI alike worked through a text-based "oracle" rather than electronic health records or real patients, with no imaging, waveforms or pathology slides. The judge scores the diagnosis, not "therapeutic appropriateness, safety, or longitudinal management quality." The authors expect "the performance gap to narrow when comparing DxO with expert specialists or multidisciplinary specialist teams." Their summary: SDBench "measures diagnostic reasoning under idealized information conditions rather than bedside performance or patient outcomes."

TechCrunch reported the study on 22 October 2026 and noted that "Cases this complex are usually handled by specialists or physician teams, who weren't part of the comparison." TechCrunch doesn't test the figures. Microsoft's own post rounds them to 83%, 36% and 22% and calls MAI-DxO "about 3.7 and 2.3 times more accurate" than the doctors without and with references. The paper itself doesn't give those ratios, and they set the AI's score against the doctors' without saying the two cover the same cases. Microsoft also frames the work as a step toward what it calls medical superintelligence. That's Microsoft's goal, not a finding of the paper.

What would make this a graded case study: results from clinical use. Microsoft says it will run a prospective study with Beth Israel Deaconess Medical Center in Boston, where "patients will interact with the system directly, with performance measured against physicians' independent assessments." It gives no start date. A graded case would need accuracy, time or cost per case measured with clinicians using the system, against a stated baseline. Until then, no graded Index case backs this pattern.