Medical EvalsDossier01
2022 benchmark; English release tracked separately
DDXPlus ↗
The most likely disease and the useful differential are different targets.
UnitOne synthetic patient interactionMeasureDifferential recall, precision, F1 and pathology accuracy
Diagnostic evaluation studies
Follow the evidence into the diagnosis.
Independent analysis of DDXPlus and MedR-Bench: synthetic differentials, case-report reasoning, examination requests, oracle conditions, metric denominators and published results.
The path to a conclusion is part of the task
Initialize a synthetic patient
Start with the supplied initial evidence.
Collect structured evidence
Ask about symptoms or antecedents.
Stop and predict
Return an underlying pathology or differential distribution.
Choose information availability
Use one-turn, free-turn or oracle access to examination evidence.
Generate clinical outputs
Request examinations or produce a diagnosis/treatment plan.
Evaluate outcomes
Match outputs to the versioned references with the source judge.
Simulator-derived differentials and case-report judgments have different reference standards. Follow each dossier's information setting and metric.
01 /
Diagnostic target
Top-pathology accuracy and differential coverage answer different questions.
02 /
Information availability
Requested examinations and oracle evidence produce different tasks.
03 /
Reasoning evidence
Written-rationale scores use separate denominators and an automated judge.
Our analytical question
DDXPlus separates a simulated pathology from its reference differential. MedR-Bench separates examination gathering, diagnosis, treatment and written reasoning. We analyze those task boundaries and the original scoring rules, with selected version-matched historical results. Our coverage explorer and guides are original analytical resources. The benchmarks and experiments belong to their credited authors; this site has not run the displayed models or created a new patient dataset.
Read the original evidence closely
The benchmark, unpacked.
Dossier02
Nature Communications 2025, seven-model study
MedR-Bench ↗
Supplying the examinations changes what a diagnosis score means.
UnitOne case-report-derived patient caseMeasureOutcome accuracy and written-reasoning dimensions
An original analytical tool
Map the diagnostic target and information path
Compare actual simulator and case-report tasks. Filter by diagnostic target, information gathering, clinical stage or written-reasoning evaluation.
9 of 9 evidence entries shown
- Input
- Collected simulator evidence
- Output
- Most likely pathology
- Measure
- GTPA@1
- Input
- Collected simulator evidence
- Output
- Set of plausible pathologies
- Measure
- DDR / DDP / DDF1
- Input
- Initial symptom and interactive evidence
- Output
- Evidence requests
- Measure
- Positive evidence recall; interaction length
- Input
- Case without ancillary test findings
- Output
- Recommended examination list
- Measure
- Precision and recall
- Input
- Case with iterative examination access
- Output
- Sequence of requests
- Measure
- Precision and recall
- Input
- Case plus once-requested examinations
- Output
- Diagnosis
- Measure
- Semantic accuracy
- Input
- All reference examinations
- Output
- Diagnosis
- Measure
- Semantic accuracy
- Input
- Complete case including diagnosis
- Output
- Treatment plan
- Measure
- Semantic plan accuracy
Interpretation boundary
Case-report treatment is the reference; not a prospective intervention comparison.
[4]- Input
- Generated rationale and reference reasoning
- Output
- Judged steps
- Measure
- Efficiency, factuality, completeness
This is our independent analytical map of published tasks. It does not execute an evaluation, predict a model’s performance or establish clinical benefit. [1][2][3][4][5][6]
Original analysis / methods and interpretation
What the score leaves unsaid.
Benchmark analysis
DDXPlus: a correct top diagnosis can hide a weak differential
Interpret differential recall, precision and pathology accuracy as different diagnostic targets.
Benchmark analysis
MedR-Bench: read the gap between requested and supplied examinations
Understand what changes between one-turn, free-turn and oracle diagnosis conditions.
Benchmark analysis
MedR-Bench reasoning scores: three denominators, three questions
Analyze efficiency, factuality and completeness without treating written explanations as hidden model reasoning.
Questions, answered
Read the result in context.
Specific tasks. Stated conditions.
Inspect every source.
Does DDXPlus contain real patient records?
No. It contains synthetic patients generated from a knowledge base and rule-based diagnostic system. Its diagnostic scores measure agreement with that constructed reference.
What is the MedR-Bench oracle condition?
The model receives all ground-truth examination evidence before diagnosis. This differs from identifying which examinations to request.
Do reasoning scores reveal a model’s internal thinking?
They evaluate the written reasoning that is supplied to the evaluator. They do not directly establish which internal computation caused the answer.
Which MedR-Bench version do the result panels use?
The November 2025 Nature Communications study and its supplementary tables, rather than mixing those results with the earlier five-model preprint.
Working tool / saved on this device
Prepare a benchmark comparison brief
Use this secondary checklist to document a run or literature comparison after inspecting the named benchmark conditions. Completion records documentation, not performance.
Evidence you can inspect. Benchmark dossiers distinguish published facts from our interpretation, with source versions and access notes attached.
Download the evidence ↗