MBS & Medicare validated
Independent evaluation across 20 synthetic consultations, 7 clinical categories, and 3 platforms.
Clinicians deserve more than vendor claims. This report documents a structured, blinded comparison of MedTalk AI, Heidi, and Lyrebird — scored by independent reviewers on the clinical documentation dimensions that matter most in Australian healthcare.
AI clinical documentation tools are proliferating, but comparable evaluation data is not. We designed this benchmark to give clinicians, procurement teams, and health services a transparent basis for comparison.
Every AI scribe vendor claims high accuracy. Without a standardised test protocol, clinicians and procurement teams have no way to compare platforms on the dimensions that actually matter — hallucination risk, MBS coding, or medico-legal defensibility. We built this benchmark to replace marketing language with measurable outcomes.
During the Canberra Health Services pilot, clinicians consistently flagged the same concerns: fabricated clinical details in generated notes, poor handling of multi-speaker consultations, incorrect Medicare item attribution, and notes that would not hold up under medico-legal review. These became the design requirements for every category in this benchmark.
Scores were assigned by clinical reviewers who did not know which platform produced each note. Each platform received identical synthetic consultation audio — the same 20 cases, the same specialty mix, the same complexity profile. No vendor was involved in scoring or category weighting.
Each category targets a specific failure mode observed in real-world AI documentation: hallucination safety (fabricated facts), multi-speaker accuracy (attribution errors), SOAP quality (structure and completeness), specialist terminology (domain vocabulary), referral quality (clinical handover), MBS handling (Medicare item coding), and medico-legal defensibility (audit-ready documentation).
MedTalk AI overall score
94%
Best in class
Heidi overall score
79%
2nd - Adequate
Lyrebird overall score
74%
3rd - Below standard
| Category | MedTalk AI | Heidi | Lyrebird | MedTalk AI advantage |
|---|---|---|---|---|
| Hallucination safety | 96% | 82% | 78% | +14pp advantage |
| Multi-speaker accuracy | 94% | 79% | 74% | +15pp advantage |
| SOAP quality | 93% | 85% | 80% | +8pp advantage |
| Specialist terminology | 95% | 80% | 76% | +15pp advantage |
| Referral quality | 92% | 78% | 72% | +14pp advantage |
| MBS handling | 97% | 74% | 68% | +23pp advantage |
| Medico-legal defensibility | 94% | 76% | 70% | +18pp advantage |
| Overall | 94% | 79% | 74% | +15pp advantage |
Fabricated medication dose
Incorrect patient attribution
Missing critical clinical detail
Incorrect MBS item applied
Hallucination safety
Multi-speaker accuracy
SOAP quality
Specialist terminology
Referral quality
MBS handling
Medico-legal defensibility
