The new Black Mirror has just been released. The season overall was pretty mid. Not been the same since they Americanised it.
But I really loved that episode where the patient opened up to a super empathetic doctor, only to find out at the end… the doctor was an AI the whole time?
Wait, that wasn't an episode? Ohh… that was actually real life.
Another AI vs Doctor study just dropped in Nature. And this time, the LLM isn't just smarter than doctors, it's also apparently more empathetic. And there goes that "human connection" moat we thought we had.
Introducing AMIE (Articulate Medical Intelligence Explorer). This is a custom LLM which has been trained and optimised for diagnostic dialogue. This includes history-taking, differential diagnosis, management and escalation. The researchers are trying to give GPs a run for their money by pitting the AI against primary care providers.
Method: This was an OSCE-style RCT. They took 159 case scenarios from the UK, Canada and India, from a multitude of specialities. They compared the performance of the AI to that of 20 board-certified primary care physicians. The performance was then evaluated by patient-actors and specialist physicians.
The consultations were conducted over text message (which obviously isn't how things go down in real life).
So…the AI beat the physicians in a variety of clinical domains. Across accuracy, information acquisition, and differentials, we only matched it on escalation recommendations. But how on earth is it more human than us?
Really, the patient-actors rated it on politeness, attentiveness, rapport building, honesty, and comfort. We lost in all domains.
So is it time to hang up the boots and leave the game before the game leaves us? No. Why? Because the study has some issues.
Doctors don't talk in text: Unless you're trying to get a Viagra prescription from Superdrug, we don't communicate over text. This unfamiliar text-chat interface handicapped the physicians. Additionally, the AI had been trained to be good in this environment, unlike the physicians.
Read between the lines: Patients don't tell you everything. The intricacies of non-verbal communication were not, and cannot be, explored in this study.
It's a simulation: The simulated environment had an array of limitations. Assumes an underlying disease state (as OSCEs always have a diagnosis), thus neglecting patients who are really just fine. No space for the worried well.
Examinations: AMIE can't do examinations; the system reported all its investigations. Which is good for clinicians (for now). Until they fit GPT into a stethoscope…
So before you change your LinkedIn profile to "former doctor, future barista", remember that real-life medicine isn't the clean back and forth that an OSCE simulates. Until an AI can navigate a jam-packed Monday morning with a toddler screaming in one room and a patient who should have really gone straight to A&E at reception, we've still got the advantage 💪.
