It is hardly surprising that I have been thinking about AI a lot recently. It’s everywhere, isn’t it? It seems most days there’s a news story about AI in some shape or form - be it related to a hacking incident, a new development in the technology, or even the data centre infrastructure the systems run on. So I’ve been reflecting on the value it brings in a variety of ways, particularly when it comes to healthcare. Overall, I feel sceptical. Not of AI in medicine as a whole, but of one particular claim - that a microphone plugged into a computer can write my notes better than I can.
I wonder if feelings about the utility of AI are negatively impacted by familiarity with the area in which it is deployed. Much like when reading a newspaper article on a topic you are knowledgeable about, you are very conscious of the knowledge gaps of the author.
Programmers can be critical of the role of AI when it comes to coding, and yet as a non-programmer, I have found it helpful in tweaking my home server setup as well as some small changes to this very website. Without AI, it would have taken me far longer to get things sorted with a large amount of web research needed. Writers, too, are critical of AI’s writing style1 and yet, a large number of people use it to write their emails or to get them started for first drafts. Just this month, there has been controversy about one company’s AI model solving a mathematics problem - with criticism from professors in mathematics - yet students of the field use it to help them understand principles. Here the expert anxiety is about verification; mathematicians know how proofs can look right yet be wrong, whereas a student is happy with an explanation that clicks for them.
This could, of course, be a matter of the stakes when it comes to errors. A programmer knows the cost of a slightly wrong line of code in a production environment; the novice with a home server just has to deal with their own documents being inaccessible for an hour or so while they figure the mess out2. AI's error rate is acceptable there. But it is not only the cost of an error that shapes how we judge it - it is whether we can see it at all. The expert programmer catches flaws the novice never would, which is why familiarity breeds scepticism even when nothing important is on the line.
Medicine, though, is where accuracy matters. The health of the person sat in front of you is critical, and the stakes could hardly be higher. This is not simply tweaking with a personal blog; this is ensuring the care that is being delivered is safe and effective.
The Different Types of AI
Before anything else, I think it would be handy to talk about the two main types of AI. When most people talk about AI currently, they are generally referring to LLMs3 such as ChatGPT, Claude, Gemini, and many others besides. However, AI can also refer to rule-based systems. These are forms of AI that rely on logic and “if-then” statements to follow rules. You may be surprised to learn that Pac-Man used a form of rule-based AI for the ghosts to follow when they were chasing Pac-Man4! This form of AI requires quite exhaustive programming to provide a robust set of rules, whereas LLMs are given a large set of data to analyse and they learn from patterns. This requires far fewer specific rules to be given at the outset.5
Rule-based AI certainly has its place - at your bank when it comes to credit scoring and checks for money laundering, spam filters on your email provider. LLMs certainly have their place too, but the exact method of implementation of them is important.
It is mainly LLM use I plan to focus on - this is where the most marketing seems to be, specifically for ambient AI scribes.
AI Scribes
In healthcare, this is where I have seen a lot of the discussion get focussed. It’s easy to see why - as clinicians we spend a large amount of time documenting clinical encounters. Something that takes away or even reduces that burden is only a good thing, right? This is where I’m not so sure - for the reasons I’ll discuss below.
Verbosity
This is my first bugbear with the current AI scribes that are out there. They record a lot of information in a consultation note. If you’ve ever had the pleasure of scrolling through someone’s medical records, it’s usually easy to tell which entry was written using Accurx Scribe or Heidi - it’s about three times as long as the other entries in their notes and does waffle on a bit.6
This makes it quite difficult to tease out the relevant information from a clinical note - what was the person seeing the patient thinking? What did they (not an AI tool) feel was important and pertinent to the diagnosis made and plan given? The natural summarisation that goes on in a clinician’s head, which is then translated to their typed note, helps future readers see into the consultation in a way a wall of text never could.
Style
Linking to the above point, an individual clinician’s style is lost when using an AI scribe. I think that most GPs who have worked in one practice for a year or more would be able to identify which of their colleagues wrote an entry in the notes without needing to see the name at the top. I certainly would, and I suspect my colleagues would say the same. Everyone has their own way of writing. This is in part the internal summarisation that goes on before fingers even hit a keyboard, and in part our own little quirks - certain words we use, certain abbreviations we use and even certain spelling mistakes7.
I like that individual style. There are subtleties to it as well - if you’ve read enough of a doctor’s entries you could probably tell when they were worried about someone by how the entry was written. The loss of that would be, I think, a real shame.
Involvement
This is slightly different to the above points, and maybe contrary to how other people see the dawn of AI scribes. They are billed as being able to reduce workload, and some of that is an aim of reducing cognitive load - being able to dedicate your brain to listening to someone rather than thinking about how you might document the encounter after the patient leaves. However, I see it differently. If I see someone, I am engaged in the consultation whilst internal summarisation is ongoing, and I am thinking about how I might approach the next question with a view to a diagnosis and plan. My internal summary keeps me right - I can quickly think through what I have asked and make sure I’ve not missed anything.
In contrast, using an ambient AI scribe removes this process. Whilst some people may be more engaged in a consultation, I find it leads to less structure and an over reliance on the generated note for a summary rather than having to think about this. For me, this is less safe - who is to say the note summarises things accurately…
Hallucination
…Which leads us nicely onto the next step. LLMs are notorious for hallucinations, which is to say inventing details that are not real to fill in gaps. This is a critical issue with their use in healthcare - someone may later rely on the detail in your consultation to make a diagnosis or prescribe a medication. It must be as accurate as possible to enable this. Hallucinated information has a real risk of causing errors down the line.
Time
The time savings from AI scribes are oft-touted on their websites. Whilst I have no reason not to believe their claims that, for many people, they may well save time, I personally want to ensure my notes meet my style. This will naturally require some degree of editing, and to ensure the details included are accurate, and the note is stylistically mine takes time. I am therefore not convinced that, for me, the time savings are as groundbreaking as the marketing information makes them seem.
Summary
My scepticism, then, is not that AI cannot help in medicine; it can, and in places already does8. It is that the consultation note is not merely a transcription task, but a thinking task. AI scribes automate the transcription, yes, but they erode the thinking.
AI can genuinely be useful when the stakes are low and errors are easily visible. Healthcare is neither. The consultation isn’t just a record of an encounter - it is a clinician’s thinking made visible. It is summarised, styled and structured in a way an ambient scribe cannot replicate. Until AI-generated notes can match that (and be consistently accurate), the time they save will come at the cost of the very thing that makes medical notes worth reading - and worth trusting.
Footnotes
-
On this one I am firmly in the writers’ corner. My son has been given a book which is clearly AI slop. It is terrible to read. He doesn’t mind though and keeps wanting me to read it - the soullessness of the text (not to mention the inconsistent illustrations) pains me every time he pulls it from his bookcase. ↩
-
Possibly, of course, with the help of the very AI model that got them there in the first place. ↩
-
Large Language Models. ↩
-
Although the LLMs do still need instructions on how to respond to certain prompts. ↩
-
Yes I know, current company notwithstanding etc etc. ↩
-
Not everyone can spell amitriptyline easily. ↩
-
For example, lung cancer screening with AI triage of chest CTs. ↩