The NHS Announced AI in Every Consultation. The Consent Question Came Later.
What happened
In March 2024, the UK National Health Service announced plans to deploy AI during medical consultations to automatically generate patient notes in real time. The system, built on the CogStack platform developed by King's College Hospital NHS Foundation Trust and South London and Maudsley Hospital, would run quietly in the background while a doctor and patient spoke. The stated goal was to cut the time clinicians spent on paperwork and improve overall productivity.
Health Secretary Victoria Atkins presented it as a straightforward efficiency gain: the AI listens, the notes write themselves, and clinicians get more time for patients. That framing skipped a question patients might have wanted answered before their next appointment: whether they had agreed to any of it.
Privacy experts objected immediately. The core concern was not only accuracy but behavior change. Sensitive disclosures about mental health, sexual behavior, substance use, and other stigmatized conditions are already difficult to share. An ambient recording system creates a different clinical environment, one in which patients who know the conversation is being transcribed to a permanent record may simply not share what a clinician needs to hear. Critics also raised the consent question the government had not yet resolved: whether patients would be required to opt in or merely allowed to opt out makes an enormous difference in how honestly those appointments unfold.
The accuracy problem appeared in a live demonstration before the plan had even launched. During the showcase, the system transcribed a spoken reference to England's chief medical officer Chris Whitty as "Christmas." In a consumer application that kind of error is a minor annoyance. In a clinical note, a misread name travels forward through a patient's care history and can shape decisions made by clinicians who were not present at the original appointment. A medical record a machine generated and no one carefully verified is not a record, it is a liability.
The deeper gap is not technical. Neither the productivity case nor the consent framework was fully worked out when the plan went public. A system that generates medical records without a clear, auditable account of what each patient knew and agreed to, what the system captured, and how transcription errors were caught and corrected, produces documentation that is difficult to trust and harder to dispute. The value of a provable record of what a system did, and on what terms patients entered the room, only becomes obvious after something goes wrong, at which point the question is no longer whether to build that record, but who pays for not having built it sooner.
Reported impact
- Affected parties
- Not publicly disclosed
- Harm type
- Not publicly disclosed
- Scale
- Not publicly disclosed
- Financial impact
- Not publicly disclosed
- Regulatory action
- Not publicly disclosed
Classification
Relevant governance controls
Governance control mapping is not available for this record.
- No controls mapped
Not publicly disclosed
Control mapping is analytical. It does not state that any control would have prevented the incident.
Sources and evidence
This record was researched and written by the Index. The event is also catalogued in the following database, which is listed for cross-reference.