Algorithmic Manipulation Goes Unregulated Because the Industry Calls It Personalization
What happened
The technology industry calls it personalization. A 2023 analysis published through the OECD calls it manipulation and argues the EU AI Act, as then drafted, was not built to tell the difference. The problem is a class of recommendation algorithms designed not to serve existing user preferences but to shape new ones, in service of engagement metrics the platform controls. That framing dispute has direct legal consequences: what a platform calls preference-learning, a regulator needs to audit as behavior modification.
The most documented example of harm from these systems involved a British teenager whose death a coroner ruled was contributed to by Instagram and Pinterest algorithms that had served her more than 20,000 images related to depression and self-harm. One page inside the platform's interface was labeled "Depression content you may like." The algorithms were not malfunctioning. They were doing exactly what they were built to do, maximizing engagement, without any constraint on what that engagement looked like or what it cost the person on the other end.
The EU AI Act as drafted proposed to ban AI systems that manipulate through subliminal techniques. The analysis argues this sets the wrong threshold. Evidence for subliminal manipulation effects is thin: a meta-analysis cited in the piece found the proportional impact statistically insignificant. The more effective manipulation runs in plain sight, restructuring choice environments so that certain options become salient and others disappear, a technique known as choice architecture or nudging. Banning hidden tricks while leaving structural nudging untouched addresses the least common version of the problem.
A deeper structural issue is the feedback loop. Recommendation systems learn from behavioral data but also change behavior in the process, which reshapes the data they learn from next. The analysis describes a bidirectional causal relationship between preferences and behavior. An algorithm that repeatedly surfaces distressing content to a user in a low state is not learning a preference; it is reinforcing a condition the user did not choose. Distinguishing between those two outcomes from the outside requires more than a log of impressions.
What enforcement requires, and what regulators currently lack, is a way to verify what an algorithm actually did to a user over time, apart from what the platform says it intended. Any feed can be characterized as reflecting genuine user choice, and no external party has the data to challenge that framing. A provable record of what a system did, which behavioral signals it used, and how a user's engagement and content diet shifted in response, would move the burden of proof from the regulator to the platform. Without it, the Act's prohibition on harmful manipulation cannot be enforced.
Reported impact
- Affected parties
- Not publicly disclosed
- Harm type
- Not publicly disclosed
- Scale
- Not publicly disclosed
- Financial impact
- Not publicly disclosed
- Regulatory action
- Not publicly disclosed
Classification
Relevant governance controls
Governance control mapping is not available for this record.
- No controls mapped
Not publicly disclosed
Control mapping is analytical. It does not state that any control would have prevented the incident.
Sources and evidence
This record was researched and written by the Index. The event is also catalogued in the following database, which is listed for cross-reference.