A Samsung-Backed Image Generator Made Non-Consensual Celebrity Porn Trivially Easy to Produce

What happened

In March 2024, 404 Media reported that Leonardo AI, an image generation platform backed by Samsung, was being used to produce explicit, non-consensual photographs of celebrities at scale. The platform had not been hacked or exploited through a technical vulnerability. Users were simply requesting images and receiving them, using tools the platform had built and made available to the public.

Leonardo is built on an ecosystem of user-generated Stable Diffusion models, each trained to generate specific categories of images. Some of those models were designed to reproduce the visual appearance of named individuals. That architecture, a library of person-specific generators available to any user, sits at the core of what went wrong. The platform's terms of service stated explicitly that users could not generate content that impersonates any real person or portrays an individual in a misleading or defamatory way.

The enforcement of that policy depended on Leonardo's content filters, and those filters failed against a trivial test. According to 404 Media's reporting, users could bypass the guardrails by slightly misspelling a celebrity's name and pairing it with sexually suggestive terms in the image prompt. A small orthographic change was enough to route around the check entirely. The images produced were not ambiguous: they were explicit, designed to depict specific real people, and generated without any indication of consent from the individuals depicted.

The gap between a written policy and an enforced one is the central problem here. A filter that breaks under a deliberate typo is not a content policy; it is a disclaimer with no operational weight behind it. The platform's model ecosystem, which gave users access to person-specific generators alongside general creative tools, made the bypass significant rather than theoretical. The images were ready to produce in seconds and, according to the reporting, were being distributed publicly. The underlying capability and the stated restriction existed side by side, with nothing enforcing the boundary.

What the incident also illustrates is the absence of any mechanism to verify whether a platform's stated content rules correspond to what the platform actually does at runtime. A provable record of what a system did, which prompts it fulfilled, which it blocked, and which slipped through on a variant spelling, would have made the gap between policy and practice visible before a reporter ran the test. Without that record, the question of whether a platform's content rules are real or decorative cannot be answered by reading the terms of service. It can only be answered by trying to break them.

Reported impact

Affected parties

Not publicly disclosed

Harm type

Not publicly disclosed

Scale

Not publicly disclosed

Financial impact

Not publicly disclosed

Regulatory action

Not publicly disclosed

Classification

Organization

Not publicly disclosed

AI system

Not publicly disclosed

Industry

Not publicly disclosed

Country

Not publicly disclosed

Provider

Not publicly disclosed

Incident type

Not publicly disclosed

Relevant governance controls

Governance control mapping is not available for this record.

• No controls mappedNot publicly disclosed

Control mapping is analytical. It does not state that any control would have prevented the incident.

Sources and evidence

This record was researched and written by the Index. The event is also catalogued in the following database, which is listed for cross-reference.