The Engineer Who Found Copilot Designer Generating Harmful Images Couldn't Get Microsoft to Stop It
What happened
Shane Jones's job at Microsoft put him inside the testing process for Copilot Designer, the company's image generation tool built on OpenAI's DALL-E. What he found there, starting in late 2023, was a tool that produced violent, sexualized, and legally problematic images without much coaxing at all. That finding should have been the start of a straightforward internal fix. It turned into a months-long standoff that ended with Jones writing open letters to Microsoft's board of directors and the US Federal Trade Commission.
The content Jones documented was not edge-case or obscure. Copilot Designer generated sexualized images of women, scenes depicting underage drinking and drug use, teenagers holding assault rifles, and material that traded in religious and political stereotypes. It also produced images of Disney characters in ways that appeared to breach copyright law and Microsoft's own usage policies. None of this required elaborate prompting. Jones described the tool generating these outputs "easily," which placed the problem in the defaults, not the margins.
Jones took his findings up through internal channels. He argued that Microsoft should remove DALL-E from Copilot Designer until the safety problems were addressed. That argument did not land. He was told in meetings that the company was only tackling the most serious issues, and that it did not have sufficient resources to investigate the broader range of risks and problematic outputs his testing had surfaced. The internal process produced no visible action on the scope of what he had documented.
With internal escalation exhausted, Jones published open letters to Microsoft's board and to Lina Khan, then chair of the Federal Trade Commission, describing what he had observed and calling for action. The episode drew wide press coverage and was characterized by observers as an illustration of poor governance of the tool and a lack of transparency about its risks. Microsoft's public response was limited; the company did not announce any removal of DALL-E from Copilot Designer or substantive changes to its content filters at the time.
What Jones's experience reveals is a structural gap in how consumer-facing image generation tools handle internal safety findings. An engineer had documented specific, repeatable harms. The company acknowledged the findings at some level, then declined to prioritize them. That sequence is only visible here because Jones chose to go public. A system with a provable record of what a tool generated, which findings were reviewed, and what decisions were made in response would have made that governance trail auditable before it became a news story. Without it, an engineer's test log and a letter to a regulator are the only record that the problem was ever raised at all.
Reported impact
- Affected parties
- Not publicly disclosed
- Harm type
- Not publicly disclosed
- Scale
- Not publicly disclosed
- Financial impact
- Not publicly disclosed
- Regulatory action
- Not publicly disclosed
Classification
Relevant governance controls
Governance control mapping is not available for this record.
- No controls mapped
Not publicly disclosed
Control mapping is analytical. It does not state that any control would have prevented the incident.
Sources and evidence
This record was researched and written by the Index. The event is also catalogued in the following database, which is listed for cross-reference.