In a landmark case, a group of six individuals has been convicted for using generative AI to create and profit from obscene materials. The court’s ruling underscores a crucial distinction: while AI technology itself may be neutral, its misuse for illegal purposes carries severe legal consequences. The case highlights the evolving challenges in regulating advanced AI technologies.
AI “Jailbreak” Leads to Criminal Conviction
The individuals, led by a former game company programmer named Zheng, were found guilty by the Yuyao City People’s Court in Zhejiang Province of manufacturing obscene items for profit. Zheng, who had a notable presence in the domestic AI field and had pursued advanced studies abroad, was inspired to develop a similar software after encountering an AI “sexting” robot from overseas in July 2024. This marked the beginning of an illegal operation aimed at financial gain.
Zheng assembled a team of six, including former colleague Xie and online acquaintance Yan. Their illicit venture involved developing a generative AI model that was launched on an offshore platform in January 2025. The team employed sophisticated techniques to “jailbreak” the AI, bypassing its built-in safety filters designed to prevent the generation of inappropriate content. Users who paid for access could engage in explicit chats with AI robots and, by consuming virtual currency or points, generate obscene static and dynamic images.
Sophisticated Technical Manipulation
The operation utilized “prompt engineering,” a technique involving the meticulous crafting and modification of system prompts. This, along with setting up specific “character cards” and dialogue scenarios, effectively tricked the AI into bypassing its ethical and safety protocols. The AI model was thus transformed from a general-purpose tool into a specialized engine for producing illicit content.
Payments were processed through various channels, including “fourth-party payment” systems and cryptocurrencies, to obscure the financial transactions. The scheme continued until July 2025, when law enforcement, acting on criminal intelligence, initiated an investigation. After a two-month probe, all six defendants were apprehended.
Scale of Illicit Content and Financial Gain
During the AI model’s operation, a staggering volume of explicit content was generated. According to Sun Yuping, an assistant judge at the Yuyao City People’s Court, over 370,000 static images and more than 6,000 dynamic images were produced. Users paid a total of approximately $24,000 USD, equivalent to about 170,000 RMB.
Forensic analysis confirmed the nature of the generated content. Out of 12,500 randomly selected static images and all dynamic images, over 9,000 static and 5,000 dynamic images were officially classified as obscene materials.
Judicial Interpretation: “Technology Neutrality” vs. “Criminal Responsibility”
Judge Zhu Zhanglin of the Yuyao City People’s Court emphasized that generative AI models are equipped with content filtering mechanisms to prevent illegal and harmful information. These tools, in their original state, are not inherently criminal. However, the defendants’ actions involved actively altering the AI’s safety features and creating an environment conducive to producing obscene content. This demonstrated a substantial level of control over the generation of illegal material.
The court rejected the defense that the content was merely AI-generated and therefore absolved the developers of responsibility. Judge Zhu stated,
“Technology neutrality” has never been a shield for illegal activities. We cannot deny the developers’ “producer” status simply because the content was generated by AI. The developers actively tampered with the model’s safety mechanisms and built a technical environment that facilitated the production of obscene content, thereby exercising substantial control over the generation of illegal content. Although this content was generated at the user’s request, the developers are still considered “producers” of obscene materials in the context of criminal law and must bear corresponding criminal responsibility.
Legal Precedent and Future Implications
This verdict aligns with recent judicial guidance. A Supreme People’s Court opinion issued on September 7th mandates strict legal action against activities constituting crimes, including fraud, insult, defamation, infringement of personal information, and the production, sale, or dissemination of obscene materials, when these acts involve the use of deepfake or similar AI technologies.
The case serves as a potent reminder that leveraging advanced technologies like generative AI for criminal enterprises will be met with firm legal repercussions. The court’s stance reinforces that the creators and facilitators of AI-driven illicit activities are held accountable for their actions, regardless of the technological means employed.
