Generative AI can produce a convincing image in the time it takes to type a sentence. That has unlocked real creative and commercial possibility. It has also removed nearly every barrier that once stood between a bad actor and the material they want to make. No technical skill, no equipment, no budget. Exploitative content can now be created and distributed faster than platforms or authorities can respond to it.
The consequences are already in front of us. The Washington Post recently reported on a federal lawsuit against xAI brought by plaintiffs who allege that childhood photographs of them were used to generate sexual imagery. Cases like it are the leading edge of a category of harm that did not meaningfully exist three years ago.
The response across the industry has largely been to tighten access and remove material once it surfaces. Both are necessary. Neither is prevention. By the time an image is reported, reviewed and removed, it has usually been copied, reposted, and moved somewhere the original platform cannot see it.
The Scale Has Changed, Not Just the Volume
The Internet Watch Foundation identified 3,443 AI-generated child sexual abuse videos in 2025, against 13 the year before. 65% were assessed in the most severe legal category, compared with 43% of the non-AI criminal videos the IWF reviewed in the same period.
That is not a trend line, it is a step change. And the severity figure is the part that should stop people: this material is not milder because it is synthetic.
It also does not stay where it starts. An image is generated in one place, shared in another, cut into a video, referenced in a third, and gone from its origin before anyone responsible for finding it knows it exists. Every organization involved sees a fragment. None of them sees the shape of the whole thing.
Beyond Keywords and Reports
Moderation has historically leaned on a narrow set of cues: reported posts, flagged terms, account names. Those still matter. They are no longer sufficient on their own.
Bad actors adapt to enforcement faster than enforcement adapts to them. They learn which terms trigger review and stop using them. They carry meaning in images and audio instead of text. They move between accounts and platforms, treating each one as disposable. Detection anchored to a fixed vocabulary is detection they have already routed around.
What breaks the pattern is context. A single signal in isolation usually means very little. The same signal alongside coordinated accounts, recurring imagery, or activity that resurfaces on a second platform after removal from the first is worth a closer look. For trust-and-safety and public-safety teams, the gap between one post and the pattern it belongs to is the gap between reacting and getting ahead.
Guardrails Are Only Half the Answer
AI companies and platforms have a real obligation to build safeguards into what they ship, and to enforce the rules they publish. That work matters, and it should be held to a high standard.
But no guardrail catches every actor, and no policy catches every piece of material the moment it appears. Building only on that side of the equation means accepting, permanently, that harm gets discovered after the fact.
The other half is visibility for the teams whose job is to stop it. That is not an argument for casting the widest possible net, and it is not an argument for treating every user as a suspect. It is closer to the opposite. Visibility of this kind only holds up when it is narrow on purpose: a defined mission, controlled access, and a clear answer to who is using it and why. The restraint is what makes it legitimate.
Within those limits, the problem is tractable. The signals that indicate this activity are already there. They are simply scattered across platforms, formats and noise in a way that costs investigators the resource they have least of, which is time. This is the work we do at Vetric: we help trust-and-safety and public-safety teams to see the activity that matters, early enough to act on it.
Public Safety Is the Line
New technology is usually debated as a choice between innovation and restriction. With AI, it is increasingly framed as a choice between expression and censorship. That framing puts the argument in the wrong place.
We can back technologies that let people create and still draw a hard line around tools used to exploit people. Those two positions have never actually been in conflict. But a line only exists if someone can see it being crossed. Policy alone does not do that. It needs trusted organizations able to recognize dangerous activity as it moves between platforms.
Generative AI is going to keep getting faster and more capable. The open question is whether the people responsible for protecting others are equipped to move at the same speed.