AI and child safety

This time around, can we build in some child protections from the start?

“The internet was not built for kids.” This nearly 10-year old quote from my colleague and friend Dylan Collins continues to be repeated, often by regulators looking to understand advertising technologies that track users, or social media algorithms that pull consumers into ever-darker rabbitholes.

Since then, many parts of the internet have been made a lot safer for kids. Contextual advertising (that doesn’t track) is now the norm on kids’ content; most ad platforms are able to switch off the collection of persistent identifiers (in order to comply with COPPA or GDPR-K); more and more adult services are age-restricted (though we can debate how effectively); and content labelling and moderation are now common practice (and increasingly required by law).

But then came generative AI — transformative new technology that creates next-level of risk for consumers, and for children in particular. And we are — once again — in the position of asking ourselves: are we going to build in protections from the start or — as with the internet — ignore the risks and wait until we’re forced to retrofit protections later?

This time at least, the harms are more obvious and more widely reported. Schools everywhere are struggling to contain an epidemic of deepfake nudes that cause immense distress and even suicide among teens. The explosion of AI-generated misinformation puts kids and teens at particular risk, as they are both more likely to get their news from non-verified media and to believe in conspiracy theories than older generations.

But perhaps most distressingly, AI-generated content is overwhelming the existing systems for controlling child sexual abuse material (CSAM). Little known outside the specialist community that runs it, the CyberTipline operated by the National Center for Missing & Exploited Children (NCMEC) plays a critical role in reducing the spread of CSAM and — more important — enabling law enforcement to rescue victims and arrest perpetrators. That safety net is now falling apart.

All major internet platforms report suspected CSAM to the CyberTipline. NCMEC NCMEC screens the content to determine whether a real child is being harmed and tries to extract clues that might help law enforcement take action. As of February, NCMEC is receiving some 99,000 reports every day. Less than 1% of these lead to investigation by law enforcement.

Once verified, unique instances of CSAM are entered into a database of digital fingerprints (hashes), which online platforms can use to automatically identify and remove the same content. At the end of 2023, NCMEC had some 7.7m hashes on its list, and all major traditional online platforms rely on it to automate CSAM removal.¹

But NCMEC has delivered this service in spite of being seriously hampered by legal, technical and financial hurdles. According to a recent Stanford Internet Observatory report, online services often fail to provide relevant context information with reports they file, making it harder for NCMEC to prioritise. The whole system is subject to arcane (and complex) legal precedents that restrict NCMEC’s ability to train platforms on how to improve their reports, and that limit its ability to help law enforcement get a warrant to investigate a hash match.² Because of the rapid growth of viral and meme content, the volume of hash matches and new reported content is overwhelming the police’s ability to follow up within the 90-day period that platforms typically retain preserved content (ie, evidence).

Into this scenario enter AI deepfakes. If ‘nudify’ apps used by teens to embarrass their classmates are at one end of the awfulness spectrum, AI-generated CSAM is at the other. NCMEC was able to identify 4,700 AI-CSAM reports in 2023 — which is a tiny fraction of the 36m reports received in the year — but the volume is increasing rapidly, as reported by NCMEC SVP John Shehan in recent congressional testimony. AI-CSAM might involve purely AI-generated (ie, fictitious) children, or use AI to alter non-sexual images or videos of children to be sexually explicit, or to generate new imagery from older CSAM content. One way to prevent AI models from generating CSAM is to ensure their training data does not contain any. But for that it is arguably too late: a number of AI image models were trained on a public dataset known as LAOIN-5B, which is known to include CSAM images.

A dispassionate philosopher might contrive an argument that AI-CSAM is better than CSAM because its creation does not involve the abuse of children. But this is wrong. AI-CSAM floods the existing reporting mechanisms and leads law enforcement to look for children that do not exist, making it harder to rescue victims. It victimises or re-victimises real children, either because earlier abuse material is being recycled into new images and redistributed, or because their benign images are being used to generate abuse materials, including for sextortion attempts. And it is likely to generate more demand for such material, helping to normalise child abuse.³

Two organisations have done incredible, forward-looking work on this problem: child abuse advocacy group Thorn, and responsible tech non-profit All Tech Is Human. They recently announced a set of Generative AI Principles to protect children, which has been endorsed by a number of AI platform vendors, including OpenAI, Anthropic, Google, Meta, Amazon, Microsoft, Mistral and Stability AI. More important, their paper includes safety-by-design recommendations for each stage of the AI model lifecycle to mitigate the likelihood of misuse for CSAM creation.⁴

This is ambitious, and it will help, but will not be sufficient.

NCMEC (and its international equivalents) are the only viable infrastructure to reduce the spread of CSAM and support the rescue of children at risk. And yet the organisation is woefully underfunded. The government last year provided $45m of its $60m budget, a rounding error in government spending terms.

Some steps in the right direction are underway. Virtually unreported, last week President Biden actually signed the bipartisan REPORT Act. This is a very narrow piece of legislation. It addresses the retention period and extends NCMEC’s liability shield in ways that will help it be more effective. But it also significantly increases the pressure on platforms to report suspected CSAM. One might assume that’s a good thing, but in fact it is likely to exacerbate over-reporting, where the operators file anything that could be interpreted as illegal and push the burden over to NCMEC. Kashmir Hill has reported on what can go wrong with that approach.

But the new law falls short of providing new funding for NCMEC, basically guaranteeing an increase in its workload without committing any additional resources. For some reason, the bill that addresses that, the Invest in Child Safety Act, has been bouncing around the Senate for three years. Its most recent incarnation would provide $5bn in mandatory funding to law enforcement, NCMEC and various organisations that support victims. I get that Congress is not exactly operating at peak peformance, but was it really so hard to combine these bills and do the right thing for the heroic alliance facing down CSAM just as AI is about to change everything?

1. For more stats, see NCMEC’s 2023 report.

2. From How to Fix the Online Child Exploitation Reporting System: “A federal appeals court held in 2016 that NCMEC is a governmental entity or agent, meaning its actions are subject to Fourth Amendment rules. As a result, NCMEC may not tell platforms what to look for or report, as that risks turning them into government agents too, converting what once were voluntary private searches into warrantless government searches (which generally requires suppression of evidence in court). Consequently, NCMEC is hesitant to put best practices in writing. Instead, many trust and safety staff who are new to the CyberTipline process must learn from more established platforms or industry coalitions. 

Another federal appeals court held in 2021 that the government must get a warrant before opening a reported file unless the platform viewed that file before submitting the report.  Platforms often do not indicate whether content has been viewed; if they have not so indicated, then NCMEC, like law enforcement, cannot open those files. Platforms may automate reports to the CyberTipline on the basis of a hash match hit to known CSAM instead of having staff view each file, whether due to limited review capacity or not wanting to expose staff to harmful content. Where reported files weren’t viewed by the platform, law enforcement may need a warrant to investigate those reports, and NCMEC currently cannot help with an initial review.”

3. The effects are clearly spelled out in the report accompanying a recent announcement by Thorn and All Tech Is Human of new child safety principles for AI.

4. A summary of some of their recommendations:

• remove CSAM from training data and avoid having adult sexual content alongside non-sexual depictions of children

• remove known models for CSAM creation from public directories

• conduct adversarial training to remove the capability of creating CSAM

• app stores / search engines, etc - remove from results models that produce unwanted sexual imagery, ‘nudify’ apps, etc.

• implement a system of content provenance, eg watermarking; and align content provenance research with the systems already in use by NCMEC

• require use of the “generativeAi” file annotation when filing a report with NCMEC