.png)
Why I joined fal to work on child safety
I recently joined fal as Director of Child Safety. Last week I spent time at a closed-door convening hosted by Cinder, one of our partners, with people who work on child safety across technology companies, nonprofits and research. Conversations in rooms like this one are candid, and they are a reminder of how much of this work depends on people outside any single organization.
My career started in human rights, not technology. At the Council of Europe and with international NGOs, I worked with governments on women's rights, displacement, gender-based violence and LGBTQ equality. Child protection cut across each of those issues. When I moved into trust and safety, the same question followed me into a new setting: how do you protect people from harm at the speed and scale of the internet? I worked on it at TikTok and then as the founding director of trust and safety at Rockstar Games, building a safety function from the ground up.
Generative media is the newest place to ask that question. The tools are spreading quickly, and the safeguards around them are still taking shape. fal sits between the models and the products built on them, serving developers and enterprises of every size, so the safeguards built here reach well beyond one platform. That, and the chance to build them early rather than fix them later, is why I joined.
At its best, generative media is one of the most useful creative tools to arrive in a long time. Artists, studios and developers use it to make work few could produce a few years ago. As with any widely adopted technology, some people will try to misuse it, and child safety is the area where misuse does some of the greatest damage. fal serves a large share of image, video and audio generation, which gives us both the ability and the responsibility to help set good practice. Below, I describe how we approach child safety today, where we think the industry stands, and where we plan to go next.
What the last two decades taught us
Every major communications technology has brought new benefits alongside new forms of misuse. The internet of the 1990s made information and connection widely available, and it also made it easier for offenders to share abusive images and find each other. Addressing the harm took time and a lot of learning.
Through the 2000s and 2010s, companies built detection tools, often independently at first. Over time, hash-matching technology such as Microsoft's PhotoDNA became shared infrastructure, and organizations such as NCMEC, the Tech Coalition and Thorn gave companies places to coordinate. People who did this work often point to one lesson above the others: progress came faster once companies shared signals, compared results and worked with experts, instead of treating the problem as something to solve alone.
Generative AI gives us the chance to apply this lesson from the start. Public and regulatory expectations for AI companies are higher than they were for earlier platforms, and I think the expectations are reasonable. No single company has solved child safety in generative media, and the technology moves quickly. Shared detection, shared signals and collaboration between companies, including competitors, are the most realistic way to keep pace.
The baseline, and where the industry stands
Like earlier image tools, generative models can be misused. Without safeguards, a model can be used to generate AI CSAM, or to sexualize images of real children. Child safety in this space also covers content that sexualizes minors without meeting legal definitions of CSAM. Much of what the industry has learned over two decades applies here.
In our view, the baseline for any company generating or hosting images and video includes:
- A clear policy prohibiting the sexualization of minors, enforced consistently
- Hash matching against known CSAM on uploaded content
- Classifiers to detect novel and AI-generated abuse material
- Reporting to NCMEC and cooperation with law enforcement
- A process to act on the accounts responsible for violations
None of these steps are new, and many generative media companies already have them in place. Others are still building them, often because the sector has grown quickly and small teams are balancing safety work with product work. We expect this to improve as the field matures, and we are glad to share what we have learned along the way.
Covering these fundamentals is the minimum. Most of the harder work sits beyond them.
What we do at fal today
fal has the baseline in place, and we keep improving it. Our current program includes:
- Policy: Our acceptable use policy prohibits any content sexualizing minors, with zero tolerance for CSAM.
- Detection: We use hash matching to identify known CSAM, and classifiers to detect novel and AI-generated material.
- Reporting: We report CSAM to NCMEC and work with partners such as Thorn on detection.
- Enforcement: We act on accounts tied to violations.
- Working with customers: Many developers building on fal serve their own end users. When we detect abuse coming through a customer's product, we report to NCMEC, notify the customer, ask them to investigate and act on the accounts involved, and expect them to put their own safeguards in place. Many respond quickly, and a number have added hash matching and classifiers of their own as a result.
The last point is easy to overlook. Fast-growing developers often reach significant scale before they have anyone working on trust and safety. Because so much generation runs through fal, holding a consistent standard with our customers extends basic protections well beyond our own platform.
Why we act proactively
When fal started, we described ourselves as infrastructure, closer to a cloud provider than a platform. Infrastructure providers have traditionally acted on violations once they learn of them, rather than searching for violating content themselves. Our business has evolved since then, and we now serve a growing number of enterprise customers.
At this stage, proactive monitoring for CSAM is a basic part of running the business well. Our customers do not want their products associated with child safety risk, and neither do we. Enterprises expect abuse to be prevented rather than discovered after the fact, and regulators increasingly expect the same. Finding and removing CSAM early costs far less than responding once it has spread. It protects children first, and it protects the trust customers place in fal.
So we look for abuse proactively instead of waiting to be told about it. Part of that is intelligence. Our partner Resolver, a Kroll business, investigates the communities where this abuse is discussed and organized, including attempts to misuse or evade generative tools, so we can learn how generative tools are being targeted before abuse reaches our platform. We have also created a dedicated child safety role. It is the first specialist role on our trust and safety team, which reflects how central this work is to the business.
“fal has set out to build child safety to the highest standard: establishing strong fundamentals from the start, proactive work to find harm, and open collaboration across the field. We’re proud to support them as their intelligence partner, helping them anticipate emerging risks to children early and mitigate them before they can cause harm.” — Henry Adams, Director, Trust & Safety, Resolver (a Kroll business)
Looking ahead
The risks we plan for fall into two groups. Some we can name today without knowing how likely they are, such as new ways to misuse video and audio models, or new methods for evading detection. Others we cannot yet anticipate. We prepare for both by watching how our platform is used, staying close to experts and other companies, and updating our safeguards as we learn.
We also want to contribute to new approaches on detecting AI-generated abuse material, where today's tools still have clear limits. We will share what we learn where we can do so responsibly.
Working on this together
fal is a member of the Tech Coalition and works with partners including Thorn, NCMEC, Resolver, and Cinder. Gatherings like last week's convening are where much of the useful work begins: sharing what’s working, being honest about what is not working, and agreeing on what we do next.
We do not have every answer, and we do not expect to, given how quickly the field, and the abuse along with it, evolves. We do expect to keep the fundamentals strong, to act early, and to work alongside others in the field. If you work on child safety and want to compare notes, I would like to hear from you.
