The ads pointed people towards almost 50 apps combined across Google’s Play Store and Apple’s App Store. Of the 350 ads, the researchers say that around 300 of them directed people to AI image or video generator ads, with most of these apps being linked to China-based developers. Some apps had already been removed by the time WIRED started reporting this story.
Among those that would generate explicit imagery of adults were eight identical apps listed on Apple’s App Store, multiple of which were linked to Chinese developer names. When the apps opened, they would show explicit AI-generated content, including a menu of sexual scenarios, including one called “threatening knife” where a knife is held to a woman’s throat. WIRED identified another seven identical apps that shared the same design, explicit content, and listed the same developer contact details.
“The App Store was designed to be a safe and trusted place for users, and we have zero tolerance for developers who deliberately attempt to evade our review process,” says Apple spokesperson Adam Dema. “The developers mentioned in this report initially submitted apps that are compliant with the App Review Guidelines, only to later surface nudify capabilities, which are against our rules.” The company says that 17 of the 20 apps included in the research had already been removed from its store and it has taken action against the remaining three, as well as removing the seven apps WIRED found.
Google spokesperson Dan Jackson says the company does not allow apps that contain sexual content or the ability to nudify. “We continually take proactive steps to detect and remove apps with harmful content,” they say, adding that it removed all the apps included in the report. “When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like ‘nudify’ on our store.”
“The share of Chinese developers creating nudify apps has grown year over year,” says David Pham, a former Meta employee who investigated nonconsensual imagery and abuse on its platforms. Pham says that while at Meta he observed similar ads to those discovered by the researchers and those uploading the content would constantly try to get around safety systems.
“Every time the platform or investigators would build a new detection model or a new way to have some type of enforcement, they quickly—within a day, maybe five days—built a new way to avoid detection,” Pham says. Those uploading the ads would change domains they used, obscure who posted them, and include measures such as blurring imagery or subtly changing images to avoid detection. However, Pham, who left Meta last year, says it has the funding and resources to be able to tackle the uploads. “I would like to live in a world where we’re not getting CSAM or NCII material sent directly to our phones,” he says.
Meta’s current protections against ads containing CSAM and NCII appear to need work. More than 11,000 undress ads have appeared on Meta’s platforms since February this year, according to a recent Indicator Media report, with many of these being reportedly posted by “repeat offenders.” Analysis by WIRED also found hundreds of ads including explicit content that contain similar language to those ads found by the TTP researchers. (Meta says malicious advertisers use adversarial tactics to continually try to avoid its detection systems and it has “incorporated” the language in the ads highlighted by WIRED into its moderation systems.)
One ad containing one of the teenagers that the researchers found the real-world identity of was reported to Meta when it had reached four accounts in Europe, according to the TTPs research. However, when it was removed days later, it had been seen by more than 330 users. “This is not something that it should take a small nonprofit doing research to stumble across and then have to investigate and report,” Paul from TTP says. “We’re talking about one of the most well-resourced— technologically and financially—companies on the planet.”