Last Friday (17), the office of the San Francisco city attorney sent formal notifications to both Apple and Google. The objective was to demand the removal of thirteen face-swapping applications linked to the creation of artificially generated nudity images without the consent of the people depicted.
The municipal action specifically targets tools that use artificial intelligence to modify individuals' photographs, resulting in sexualized material. According to the city hall, these digital platforms profit from such services through payments made within the applications themselves.
This initiative arises after investigations revealed that applications initially presented as simple image editors could be used in the production of sexually explicit deepfakes. The victims most affected by this type of content are women and girls.
The letters addressed to the two tech giants were signed by San Francisco City Attorney David Chiu. He emphasized that generating intimate images without permission constitutes an illegal and harmful practice, requiring Apple and Google to prevent the use of their digital ecosystems to facilitate abuse.
The city representative also requested that the companies terminate any business relationships with developers responsible for these types of applications. Furthermore, the demand extends to the payment systems of the virtual stores, where the companies receive part of the revenue generated by this software.
Both Apple and Google stated they have strict guidelines against pornography, harassment, and any abusive content on their respective platforms. Apple communicated that it has removed three of the applications pointed out by the city and has already begun the process of suspending the accounts of the involved developers. The company warned that other creators may also be excluded if they do not correct the detected infringements.
For its part, Google announced the removal of hundreds of applications containing 'nudification' features for violating its policies. The company also reported suspending similar tools and implementing restrictions on search terms related to this service within the app store.
The surveys cited in the municipal investigation indicated that the problem is not limited only to applications clearly labeled as artificial nudity tools. The studies discovered face-swapping programs that, despite being presented as common editing features, enabled the creation of sexualized images without effective protections.
A survey mentioned in the text identified hundreds of face-swapping applications available on the Apple and Google stores. A considerable portion of these programs allowed the creation of nude montages, even if they were not officially advertised for that purpose.
This proliferation of tools occurred parallel to the advancement of generative artificial intelligence systems, which significantly reduced the technical complexity required to produce realistic fake images. Experts cited in the article noted that just one reference photo and a few commands are enough to generate this type of content.
The City of San Francisco declared that it will continue to evaluate legal actions against companies that allow the circulation of these tools. The stated purpose of the agency is to force the platforms to improve their analysis mechanisms before making new applications available.
Adam Mosseri, head of Instagram, stated that content generated by artificial intelligence should not be shown to users who do not show interest in it. The executive argues that the platform's algorithm must be adapted to respect this preference, although he dismisses the possibility of simply blocking all AI content on the social network.
With the proliferation of AI-created materials in recent years, there is currently no simple mechanism for users to hide this type of content. Mosseri explained in an interview with Lenny Rachitsky's podcast that Meta's goal is to improve the ability to identify AI-generated videos and images, allowing recommendation systems themselves to understand the user's level of interest.
Mosseri had previously admitted publicly that the Instagram feed had stopped reflecting users' real interests due to the growing power of the algorithm. This observation arose after the expansion of the 'Your Algorithm' feature, which shows users the types of posts the technology deems relevant and offers the option to decrease the visibility of certain topics, although it does not completely prevent the delivery of specific content.
According to Mosseri, detecting AI content is becoming progressively more complex as language models evolve. For this reason, he proposed that Instagram could adopt less definitive responses regarding the origin of media. Instead of rigidly categorizing whether something was or was not made by AI, the network could indicate confidence levels, such as 'probably yes,' 'we are not sure,' or 'definitely no.'
Currently, the platform has a flagging system for the use of generative tools, as do competitors like YouTube and TikTok. These posts may display an 'AI Information' notification in the feed, detailing that the content 'may have been created by AI.' The pop-up clarifies that this flag generally applies to content entirely generated by AI, and not just modified with AI assistance.
During the podcast, the executive presented two main ideas. The first is to create a dedicated space to segregate content entirely generated by AI—including spam, fraudulent montages, and other 'AI junk'—from posts that used creative editing or content remixing tools.
Meta has already experimented with similar concepts in the Meta AI app, presenting feeds of images generated by the company's models and even a Reels format with synthetic videos. The second proposal, advocated by Mosseri since December of last year, is the implementation of certification for authentic content. The argument is that it would be simpler to attest to the provenance of a real photograph than to prove the use of AI.
Large AI corporations developing image and video models, such as OpenAI, Google, Microsoft, and Meta itself, are using digital certificates, such as C2PA, and invisible watermarks in content, such as SynthID. These marks are generated by the tools themselves to aid identification. In addition to the declaration made by the user at the time of posting, Instagram can read this information to alert about possible AI use. However, this practice is not yet standardized and these identifiers are not impossible to bypass or remove.
