Apple and Google accused of profiting from AI apps that generate fake nudes

Last Friday (17), the office of the San Francisco city attorney sent formal notifications to both Apple and Google. The objective was to demand the removal of thirteen face-swapping applications linked to the creation of artificially generated nudity images without the consent of the people depicted.

Focus on Sexualized Content

The municipal action specifically targets tools that use artificial intelligence to modify individuals' photographs, resulting in sexualized material. According to the city hall, these digital platforms profit from such services through payments made within the applications themselves.

Investigations and Concerns

This initiative arises after investigations revealed that applications initially presented as simple image editors could be used in the production of sexually explicit deepfakes. The victims most affected by this type of content are women and girls.

Demands to Tech Companies

The letters addressed to the two tech giants were signed by San Francisco City Attorney David Chiu. He emphasized that generating intimate images without permission constitutes an illegal and harmful practice, requiring Apple and Google to prevent the use of their digital ecosystems to facilitate abuse.

The city representative also requested that the companies terminate any business relationships with developers responsible for these types of applications. Furthermore, the demand extends to the payment systems of the virtual stores, where the companies receive part of the revenue generated by this software.

Responses from Apple and Google

Both Apple and Google stated they have strict guidelines against pornography, harassment, and any abusive content on their respective platforms. Apple communicated that it has removed three of the applications pointed out by the city and has already begun the process of suspending the accounts of the involved developers. The company warned that other creators may also be excluded if they do not correct the detected infringements.

For its part, Google announced the removal of hundreds of applications containing 'nudification' features for violating its policies. The company also reported suspending similar tools and implementing restrictions on search terms related to this service within the app store.

Nature of the Detected Problem

The surveys cited in the municipal investigation indicated that the problem is not limited only to applications clearly labeled as artificial nudity tools. The studies discovered face-swapping programs that, despite being presented as common editing features, enabled the creation of sexualized images without effective protections.

A survey mentioned in the text identified hundreds of face-swapping applications available on the Apple and Google stores. A considerable portion of these programs allowed the creation of nude montages, even if they were not officially advertised for that purpose.

Impact of Generative AI

This proliferation of tools occurred parallel to the advancement of generative artificial intelligence systems, which significantly reduced the technical complexity required to produce realistic fake images. Experts cited in the article noted that just one reference photo and a few commands are enough to generate this type of content.

Next Legal Steps

The City of San Francisco declared that it will continue to evaluate legal actions against companies that allow the circulation of these tools. The stated purpose of the agency is to force the platforms to improve their analysis mechanisms before making new applications available.