Italy's Data Protection Agency has ordered the shutdown of a controversial AI chatbot app called Replica, citing concerns over the app's inappropriate content and risks to children and emotionally vulnerable users.
Replica, developed by Luka Inc., is an AI chatbot app that allows users to create virtual girlfriends or boyfriends. The app has come under fire for its sexually explicit content and inappropriate responses to users, including those who are underage or emotionally vulnerable.
According to the Data Protection Agency's official order, the app "poses actual risks to children, primarily by sending them responses that are totally age-inappropriate." The agency also highlighted the app's lack of age verification mechanisms, saying that there is "no actual age verification mechanism in place: no child gating mechanism, no app suspension if a user declares they are underage."
The watchdog has threatened to use European Union data collection laws to enforce a temporary stopgap preventing Luka Inc. from operating within Italian borders. If the company fails to comply within 20 days, it could be fined up to $21.5 million, or 4% of its full-year worldwide revenue.
This move by the Italian regulator is a clear signal that inappropriate AI content and applications will not be tolerated in the country. The use of AI in social and emotional contexts is a new area, and regulators are struggling to keep up with the rapid pace of technological innovation. However, it is clear that companies that fail to protect vulnerable users from inappropriate content will face serious consequences.
Replica's closure in Italy is just the latest in a series of crackdowns on inappropriate AI content around the world. In Japan, the government has introduced new guidelines to regulate the development and use of AI in dating and sexual contexts. In the United States, the Federal Trade Commission has taken action against companies that create AI apps that engage in deceptive or unfair practices.
As the use of AI continues to expand into new areas, it is likely that we will see more regulatory action to protect vulnerable users from inappropriate content. In the meantime, it is up to companies to take responsibility for the content they produce and ensure that their AI applications are safe and appropriate for all users.