OpenAI won’t be the first chatbot to try to profit from sexual content

ChatGPT will be able to have naughtier conversations after OpenAI CEO Sam Altman announced that the artificial intelligence company will soon allow its chatbot to engage in “verified adult erotica.”

OpenAI won’t be the first to try to take advantage of sexualized AI. Sexual content has been a major draw for AI tools almost as soon as the boom in AI-generated images and words erupted in 2022.

But companies that were early adopters of mature AI also faced legal and societal minefields and harmful abuses, as growing numbers of people turned to the technology to join or titillate.

Will a sexier ChatGPT be any different? After three years of largely banning adult content, Altman said Wednesday that his company was “not the world’s elected moral police” and was prepared to allow “more freedom of use for adults” while setting new limits for teens.

“In the same way that the company differentiates other appropriate boundaries (R-rated movies, for example), we want to do the same thing here,” Altman wrote on social media platform X, whose owner Elon Musk also introduced an animated AI character who flirts with paying subscribers.

For now, unlike Musk’s Grok chatbot, paid subscriptions to ChatGPT are primarily intended for professional use. But letting the chatbot become a friend or romantic partner could be another way for the world’s most valuable startup, which loses more money than it makes, to turn a profit that could justify its $500 billion valuation.

“They don’t really earn much from subscriptions, so having erotic content will make them money quickly,” said Zilan Qian, a researcher at the China Policy Lab at the University of Oxford, who has studied the popularity of dating-based chatbots in the United States and China.

There are already about 29 million active users of AI chatbots designed specifically for romantic or sexual connections, and that’s not counting people who use conventional chatbots in this way, according to a study published by Qian earlier this month.

It also does not include users of Character.AI, which is conduct a trial which alleges that a chatbot inspired by the “Game of Thrones” character Daenerys Targaryen entered into a sexually abusive relationship with a 14-year-old boy and drove him to commit suicide. OpenAI faces lawsuit from the family of a 16 year old ChatGPT user died by suicide in April.

Qian said she worries about the consequences for real-world relationships when mainstream chatbots, already prone to sycophancy, are willing to be available 24 hours a day to distribute sexually explicit content.

“ChatGPT has versions of voice chat. I would expect that in the future, if they were to go that route – voice, text, visual – it would all be there,” she said.

Humans falling in love with human-like machines has long been the subject of a cautionary tale, from popular science fiction of the last century to the ancient Greek legend of Pygmalionobsessed with a woman he carved in ivory. Creating such a machine seems like an unusual detour for OpenAI, founded a decade ago as a nonprofit dedicated to safely building better-than-human AI.

Altmann said on a podcast in August, OpenAI tried to resist the temptation to introduce products that might “drive growth or revenue” but would be “very misaligned” with its long-term mission. When asked for a specific example, he gave one: “Well, we haven’t put a sexbot avatar in ChatGPT yet.”

Idaho-based startup Civitai, a platform for AI-generated art, learned the hard way that making money from mature AI wouldn’t be an easy path.

“When we launched the site, it was an intentional choice to allow adult content,” Justin Maier, the company’s co-founder and CEO, said in an interview last year.

Backed by prominent venture capital firm Andreessen Horowitz, which also invested in OpenAI, the Idaho startup was one of several that tried to capitalize on the sudden popularity of tools like Stable Diffusion and Midjourney, which let people type in a description and conjure up almost any type of image. Part of Stable Diffusion’s initial popularity was the ease with which it could generate a new type of synthetic, highly personalized pornography.

“What we found was that there was a lot of interest in adult content,” Maier said. Training these AI systems, called models, on “mature themes has actually made these models more capable of understanding human anatomy and has resulted in actually better models,” he said.

“We didn’t want to prevent the type of growth that has actually increased everything for the entire community, whether you’re interested in adult content or Pixar,” Maier said. “So we allowed it from the beginning and always had that battle so we could keep things filtered and secure, if that’s not what you’re interested in. We ultimately wanted to give control to the user to decide what they would see on the site and what their experience would be.”

It also encouraged abuse. Civitai implemented new measures last year to detect and remove sexual images depicting children, but the company remains a hub for AI-generated pornography, including fake images of celebrities. Facing increasing pressure, particularly from credit card processors and a new law against non-consensual images Signed by President Donald Trump, Civitai earlier this year blocked users from creating fake images of real people. Engagement plummeted.

Baltimore-based Nomi.ai is another company that hasn’t shied away from adult content, although founder and CEO Alex Cardinell has said its companion chatbots are “strictly” aimed at users over 18 and have never been marketed toward children. They’re also not designed for sex, although Cardinell said in an interview earlier this year that people who build platonic relationships with their chatbot might find it veers toward romanticism.

“It kind of depends on the user, because they’re kind of missing the human gap in their life. And I think it’s different for everyone,” he said.

He declined to guess how many Nomi users have erotic conversations with the chatbot, comparing it to real-life partners who might do “adult content things” for part of their lives, but “all kinds of other things together as well.”

“We don’t monitor user conversations in this way,” Cardinell said.

Altman’s announcement that adult erotica could be coming to ChatGPT in December came a day after California Gov. Gavin Newsom vetoed the legislation it would have prohibited companies from making AI chatbots available to anyone under 18 if it was “foreseeable” that they would engage in “erotic or sexually explicit interactions” with children or encourage them to harm themselves. The tech industry lobbied against the bill, which Newsom said was too broad, but OpenAI, Meta and others introduced new age restrictions and parental controls. for AI-teen interactions.