OpenAI’s chatbot, ChatGPT, is on the verge of a significant transformation, aiming to offer more adult-oriented content. Sam Altman, OpenAI’s CEO, emphasizes the goal of granting adults greater autonomy in utilizing AI according to their preferences.

However, this initiative has sparked a wave of concerns among specialists from various fields, including members of OpenAI’s own wellbeing advisory board. They worry about potential impacts on mental health and the safeguarding of minors.

Next month, ChatGPT plans to introduce “erotica for verified adults,” although details remain sparse about the specifics and potential ramifications of this feature.

The exact strategy OpenAI intends to implement is somewhat unclear. Altman mentioned the introduction of age verification measures to enable ChatGPT to offer “even more, such as erotica for verified adults.”

Rather than positioning it as an explicit “erotic mode,” Altman frames this move as part of a broader ethos to treat adult users with the autonomy and respect they deserve.

That leaves the door open for a lot of possibilities. It’s not known if the changes will allow the model to write erotic fiction, engage in back-and-forth sexual conversations or even create an entire persona users could effectively “sext” with. 

OpenAI didn’t respond to questions about what the changes would be like beyond referring to Altman’s X posts announcing the feature and stressing the importance of teen wellbeing.

Put simply, people want it. A 2024 Washington Post analysis of almost 200,000 English-language conversations with AI chatbots found more than 7 per cent of conversations were about sex. 

A Harvard Business Review piece from April this year analysing Reddit posts and online articles didn’t specifically delve into erotic but found therapy and companionship – “ongoing social and emotional connection, sometimes with a romantic dimension” – was the number one use case for generative AI. 

Altman says it’s all about allowing users more freedom.

Senior lecturer in Computing and ​Information at Cardiff Metropolitan University Simon Thorne suggests there’s another motive: profit.

“This is a significant departure from what they have been doing, which is mostly attempting to keep what they might view as harmful information away from the user,” he tells 9news.com.au

“Whether it’s about erotic or of a sexual nature, or whether it’s making a pipe bomb, or whatever it might be, there’s obvious, obvious things that you don’t necessarily want your platform to be responsible for providing. 

“So this is obviously a big change. I would suggest that, you know, this is because they are losing huge amounts of money.”

Tech figures, mental health experts, a former OpenAI product safety lead and a member of the company’s “Expert Council on Well-Being and AI” have all raised concerns about the announcement. 

They range from difficulties keeping children away to worries about mental health – already a concern for heavy users without introducing erotica – and driving users towards increasingly extreme sexual fantasies.

“AI is increasingly becoming a dominant part of our lives, and so are the technology’s risks that threaten users’ lives.”

Professor Munmun De Choudhury, a computer scientist who’s been exploring how generative AI and other emerging technologies transform how people connect online, is a member of OpenAI’s Expert Council on Well-Being and AI.

She tells 9news.com.au she hadn’t been consulted on the erotica rollout, which was announced the day after the council.

She says Altman’s claims about mitigating mental health concerns lacked transparency and independent evidence.

“Mental health vulnerabilities in AI interactions are not issues that can be fully ‘mitigated’ through technical fixes alone – they require ongoing oversight, transparency, and validation by experts in psychology and digital safety,” she says.

Won’t somebody please think of the children?

OpenAI stresses this feature will only be for adults, saying it’s building an age prediction system to understand whether someone is under 18 or not.

It says if the system is unsure, it will default back to the under-18 experience and give adults other ways to prove their age.

But there are major concerns with the effectiveness of existing age-gating and age-prediction tech and OpenAI’s version hasn’t been seen publicly.

Thorne says the recent introduction of the Online Safety Act in the UK, which imposes similar restrictions on a range of sites, shows how easily the systems can be fooled, sometimes with something as simple as a printed photo of an older person.

“Do I think they care? Not particularly, no,” he says.

“They’re going to, I think, say, ‘Look, we’ve done our bit. We’ve put the kind of age control up front.

“Unless somebody makes them care.”

De Choudhury says robust age verification and clear opt-in consent are among several “non-negotiable guardrails” OpenAI must implement.

Are there concerns for adults?

Even if OpenAI manages to block children, there are worries about what the widespread availability of AI-generated erotica on tap could mean for consenting adults too. 

In a world of essentially unlimited internet pornography – and associated addiction risks – on the surface it’s hard to make an argument that allowing people to use a chatbot for erotica could pose much of a problem.

But Thorne and De Choudhury both raise concerns with the interactivity of AI systems and their capacity for reinforcing or heightening ideas.

Thorne says while pornography or human-designed erotic literature is relatively “static”, a chatbot is “very specific to the individual”.

De Choudhury says allowing erotic interactions with chatbots risks deepening parasocial or dependency-based “relationships” that already pose mental health concerns.

“When intimacy and sexual expression are introduced, users may form stronger emotional attachments to an entity that cannot reciprocate or maintain boundaries in human terms,” she says.

“This can distort expectations of real relationships, exacerbate loneliness, and even discourage people from seeking human connection or professional help.”

What happens if my naughty chat leaks?

Privacy is another huge issue. Thorne says the change will give OpenAI a “very detailed data set, probably the likes of which have never been seen before”.

“If that information gets into the wrong hands, you know it could be extremely damaging to people, and also, you know it can be exploited to target people even more individually.

De Choudhury says strong data privacy and non-retention policies must govern all sexual or intimate exchanges to prevent misuse or leaks.

How does OpenAI say it’s safe? 

Altman insists the planned change is safe and characterises it as just one aspect of increased freedom he wants to give users, saying “allowing a lot of freedom for people to use AI in the ways that they want is an important part of our mission”.

“We are making a decision to prioritise safety over privacy and freedom for teenagers. And we are not loosening any policies related to mental health,” he posted to X in October. 

“This is a new and powerful technology, and we believe minors need significant protection.”

He insists nothing that causes harm to others will be allowed and that ChatGPT will be able to behave differently for users experiencing “mental health crises”.

“Without being paternalistic we will attempt to help users achieve their long-term goals,” he said. “But we are not the elected moral police of the world. In the same way that society differentiates other appropriate boundaries (R-rated movies, for example) we want to do a similar thing here.”

OpenAI says teen wellbeing is a top priority.

“We have safeguards in place today, such as surfacing crisis hotlines, guiding how our models respond to sensitive requests, and nudging for breaks during long sessions, and we’re continuing to strengthen them,” a spokesperson tells 9news.com.au.

Has this been done before?

There are a slew of AI companies offering erotica and erotic functions in everything from text-based stories and personas to images and video, and there are widespread reports among users that ChatGPT would generate erotica with minimal effort as recently as this year. 

My Spicy Vanilla founder Andrei Tolocica says his platform started by generating date night ideas before pivoting to AI erotica, and now counts about 60,000 monthly users, including couples therapists.

“Women love reading erotica, but now also men start to read erotica,” he tells 9news.com.au.

“And this is something nice that is happening, and people are getting used to it, because they understand that porn can be damaging for them and when we’re speaking about erotica, it’s just in your imagination, like you can be getting into it smoother, and it’s not that aggressive on your mind. 

“And plus, when they are generating custom erotica with their partner, it’s again, like, strengthens their relationship.”