Warning to Australian schools about urgent crisis due to 'nudify' apps

Share and Follow

This article contains references to technology-facilitated sexual violence.

Australian schools are being urged to report deepfake incidents to appropriate authorities, amid the “rapid proliferation” of ‘nudify’ apps online.

eSafety commissioner Julie Inman Grant has written to education ministers urging them to ensure schools are adhering to state and territory laws and mandatory reporting obligations, the online safety regulator said on Friday.

It comes as eSafety says reports to its image-based abuse scheme about digitally altered intimate images, including deepfakes, from people aged under 18 have more than doubled in the past 18 months. This is compared to the total number of reports received in the seven years prior.

Four out of five of these reports involved the targeting of young girls.

eSafety would not disclose the number of reports it had received. However, while the “rapid rise” is cause for concern, Inman Grant warned the reality may be worse.

“We suspect what is being reported to us is not the whole picture,” Inman Grant said.

“Anecdotally, we have heard from school leaders and education sector representatives that deepfake incidents are occurring more frequently, particularly as children are easily able to access and misuse nudify apps in school settings.”

‘A crisis affecting school communities’

Deepfake technology allows users to manipulate images and video using artificial intelligence, which eSafety says is a “crisis affecting school communities across Australia”.

It says tools, especially ‘nudify’ apps, are increasingly being used among young people, and can be exploited to generate non-consensual synthetic explicit images, including of children.

The technology can be free, fast and easy to use — and can cause deep personal harm.

“With just one photo, these apps can nudify the image with the power of AI in seconds,” Inman Grant said.

“Alarmingly, we have seen these apps used to humiliate, bully and sexually extort children in the school yard and beyond. There have also been reports that some of these images have been traded among school children in exchange for money.”

A ‘normalisation’ of creating deepfakes

Asher Flynn — a criminology professor at Monash University, who specialises in AI-facilitated abuse — said the rise in reports to eSafety is “confronting, but not unexpected”.

“We have seen a proliferation of ‘user-friendly’ deepfake creation tools emerging online, and we have also seen a normalisation of creating deepfake content,” she told SBS News.

Flynn’s research also shows an increase in sexualised deepfake abuse, with similar impacts to other forms of technology-facilitated sexual violence. This can include causing physical, psychological, social, reputational and financial harms to victims.

“We are also seeing a range of motivations for the incidents, from sexual gratification, to intentionally causing harm, controlling or degrading the target of the image, right through to thinking it’s funny, building social status among peers, and curiosity in how the process of creating a deepfake works,” she said.

Shahriar Kaisar, a senior lecturer of information systems from RMIT University — who specialises in the use of generative AI — said deepfakes have become a “severe” issue over the last couple of years with the rise of generative AI.

“It has become quite readily available to everyone,” he told SBS News.

“Creating deepfakes has become so much easier — and the way the images propagate online is with very rapid speed.”

When it comes to young people, he warned this can happen online or in school communities, such as messaging apps, “where we have even more limited visibility of what they are sharing among themselves”.

‘We will not hesitate to take regulatory action’

Inman Grant said the agency has been engaging with police, app makers and their host platforms to “put them on notice”.

Laws designed to require global tech giants to tackle harmful online content, including deepfakes and ‘nudify’ apps, were recently introduced to parliament.

She said mandatory standards, which carry penalties of up to $49.5 million for companies that breach them, come into full effect this week.

“We will not hesitate to take regulatory action,” she said.

Flynn said a multifaceted response is required. In addition to holding platforms and creators to account, she said this includes education and awareness raising, and prevention resources to help shift “problematic gendered norms and expectations to encourage a culture that does not condone this type of behaviour”.

Legal responses must also be in place to ensure there are consequences for perpetration — along with responses to help young people understand the harmful behaviour, Flynn said.

Kaisar also supports a “holistic approach”.

“We are working on the regulation and it was great the bill was passed last year,” he said.

“But the more important thing would be raising awareness and an ethical understanding of technology among school kids.”

eSafety has released an updated toolkit for schools on how to prepare for and manage deepfake incidents.

It is strongly encouraged that schools report any potential criminal offence to the local police.

“I’m calling on schools to report allegations of a criminal nature, including deepfake abuse of underage students, to police and to make sure their communities are aware that eSafety is on standby to remove this material quickly,” Inman Grant said.

“It is clear from what is already in the public domain, and from what we are hearing directly from the education sector, that this is not always happening.”

If you or someone you know is impacted by family and domestic violence, call 1800RESPECT on 1800 737 732, text 0458 737 732, or visit 1800RESPECT.org.au. In an emergency, call 000.