eSafety would not disclose the number of reports it had received. However, while the “rapid rise” is cause for concern, Inman Grant warned the reality may be worse.

“Anecdotally, we have heard from school leaders and education sector representatives that deepfake incidents are occurring more frequently, particularly as children are easily able to access and misuse nudify apps in school settings.”

‘A crisis affecting school communities’

“Alarmingly, we have seen these apps used to humiliate, bully and sexually extort children in the school yard and beyond. There have also been reports that some of these images have been traded among school children in exchange for money.”

A ‘normalisation’ of creating deepfakes

“We are also seeing a range of motivations for the incidents, from sexual gratification, to intentionally causing harm, controlling or degrading the target of the image, right through to thinking it’s funny, building social status among peers, and curiosity in how the process of creating a deepfake works,” she said.

When it comes to young people, he warned this can happen online or in school communities, such as messaging apps, “where we have even more limited visibility of what they are sharing among themselves”.

‘We will not hesitate to take regulatory action’

“We will not hesitate to take regulatory action,” she said.

“But the more important thing would be raising awareness and an ethical understanding of technology among school kids.”