Content warning: suicide, grooming.
An AI-powered tool promises to detect signs of grooming in voice chat in near real-time, enabling moderators to intervene before situations can escalate. The tool, named Amanda, has been developed by the Norwegian company Aiba, and it’s being made available through the Odin voice chat platform from German firm 4Players.
Hege Tokerud, CEO and co-founder of Aiba, tells GamesIndustry.biz that Amanda picks up more than just signs of grooming. “It’s grooming, bullying, radicalisation, all kinds of social manipulation,” she says.
The tool was named after a Canadian girl called Amanda Todd. “She was groomed 15 years ago: she met the boy online, and it ended tragically,” explains Tokerud. “She took her own life. But that was the reason why we started working on this. That was because of Amanda Todd.”
Tokerud says Amanda’s mother is supportive of the project. “I’ve been chatting with her mother about this. She’s supporting what we’re doing, and she’s also a voice in child safety in Canada as well.”
Aiba’s system has emerged at a time when increasing attention is being placed on online safety by lawmakers. For example, in the European Union, the 2022 Digital Services Act outlined rules for protecting online users from illegal or harmful content, and earlier this year, Dr Andreas Lober from the law firm ADVANT Beiten told GamesIndustry.biz that he expects “enforcement of the DSA against gaming companies to become increasingly likely in 2026.”
Meanwhile, the UK has introduced the Online Safety Act, which demands that online services have “a higher standard of protection for children,” while the European ratings board PEGI has said that games with completely unrestricted online communication will be rated 18, among other changes. In the US, Roblox recently paid more than $12 million to the state of Nevada as part of a settlement that includes additional protections for child safety, and the company is currently facing over 140 lawsuits related to child safety concerns.
How it works
Aiba’s technology emerged from research at the Norwegian University of Science and Technology around ten years back. “Then we spun out as a company four years ago,” says Tokerud.
In addition to collaborating with researchers at the university, Aiba has been working with the Norwegian police. Tokerud says that the machine learning models that underpin Amanda have been trained on police records of cybergrooming cases, enabling the technology to detect early signs of grooming in online conversations.
Crucially, the technology takes into account the context of the conversation: Aiba claims that Amanda can identify harmful patterns even if individual sentences seem harmless. If the system picks up signs of harmful speech, the case is routed to a human moderator for checking. “Humans still need to be involved,” emphasises Tokerud.
She adds that the sensitivity of the system can be fine-tuned by customers. “Should we pick up on everything, or should we pick up on almost everything? And the more we want to pick up on, the more false negatives we will have, of course. But this has not been an issue, because there are humans that are looking into the [cases] and deciding if the AI is right or not right.”
Tokerud says Amanda is based on several AI models. “The first one automatically detects the language, and then we do different violation categories,” she says. “Is it grooming, radicalisation?” At the moment, Amanda supports nearly 40 languages, but Tokerud says “we can support over 100 languages without a problem.”
In practice, Odin Cortex, the intelligence layer behind Odin Voice, captures speech and transcribes it to enable Amanda to analyse the context, language, and nuances. Then the results are shared with moderators.
In order to comply with the strict rules of EU General Data Protection Regulation (GDPR), the voice data is encrypted, and only stored when harmful speech is detected. “We are completely GDPR compliant,” says Marco Hirt, head of business relationships at 4Players. “We as 4Players, we save no kind of data. What we have to do – to be fully transparent – we save IP addresses, but when the call is done, it’s deleted.”
Gathering interest
The anti-toxicity system comes at an extra financial cost – but Tokerud says it’s something that plenty of companies have shown interest in. “With the new regulations now … we see that more and more companies are actually looking for such solutions, so we see the interest is increasing. But also from the market perspective, [players] don’t want to be in a community where there are only toxic people. And we see they escape from that community and go somewhere else.”
Aiba isn’t the only tech company working on anti-toxicity and anti-grooming solutions. For example, last month, Snap, Meta, and Roblox confirmed to the UK regulator Ofcom that they will bring in new safety measures designed to better protect children from online strangers.
Tokerud says that 4Players and Aiba have been talking with a number of potential clients for their system. “We cannot talk openly about the latest, biggest ones we have,” she says, but adds that Aiba has been working with the Danish free-to-play online social game MovieStarPlanet for two years. “They are using our system and are very happy about it.”
She emphasises that the ability to monitor voice chat for signs of toxicity using AI isn’t a work in progress or a future dream. “This is something that’s working now. And the problem with online toxicity, that’s something that we can solve today.”
