Sony is working on a prototype AI version of at least one of its PlayStation game characters. An anonymous whistleblower shared an internal video from Sony’s PlayStation team with The Verge that shows off an AI version of Aloy from Horizon Forbidden West. After we published the story, the video was removed from YouTube due to a copyright infringement lawsuit from Muso, a copyright enforcement company that advertises Sony Interactive Entertainment (aka PlayStation) as a client.
The video is narrated by Sharvin Raghobardajal, a software development director at Sony Interactive Entertainment who works on video game technology, artificial intelligence, computer vision, and facial recognition technology for Sony’s PlayStation Studios Advanced Technology Group. We watched Raghobardajal demonstrate an AI version of Aloy that can converse with the player using voice prompts during gameplay.
Aloy responded to requests with a synthesized voice and facial movements powered by AI in both the demo and the full version of Horizon Forbidden West. Raghoebardajal makes it clear that this is just a prototype that was developed in conjunction with Guerrilla Games to demonstrate the technology internally at Sony.
The technology demo uses OpenAI’s Whisper to convert speech to text, as well as GPT-4 and Llama 3 for conversation and decision-making. According to Raghoebardajal, Sony has its own internal Emotional Voice Synthesis (EVS) system that it uses to generate speech, and the facial animations are based on Sony’s own Mockingbird technology.
While the demo can be seen on PC, Sony has also been experimenting with running parts of this technology directly on PS5 consoles with “little overhead,” according to Raghoebardajal. Sony first demonstrated the technology a year ago, then showed a more advanced version behind closed doors at the Sony Technology Exchange Fair (STEF) in Tokyo in November.
“This is just the first glimpse of what’s possible,” Raghoebardajal says. The tech demo doesn’t answer the more obvious questions of whether it makes sense to talk to Aloy when you play as her in Horizon Forbidden West, or what impact such technology could have on voice actors and game developers.
Nvidia is working on its own similar technology for AI NPCs in games, where you can freely converse with video game characters. Ace, as Nvidia calls it, was shown off several times throughout 2024, and the GPU maker even worked with Inworld AI to create Covert Protocol, a gaming demo of Nvidia’s AI technology in games.
Microsoft is also working closely with Inworld AI to eventually bring AI characters to Xbox, allowing game developers to use generative AI characters, storylines, and more. Microsoft has also created its own Muse AI model, which generates gameplay, originally designed for game developers to create a game environment for prototyping and ideation.
There are already concerns among game developers and studios about how AI could impact the creative process of video game creation, especially with so many layoffs happening in the gaming industry. Almost half (49%) of the 3,000 respondents to a survey conducted at the Game Developers Conference (GDC) 2024 said that generative AI tools are currently used in their workplace, and 31% said they use them personally.
Sony’s experiments with PlayStation AI characters are sure to spark new conversations about the role of AI in video game development ahead of next week’s GDC conference in San Francisco. We’ve repeatedly reached out to Sony for comment on the internal video, but the company hasn’t responded. We’ve reached out again to confirm whether Sony has removed the video and whether it has any further comment.
