A developer who worked on Phoenix Wright: Ace Attorney has commented on recent research that pitted various AI models against the classic courtroom series’ first installment.
As reported by Automaton, Hao AI Lab used Ace Attorney to try and benchmark four AI models’ memory, reasoning skills, visual understanding and strategic decision-making – with some performing better than others.
While none of the AI models managed to beat the game, Google Gemini and Open AI’s o1 managed to reason their way through to its penultimate episode.
“This index uses Ace Attorney to evaluate the AI’s practical ability to ‘find inconsistencies in testimony, select appropriate evidence to support them, and refute them most effectively’,” AI researcher K Ishi wrote. “As a result, the best ‘lawyer’ was o1.”
The research was designed to test each AI’s ability to consider “the entire flow of the case and make decisions without being limited to current testimony”, Ishi added, as well as having the “ability to consider the timing of ‘meaningful’ situations and create the most effective strategy”.
Each AI’s understanding would also have to remain flexible enough to react to fresh evidence coming to light mid-trial, he continued. You can watch each of the four AI models tested play the game via the video in the embed below.
“How should I put this, I never thought the game I worked on so desperately 25 years ago would come to be used in this way, and overseas at that (laughs),” wrote Ace Attorney developer Masakazu Sugimori, who composed the game’s soundtrack and voiced antagonist Manfred von Karma.
“I find it interesting how the AI models get stumped in the first episode,” Sugimori continued, in response to seeing the research. “[Ace Attorney director Shu] Takumi and [executive producer Shinji] Mikami were very particular about the difficulty level of Episode 1 – it’s supposed to be simple for a human. Maybe this kind of deductive power is the strength of humans?”
“The reason why Takumi and Mikami were so particular about balancing the difficulty level of Ace Attorney’s first episode was because ‘there was no other game like it in the world at the time’,” Sugimori added.
“It had to be a difficulty that would be acceptable to a wide playerbase, but it had to avoid being insultingly simple too. They were going for the kind of difficulty that gives you a sense of satisfaction when the solution hits you.”
And as far as we know, no AI has been programmed to feel that same sense of satisfaction – yet. Come back in five years, though, and it’ll be interesting to see whether the AI models of 2030 can beat the game.
Eurogamer recently visited Microsoft to see more of Muse, its generative AI game tool everyone hates. What’s really going on and why is it being marketed at Xbox fans? We took a look.