ChatGPT flaw means it can’t win at Wordle, other word games

Artificial intelligence is about to turn the world upside down — but don’t ask it to play a game of Wordle.

ChatGPT’s “performance on these [style] puzzles was surprisingly poor,” according to computing science professor Michael G. Madden, who tested the most recent version of the AI — one posed to reshape industries and take jobs — against the popular vocabulary guessing game.

“You might expect word games to be a piece of cake. ChatGPT-4 [the latest version of the tool] was trained on about 500 billion words: all of Wikipedia, all public-domain books, huge volumes of scientific articles, and text from many websites,” Madden, based at Ireland’s University of Galway, wrote in an article on The Conversation.

But feed it the entire Library of Congress and that still wouldn’t improve its chances of winning — at least not for now, according to Madden.

That’s because ChatGPT’s immense neural network uses a complicated mathematical function that that maps its inputs and outputs. In order for the program to function, these inputs and outputs must be numbers.

“Since ChatGPT-4 works with words, these must be “translated” to numbers for the neural network to work with them,” Madden wrote.

They are converted into a program called a tokenizer, which ChatGPT’s parent company OpenAI describes as “common sequences of characters found in text,” according to the professor.

The models understand the statistical relationships between these tokens, and excel at producing the next token in a sequence of tokens.

But Madden has found a flaw in the system — the one that ruins its chances of winning a simple word game.

“A word such as “friend” has a token ID of 6756, so a word such as “friendship” is broken down into the tokens ‘friend’ and ‘ship.’ These are represented as the identifiers 6756 and 6729,” he wrote.

“When the user enters a question, the words are translated into numbers before ChatGPT-4 even starts processing the request. The deep neural network does not have access to the words as text, so it cannot really reason about the letters.”

In essence, you’re on your own trying to guess today’s Wordle — for now.

“There are two ways [to] overcome this,” Madden wrote. “First, ChatGPT-4 knows the first letter of every word, so its training data could be augmented to include mappings of every letter position within every word in its dictionary.”

Or, he said, future large language models like ChatGPT could also be programmed to generate specific code for games like Wordle.

“The second is a more exciting and general solution.”