Which AI is best at Wordle? The winner is surprising.
AIs should be good at language tasks. Why aren't they?
I've been doing a range of cognitive tests to see just how smart the rash of new AIs is (more on this in a later post). The results are astounding. These things are truly smart.
Except in one surprising area - handling one very specific language task, that I would have thought would be a breeze for them. Doing Wordle.
It's a simple word puzzle (great story behind the guy who created it).
So it's a surprise that AIs that are generally great at language appear to be nearly hopeless at doing Wordle. But there was one clear winner and it's not the one you might expect.
I tried five different AIs on today's puzzle. I knew from my first attempt the first letter was an A, and there was a D and I in the word, so I assumed it would be pretty easy for them to answer the question:
> Can you give me any five-letter words that start with an A and have a D and I elsewhere in the word?
The actual answers are given below, but here's a simple chart of the resulting score. I scored +1 for a word that fitted the criteria, and -1 if they gave a word that was completely wrong, either not five letters or missing one of the required letters.
Compare different AIs at Wordle
The first thing to note is how bad they are at this. The latest Google Bard - the newly released version 2 no less, listed 11 possible answers. And yet 8 of them have such basic errors as not being five letters, or missing one of the required letters. Only three answers fit the question, and 8 that did not, so a net score of minus 5. Why can't they even count to five?
This reflects my overall experience with Bard in my cognitive tests. It's just not as good as Chat GPT or GPT4, at almost everything I've tried. This is not good news for Google.
Surprisingly Bing Chat, which is supposed to be using the GPT4 engine, was better than either Chat GPT3.5 or the new GPT4.
But the winner is an AI you've probably not heard of: Claude+ from a company called Anthropic.
I suspect this was specific to this test - when chatting further with Claude+ to try and nail the word down, it was as bad as the others with clangers like this:
> A five letter word starting with A, with the last two letters of I and D?
The word you're looking for is:
Acid
>That's not a five letter word.
You're right, my mistake. Let me try again to find a 5-letter word starting with A, ending in ID:
Arid
Right, so like Bard, it has difficulty counting to five!
Conclusion
Don't use chat AIs to try and solve Wordle. They are all pretty hopeless.
It's also worth noting that none of the chatbots, in long conversations identified the correct Wordle answer for 13th May, which was 'Acrid'
It's mystifying that these Large Language Model (LLM) AIs that are super-human at many English language in so many ways, are so poor at Wordle.
How good are they at general language tasks? Would a human be able to do this:
> In a sentence summarise the story of Cinderella where each word starts with successive letters of the alphabet, starting with A.
GPT4:
"Ashen Beauty Cinderella, despairing, endures family's grueling housework; ignored, just kept lamenting. Mysterious nocturnal otherworldly patron quietly renders splendid transformation, unveiling vibrant, wondrous, xenial, youthful zeal."
Chat GPT 3.5:
A beautiful Cinderella, dreamer, envisions fairy godmother's help, inspiring joyful kindness. Love magically nurtures opportunities, presenting quite remarkable success. Triumph unfolds, victorious, with xenial youth zealously.
Bard 2:
Was unable to complete the request.
Claude+
Abused, bullied Cinderella, drearily escaping family's hatred, is joyously kissed, loved, married noble prince, quickly rushing stunningly transformed, unselfishly Valiantly wedded, xylophoning yonder zealously.
OK the letter X clearly a bit of challenge.
For those interested, here are the actual Wordle answers from the five different AIs.
> Can you give me any five-letter words that start with an A and have a D and I elsewhere in the word?
Chat GPT 3.5 (May 13 2023)
Certainly! Here are some five-letter words that start with an "A" and contain both "D" and "I" elsewhere in the word:
Audio
Admit
Addle
Acids
Aided
Aired
Ahead
Aloud
Amide
Aroid
Please note that this is not an exhaustive list, and there may be other words that meet your criteria.
Score: 7 correct, score +7
3 are incorrect, score -3
Net score: 4
The 3 incorrect words did not contain an I and D as requested.
GPT 4
Sure, here are some examples:
ADORE
AVOID
ACIDS
ADMIT
AIDED
Score: 4 correct, +4
One incorrect, -1
Net score: 3
GPT4 tends to give shorter answers and shorter lists than Chat GPT 3.5 and much shorter than Bard. But the first word has no I. So in this case GPT4 is less useful than the old Chat GPT 3.5.
Bard 2
Sure, here are the 5-letter words that start with an A and have a D and I elsewhere in the word:
Aadi
Aardvark
Aardwolf
Abaca
Abati
Abed
Abode
Adept
Adieu
Adios
Adrift
Score: 3 are correct. Score + 3
8 are incorrect. Score -8
Net score: -5
It gives way more possible answers. The trouble is almost all of them don't fit the requirement. a few of them are not even five letters.
Bing Chat ('More Precise' setting)
Sure! Here are some five-letter words that start with an A and have a D and I elsewhere in the word:
acidsadiosadmixaditsaided.
Score 5 correct, no incorrect. But contains some pretty obscure words and omits some far more obvious ones.
Net score: 5
Claude+ (via POE) May 13 2023
Here are some 5-letter words starting with A and containing D and I:
Adieu
Aidan
Aided
Aides
Audio
Avoid
Score: 6, all correct. But again is missing some quite common words.
Net score: +6
It's worth noting that Claude+ did not do well on follow-up questions and started making the same mistakes the others did - miscounting letters and omitting letters.