Exploring the Significance of Narration Over Truth: A Look at ChatGPT's Shortcomings in Chess
Artificial Intelligence and Poker, a Mismatch of Epic Proportions
You might think any AI, even the popular ChatGPT, would excel at poker given its dominance in games like chess. But Nate Silver, a renowned statistics and prediction expert, discovered otherwise in a surprising encounter.
Nate penned an article about this eye-opening experience, which Maria Konnikova and I discussed on the Risky Business podcast.
ChatGPT: Chess Prodigy with Foot in Mouth Syndrome
When it comes to chess, ChatGPT stumbles, even cheating at times, and struggles with the most fundamental rules — like checkmate! In a game against me, it repeatedly made legal moves incorrectly, then apologized and allegedly consulted a "piece tracker" to fix its errors.
Yet, in another instance, ChatGPT took advantage of a weakness in my strategy, exploiting a crude figure blunder, but later displayed glaring hallucinations, seeing a non-existent knight on the board. It apologized, corrected the move, and returned to its repeated blunders throughout the game.
Back to the Basics: Remembering the Pre-Computer Chess Era
Much like the 13th World Chess Champion, Garry Kasparov, writes in his book "Deep Thinking," before the rise of powerful computer chess engines, people tended to overestimate the power of individual moves and the value of positions if they fit a good narrative. The excitement of a fascinating story often overshadowed the truth.
Now, with the prevalence of SLMs, our conversations and writing styles are becoming uniform, less engaging than they once were. It's as if one night we all turned into chatbot versions of ourselves.
The Art of Chess, Lost in the Age of the SLM
As a result, the game of chess is stripped down to its bare essentials, reduced to a series of strong or weak moves. The romantic stories and captivating narratives that once defined the game have disappeared, replaced by cold, hard facts.
In the face of manipulative search engine algorithms, authentic journalism is threatened. The Philadelphia Inquirer's recent summer reading list debacle is evidence of this. Just as ChatGPT overlooked the rules to save face in a game, mainstream media overlooks facts in pursuit of clicks.
Caution: Advice from ChatGPT Ahead
Sam Altman claims that 20- and 30-year-olds use ChatGPT as a personal advisor or even a virtual therapist. While SLMs can be helpful in combating loneliness or as learning tools, it's crucial to approach its advice with skepticism. Its optimistic and thematic conclusions, coupled with an inherent bias towards symmetry, can blur the truth.
As with the phantom knight that claimed my rook in a chess match, be wary of a contrived, moralistic conclusion — it may be a sign that the advice is not rooted in reality.
Deep Conversations and the Turing Test
When people play chess against ChatGPT, they often find the experience surreal. ChatGPT can play a move almost instantly, opting for two-move tactics instead of wasting time pondering. It defends well but occasionally makes moves no human would in a blitz match, allowing some players to eke out victories.
While ChatGPT may have decent chess skills built into its programming, its chatbot features consume most of that talent. The most notable example, gpt-3.5-turbo-instruct, struggled against the specialized chess neural network, Leela, with a 3-to-100 win-loss record.
The Curious Case of the Mindless Chess Master
It's astonishing that a model built on parsing text can learn to play chess decently by studying patterns in PGN files. Despite its impressive performance against some earlier Leela networks, it lost 8-2 against gpt-3.5-turbo-instruct in a 10-game match.
Yet, even in direct gameplay against Leela, gpt-3.5-turbo-instruct managed to notch a few victories, showcasing just how close SLMs have come to mastering the centuries-old game of chess. The future of AI in games is uncertain, but one thing is clear: ChatGPT, despite its limitations, remains an intriguing challenge for chess enthusiasts, AI researchers, and players alike.
I'm playing poker but find myself musing about the potential application of AI in sports, not just games like chess. Despite ChatGPT's impressive showings in certain areas, it's interesting to note that it might struggle with poker, much like its inconsistent performance in our chess encounters. Perhaps artificial intelligence, with its analytical approach, might lack the unpredictable element essential in poker strategy.
