Can You Tell if You're Playing a Bot?
Introduction
You sit down at the chessboard. But who is seated across from you?
From the days of fraudulent automatons, concealing sly masters rudely crammed in claustrophobic boxes, chess players work to distinguish human and inhuman opponents. The Turk's history of fooling even the brightest minds shows that this challenge isn't easy.
Such deception hasn't been left in the past. Today, advancements in neural networks, move-matching and other alliterative annoyances make differentiation more confusing. The anonymity of online websites adds another layer of complexity. With the click of a button we lock ourselves into a five-minute death match with anyone... or anything. Your opponent may be a six-year-old on their mom's iPad, or a middle-aged man named Steve peering at his phone, escaping the world holed away in his workplace's bathroom stall.
I AM STEVE!
-Steve, probably
But what if Steve was never real... could it instead be a bot, carefully designed to imitate Steve's moves and mistakes? Moreover, can we really tell apart games played with a human touch from those of our silicon counterparts?
In this blog, I seek answers through profoundly unprofessional experiment. We'll find out together if man and machine can be separated by mere logic.
Setup and Results
I tasked four daring participants with playing a series of games, obviously keeping each of them in the dark about my true intentions, like an evil mastermind. Afterwards I revealed (with a cartoonish internal MWAHAHAHAHA) that one of their opponents was a bot.
I made them guess the impostor, just like a Turing test. Or like the hit game Among Us, if you prefer that analogy. After faking resignations as impostors fake tasks, and sabotaging my participants' plans of having a normal day, I called the emergency meeting.
It was hard to gather participants for a secret psychological study while maintaining the lie that they were just playing casual games. Each misinformed recruit played three 5-minute games. Two were against humans, and one was against a bot.
Since the games needed to be anonymous, I had the fatiguing role of relaying moves back and forth between the bot and its human opponents. My toil as the middleman meant the bot folded like a newspaper at the onset of time pressure. I hoped that I could pass it off as just a slow human.
The next day, I dragged in two more unsuspecting players and repeated the procedure, but this time even more villainously. I camouflaged myself amongst them, and played while pretending to be a bot, to further mess with my poor participants.
I'm sure you guys are all dying to know how many guessed correctly and incorrectly. The whole blog sort of hinges on this one figure:
Correct Guesses - 73%
Incorrect Guesses - 27%
I had a low sample size of just five people, but nevertheless, I was surprised at how many guesses were incorrect. Before the experiment, I assumed that stronger and more experienced players would identify bots more accurately, but we had participants as high-rated as 2100 messing up.
Interestingly, numerous participants guessed incorrectly, not only because the bot was exceptionally human-like, but more so because their human opponents seemed like bots! I also saw a lot of prejudice towards fellow humans: "that was such a good move, no way a human could've played it." Bots only play good moves, or so we thought...
Meet the Bot: Maia
An ordinary bot could not withstand the pressure of imitating a mighty human. Famous bots like Stockfish, who far outgun us in skill, practically scream "I'M A ROBOOOOT" with each disgustingly precise move.
To properly trick my participants, I needed to use a bot that made errors. Not just random evaluation-dropping aberrations, but authentic human errors: realistic tactical oversights and common misguided plans. I needed to use the most ‘human’ bot: Maia.
Maia is a neural network, like Leela, but instead of playing against itself a mind-numbing amount of times, it gets trained on the games of mere mortals. This process produces exceptionally humanlike play. Unlike traditional engines, which crave victory at all costs, its objective is to predict what move a human would play.
Maia is perfectly cast for a chess Turing test. While it doesn't have the raw power of top level engines, its predictive capabilities are more important. Maia strives to play moves that we would. And, as we saw in the results, it certainly delivers on this promise.
Games
While haranguing participants about appropriate game times and organizing data in a spreadsheet was truly enthralling, I also found a little excitement in the games played throughout the experiment. I'll highlight six moments that I felt were particularly funny or interesting.
My Mouse's Betrayal
Since all the games were anonymous, I couldn't just pair a participant with a bot and sit back munching on popcorn. Instead I suffered, relaying moves back and forth between the bot and its opponent. The high-speed nature of blitz and my floppy mouse skills resulted in several unfortunate slips.
My mouse zigged and zagged across the screen in a frenzy to relay moves quickly to the bot. After the first game passed with no major bumps in the road, I drove with one hand on my steering wheel, taking my laborious task more lightly. The punishment for my hubris was swift and brutal. As soon as game two, I grazed across the wrong piece and disaster struck.
I made a mouse slip: Ke7.
My soul left my body. I stared at my screen in pure shock, crushed beneath the weight of my actions. Did I really just do that?
As you should always do in the midst of a chess game, I began to panic. The world unfroze, to my dismay, and the Bongcloudian error stared me down. After immediately resigning against the bot, I tried sending it a rematch offer and replaying the same moves, but the bot mercilessly deviated.
I resigned and rematched again. And again and again. It didn't work. Finally, after what seemed like hours of tech support, I remembered I could challenge the bot from a custom position. I hurriedly pasted in the FEN, and got a second chance to play the correct Be7. By then, I had burned a whole minute in a blitz game over a trivial developing move.
Interestingly, @outwittedyou spent a full minute on his next move as well, returning the favor, and also guessed the bot's identity incorrectly. Perhaps he felt that long thinks in odd positions was a human characteristic. Either way, if I ever repeat this experiment, I'll find a way to connect the bot directly instead of fumbling with my mouse and potentially ruining entire games.
The Contemplation
I took on some participants while pretending to be a bot. My first victim: @Anonymousplayer_796.
I felt the above was an Oscar-worthy performance, while he feels that it's his new highest-rated blitz win. In my defense, dude... it was an unrated anonymous game where I was imitating a weak bot (all my mistakes were definitely on purpose). But if it brings you guys joy, I'm happy be an ego charity with my inflated blitz rating.
After the other games were finished, I posed to him the ultimate question: Who is the bot? His instinct seemed to point towards his first opponent, @ALondonInVienna. He said the following (note that Game 3 was the bot):
Game 3 was definitely not the bot.
-Anonymousplayer_796
Excited beyond measure by even more incorrectness, I diligently logged it in my spreadsheet, joyfully highlighting his guess in bright red. But I didn't leave him be. I couldn't take his answer for an answer. Instead, I pressed him for some deeper explanation, trying to squeeze out more eloquence for the blog's direct quotes. With hands nearly trembling from praying on my participants' downfalls, I typed out two follow-up messages:
Apparently, this prompted @Anonymousplayer_796 to think about the situation even more deeply. I understand why... looking back, the message does indeed seem to imply, "please fortify your wrong answer so I can publish your failure". He requested some extra time to ponder the situation, promising to return later with a revised reply. And in mere hours, he re-emerged with a brilliant new hypothesis:
Game 3 was the imposter because the moves were really good... the opponent was up a piece and lost on time while the first two games I won was by resignation... bots do not resign.
-Anonymousplayer_796
The Game 3 he spoke of came out of the Grunfeld, where White (Maia) played a quiet e3 system. The middlegame featured Black ruthlessly ganging up on White's isolated d-pawn, as White defended it with his life (and his pieces). Later on, White managed to push that pawn forward to d6. Black captured it, failing to notice that doing so hung a piece. The position was simplified into a winning endgame for White in which Black won by timeout, despite my fastest relaying yet.
How dare he realize that the bot game was played like a bot? My joy melted away into despair and regret. Why did I ask him to explain further? Later, I even had the indecency to discourage his correction, 'casually' mentioning that I resigned on behalf of the bot in some other games. However, he had the right answer and sadly could not be budged from it. Jokes aside, good job, @Anonymousplayer_796!
Checkmate in One
One of Maia's strangest habits soon emerged, and was noticed by both me and the participants: it would hang the simplest of checkmates, even while having huge advantages. One participant, @VOB96, saw this as reason to assume it was human:
[Allowing me to play] mate in one was weird. A bot, even a weak one, would probably try to survive longer.
-VOB96
This would've been a correct assumption for a less humanlike bot, such as weaker versions of Stockfish, since they are often coded to delay checkmate for as long as possible. However, Maia is not afraid to make even the most grievous errors, and will happily hang mate-in-one if it thinks humans would be stupid enough to do so as well.
Another participant, @crspychkn122, had an opposite viewpoint towards the same issue. In his game, the bot allowed checkmate to take place while up a full queen!
The above game was the first played by our protagonist @crspychkn122. While relaying moves against him, I mouse-slipped again, this time in the actual game. I was forced to resign, and begin a new game with him. Though I feared that he'd consider the whole situation suspicious, his detective work focused on the moves:
I think the first opponent was a bot because I felt as if a very nice tactic was played at the beginning of the game and was followed up by hanging an easy mate.
-crspychkn122
@crspychkn122's guess was correct and his logic is too.
The version of Maia used was trained on the games of players rated about 1500-1600. Having been stuck at this exact elo for many months in the distant past, I can reassure you guys that nobody at such a high level allows simple mates like that. I can't help but feel this is some sort of bug in the algorithm, or at least a dead giveaway for experienced Maia hunters.
In the very next game of the study, these two players with conflicting perspectives were paired with each other, resulting in an intriguing game:
Quiet maneuvers in the Slow Italian quickly transmogrified to a full-blown kingside attack for White. Black's queenside counterplay never materialized, and instead a tactic was hung allowing White the win of a queen. From there it was simplifications, as the middlegame faded into the endgame. Despite Black's surprising resilience, there was little to be done against the mighty White queen, and after another was made through pawn promotion, checkmate arrived.
I got destroyed by @VOB96... she played very well.
-crspychkn122
The Bot Whisperer
One participant stood out to me for their correct guess, full confidence and lucid explanation: @nova-stone.
Unlike others who hesitated or changed their answer, Nova got it right on the first try. As an instigating organizer who shamelessly roots against all participants, his guessing skills irked me very much.
His game against Maia began with the ponderous English Opening. Black played aggressively and swiped a pawn early on, but later played some risky moves with the pawns in front of the king. These advances backfired as White smashed open the kingside with two pawn breaks: f4 and e5. Black's position came crashing down and resignation followed after a queen hang.
After the game, this is what Nova said:
The second one struck me as quite bot-like, mostly because of the automatic recaptures and the fact that they didn't speed up when the time was running out. Additionally, this opponent spent a long time pondering before taking white's rook on move 28 and then resigned within two seconds after I took the queen.
- nova-stone
Unfortunately, the bot's failure to speed up in time pressure was entirely my fault. As the move relayer, I was slowing Maia down with my sluggish reaction time and mouse speed, leading its inhuman identity to eventually be figured out. After all, it's human nature to panic and throw out random moves with thirty seconds left on the clock.
Apparently, Nova is so familiar with robot play that his opponent, @outwittedyou, guessed he was a robot himself! The latter has earned the following, less flattering title:
The Not Whisperer
One of the shortest games throughout the whole study came from an Exchange Caro-Kann. @outwittedyou was far outmatched by his opponent, @nova-stone, who was over double his elo. On just move seventeen, the first player hung a full queen and promptly resigned.
In his answer, @outwittedyou decided his above opponent was a robot. Then, to add insult to injury, he guessed his opponent's rating to be 1500.
No human would see 15...Rc8 in 5 seconds. It's a good move, but it's not one somebody would instinctively play.
-outwittedyou
I disagree, actually. I think Rc8 is the most natural move in that position. From doing nothing on the a8-square it moves to the only open file available and helps to control the c4- and c2-squares. Also, I'm really not sure how such a clean and quick win could've been executed by a 1500 player. If a 1500 played like that against me, I would begin mumbling something about a procedure...
(I apologize for the slander throughout this section — towards both @outwittedyou and the Caro-Kann)
Revenge of the Robots
Throughout this experiment, our poor hero Maia has been crushed left and right, like a baseball getting whacked repeatedly for the sake of sport and entertainment. However, sometimes a baseball zooms into the stands to hit an unsuspecting spectator square in the face, and sometimes our robot does achieve victory. Maia won one game throughout this challenge, and it came against @ALondonInVienna:
Just a few moves deep into Maia's odd system opening, @ALondonInVienna plays one bad move and takes a neural-net fastball to the noggin. Despite the horrible position and material deficit, I still think this resignation was premature, especially because @ALondonInVienna is quite a good player. Had he not thrown in the towel, I would've honestly bet on him to pull off the comeback!
Humans and Bots
In a traditional Turing test, chatbots fool humans using their conversational skills. A human speaks with someone else and has to determine if the other side of the conversation is man or machine. The machine passes the test if the humans can't tell them apart. A popular online Turing game is playable at www.humanornot.ai.
In my opinion, chess is a great setting for a Turing test. Our game is finite, with a set of rules enclosing its play, and also focuses on logic, which machines are usually great at. Despite that, each game played by humans is imbued with emotion. Our mood and feelings are reflected in every move. For example, when I'm calm, I tend to play solidly, and when I'm confident, I go forth with attacks with reckless abandon.
For chess players, the Turing test poses interesting questions about whether humans truly have distinctive styles that can't be copied. As we saw in the above games, Maia is great at predicting human moves. In fact, if we were handed only games with no context, and had to differentiate between humans and bots (foreshadowing), I predict we'd get it wrong more often than we think.
Maia's weakness, though, is playing chess like a human. Similar to how a chatbot can string together the correct words but speak with an imperfect (even unsettling) tone, Maia still can't quite grasp behavioral patterns in online chess. Maia can't panic in time pressure or make wrong premoves or throw the kitchen sink at the opponent as counterplay in a losing position. It can't even resign or offer draws!
In the study, the most effective method of sleuthing involved these off-the-board elements. For example, @Anonymousplayer_796 noticed that Maia couldn't resign, and @nova-stone described Maia's inability to play in time pressure.
I think in the not-so-distant future, chess bots will consistently pass the Turing test against experienced human players. Maia is just a couple code tweaks away from being exponentially harder to detect. Imagine being offered a draw mid-game, giving you clear intel on your opponent's humanness, only to guess incorrectly and find out the hard way of some new Maia update!
Through the Turing test, we searched for cold computer logic, but instead stumbled upon human nature. We learned what Maia was built to teach: to be human is to be imperfect... as @outwittedyou expertly demonstrated.
Interactive Guessing
Now, it's your turn.
Reading about the difference in human and bot chess, and playing through the example games, you should have a sharper eye for silicon strategies by now. You have the chance to test it out and compare your guessing skills to those of the participants.
Below you'll find three carefully chosen games, and your task is to guess which player is human and which is a bot. Good luck...
Game #1
The first game is staged in the mainline of the Bogo-Indian Defense, featuring an interesting bishop retreat. White manages to thwart some of Black's aggressive plans, and the position is simplified until both sides only have a pair of rooks and a handful of pawns. Black successfully holds the rook endgame despite a nagging White edge.
Lock in your guesses now...
Player 1 is a...
.
Good choices! Let's keep going.
Game #2
In game two we wade over to the theoretical and complicated waters of the English Attack in the Najdorf Sicilian. Opposite side castling, a legendary declaration of war, brews a series of rook-pawn pushes. Both sides spend tempi on careful defensive measures, and the continuous piece trades slow down Black's attack, while their opponent marches on unperturbed. White's daring h-pawn opens up Black's king, eventually leading the second player to throw in the towel in the face of imminent checkmate.
No more tricks! Ready to check your answer?
.
You're doing great, one more game!
Game #3
For our final destination, we return to the Open Games, and see more opposite side castling. However, this time both sides' attacks fizzle out quickly due to rapid simplifications. Black errs by placing all his pieces in one corner passively, allowing White the chance to decisively infiltrate with major pieces. A pawn march and eventual promotion delivers White a winning material advantage and later checkmate.
This time, I promise, the answers are not the exact same!
.
Thanks for playing!
Conclusion
Computers in chess have been labelled, often by angry old men and gambit players, as harbingers of doom. Amateurs and grandmasters alike uncork brilliancies only for engines to uncaringly point out the tiniest of errors, staining beautiful sacrifices and ingenious combinations.
Nowadays, kibitzers intently watch calculated black-and-white bars slide back and forth to determine a positions evaluation: “Wow, that GM is so dumb — his last move had a centipawn loss of 143!” Advancements in computer strength have also introduced the plague of engine-assisted cheating and removed our beloved adjournments.
Most computers coldly crunch numbers, with powerful algorithms designed to find the best possible move in any position. Maia doesn’t strive for perfection. It emulates human play, and as we saw from numerous participants mistaking it for a human opponent, it’s highly successful. Maia exposes the predictability beneath our supposed creativity, and perhaps that makes it the most unsettling bot of all.
Comment your score on the interactive section below! I will reward any 6/6 scorers with my prestigious No-Prize, as inspired by the great Stan Lee.
I know this post was a bit shorter, but you guys deserve a breather after my previous mammoth of a blog. Thanks for reading, and see you in the next blog (part 2 perhaps!) 
