Are Chess.com Bots' Ratings Accurate?
In the second half of 2020, chess.com has introduced close to a hundred bots to their new version of “Vs Computer”. This army of droids has generated much buzz within the chess community. Some of the most frequently asked questions in the forums include “How accurate are the ratings of these bots?”, “Should I play the bots as part of my chess training?” and “Do the bots really have specific playing styles?”. I have personally played dozens of games to find the answers.
Chess.com bots are classified into three types:
- Standard bots which span across a wide variety of ratings, from 250 to 2450,
- Adaptive bots whose strength will change dynamically depending on the quality of the user’s play, and
- Celebrity bots which play like various chess supremos such as Hikaru Nakamura, Alexandra Botez, Robert Hess, and even Beth Harmon from The Queen’s Gambit.
I will review each type in a three-part series, starting with the standard bots in this article.
The Standard Bots
This makes up the bulk of the chess.com bots. There are 60 of them in total (15 Beginner, 15 Intermediate, 20 Advanced, 10 Master). Obviously, it would have been quite an onerous task to analyse each bot thoroughly. What I have decided to do instead was sample a few bots with interesting personality descriptions.
Within each rating range, I selected at least two bots that claim to have contrasting play styles as far as possible. I then compared the games and assessed if there was indeed a difference. To gain a nuanced understanding of the bots, I deliberately altered my play style. For example, if a given bot was booked to the teeth, I would make non-theoretical moves very early in the game to see its response. At the end of the games, I also gave each a bot a rating estimate based on a 3+0 time control.
Beginner Bots (250 – 850 Rating)
Wayne (250) & Fabian (250)
Wayne and Fabian are two of the lowest rated bots on chess.com. Wayne’s description reveals his penchant for using his Queen whereas Fabian has no preference for any piece. I played a few games with them, and here is what happened.
Assessment
Wayne was only able to bring out his Queen when a common opening was played. For example, against e4, he would respond with e5/e6, followed by Qh4, Qg5 or Qf6, depending on which squares were protected. However, after taking him out of “book” with 1. g3, and 2. f4, he did not move his Queen for 10 full moves. Both the bots were capable of spotting mate in 1, whether in delivering or defending it.
The bots’ play was extremely far from a 250-rated human. They insanely went out of their way to blunder. For example, they ignored a hanging piece that could be taken in three different ways. At the same time, both bots could always stop an unforced mate in 1, which prevented quick and easy wins such as the scholar’s mate.
I would say the bots are only useful for players who have just learnt the rules and want to get a feel for playing and winning a full game.
Intermediate Bots (1000 – 1400 Rating)
Elena (1200) & Maria (1200)
Elena coaches an elementary school chess club. For this reason, I suspected she would play more principled chess compared to Maria, an equal rated opponent with an aggressive style. Let’s find out if this is really true.
Assessment
True to her form, Elena stuck to sound openings for common main lines in the first 6 moves or so. However, she played significantly worse as soon as she was out of book. On the other hand, Maria loved to gambit very early on in the game. Even in openings where there were no sound sacrifices, she readily gave up at least a pawn, creating very novel positions.
While they rewarded me when I set up good traps, some of their moves were clearly non-human. If you are rated between 900 and 1200 chess.com rapid rating, they would make good sparring partners. Play with both of them to have a feel for both principled and unprincipled chess styles.
Advanced Bots (1500 – 2100 Rating)
Pierre (1500) & Antonio (1500)
Pierre’s seems eager to push the f-pawn early while Antonio prides himself as a balanced player. Let’s check out how adventurous Pierre really is and whether both are worth their ratings.
Assessment
Pierre always played 1. f4 as White and played 1…f5 as Black only if he was in “book”. After pushing his f-pawn, he would often gambit a few pawns. As for Antonio, he was indeed a balanced player who played much sounder openings. Given that Pierre conceded such a huge advantage in the opening, I expected him to compensate in other areas of the game. However, that was not the case. Pierre failed to play any better in the middlegame and endgame than Antonio. On the whole, Pierre’s characteristic two pawn “sacrifice” would put his rating below his fellow 1500 bots.
Apart from Pierre’s dubious opening choice, the bots’ play was quite human. As training partners, Pierre makes a good opponent for 1200s – 1400s to practice converting a two-pawn advantage. Antonio, on the other hand, is suitable for anyone who wants a solid 1500 opponent.
Fatima (2000)
Fatima enjoys trading pieces quickly and setting the battlefield in the endgame. In most instances, rushing to trade pieces is a poor strategy. Hence, I was curious to see how an engine bent on trading pieces could play at a 2000 level.
Assessment
Fatima was capable of playing an extremely strong game. However, there is a quirk I could sometimes exploit to get a better score against her. All I needed was a good opening choice that did not allow easy trades, which frustrated her logic. Her willingness to trade at a loss gave me a moderate advantage in the endgame. For this reason, I would put Fatima at an 1800 rating. Furthermore, her reckless tendency to trade robs her of her “humanness”. She would, therefore, only be suitable for those who would like to flex their creative muscle in encouraging unfavourable trades.
Master Bots (2200 – 2450 Rating)
Noam (2200) & Nora (2200)
Unlike Noam, a veteran player who plays very solidly, Nora prefers sharp games. She has “learnt a lot from Magnus Carlsen’s tactics but hasn’t quite mastered his patience”. Let’s have a taste of their play styles and strength.
Assessment
Noam played classical and solid lines while Nora veered off theory with unsound Queen moves as early as move 2. The way Nora brought the Queen out early was identical to Wayne’s – Qh5, Qg4, Qf3 as White or Qh4, Qg5, Qf6 as Black. Beyond the opening, I could detect no difference between the two bots. On this count, Nora’s poor opening choice places her about 100 rating points below Noam.
Both their playing strengths were inconsistent across games, with a mix of excellently executed combinations and unacceptable blunders. All in all, Noam’s rating seemed to average at 2000 while Nora’s was around 1900. Still, they make good training partners for anyone who wants a “blind pairing” with opponents from 1600 to 2300.
Conclusion
Are the ratings accurate?
At the Beginner or Intermediate levels, the ratings are quite accurate. Nevertheless, the higher the purported rating of the bots, the more overrated they are. On top of that, bots powered by an “aggressive” opening must be discounted by another 100 – 200 rating points since they didn’t seem to make up for their initial folly during the middlegames and endgames. Again, this assumes that the bots’ game quality is based on 3+0 chess games.
Should I play the bots as part of my chess training?
While the bots should not completely replace your human opponents, they are certainly decent options if you cannot commit to a full game. If you wish to play the bots exclusively, it is best to select a mix of classical and caveman styled ones. This will give you the experience of different flavoured games.
Do the bots have specific or unique play styles?
Yes, but for the most part, their playing styles were limited to the initial part of the game. If you force the bots out of book with an unusual first move, their opening choices will not show any stylistic difference. One exception I found was Fatima, whose unique style of trading pieces swiftly was apparent even beyond the opening.
If you’d like bots that can adjust their styles according to your play, chess.com offers another line-up. The “Adaptive bots” which are programmed to vary in strength dynamically as the game progresses will be explored in the next segment of this three-part series. Feel free to follow my account and be notified when I publish the article entitled “Are Chess.com Adaptive Bots Truly Adaptive?”. Take care and have a happy new year!
