How to Catch a Chess Cheater

My longform profile of IM and computer science professor Ken Regan was posted today in front of the USCF Chess Life online paywall for all to read. Regan is the #1 authority for  how to catch a chess cheater. I profile Regan, the technical details behind his methods, some of the non-cheating implications of his work, and discuss a bit about how humans and computers think about chess positions.

http://www.uschess.org/content/view/12677/763/

That's quite a coincidence OneArrow. Look at my post earlier today. Haha.

http://www.chess.com/forum/view/general/an-interesting-article-from-the-uscf-website-that-yall-may-want-to-read

Was my description somewhere close to explaining it? I actually read about half of it. 

Thanks.  That was captivating.

An excellent article. 

The concept of Intrinsic Performance Rating (IPR) is captivating indeed. Imagine the day when we'll have at home a program capable of evaluating our own IPR... that would be a great aid in analysis and self-improvement. 

What a long and meandering article. It took 2600 words to get to the point! It then continued with 4000 more words of useless drivel.

Here is the essential thrust of the method. It assigns a performance rating to your moves based the difference in 'centipawns' between your moves and the computer's best moves. It then compares the performance rating with your current Elo rating. You are declared a cheater if your performance rating is too high.

The problem is the reliance on the current Elo rating. While this method would be useful against Borislav Ivanov, who had a low Elo rating, it would be useless in detecting cheating in the Kramnik vs Topalov match (which the article extensively discusses), where the players already have a high Elo rating.

<r_k_ting> it wouldn't be useless, as an IPR can be detected which is 400-500 higher than Kramnik and/or Topalov's. 

One of the points that made Ivanov suspicious was not so much that he beat 2600+ players - but that he CRUSHED them, beat them to a pulp, basically out of the opening, while giving them absolutely no chance, tactically and positionally just stomping on them and strangling them at the outset. 

Nobody beats 2600+ players like that. Only a strong engine does. 

At the 1975 World Open, Alan Trefler, a mere expert with a lowly 2045 rating, tied for first with GM Pal Benko.  He finished ahead of GM Nicolas Rossolimo, GM Walter Browne, and future GM Michael Rohde, beating the latter in the last round.

Man, it's a good thing Trefler had his dream tournament in 1975, when computers still played like beginners.  If he had done this today, he'd have been labelled a cheater for sure. 

The other problem I see with the statistical detail of the method is that it's actually very easy to fool. If this becomes the official cheat detection method, it is easily bypassed.

The curve fitting process discards much of the raw data collected by the method. In particular, it doesn't place sufficient emphasis on mistakes that actually lose the game. A smart cheater will simply select a suboptimal move so that their performance rating corresponds with their Elo rating. Sure you give away a few centipawns each move, but so long as you don't make any game losing moves and your opponent does, then you will win the game.

It's more than a few centipawns... a few centipawns won't lower your IPR that much. Play a lot of nonoptimum moves, you won't beat that player... you'll play closer to your actual strength, and get more or less your expected results - so where's the cheating?

But now wait a minute - wouldn't it be funny if you would be cheating with a device that during the game also tells you the IPR of the moves you'd use, and the "z" factor, or how far your IPR is from your rating - so that you can control not only the extent of your cheating, but also know in advance how detectable it would be... 

A nice long-term project of cheating. Say you're 1700 - so with this machine, you'll just play as an 1850. You don't care about your results in a single game - but you know that you'll get 1850ish results and won't be detected. 

If you play frequently enough, very soon you'll be rated 1750 - now play as an 1900, and so on... you will rise to the top, invites to top tourneys will start arriving...

That would be something! The IPR machine used by cheaters to "tweak" their cheating... :-)

In a few years, you'll have legions of "genuine" 3260 players, human mind you - who yet won't be able to answer simple chess questions while "unplugged", or do a decent post-mortem right up there with Kramnik, in front of the cameras...

<OBIT> and how did Trefler fare after that tourney? Did he remain a 2100-ish expert, or did it actually foreshadow a significant improvement in his play and results?

Another angle is that a 2100 player can get lucky and achieve 2500ish results in a tourney - despite having an IPR in the 2100 or 2200 area - just because maybe people blundered against him too much that week... 

<solskytz> Well, since you ask, Trefler had no noteworthy chess results after that.  However, he founded Pegasystems, Inc. and became a billionaire.

Four of Trefler's games from the 1975 World Open were published in the September, 1975, Chess Life & Review.  It might be fun to enter those games into a chess engine to see what the IPR would register.  

Well, maybe something did happen to him in that tournament then... he seems to have made some kind of a qualitative leap...

So yes, cheating would become a long term project rather than one of instant gratification, and would require mathematical as well as computing expertise. Ivanov would probably be excluded on both counts, but as a certain seven time consecutive winner shows, easy to evade methods won't deter all cheaters.

When I see these cheating threads, I ask myself the question that security folks ask, "what is the threat model?" What exactly are we worried about and trying to stop? Agreeing on that question is absolutely necessary for evaluating any solutions proposed.

Fortunately for me, I don't worry about cheating much at my level. If someone is using a chess engine to make their decision, but still in my rating level (1600 for days/turn chess), clearly their computer program is awful! 

So are we worried about the 2500+ rated player using the latest stockfish every time? Are we worried about a 1300 player who just downloaded stockfish, using it to quickly rise to 2000+? Or somebody who uses the program only when playing online tournaments? Or something else entirely?

http://www.bloomberg.com/news/2013-11-26/pegasystems-founder-becomes-billionaire-as-stock-surges.html

http://www.bbc.com/news/business-24954340

It doesn´t matter that you don´t have a title, but for the media you can be a chess master if you have had a good result in your career.

The method described by r-k-ting seems to place a lot of reliance on rating. It also does not seem to take any account of the differences between different programmes and first, second and third choice moves which programmes rate so close as to be indistinguishable nor the effect of opening theory/books.

Maybe the 3,000 words if I read them would make the idea seem less naive but as presented above it would clearly need an awful lot of work.

The method does take into account nearly equal moves, which are assigned nearly the same score.

You are right about book moves though. Nowhere in the article is the effect of book moves mentioned. Though I suppose it's one of the things that's supposed to come out in the wash when you perform the curve fitting to find the performance rating.

Trouble is, you don't need a performance rating 400-500 higher than a super-GM to beat one. Take for example the Anand-Gelfand match. If Gelfand had a computer to warn him about that single tactic where he got his queen trapped, he would be the world champion.

This sort of cheating by already strong player is probably the most insidious, and something which this method would fail to detect. I think it's not unlikely either. There must be some washed up IM out there who realises that they will never become a GM.

Speculating on the ratings of fictional characters is fun too.  He did it in that article, I say Superman is 4000. 

No where in the wiki article does it mention that Josh Waitzkin was frustrated about not acheiving GM. I have read his books, listened to his lectures, and it seems nothing could be further from the truth. He left chess for entirely different reasons than this.

He is a world champion martial artist (Tai Chi Chuan) and made a decision to take his life in a different direction than chess. He is one of the best authors on "learning" and self-improvement that I have ever read.