Site problem: Rated puzzles with very low passrates

Sort:
Martin_Stahl
Torsen2 wrote:

https://www.chess.com/puzzles/problem/1190562

This puzzle is 2023 rated and has a 0% pass rate. What is going on? I have seen many like this.

This 2200 one has a 3% pass rate: https://www.chess.com/puzzles/problem/1197046

Why are these puzzles being added with the wrong rating?

 

Both of those are fairly new puzzles with less than 500 attempts each. New puzzles take a while to get a stable rating, just like players.

Martin_Stahl

It's also very possible that the puzzles are still considered Pending

KMMCS88

But pending puzzles don't have a rating.

Torsen

These puzzles were not pending when I had them and with a 0% pass rate it seems insane to set the rating at 2023 so there must be some errors in the process..

Torsen

This is a recent problem caused by recent changes.

shirazu0

Here is a 471 level puzzle with a 9% pass rate after 3659 attempts. How can it not be possible to adjust the ratings better than this? https://www.chess.com/puzzles/problem/1018032

Some of the problems you could say "low rated players attempted these puzzles so the pass rate is very low hence the low rating" but there is simply no explanation here and let's not get into very bizarre ratings both too high and low all across the board...I have seen puzzles 2200 where there were almost no reasonable moves and the pass rate was 60+% so it is not just in one direction

shirazu0

The problem is that the ratings don't adjust anymore. There are puzzles with 10k attempts and pass rates in the teens with ratings <1000 and the comments are all "wtf is this" and the rating never changes.

Martin_Stahl
KMMCS88 wrote:

But pending puzzles don't have a rating.

 

I verified on a recent puzzle that was pending and they do have ratings, it's just that they are listed as pending on the puzzle history and do not cause a rating loss when missed.


You did this one yesterday and it's listed as pending on your stats, and it has a rating: https://www.chess.com/puzzles/problem/1243429

 

Martin_Stahl
shirazu0 wrote:

Here is a 471 level puzzle with a 9% pass rate after 3659 attempts. How can it not be possible to adjust the ratings better than this? https://www.chess.com/puzzles/problem/1018032

Some of the problems you could say "low rated players attempted these puzzles so the pass rate is very low hence the low rating" but there is simply no explanation here and let's not get into very bizarre ratings both too high and low all across the board...I have seen puzzles 2200 where there were almost no reasonable moves and the pass rate was 60+% so it is not just in one direction

 

With a rating that low and that many attempts, it won't go up very much when attempted, since the players attempting it are lower rated. It's a pretty standard mate, but I can see that it might be missed a lot.

Torsen

Martin the problem is with how ratings are being assigned. I can guarantee the 2 puzzles I gave were not pending and I did lose rating for them.

This problem may have emerged recently due to chess.com changing how puzzles are approved as they said they have done in this post:  https://www.chess.com/article/view/chess-com-month-in-review-january-2021

 

Martin_Stahl

The second puzzle you posted has went up since your post as has the one from post 9.

 

The ratings still appear to be adjusting. It's possible that the number of attempts before being removed from pending status has changed, and a few early attempts are heavily influencing the rating, even on the low pass rate ones.

 

 

Torsen

The 2nd one hasn't actually gone up, I put 2200 as a rough rating rather than exact rating. 

I don't think a puzzle's rating ever changes once it's been determined unfortunately. The one in the 9th post is from before November 27 2020. That sort of Boden's mate puzzle is usually around 1600.

Martin_Stahl

Actually, I was wrong about the one on post 9.

 

I'm asking about the process to verify.

DeepGreene

Thanks, all! The problem with puzzles that have very low pass-rates, even after many many attempts, is that there is no accurate was to programmatically determine a decent starting rating. Fortunately, these cases are relatively rare, but we are having to fall back on getting human reviewers to weigh in on the most problematic cases. These examples may already be slated for review, but thanks for shining a light on them in any case!

Torsen

Thanks for the reply. Maybe it would be programmable to delay the addition of puzzles with very low pass rates so that they stay pending for longer.