Puzzle Rush Survival repeating puzzles too much

Hi everyone,

I wanted to open a discussion about something I've noticed in Puzzle Rush Survival. As I've been playing and improving, I've started to get higher scores, often breaking past 120 puzzles. However, I've realized that after this point, the puzzles seem to repeat quite a bit.

Given that Chess.com claims to have over 500,000 puzzles on the site, I was expecting a lot more variety in Puzzle Rush Survival, but it feels like I'm seeing the same puzzles in a loop after reaching certain milestones. It's made it easier for me to improve my scores, but it also leads to some unexpected challenges.

Because I often get similar puzzles, my scores have risen, and as a result, I've received negative comments and accusations of cheating from other users. It's not something I enjoy dealing with, especially when I'm playing fairly and just working with what I'm given.

I'm curious if others have had similar experiences? Does anyone know if Chess.com has plans to expand the puzzle pool for Puzzle Rush Survival or if there's a specific reason for the repetition? I’d love to hear from other high-scorers about their experiences, and any insights or suggestions on what we could do to address this issue.

Just a note: this isn't a place to argue about high scores or whether someone is cheating. I just want to understand why there isn't more variety in the puzzles beyond a certain point, and what Chess.com might be able to do about it. Thanks!

Looking forward to your responses,
Joseph Reid.

It sounds like they need more high rated puzzles!

Most of the puzzles are at lower levels. The higher you get, the fewer puzzles there are at that level.

Tell me about it! The actual puzzle ratings themselves go up to 4000, and I find them the easiest, no I'm not just saying this to be contrary 🤨

OK, interesting. As I said, I find the 4000-rated puzzles the easiest, whether it is just that I've memorised them the most or something else. But it is worrying (in a good way) that once I've passed the 100 or 120th mark I fly through the puzzles. For example: I started another Survival on Sunday morning (New Zealand time) and by the end of that day I had gotten to 250+ with breaks! Ridiculous!

Wow! My best puzzle rush survival score is a 53. You must spend a lot of time developing intuitive accuracy. Chess.com should label all 4000 puzzles as Joe's. I'm joking but kind of serious too. How many people can say they've solved those? Excellent, Joseph. 🙂♟️

It seems to me that the small number of puzzles at the highest rating inflates the rating of the top puzzle players, and pulls everyone else up with it. I think that's possibly why puzzle ratings seem way too high. If that's the case, the low number of of puzzles at the high-rating levels affects all users, not just the high-rated ones; so they should fix it.

I understand chess.com generates puzzles automatically, but has a filtering system that removes puzzles which don't fit certain criteria. What they should do is be more lenient with these criteria for puzzles that are expected to be higher-rated, so there are more puzzles at the higher rating levels.

Also, it really shouldn't be all that hard to make higher-rated puzzles, since many lower-rated puzzles have additional moves from the game before and afterwards which (if lumped with the puzzle) would make it harder.

You can't simply relax criteria about things like only one solution. Adding extra moves before isn't always viable, because again of the issue that there must be only one viable solution; the puzzle can't start before a tactic becomes possible. And adding extra moves to the end runs into the same issue, but in cases where it doesn't, the extra moves at the end are often going to be easy.

But I think you can.

Presumably, "only one solution" means something quantitative; like a solution being worth at least +1.0 in stockfish evaluation compared to the other possible moves. Well, you could just make it e.g. +0.5 in stockfish evaluation compared to the other possible moves for harder puzzles.

If (as I suspect) this is warping the rating of all puzzles, due to it creating people like the OP with a puzzle elo of 65535, it affects everyone, not just people with 100+ on puzzle rush.

+0.5 usually means there's two good moves and one is a bit better. That's not enough to make it a good puzzle.

Puzzles are about finding difficult tactics that are hard to spot, not about correctly guessing the engine's evaluation of two positions where the computer thinks one is slightly better.

Correct, it means there will be lower quality puzzles on average at higher elo.

But so what?

To me, many low-quality puzzles is better than having a very small number of high quality puzzles that some people can just memorise, massively skewing their own rating and subsequently the rating of every other puzzle on the website.

Judging by his sentiments in this thread, I suspect @JosephReidNZ would agree with me, and he's the one actually solving these puzzles.

The problem with a lower-quality puzzle is that you can get it wrong not by missing a tactic but by guessing incorrectly which ending position is slightly preferred by the engine.

If you have a choice between checkmate and winning the queen, you take checkmate, and winning the queen is incorrect. If you have a choice between winning a queen and winning a rook, you take the queen, the rook is incorrect.

But if you have a choice between two positions where the difference is about half a pawn - sometimes the engine doesn't even evaluate that small a difference correctly (it usually does, but not always).

Tactics puzzles should have a single clear, right answer. Hard puzzles should be ones where that single right answer is very hard to find, not ones where it's easy to find two decent moves and hard to figure out which one is slightly better. I don't think anyone wants bad puzzles.

Interesting discussions going on, thank you for everybody's insights into this.

If it's getting to a point where engines can be inaccurate, they can just run the engine at a deeper depth for those puzzles: or have GMs flag puzzles where the evaluation is clearly incorrect.

Although to be honest I don't know what the current threshold is and whether reducing the amount the evaluation drops by that much is neccesary.

In the vast majority of (low rating) puzzles I've looked at, you go +2 or more in material. If the threshold is actually 2.0 in stockfish evaluation, would dropping it to 1.0 for puzzles (say) over 3500 rating work?

But if it is a theashold of 1.0, you may not need to drop it all the way to 0.5. My suspicion is there are more potential high-rated puzzles in the 0.75-1.00 range than above 1.0. If that's the case, just dropping it to 0.75 could be enough. That could describe either a pawn up and a worse position or a heavily superior position to rival moves. Seems good enough to me. The people doing these are 3000+ puzzle elo, that should not be hard to distinguish for them.

Simply lowering thresholds is not a good way to generate more puzzles.

More bad puzzles is not necessarily better than fewer good puzzles. When doing puzzle rush, you will eventually see puzzles start to repeat. But is that really worse than having your puzzle rush cut short because you guessed wrong on a bad puzzle where the "right" answer was only a little better than a "wrong" answer?

Very few people are actually doing survival puzzle rush slowly enough to reach the point where high-level puzzles start to repeat. It's very much a rare edge-case situation. Compromising the quality of the puzzles to try to improve that edge case situation (and I would argue it's not an improvement) would actually make puzzles worse for many people who never do survival puzzle rush in that way.

But if you only let it happen with high-rated puzzles, it shouldn't be an issue. Puzzle rush puzzles increase by 50 elo each turn. This means if you only loosened the requirements for puzzles expected to be 3500+ elo, this would only affect people with a puzzle rush score of above 68 or so. I feel like it would affect the higher number of people who have a normal puzzle rating of 3100 or more, (roughly), but this is still only a small proportion of the people who use the site.

If I'm right and people memorising puzzles is skewing the puzzle ratings of all puzzles, though, that affects everyone on the site.

Thank you for your insightful response. You raise some important points about puzzle quality and the impact of lowering thresholds to increase puzzle variety.

Puzzle repetition in certain modes like Puzzle Rush Survival is indeed an edge case, and most users might never reach the point where they notice recurring puzzles. From this perspective, focusing on a smaller pool of well-crafted puzzles might be better than expanding the pool with potentially lower-quality puzzles.

Additionally, a poorly constructed puzzle, where the "right" answer is marginally better than other alternatives, can frustrate users and undermine the puzzle-solving experience. Quality puzzles should offer clear solutions and be designed to challenge users meaningfully.

A balanced approach to creating more puzzles without sacrificing quality could involve improving puzzle generation algorithms to ensure consistent quality while expanding the puzzle pool. This way, the potential for repetition is reduced without compromising the integrity of the puzzles.

Ultimately, it comes down to maintaining a high standard of puzzle quality while addressing user needs. By focusing on quality, users are more likely to have a positive experience, leading to better engagement and satisfaction.