HTTP Error 429: Should I Have Received This?

I was carrying out a task as a favour for one of the admins of The Empire Of Assassins, involving sequentially downloading the club's list of team matches and unexpectedly got this response after the 772nd. match:-

match_id=444708 / HTTP error code: 429 - 'Too many requests'

I've read responses from staff members on the subject of limits on parallel endpoint requests but I've always understood sequential (i.e. in series, one after the other) are absolutely fine.

Has something changed?

I've cleared that error & my program's now continuing normally but I've lost any data on that one match which is annoying.

The limit is on the total number of requests you're making. it doesn't matter if they're in serial or parallel.

Please see:-

https://www.chess.com/news/view/published-data-api#pubapi-general-rate-limits

Rate Limiting
Your serial access rate is unlimited. If you always wait to receive the response to your previous request before making your next request, then you should never encounter rate limiting.

However, if you make requests in parallel (for example, in a threaded application or a webserver handling multiple simultaneous requests), then some requests may be blocked depending on how much work it takes to fulfill your previous request. You should be prepared to accept a "429 Too Many Requests" response from our server for any non-serial request that you make.

All of my requests are serial.

@stephen_33 Adding time.sleep(1) in your loop solves this problem once and forever. I understand that it is not, what you expect, reading "Your serial access rate is unlimited.", but API automatically temporary blocks users, who overload it too much.

It happened only once in a club archive of 1624 matches, so it seems strange that I was simply overloading the servers.

You may be right but I'll wait to see what any of the staff admins have to say.

I've seen this in my code where it makes a number of requests in rapid succession.  My solution was to catch this particular exception, sleep for a short time, and retry with the same request.

It has to do with how this is coded on the server side. The server doesn't actually know if you're serial or parallel, they just have a "n requests per second" limit, and if the server responds fast enough and your app queries fast enough, you can still hit it.  Notice it says "should never encounter rate limiting".

This is going beyond my area of knowledge but I understood problems begin when the server receives the next endpoint request before it's fully serviced the previous one?

But when making serial requests that can never happen.

@stephen_33 I can't find the related post, but I read somewhere here, that currently API can handle 3 simultaneous requests. So I assume that you are fine, while API is not overloaded, but server kicks you out, when several other people make a request at the same time as you, because you were just unlucky to make 4th request, when the first is not yet fulfilled, AND you are the one, who loads API at most in the last several minutes.

That makes more sense. So you're saying the 'too many requests' error doesn't necessarily mean a particular user has been making too many endpoint requests, the server is indicating that it's receiving too many?

I don't know. It is just my assumption. And you miss just one detail. Not only "the server is indicating that it's receiving too many", but "the server is indicating that it's receiving too many from you". It could be that receiving 4th request, when the first is not yet fulfilled, server can choose: to add your request in a queue and perform it with a small delay or refuse to execute, because you are really making too many of them lately. So you do not encounter Error 429:

• case 1: when you make requests too often, but server is not overloaded;
• case 2: when you make your requests with acceptable frequency.

As @imperfectage said you can use try...except construction to handle this error and wait a bit before repeat, when you got it. Or you can add small constant delay between your request to reduce server load.

It would be great to hear developer's opinion about preferred approach. First one seems to be quicker, but if freeze time increases each time you hit this error (like 5 second in first time, one minute in second, etc), it will seriously slow down your data retrieval. I use the second one and have never met this error, but I never was in situation, when I need a lot of data urgently. In this case, the first approach is preferable.

Not sure what this means: "..but "the server is indicating that it's receiving too many from you". It could be that receiving 4th request, when the first is not yet fulfilled, server can choose: to add your request in a queue and perform it with a small delay or refuse to execute, because you are really making too many of them lately"

Since my program is making serial requests only, it makes the 'nth.-plus-1' request only when the results of the nth. request have been provided in full. There's no risk of any 'overlap' between the requests I'm making.

I thought you understood that? Only when my program has received an HTTP 200 status code, to indicate a normal, complete endpoint data-set, does it proceed. There's no possibility for it to send a request before the previous one has been fully executed & results returned.

"My program is making serial requests only <...> I thought you understood that?"

Sure. Here is simplified description of what I meant:

moment 1: request from user1 -> return result to user1 

moment 2: request from user1 -> return result to user1 

moment 3: request from user1 and from user2 -> return result to user1 and user2

moment 4: request from user1 -> return result to user1 

moment 5: request from user1, user2, user3, user4, user5 -> kick out user1, return result to user2, user3, user4, add user5 to queue

moment 6: request from user1 -> return result to user1 and user5 (with a small delay)

As I already said, it is not necessary working like that. I'm just trying to explain one of the possible reasons, using my personal beliefs on the basis of which I would develop my own client-server application. That's why "It would be great to hear developer's answer."

"It would be great to hear developer's answer."

I agree which is why I PM'd bcurtis (the SA of the club) about an hour ago to ask him to take a look at this & give his expert opinion  😊

This is interesting ...

https://www.chess.com/news/view/non-breaking-change-club-matches-endpoint

"Starting from 8th of March, we will start to trim the list of finished matches for endpoint /pub/club/{ID}/matches to the most recent 1000, for performance purposes."

Could there possibly be a connection between the problem I had and this change to the club-matches endpoint? I can see the need for it but it's going to put a spanner in the works for some users of the API I imagine.

I thought the plan was to restrict users to the most recent 1000 finished matches but I just noticed this on the end of one club-matches endpoint:-

"comment":"Results are temporarily reduced to the most recent 500 matches for performance reasons"

I'm guessing that the servers are becoming a little over-loaded?