We updated the API today with a new 503 error status code.
Context – there are two reasons why you will get an error with the status code of 429. And it has been confusing for users because users assume they made a mistake.
The first reason is that your account has hit an account-level rate limit of 300 requests/minute. However, most users do not hit that limit.
The second reason is that scraping failed. And this is a limitation on our end—a backend error. And backend errors should not return a 4xx error code, given that 4xx errors signify a client-side error.
Today, we deployed an update to the API, which will return a status code of 503 if a scrape failed instead of 429. The way you should handle it is the same as is with error 429. If you get a status code of 503, do retry the request immediately, and you're good to go.