All of a sudden today, my web app has gotten several RATE_LIMIT_ERROR responses to UpsertCatalogObject calls. I see from searching on the forums that you don’t publish your rate limits, but can you provide any sense of the scale? And is the rate limit specific to a given call or is an aggregate limit for all calls? It’s a little disconcerting because the last RATE_LIMIT_ERROR I got was on a single UpsertCatalogObject call when the previous UpsertCatalogObject call was 60 seconds earlier (according to my API log). I can’t imagine that every 60 seconds is too frequent…
When you get the RATE_LIMIT_ERROR how many queries per second is your application calling Square?
As for rate limits you are correct, Square does have rate limits but we don’t document the limits. We encourage you to handle the limit gracefully by building a retry mechanism. This mechanism should have an exponential backoff schedule to reduce requests when volume is necessary. Also some randomness wouldn’t hurt to avoid a thundering herd effect. 
@Bryan-Square Hi Bryan. The calls to UpsertCatalogObject were several seconds apart when I got the error. Here’s a screenshot of a relevant section of my API log:
There are calls to other endpoints being made in between, but they’re filtered out in that screenshot of the log. That said, in the full log I see my app making at most 4 or 5 calls per second across all endpoints.
Thanks for providing all the examples. Are you still experiencing this issue or has it recovered to normal now?
It has recovered to normal now. There were in total 10 RATE_LIMIT_ERROR responses over the course of an hour or two on 7/22 with a little over half of them occurring within a three minute span starting at 17:46:42 GMT-5.
Mainly, I want to understand if this was possibly an anomaly on the Square server side or if my app was actually exceeding rate limits, because the occurrence seemed fairly random from looking at the logs.
I checked in with the team and it does look like there were some issues at that time that could have caused degregated performance. The team was able to quickly fix the issues.
Cool, thanks @Bryan-Square. Not too big of a deal. Since my app doesn’t generally make API calls at a rapid rate, knowing it was likely on the server side helps me determine better what kind of adjustments I may want to make to my code.
@Bryan-Square I got another random RATE_LIMIT_ERROR today. What’s very weird this time is that it isn’t shown in the Square API Log (should be at ~11:05):
But my web app caught it and logged it:
Any insights would be helpful. Thanks!
FYI I got another random RATE_LIMIT_ERROR today that my app caught and logged, but that doesn’t show up in the Square API log. I’m a little weirded out now that these aren’t showing up in the Square API log anymore.
What’s the application ID your getting this error with?
Here you go:
sq0idp-F4ya7v-DG12NERCMHQDWXw
Hmm, I’m not seeing as well in our internal logs. Can you confirm that the error came from Square?
Sure. Here’s what my app logged:
This was on a SearchOrders call that occurred at 7:42 Pacific/9:42 Central. My app makes that call nominally once every minute. Just like the previous instance I posted, you can clearly see in the Square API log by looking at the timestamps that the call is simply absent from the log (it should be where the arrow is pointing):
@Bryan-Square Just wanted to follow up and see if the above information was helpful in digging into this further on the Square side?
Hey @cbstl, I took a look into this and I wasn’t able to find the request in our logs either. By any chance, do you log the idempotency_key that is used for your requests? That might help us find this on our side.
@josh-square I posted this original thread about RATE_LIMIT_ERROR responses that I got on UpsertCatalogObjects calls in a fairly isolated period of time and that was chalked up to some issues on the Square side that were noticed and resolved fairly quickly.
These last two oddities that don’t show up in the API log were seemingly random, isolated RATE_LIMIT_ERROR responses that I got on SearchOrders calls and of course, that API call does not have an idempotency_key in it. I think the best I can provide is the exact time stamp of the response at my web app’s server which is in my logged info above (Pacific time).
FWIW, this isn’t really causing my web app any catastrophic issues. I handle these API exceptions and end up retrying the failed SearchOrders call a minute or so later. Mainly I’m interested because I don’t believe these calls could really have exceeded a retry limit and the oddity of the call missing from the Square API log is very strange. I hate to be the, “it’s your obscure, intermittent bug, not mine” guy but the code on my side in this case is pretty straightforward, so if there’s a possible bug on the Square side I want to help provide as much info as I can.
1 Like
Just a shot in the dark here - is there any possibility that the .NET SDK SearchOrdersAsync call might possibly have pre-emptively (and incorrectly) thrown a RATE_LIMIT_ERROR exception before actually sending the call out on the wire? That would at least explain why it would be missing from the log…
My mistake, totally glossed over the fact that there’s no idempotency_key for us to track down in this case! It’s definitely strange that there appears to be no record of it on our side. The SDK seems like a likely culprit here, but we’ll need to coordinate with the team on our side to verify whether the .NET SDK throwing a rate limit error independent of any response from Square’s servers is possible in the first place.
Thanks for flagging this and providing your own logs, we definitely appreciate it!
@josh-square Cool, if there’s anything else I can provide that would help, let me know. Otherwise, I’ll leave you guys to it unless I see it again. It might be a tricky thing to track down, because it’s been extremely intermittent. My web app has been calling SearchOrdersAsync ~once per minute since I deployed it in July and I’ve only seen this particular oddity twice.
Good luck!