Lots of 429 Too Many Request errors for the past day

One of the companies I work with use Square for invoicing and labor, and I built a few custom integrations for them using the Square API. Things have been working great for the past six months or so, but as of yesterday, almost every single request is getting rejected with a 429 “Too Many Requests” error. Nothing has changed on my end in terms of the number of API calls that are being made. I tried to reach out to Square but so far, no response. I looked at the API logs to see if anything looked unusual, but nothing looks unusual. I looked at the logs yesterday when everything was working and getting a 200 response, the amount of API calls are the same. Maybe someone at Square at take a look at this and help?

:wave: Whats the application ID that you’re getting the error with?

The application ID is sq0idp-tj_iliMuseH7ftRZNnwhXw, but it looks like the errors stopped happening around 2 hours ago.

I’ve been having the same issue today. I haven’t been able to create new tokens for my app for the past hour.

App id is sandbox-sq0idb-mZM80IOVz0_uuaw4YYDrzQ

:wave: It appears there was an issue today with that endpoint and it has been resolved by the team. If you are experiencing any additional issues with this endpoint please do reach out. :slightly_smiling_face:

Hi Bryan, thanks for looking into this yesterday, but unfortunately, i’m still seeing 429 errors for my apps. Most notably on this one sandbox-sq0idb-v6wVp0BOapV4hPKlL_asFQ, which is our main QA app. Looking at our API logs, seems like this also just started recently (yesterday/today). Is there still something affecting sandbox apps or is it possible to relax the limits for our app?

And one thing to note, we are making calls from multiple code bases and it seems like API calls always succeed from one but not the other.

Looking at the logs it looks like the 429s started today. Have you changed the way you’re calling ObtainToken? The only other error I see is a 401 which is: The request app id sandbox-sq0idb-v6wVp0BOapV4hPKlL_asFQ does not match the app id that the refresh token links to

We haven’t changed anything with our integration on this particular sandbox app in at least a week, so I was surprised to see 429s occurring. The 401s were caused by other manual testing to this app that has since been resolved.

But, the API Logs tool does show all the requests made to the app, correct? Does the request volume look like it would trigger a rate limit response? Are there other restrictions to know about like rate limiting by IP or referrer?

We were having issues today with our services and they appear to be fully restored. We’re continuing to investigate and monitor closely. Apologies for the disruption! Please let us know if you are encountering any further issues. :slightly_smiling_face:

1 Like

Hello!

This same error cropped up in our system today on our end. Were there any issues today with this endpoint?

I’m not aware of any issues today. Were you making a lot of requests when this error happened? :slightly_smiling_face:

Not any more than our usual rate. This same endpoint hasn’t had any errors today with a similar rate of request to yesterday. Could have been a blip, we’ll continue to monitor.

Sounds good and if this happens again please include the endpoint your calling with your application ID so we can review our logs. :slightly_smiling_face:

Hi Bryan, we had more instances of this error occurring on March 22nd. Here are some details:

March 22nd, 5:40pm ET
application ID: sq0idp–BB896f7nMdcBxUucYHSHA
endpoint: GET /api/stores

In speaking with the team we found that we were able re-create is problem in our test environment while connected to our sandbox square app, getting a 429 error while sending 60 requests over the span of about 10 minutes.

The part of our software that calls this endpoint is seldom used between September and April (with zero usage some days), but sees a high volume of requests between May and July. Is it possible that a change was made to Square’s open source APIs this past winter to manage the request rate to this endpoint (or something similar)? We likely would not have noticed given the cyclical usage of this app, and we haven’t made any changes to this area of our tool since our last busy season. Please advise.

When you get this error how is the application handling it? We encourage you to handle the limit gracefully by building a retry mechanism. This mechanism should have an exponential backoff schedule to reduce requests when volume is necessary. Also some randomness wouldn’t hurt to avoid a thundering herd effect. :slightly_smiling_face: