For the last few days my web app has been getting intermittent 503 and 504 responses to API calls. I’m using the .NET SDK and my application ID is sq0idp-F4ya7v-DG12NERCMHQDWXw. I’m used to seeing one or two of these on occasion, but I’ve been getting between 50 and 60 per day since Saturday. The calls that return the error don’t show up in the Square Developer API logs at all, but here’s an example of what I’m logging when I get the ApiException:
I took a look at your API Logs and see only one 5xx error. No major outages that I’m aware of. Do you know if this is the SDK that’s throwing the error or are these requests being thrown by our server? Do you have the full error message?
As I said in the original post, the calls don’t show up in the Square API log. To give a little more detail, my application does an orders search once every minute. For example, I got a 503 error on an orders search at 7:08am this morning. As you can see from this screenshot that shows that time span, the 7:07 and 7:09 calls that were successful are there, but the call that returned a 503 is missing entirely from Square’s API log:
Here’s everything my application logged for that 7:08am call:
2024-02-05 05:08:11.243 -08:00 [ERR] SquareClientWrapper.HandleSquareApiException ERROR: ApiException occurred (see subsequent details)
2024-02-05 05:08:11.243 -08:00 [ERR] Headers:
2024-02-05 05:08:11.243 -08:00 [ERR] Square-Version: 2023-08-16
2024-02-05 05:08:11.243 -08:00 [ERR] user-agent: Square-DotNet-SDK/30.0.0 (2023-08-16) .NET 6.0.9/6.0.9 (Microsoft Windows NT 10.0.20348.0)
2024-02-05 05:08:11.243 -08:00 [ERR] Content-Type: application/json
2024-02-05 05:08:11.244 -08:00 [ERR] accept: application/json
2024-02-05 05:08:11.244 -08:00 [ERR] Status Code: 503
My application logs any Error objects that exist in the ApiException but the errors list is empty in these exceptions - they only have the Status Code. Does that suggest it’s the SDK throwing the exception?
If it’s helpful information, my application makes calls to other endpoints every minute also, but the 503 and 504 responses are always coming on the orders search call. Maybe something is funky with the Orders endpoint?
Ok. I don’t know if it’s helpful information, but the 504s that are in the Square API logs today are exactly like the 503s I’ve been getting for several days - the ApiException has the Status Code, but there are no Error objects.
I’m really confused now, because I know my code logs the error detail if it exists in the exception. Here’s what I logged this morning for a 500 that occurred at 10:45 and a 504 that occurred at 10:47:
2024-02-06 08:45:40.817 -08:00 [ERR] SquareClientWrapper.HandleSquareApiException ERROR: ApiException occurred (see subsequent details)
2024-02-06 08:45:40.817 -08:00 [ERR] Headers:
2024-02-06 08:45:40.817 -08:00 [ERR] Square-Version: 2023-08-16
2024-02-06 08:45:40.817 -08:00 [ERR] user-agent: Square-DotNet-SDK/30.0.0 (2023-08-16) .NET 6.0.9/6.0.9 (Microsoft Windows NT 10.0.20348.0)
2024-02-06 08:45:40.817 -08:00 [ERR] Content-Type: application/json
2024-02-06 08:45:40.817 -08:00 [ERR] accept: application/json
2024-02-06 08:45:40.817 -08:00 [ERR] Status Code: 500
2024-02-06 08:45:40.818 -08:00 [ERR] Error Category:API_ERROR Code:INTERNAL_SERVER_ERROR Detail:An internal error has occurred, and the API was unable to service your request.
2024-02-06 08:47:40.934 -08:00 [ERR] Exception caught in SquareClientWrapper.GetOrdersFromTimeWindowAsync: A Square ApiException was thrown - HTTP Response Not OK (more details may be available in the log)
2024-02-06 08:47:40.947 -08:00 [ERR] SquareClientWrapper.HandleSquareApiException ERROR: ApiException occurred (see subsequent details)
2024-02-06 08:47:40.947 -08:00 [ERR] Headers:
2024-02-06 08:47:40.947 -08:00 [ERR] Square-Version: 2023-08-16
2024-02-06 08:47:40.947 -08:00 [ERR] user-agent: Square-DotNet-SDK/30.0.0 (2023-08-16) .NET 6.0.9/6.0.9 (Microsoft Windows NT 10.0.20348.0)
2024-02-06 08:47:40.947 -08:00 [ERR] Content-Type: application/json
2024-02-06 08:47:40.947 -08:00 [ERR] accept: application/json
2024-02-06 08:47:40.947 -08:00 [ERR] Status Code: 504
Both of those are in the Square API logs and according to the log on Square’s side, they both had error detail in the response:
But my application only logged the error detail for the 500. My code that logs the error detail is pretty straightforward (e is the ApiException object):
var errors = e.Errors;
if (errors != null)
{
foreach (Error error in errors)
{
_logger.LogError("Error Category:{Category} Code:{Code} Detail:{Detail}", error.Category, error.Code, error.Detail);
}
}
That 504 had to come from your server right, since it’s logged in the Square API logs? Is there any chance the SDK isn’t passing along the error detail in the response for 503s and 504s?
Incidentally, I also logged a 503 at 10:48 that didn’t show up in the Square API log (just like I’ve been seeing since Saturday).
That code will log the error details. To log the full error message, including the status code and any other relevant information from the ApiException object, you can modify your existing code to capture and log all the details provided by the exception. Here’s an example of how you can do this:
try
{
// Your code that makes the Square API request
}
catch (Square.Exceptions.ApiException e)
{
// Log the status code
_logger.LogError("Square API request failed with status code: {StatusCode}", e.ResponseCode);
// Log the headers
foreach (var header in e.Headers)
{
_logger.LogError("Header: {Key}: {Value}", header.Key, string.Join(", ", header.Value));
}
// Log the error details if they exist
var errors = e.Errors;
if (errors != null && errors.Count > 0)
{
foreach (Square.Models.Error error in errors)
{
_logger.LogError("Error Category: {Category}, Code: {Code}, Detail: {Detail}",
error.Category, error.Code, error.Detail);
}
}
else
{
// Log the entire response body or a generic error message if there are no specific error details
_logger.LogError("No specific error details were provided. Response body: {ResponseBody}", e.Response);
}
}
catch (Exception ex)
{
// Log unexpected non-API exceptions
_logger.LogError("An unexpected error occurred: {ExceptionMessage}", ex.Message);
}
In this example:
It logs the status code using e.ResponseCode to identify the HTTP response code received from the Square API.
It iterates through the headers in e.Headers to log all the response headers, which can be useful for debugging.
It checks if there are any error details in e.Errors. If there are, we log each error’s category, code, and detail. If the Errors list is null or empty, we log the entire response body using e.Response to capture as much information as possible about the error.
It has an additional catch block to handle any other unexpected exceptions that might occur during the API request.
This ensures that you log all available information about the error, which can help you debug and understand the cause of the failure more effectively.
Um, as you can see from my logs, I do log all the headers and the status code for every ApiException, in addition to the error details. I was just providing the snippet of code that logs the error details for the purposes of showing that if there are error details in what I get from the SDK, they would have been logged.
Any response to my actual questions in the previous post:
That 504 had to come from your server right, since it’s logged in the Square API logs? Is there any chance the SDK isn’t passing along the error detail in the response for 503s and 504s?
We were experiencing an issue at that time as per our issquareup.com page so it could have been related to that. Otherwise the SDK is supposed to pass the error details.
Ok. If you have some insight into the issues that Square was having, would those issues really cause the Square API log to log the 504 response with error details (the Square API logs show the details in the response), the response would get to my application, but the error details wouldn’t be reported by the SDK? That seems odd if the issues were on the Square server side. It also doesn’t account for all of the 503 errors I’ve been getting that haven’t shown up in the Square API logs at all. I’m pretty skeptical that those would not be coming from the Square servers and instead are generated by the SDK due to some problem with my host. If it was an intermittent problem at my host, I would expect to see similar exceptions on calls to other endpoints that are made just as frequently, but all of these errors in the last several days have only come from the Orders endpoint and there’s been a lot of them. Is it possible all those 503s were related to the issues that Square worked on today?
Unfortunately, we don’t have any insight on the 504 and why the details weren’t logged. The logs do show that we sent them and the SDK is designed to pick them up. When you get a 4xx is your application correctly picking up the details.
As for the 503 we assume that the request is failing at the SDK level since it’s not making it to our servers.
As I’ve already shown in my logs and my code, if the ApiException contains error details, my application correctly logs them.
The last 503 I’ve received was yesterday at 10:48am - essentially the same time that I received the last 500 and 504 shown earlier which you’ve already attributed to the issues Square found and resolved in that timeframe. So I think it’s safe to assume that the 503s were directly connected to the same Square issues. I’m sure you’d agree that any other cause would represent an incredibly unlikely coincidence, given that I had been receiving at least 50 per day since Saturday up until that point.
