Anyone else seen this come back while running API connections on the production environment:
{“code”:403,“errors”:“error code: 1010”}
There is nothing I can do to fix it. All subsequent calls return this error after I’ve seen the first one. And it appears to self correct after a few minutes. I’ve not been able to pin it down to a particular API call, but I’ve been working mostly with the Customer API today, when I started seeing this behavior.
What’s the endpoint your calling when you get this error? Also are there any additional details in the error message? 
I have gotten the error 3 unique times today. Two times pull from POST /v2/customers/search with the data payload of
{“query”:{“filter”:{“reference_id”:{“exact”:“test-customer-1”}}}}
and one time from the call to POST /v2/customers (to add a new customer) with:
{“email_address":"[email protected]”,“reference_id”:“test-customer-1”,“given_name”:“Michael”,“family_name”:“Lewis”}
In all 3 cases every call after that, for a period of about 3 minutes came back with the exact same
{“code”:403,“errors”:“error code: 1010”}
(Including from my production system that has been stable for about 3 years calling the POST /v2/terminals/checkouts endpoint.
I’m going to move over to the sand box environment when I have some free cycles either later tonight or tomorrow but I can’t attempt to reproduce it now with our live system up and running.
I’m also going to be moving us on version shortly as the current one we’re calling is 2020-09-23. if this is potentially version related I’m happy to re-prioritize our upgrade first then revisit this issue and see if it’s gone away.
Best,
mike
Whats your application ID? Also our error messages should have more detail in them. Are you getting the detailed error message? It should look something like:
{
"errors": [
{
"category": "AUTHENTICATION_ERROR",
"code": "VALUE_EMPTY",
"detail": "Field must not be blank.",
"Field" : "idempotency_key"
}
]
}
I’m used to your error codes which is why I was particularly confused by these, but everything after “errors:” is what came from the error stream in the connection used to push the post data. The application id is
sq0idp-AAl5EgtVJuNBYjTS6LsWHg
For a test case, I just forced a failed authentication in the same sandbox code used from above and got:
{
“code”: 401,
“errors”: “{ "errors": [ { "category": "AUTHENTICATION_ERROR", "code": "UNAUTHORIZED", "detail": "This request could not be authorized." } ]}”
}
So i do see SQ errors “usually”.
Looking at the API Logs for the app I see one 400 error today for a call to POST /v2/customers/search. The error message is:
{
"errors": [
{
"category": "INVALID_REQUEST_ERROR",
"code": "BAD_REQUEST",
"detail": "Expected \"{\" (line 1, character 36)"
}
]
}
That’s the only error from today in production.
Yeah, that error came back to me, was before I wrapped my search value for reference id with an exact:“”. Very odd, very odd indeed. Its possible the error was thrown somewhere in the path but before it reached your app server. I’ve had nginx steal traffic from me on the head in to my applications before routing them to the app. But the oddest part is after I cause the issue from my dev laptop, a completely different machine on my production server starts immediately getting these 1010 error codes. That server is in a completely diff location at my datacenter across town. Something is fishy, I’ll continue to experiment later today, and probably just bite the bullet and upgrade the API version.
Side ask, any fear in upgrading all the way to 2023-03-15?
What language are you using? I don’t recognize a 1010 error but when I searched that error some results mentioned that it’s a client error thats terminating the connection. Is it possible that your server is somehow closing the request?
Again plausible, but it doesn’t account for the mysterious impact of a stable code release in a completely different geographical local, 100% coincidental in time. A breaks, B immediately also breaks. I’m using Java (jdk 8). I’ll keep digging and certainly update the thread if I can run it to ground. Thank you for your attention, much appreciated.
-mike
Another thought that occurs to me. Since you’re not seeing the packet, as proven by the error not matching your error format, it could be getting stopped much earlier in the process of routing. Specifically by a cloud based WAF or firewall. 1010 is listed as a Cloudflare errorcode: “Error 1010: The owner of this website has banned your access based on your browser’s signature”. That would explain everything except why its intermittent. Could be a WAF rule about the fact that packet traffic for the same API token is coming from 2 different machines and its trying preempt a DDOS. I’m not sure how to prove that one by experimentation.