Single webhook event fired 2 times in the span of a few milliseconds

Hi,

It happens quite frequently that my app receives 1 single webhook event 2 times in a row in the span of a few milliseconds.

I know about idempotency, and I also know that square will try fire again the webhook after a reasonable amount of time if it does not get a 200 back, but here it is different: I receive the initial event 2 times in a row in the span of a few milliseconds.

My app id is sq0idp-P0lN2wfKfFDEvXXupFMcNg

Example:

2023-10-26 10:58:20.903: af56340c-5e29-5a37-a190-e90f5d61a9dc booking.created
2023-10-26 10:58:20.911: af56340c-5e29-5a37-a190-e90f5d61a9dc booking.created

This is expected since webhooks can be sent more than once. You can bypass the processing of repeated notifications using the idempotency value included as the event_id field in the body of each event notification. :slightly_smiling_face:

@Bryan-Square, I’m sorry but I have to add a comment here because it’s the same issue we are having.

It makes no sense that Square webhooks notifications are sent at the same time for the same event because the 3rd party system has no time to log the notification and even less time to pull additional payloads with only milliseconds between. Without giving time to process a notification it’s impossible to have any comparison data to define if any other is a duplicate. And in addition, when sending different version notifications also at the same time makes no sense because when pulling payloads you will get always the latest version and never know what the older version had and got changed. So, this Square particular behavior is completely none sense.

I also want to chime in here. One problem may that the event_id is not a true idempotency key. Here are two webhook events that I am almost certain we successfully received (responded with 200):

{
“merchant_id”: “MLY5F08WVFJBG”,
“type”: “order.updated”,
“event_id”: “b349a949-ae52-3920-832d-610b1ae2a75d”,
“created_at”: “2024-04-02T17:41:37Z”,
“data”: {
“type”: “order_updated”,
“id”: “HGsO2YYAvEfcDcTRnkl7QwXFN17YY”,
“object”: {
“order_updated”: {
“created_at”: “2024-04-02T17:41:05.203Z”,
“location_id”: “LPWMHRSSSJBT6”,
“order_id”: “HGsO2YYAvEfcDcTRnkl7QwXFN17YY”,
“state”: “OPEN”,
“updated_at”: “2024-04-02T17:41:36.823Z”,
“version”: 6
}
}
}
}

{
“merchant_id”: “MLY5F08WVFJBG”,
“type”: “order.updated”,
“event_id”: “b349a949-ae52-3920-832d-610b1ae2a75d”,
“created_at”: “2024-04-02T17:41:37Z”,
“data”: {
“type”: “order_updated”,
“id”: “HGsO2YYAvEfcDcTRnkl7QwXFN17YY”,
“object”: {
“order_updated”: {
“created_at”: “2024-04-02T17:41:05.203Z”,
“location_id”: “LPWMHRSSSJBT6”,
“order_id”: “HGsO2YYAvEfcDcTRnkl7QwXFN17YY”,
“state”: “OPEN”,
“updated_at”: “2024-04-02T17:41:37.013Z”,
“version”: 7
}
}
}
}

How can event_id be treated as an idempotency key when your internal order state marker, version, has changed?

All the best,
Mike Price, PhD

Mike,

You’re right in that event_id isn’t truly an idempotency key; it’s just a unique id for that transaction.

If you get a second webhook event with that event_id you simply check for a later “version” value than the previous webhook hit, and do an immediate exit without further processing if the “version” is less than or equal to that of the last event with that event_id.

This does mean that you have to save the version each time you process a webhook hit.

It also means that if you have enough data in the webhook hit you can go ahead and initiate further processing. The later version event webhook posts just update things like the processing fees etc.

Square: it would probably be useful if there was some way to tell whether a particular web event post was “final”. If there’s no way to know that, the webhook processing will need to take into account that no event webhook hit can be considered final. The lack of a sense of “final” makes webhook processing harder than it should be. I can also understand that this might not be possible for internal reasons, just raising it because it would help.

In practice, the “version” value will go to about 4, with hits for about 4 of those “version” events (ie: 3 versions don’t appear to generate webhook hits).