Tips and tricks for API pagination

Welcome to the world of pagination — spreading out results over a number of pages or sections. I’ll provide an overview into some basic pagination concepts, and dive deep into how we do things at Square.

Introduction to pagination

The concept of pagination originally comes from books, but I think it’s easier to describe in terms of lists. You see pagination all the time on blogs, news sites, image sharing sites, etc. Whenever there isn’t enough room on the page to show all of the posts at once, the big list of posts is broken up into different “pages”. You are probably familiar with the UI components similar to the one below that help you navigate between pages.

This is pagination!

Pagination with APIs works a little bit differently but has the same underlying concept. When you try to retrieve a list of objects that is too large, either for the computational costs associated with retrieval (or consumption) or any of the plethora of networking concerns (waiting for a few gigs of data from an API call can have unexpected effects on your application), the service will generally respond with information about how to access the next “page”, or section, of results.

I’m going to refer to this tidbit of information as a “cursor”, but it could have many names and forms including:

• A token that you provide to the endpoint for subsequent requests either as a URL parameter, or in the header of your request.

• Standardized url parameters about which page you are on/requesting, such as &page=3 or start=100

• A link to a different URL to make the request to (this one usually has a token or URL parameter above baked in)

To help explain the concept a little more I made an animation of what a cursor added to a response looks like for our v2 List Transactions endpoint. Your code would make a request to the List Transactions endpoint, and if you have more than 50 transactions, the API would return the first 50 and attach another field to the json response called cursor. This is a pagination token, and attaching that to your next request (in the form of a URL parameter) tells the API which transactions you have already seen, and where to start when returning the next 50 results. That response includes a different pagination cursor for you to get the *next *50 results. That continues until you have “paged through” all the results that you requested, with the last page usually having fewer results than the expected page size. You can see a conceptual example of what that looks like in the animation below.

An animation of Square’s cursor based pagination

Square’s payment APIs use the token based approach, but in two different ways: The v1 endpoints return a header with a link to where to send the next request, while the v2 endpoints add a cursor token to the json response, and accept it in subsequent requests as a URL parameter.

With the 2.2.1 release of our SDKs, we’ve added additional functions to access the pagination tokens from the headers, like [getV1BatchTokenFromHeaders()](https://github.com/square/connect-php-sdk/blob/master/lib/ApiClient.php#L288)in PHP. Use those to get the pagination tokens with the client libraries more easily.

How to get them all?

In most cases, when you experience pagination, you likely want all of the items in the list, and aren’t very happy about the extra work required to page through the results. Here are some best practices to make it a little less painful to get all of your results:

Avoid it

Although a little cheeky, conceptually this is how you will be able to get the most performance. Many APIs allow you to query for items based on specific criteria, or only return certain subsets of the data. Square’s List Transactions endpoint for example allows you to query based on time ranges. If you know that the transactions you are looking for only occur in a certain time range, you can narrow down the result set from the beginning, in many cases eliminating pagination and giving you a performance boost in comparison to over fetching and then filtering in your application.

While looping

Depending on your programming language of choice, pagination can be a wonderful use case for while. The basic workflow is that while you are getting a pagination token in your response, keep making subsequent requests. In pseudo code that might look like this:

Page = GetPageOfItems();
//process the data from the page, or add it to a larger array, etc.

while( Page->cursor )
    Page  = GetPageOfItems(Page->cursor);
    //process the data again
end

As soon as your API stops responding with a pagination cursor, then you can stop looping and continue execution with the complete set of data. There are a couple important points to remember with an approach like this:

• It is easy to hit rate limits with this approach, so keep a look out for things like a 429 response code.

• Any errors from the API that don’t return a cursor might prematurely stop your execution, so always check that you get a good response from the API.

• You probably want to include some checks to make sure that you stop bringing in more information at some upper limit, just in case the API responds with much more information than you expect and keeps paging, and paging, and paging…

Hopefully this is a helpful resource for the next time you need to paginate (or avoid paginating!) your results from an API. If you have more questions about pagination with Square’s APIs, take a look at the resources for v1 & v2.