REST API Response Pagination, Sorting and Filtering

A REST API may return more results (sometimes thousands) than a client needs or can handle at a time. In this case, the API shall return paginated responses, which are essentially a subset of the results. The API can further refine the subset of responses by allowing filtering on the results.

APIs that require pagination are generally GET APIs, so the most common practice for providing the pagination and filtering criteria is through query parameters.

1. Query Parameters

RFC-8040 defines the most basic requirements about the query parameters and how the API should handle them:

• Query parameters can be given in any order.
• Each parameter can appear at most once in a request URI.
• If more than one instance of a query parameter is present, the server MUST return a “400 Bad Request” status line.
• A server MUST return an error with a “400 Bad Request” status-line if a query parameter is unexpected.
• A default value may apply if the query parameter is missing.
• Query parameter names and values are case-sensitive.

When designing the API pagination, we must make sure that rules are followed in the first place.

2. Pagination

As a best practice, REST APIs SHOULD support server-side pagination from day one, even for all collections, as adding pagination is a breaking change. Server-driven paging mitigates against denial-of-service attacks by forcibly paginating a request over multiple response payloads.

When a response is paginated, the response headers must include the link headers. The link header contains URLs that you can use to fetch additional pages of results. For example, the previous, next, first, and last page of results.

2.1. Using Link Headers

Suppose we have an API that returns the list of books. By default, it returns 10 books in each paged response. The following is a sample response when we have requested the page number 2. In this example:

• The URL for the current page is followed by rel="self".
• The URL for the first page is followed by rel="first".
• The URL for the previous page is followed by rel="prev".
• The URL for the next page is followed by rel="next".
• The URL for the last page is followed by rel="last".

HTTP GET http://localhost:8080/api/books?page=1&size=10

HTTP/1.1 200 OK
Content-Type: application/json
Link: <http://localhost:8080/api/books/paged?page=1&size=10>; rel="self"
Link: <http://localhost:8080/api/books/paged?page=0&size=10>; rel="first"
Link: <http://localhost:8080/api/books/paged?page=0&size=10>; rel="prev"
Link: <http://localhost:8080/api/books/paged?page=2&size=10>; rel="next"
Link: <http://localhost:8080/api/books/paged?page=5&size=10>; rel="last"
X-Total-Items: 54
X-Page: 1
X-Page-Size: 10
X-Total-Pages: 6

{
  "books": [
    {
      "id": 1,
      "title": "The Great Gatsby",
      "author": "F. Scott Fitzgerald",
      "_links": {
        "self": {
          "href": "http://localhost:8080/api/books/1"
        }
      }
    },
    //... more books
  ],
  "page": {
    "size": 10,
    "totalElements": 54,
    "totalPages": 6,
    "number": 1
  }
}

2.2. Adding Links in Response Payload

Some API implementations include the pagination links in the response body as well. This is also a valid design.

HTTP GET http://localhost:8080/api/books?page=1&size=10

HTTP/1.1 200 OK
Content-Type: application/json
X-Total-Items: 54
X-Page: 1
X-Page-Size: 10
X-Total-Pages: 6

{
  "_embedded": {
    "bookList": [
      {
        "id": 1,
        "title": "The Great Gatsby",
        "author": "F. Scott Fitzgerald",
        "_links": {
          "self": {
            "href": "http://localhost:8080/api/books/1"
          }
        }
      },
      // ... more books
    ]
  },
  "_links": {
    "self": {
      "href": "http://localhost:8080/api/books?page=1&size=10"
    },
    "first": {
      "href": "http://localhost:8080/api/books?page=0&size=10"
    },
    "prev": {
      "href": "http://localhost:8080/api/books?page=0&size=10"
    },
    "next": {
      "href": "http://localhost:8080/api/books?page=2&size=10"
    },
    "last": {
      "href": "http://localhost:8080/api/books?page=5&size=10"
    }
  },
  "page": {
    "size": 10,
    "totalElements": 54,
    "totalPages": 6,
    "number": 1
  }
}

2.3. Missing Links

In some cases, only a subset of these links may be available in the response payload. For example, the link to the previous page won’t be included if you are on the first page of results. Similarly, the link to the last page won’t be included if it can’t be calculated.

In the following example, we are fetching the first page so the previous page link header is unavailable.

HTTP GET http://localhost:8080/api/books?page=0&size=10

HTTP/1.1 200 OK
Content-Type: application/json
Link: <http://localhost:8080/api/books/paged?page=0&size=10>; rel="self"
Link: <http://localhost:8080/api/books/paged?page=0&size=10>; rel="first"
Link: <http://localhost:8080/api/books/paged?page=1&size=10>; rel="next"
Link: <http://localhost:8080/api/books/paged?page=5&size=10>; rel="last"

//... more response body

2.4. Page Size

The API should not enforce a hard limit on the page size. The API client should be able to change the page size parameter based on its requirements. The server SHOULD honor this preference if the specified page size is smaller than the server’s default page size.

For example, a client displaying the books on a small screen device might want to display only 5 books at a time. The size parameter must be able to alter the number of items in the response subset.

Note that if the client’s requested page size is larger than the default page size supported by the server, the expected response should be the number of results specified by the client but paginated on the server side as specified by the server paging defaults.

In the following example, we are requesting the page size of 20 while the defult page size on the server is 10. In this case, the server will internally fetch the two pages (20 items) using two SQL queries and return it to the client. As we have only 54 items in the database, there will be only 3 pages available to the client.

HTTP GET http://localhost:8080/api/books?page=0&size=20

HTTP/1.1 200 OK
Content-Type: application/json
Link: <http://localhost:8080/api/books/paged?page=0&size=20>; rel="self"
Link: <http://localhost:8080/api/books/paged?page=0&size=20>; rel="first"
Link: <http://localhost:8080/api/books/paged?page=1&size=20>; rel="next"
Link: <http://localhost:8080/api/books/paged?page=2&size=20>; rel="last"

//... more response body

3. Sorting

While implementing the mandatory pagination, it is quite essential to add a default sort order in the API. The server MUST supplement any sorting order criteria to ensure that items are always ordered consistently.

By default, the sorting-related query parameter should be optional and rely on the server-side default sort settings. When required, the client overrides the default sorting behavior by providing additional 'sort' or 'orderBy' parameter.

HTTP GET http://localhost:8080/api/books?page=0&size=10&orderBy=author

The value of the 'orderBy' parameter may contain a comma-separated list of expressions used to sort the items. In this case, the API will sort by the first field specified, then sort by the next, and so on.

HTTP GET http://localhost:8080/api/books?page=0&size=10&orderBy=author,title

The expression MAY include the suffix "asc" for ascending or "desc" for descending order. If "asc" or "desc" is not specified, the service MUST order by the specified property in the default order specified in the server-side configuration.

We may also include any special character to denote the sort direction. For example, adding a hyphen sign means that the field has to be sorted in descending order.

# sort order in one field
HTTP GET http://localhost:8080/api/books?page=0&size=10&orderBy=author,title asc

# sort order in multiple fields
HTTP GET http://localhost:8080/api/books?page=0&size=10&orderBy=author asc,title desc

# Using special hyphen symbol 
HTTP GET http://localhost:8080/api/books?page=0&size=10&orderBy=author,-title

If a service does not support sorting by a field named in an ‘orderBy‘ expression, the service MUST respond with an error message describing the root cause.

HTTP/1.1 400 Bad Request
Content-Type: application/json
Request-Id: 123e4567-e89b-12d3-a456-426614174000

{
  "error": {
    "code": "InvalidOrderByExpression",
    "message": "The property 'author' in the orderby expression is not sortable",
    "details": [
      {
        "code": "UnsupportedSortProperty",
        "target": "author",
        "message": "Sorting by 'author' is not supported. Supported sort properties are: ['id', 'title']"
      }
    ],
    "target": "orderby",
    "internal": {
      "trace-id": "00-d24f899d9c8a5a428fddd39399e7f58e-5c8a8949fcdd9a42-01",
      "timestamp": "2024-02-20T15:30:45.123Z",
      "request-id": "123e4567-e89b-12d3-a456-426614174000"
    }
  }
}

4. Filtering

An API client may not require the default list of items all the time. Sometimes, it may need a specific type of items only. For example, the client may need only the books available in the stock or the list of books written by a particular author. So it is quite important that a pagination-supported API also supports the filtering functionality. Filtering will enable the API clients to request only the required resources that it is interested in.

RFC-8040 suggests to use "filter" query parameter for filtering purposes. When available in the request URI, this mandates the server to return only a subset of all possible items.

# The 'eq' operator should execute LIKE query in the database
HTTP GET http://localhost:8080/api/books?page=0&size=20&filter=author eq 'Fitzgerald'

Similar to pagination and sorting parameters, filter parameter also is only applicable to GET and HEAD requests. A “400 Bad Request” status-line is returned if used for other methods or resource types.

The Microsoft API guidelines provide an excellent overview of set of operations that a filter expression can support. The list is not exaustive and you can add more operations based on requirements.

The filter expression SHOULD be capable of accepting more than one filter criterias. For example, the following expression will return the list of books written by author either Fitzgerald or Redmond; and the price of book should not exceed 2.55.

HTTP GET http://localhost:8080/api/books?page=0&size=20&filter=(author eq 'Fitzgerald' or name eq 'Redmond') and price lt 2.55

As a 'filter' expression may contain multiple operators so it is essential to implement the proper operator precedence when evaluating the expression.

5. Summary

Implementing pagination, sorting, and filtering enhances a REST API’s performance by optimizing the server load. It also enables API clients to retrieve precisely the data they need.

Note that we must ensure that the API is guarded against several types of injection attacks by sanitizing inputs, especially in complex filtering conditions.

References:

https://www.rfc-editor.org/rfc/rfc8040.html#section-4.8
https://github.com/microsoft/api-guidelines/blob/vNext/graph/articles/collections.md
https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api