Async API: Support for pagination of results

[aalex21]

In case of asyncQuery, retrieving results as embedded response (JSON API), What is the best approach to paginate the records ?

Use Case:
1, Result Data (responseBody) records are huge.
2, Possibility of supporting pagination, for "responseBody"
3, If yes, how to pass the page information to asyncQuery ?
4, How to be in compliance with JSONAPI spec - GET for a specific resource, but paginated !!

Example:
GET http://localhost:8080/api/v1/asyncQuery/ba31ca4e-ed8f-4be0-a0f3-12088fa9263d
{
"data": {
"type": "asyncQuery",
"id": "ba31ca4e-ed8f-4be0-a0f3-12088fa9263d",
"attributes": {
"query": "/group?sort=commonName&fields%5Bgroup%5D=commonName,description",
"queryType": "JSONAPI_V1_0",
"result": {
"recordCount": 5000,
"responseBody": "...."
}
}
}
}

[aklish]

You can add pagination parameters to your JSON API query inside the body of your async request. To get the next page, you will have to issue another (separate) async query with different parameters.

For the syntax, you can look at:
https://elide.io/pages/guide/v6/10-jsonapi.html#pagination

[aalex21]

to make it clear,

1. POST /asyncQuery --> created the required results in backend which has huge number of records
2. GET /asyncQuery --> looking to get the generated results in batches / pages

Since the result is already available, can we avoid another post with the page details ?

[aklish]

I believe I understand your use case. The async API currently stores the response in a blob in the database (there is no mechanism to paginate that). We primarily use async for analytic queries. For large responses, we leverage table export where we can offload the response to secondary storage and stream it from there (S3, redis, etc). Table export however is really only for flat models (no relationships).

Async was designed for queries that might take a long time to return but have a reasonable sized payload response. There is no mechanism today to issue a query once for everything and then paginate the result (stored in offline storage).

[aalex21]

Got it, thanks