Pagination

Endpoints that list resources (actors, runs, tasks, schedules, webhooks, listening mentions…) return one page at a time.

Page and per_page

Parameter	Type	Default	Description
page	integer ≥ 1	1	The page to return.
per_page	integer, at most 100	Depends on the endpoint	Items per page.

Second page of 50 runs

curl "https://lantfly.com/api/v3/runs?page=2&per_page=50" \
  -H "Authorization: Bearer $LANTFLY_TOKEN"

The answer carries the items in data and the position in meta:

Paged answer

{
  "data": [ { "id": "…", "status": "SUCCEEDED" } ],
  "meta": {
    "current_page": 2,
    "last_page": 7,
    "per_page": 50,
    "total": 318
  }
}

meta field	Description
current_page	The page you got.
last_page	The last page there is.
per_page	Items per page used.
total	Items across all pages.

Read every page

all_runs.py

import os
import requests

session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['LANTFLY_TOKEN']}"

page, runs = 1, []
while True:
    res = session.get(
        "https://lantfly.com/api/v3/runs",
        params={"page": page, "per_page": 100},
        timeout=30,
    )
    res.raise_for_status()
    body = res.json()
    runs.extend(body["data"])
    if page >= body["meta"]["last_page"]:
        break
    page += 1

print(len(runs), "runs")

note

A few lists return everything when you send neither page nor per_page, and page only when you ask for it: for example a listening project's sources and collections. Each endpoint's maximum per_page is in the API reference.

Storage content

Storage content can hold millions of entries, so it pages differently.

What	Parameters	Next page
Dataset items, GET /datasets/{id}/items	offset, limit, desc	offset + count. meta holds total, offset, limit, count, desc; the same values are in X-Lantfly-Pagination-Total, -Offset, -Limit, -Count and -Desc headers.
Key-value store keys, GET /key-value-stores/{id}/keys	limit (at most 1,000), exclusive_start_key	Send meta.next_exclusive_start_key while meta.is_truncated is true.
Request queue requests, GET /request-queues/{id}/requests	limit (at most 10,000), cursor, handled	Send meta.next_cursor until it's empty.

Dataset limit is capped at 250,000 items for json, jsonl and csv (streamed) and 20,000 for xlsx, html, xml and rss. A larger limit is lowered to the cap without an error. See Datasets.

Related

• API overview
• Datasets