Request queues

A request queue holds the URLs a crawler still has to visit, and the ones it has already handled. Requests are de-duplicated by a unique key (the URL by default), so adding the same page twice does nothing. Crawlers built with the Actor SDK use the run's default queue for you.

In the Console, open Storage → Request queues and a queue to see its Handled / total counts and its pending requests.

Add a request

Add a URL to a queue

curl -s -X POST "https://lantfly.com/api/v3/request-queues/$QUEUE_ID/requests" \
  -H "Authorization: Bearer $LANTFLY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products?page=2",
    "method": "GET",
    "user_data": { "label": "LIST" }
  }'

Field	Meaning
url	The URL to visit (required)
unique_key	De-duplication key; defaults to the URL
method, payload, headers	The HTTP request to make
user_data	Any JSON your crawler wants to keep with the request

Add ?forefront=1 to put the request at the front of the queue. The answer is 201 for a new request and 200 when it was already there, with request_id, was_already_present and was_already_handled.

Read requests

GET /api/v3/request-queues/:queue/requests lists requests in order, up to 10,000 per call (limit).

Parameter	Meaning
handled	0 for pending requests, 1 for handled ones
limit	Requests per call
cursor	meta.next_cursor of the previous answer, for the next page

DELETE /api/v3/request-queues/:queue/requests/:request removes one request.

Counts and management

GET /api/v3/request-queues/:queue returns total_request_count, handled_request_count and pending_request_count. Create, rename and delete queues like other storages (see Storage).

Request queues have no public links.