Web crawling API for site research

Read beyond a single page. Start with a site map, then crawl only as far as the task needs.

Discover URLs before reading pages

curl -i https://api.oassis.dev/web/v1/map \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","limit":100}'

Mapping lists URLs; it does not replace reading their content. Use discovered links to choose the pages most relevant to the agent's task.

Start a bounded crawl

curl -i https://api.oassis.dev/tasks/v1/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","limit":10,"maxDepth":2}'

The crawl starts an asynchronous job and returns a jobId. Keep that identifier, then inspect its progress and results.

curl -i https://api.oassis.dev/tasks/v1/crawl/JOB_ID

Set the scope

limit allows 1–200 pages and defaults to 25. maxDepth allows 0–5 and defaults to 2; zero reads only the starting page. See the reference for URL filters and the full status response.

Understand the cost

HTTP crawls are charged up front for the allowed page budget. The implementation refunds pages it does not read. Use a small limit first and inspect the quote before paying.

Use the results

Crawling can supply a documentation index, a research dataset or source material for retrieval. Your application still needs to select, validate and store the results. Website navigation and access restrictions determine which pages can be reached.

Continue with MCP integration, x402 payments or the OpenAPI reference.