Schedule web crawls with Python, Node.js and cron

Name: Oassis
Price: 0.002 USD
Author: Oassis

Use your scheduler to start bounded crawl jobs. Oassis crawl jobs run once; recurring execution is managed by your application.

Choose scheduled monitoring or a crawl

For a ready-made daily or weekly report, see Price Watch and Competitor Monitor. For a custom dataset, schedule the crawl endpoint yourself. Do not confuse an external cron schedule with a native recurring crawl parameter.

Python: start one bounded crawl

Install requests in your environment and provide OASSIS_API_KEY through your secret manager.

import os, requests

response = requests.post(
    "https://api.oassis.dev/tasks/v1/crawl",
    headers={"Authorization": "Bearer " + os.environ["OASSIS_API_KEY"]},
    json={"url": "https://example.com", "limit": 5,
          "maxDepth": 1, "formats": ["markdown"]},
    timeout=60,
)
response.raise_for_status()
job = response.json()
if not job.get("jobId"):
    raise RuntimeError("Crawl was not accepted: " + str(job))
# Persist this ID in your job store; poll the owning-key status endpoint.
print(job["jobId"])

Node.js: the same crawl

Use a Node.js version that provides fetch and AbortSignal.timeout.

const response = await fetch('https://api.oassis.dev/tasks/v1/crawl', {
  method: 'POST',
  headers: {'Authorization': 'Bearer ' + process.env.OASSIS_API_KEY,
            'Content-Type': 'application/json'},
  body: JSON.stringify({url:'https://example.com', limit:5,
                        maxDepth:1, formats:['markdown']}),
  signal: AbortSignal.timeout(60000)
});
if (!response.ok) throw new Error('HTTP ' + response.status);
const job = await response.json();
if (!job.jobId) throw new Error('Crawl not accepted');
console.log(job.jobId); // Persist and poll; do not start again to check status.

Run daily with cron

0 6 * * * /usr/bin/flock -n /tmp/oassis-crawl.lock /usr/bin/python3 /opt/jobs/start_crawl.py >> /var/log/oassis-crawl.log 2>&1

This is a Linux runner example using flock, not an Oassis setting. Adjust binary paths, permissions, secret loading and the scheduler timezone. Prevent duplicate in-flight crawls in your job store as well: the launcher exits before the remote job finishes. After a timeout, inspect your stored state before retrying a paid start.

Collect results

curl https://api.oassis.dev/tasks/v1/crawl/JOB_ID \
  -H "Authorization: Bearer YOUR_API_KEY"

Use the owning API key, persist results and poll with backoff until a terminal state. Treat partial results and per-page errors explicitly; do not equate acceptance with successful completion.

Reduce failures without promising to avoid blocks

Start with a small page limit, restrict paths, avoid overlapping jobs and reduce frequency when the source changes slowly. Respect access conditions and robots rules. Oassis does not guarantee unblocked access or provide an advertised built-in CAPTCHA solver. Use an authorized API or ask the site owner for access when automation is restricted.

Try Scrape & Map · API quick start · Estimate your cost

Oassis basic fresh Markdown extraction: $0.002 per page output. All operation prices and calculator.