Manage Scrapers

Create and manage agentic scrapers via the Agent API. Agentic scrapers use an AI agent that browses websites, navigates pages, and builds a reusable extraction script — ideal for complex sites that require interaction.

Generate Scraper

Build an agentic scraper by providing a URL and instructions. If template_id is omitted, a new scraper is created automatically.

The AI agent will browse the site, learn its layout, and create a reusable extraction script. This endpoint is asynchronous — it returns immediately and the build runs in the background.

Endpoint: POST /v1/agent/generate

curl https://api.parsera.org/v1/agent/generate \
--header 'Content-Type: application/json' \
--header 'X-API-KEY: <YOUR_API_KEY>' \
--data '{
    "url": "https://news.ycombinator.com/",
    "prompt": "Extract news titles with their scores",
    "attributes": [
        {
            "name": "title",
            "description": "News title"
        },
        {
            "name": "score",
            "description": "Number of points"
        }
    ]
}'

Parameters:

Response (202 Accepted):

{
    "template_id": "abc123",
    "status": "generating"
}

To check build progress, poll the scraper details endpoint.

Create Empty Scraper

Create a new empty agentic scraper. This is useful when you need a scraper ID before triggering generation — otherwise, POST /v1/agent/generate auto-creates one for you.

Endpoint: POST /v1/agent/new

curl https://api.parsera.org/v1/agent/new \
--header 'X-API-KEY: <YOUR_API_KEY>' \
--request POST

Response:

{
    "template_id": "abc123"
}

Get Scraper Details

Get an agentic scraper's details and build status:

Endpoint: GET /v1/agent/{scraper_id}

curl https://api.parsera.org/v1/agent/abc123 \
--header 'X-API-KEY: <YOUR_API_KEY>'

Response:

{
    "template_id": "abc123",
    "type": "agentic",
    "status": "ready",
    "name": "hackernews",
    "url": "https://news.ycombinator.com/",
    "sample_data": [
        {
            "title": "Show HN: A new approach to web scraping",
            "score": "142"
        }
    ],
    "credits_charged": 50
}

Generation statuses: generating, ready, failed

Delete Scraper

Delete an agentic scraper. Stops any active build process.

Endpoint: DELETE /v1/agent/{scraper_id}

curl https://api.parsera.org/v1/agent/abc123 \
--header 'X-API-KEY: <YOUR_API_KEY>' \
--request DELETE

Response:

{
    "message": "Scraper deleted successfully."
}

Running Scrapers

Once an agentic scraper is built, run it via the Scrapers API using POST /v1/scrapers/run or POST /v1/scrapers/run_async.

Related Documentation

• Agent API — One-shot extraction and building reusable scrapers
• Scrapers API — Run scrapers synchronously or asynchronously