Crawl multiple URLs based on options

POST

/api/v2/crawler/crawl

Last modified:2025-07-25 01:11:05

Request

Authorization

Add parameter in header

x-api-token

Example:

x-api-token: ********************

Body Params application/json

Example

{
    "url": "http://example.com",
    "limit": 10000,
    "excludePaths": [
        "string"
    ],
    "includePaths": [
        "string"
    ],
    "maxDepth": 10,
    "maxDiscoveryDepth": 0,
    "ignoreSitemap": false,
    "ignoreQueryParameters": false,
    "deduplicateSimilarURLs": true,
    "regexOnFullURL": true,
    "allowBackwardLinks": false,
    "allowExternalLinks": false,
    "delay": 0,
    "scrapeOptions": {
        "formats": [
            "markdown"
        ],
        "onlyMainContent": true,
        "includeTags": [
            "string"
        ],
        "excludeTags": [
            "string"
        ],
        "headers": {},
        "waitFor": 0,
        "timeout": 30000
    },
    "browserOptions": {
        "sessionName": "string",
        "sessionTTL": "string",
        "sessionRecording": "string",
        "proxyCountry": "string",
        "proxyURL": "string",
        "fingerprint": "string"
    }
}

Request Code Samples

Shell

JavaScript

Java

Swift

PHP

Python

HTTP

Objective-C

Ruby

OCaml

Dart

curl --location --request POST 'https://api.scrapeless.com/api/v2/crawler/crawl' \
--header 'x-api-token: <api-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "http://example.com",
    "limit": 10000,
    "excludePaths": [
        "string"
    ],
    "includePaths": [
        "string"
    ],
    "maxDepth": 10,
    "maxDiscoveryDepth": 0,
    "ignoreSitemap": false,
    "ignoreQueryParameters": false,
    "deduplicateSimilarURLs": true,
    "regexOnFullURL": true,
    "allowBackwardLinks": false,
    "allowExternalLinks": false,
    "delay": 0,
    "scrapeOptions": {
        "formats": [
            "markdown"
        ],
        "onlyMainContent": true,
        "includeTags": [
            "string"
        ],
        "excludeTags": [
            "string"
        ],
        "headers": {},
        "waitFor": 0,
        "timeout": 30000
    },
    "browserOptions": {
        "sessionName": "string",
        "sessionTTL": "string",
        "sessionRecording": "string",
        "proxyCountry": "string",
        "proxyURL": "string",
        "fingerprint": "string"
    }
}'

Responses

🟢200OK

application/json

Successful response

Body

Example

{
    "success": true,
    "id": "string"
}

🟠402402

🟠429429

🔴500Server Error

Modified at 2025-07-25 01:11:05

Get the errors of a batch scrape job

Cancel a crawl job