Configuration Reference

Create a JSON config file for repeatable scrape jobs:

{
    "url": "https://api.docs.example.com/products",
    "selector": ".product-card",
    "follow_links": true,
    "next_page_selector": "a.next-page",
    "max_pages": 5,
    "delay": 1.5,
    "proxies": ["http://proxy1.docs.example.com:8080"],
    "output_format": "json",
    "output_path": "products.json"
}

Field	Type	Default	Description
`url`	string	—	Target URL to scrape
`selector`	string	—	CSS-like selector for elements
`follow_links`	bool	`false`	Follow pagination links
`next_page_selector`	string	—	Selector for next-page link
`max_pages`	int	`10`	Maximum pages to follow
`delay`	float	`1.0`	Seconds between requests
`timeout`	int	`30`	HTTP timeout in seconds
`proxies`	list	`[]`	List of proxy URLs for rotation
`output_format`	string	`"json"`	`json`, `csv`, or `sqlite`
`output_path`	string	`"scraped_data.json"`	Output file path
`extract_attrs`	list	`[]`	HTML attributes to extract
`max_retries`	int	`3`	Max retry attempts per request

Included with Web Scraper

Ready for the complete Web Scraper?