⚙️ Automation Hub — Web Scraper Demo

• CSS-like selectors — Extract elements by tag, `.class`, or `#id`
• Automatic pagination — Follow next-page links across multiple pages
• Rate limiting — Configurable delay between requests (be a polite scraper)
• Proxy rotation — Round-robin through a list of proxy servers
• Retry with backoff — Exponential backoff on 429/5xx errors
• Multi-format export — Save to JSON, CSV, or SQLite

# Scrape all h2 elements from a page
python src/web_scraper.py --url "https://example.com" --selector "h2"

# Scrape product cards with pagination
python src/web_scraper.py --url "https://example.com/shop" --selector ".product" \
    --follow --next-selector "a.next" --max-pages 5

# Export to CSV
python src/web_scraper.py --url "https://example.com" --selector ".item" \
    --format csv --output items.csv

# Use a config file for complex jobs
python src/web_scraper.py --config examples/scrape_config.json

{
    "url": "https://api.example.com/products",
    "selector": ".product-card",
    "follow_links": true,
    "next_page_selector": "a.next-page",
    "max_pages": 5,
    "delay": 1.5,
    "proxies": ["http://proxy1.example.com:8080"],
    "output_format": "json",
    "output_path": "products.json"
}