Contents

Chapter 1

Features

This chapter covers the core features and capabilities of Robots.txt Generator.

Features

  • 4 built-in presets: standard, strict, open, lockdown
  • Block AI crawlers (GPTBot, CCBot, anthropic-ai, Google-Extended)
  • Multiple user-agent blocks with Allow/Disallow rules
  • Crawl-delay support
  • Sitemap references (from config or CLI flags)
  • Generation timestamp
  • Database of 20+ known crawler user-agents
  • Config validation with warnings
  • Python stdlib only — zero dependencies

Quick Start

bash
# Use a preset
python src/robots_txt_generator.py --preset standard --sitemap https://example.com/sitemap.xml

# From a config file
python src/robots_txt_generator.py --config examples/robots_config.json --output robots.txt

# Block AI crawlers + allow search engines
python src/robots_txt_generator.py --preset strict --sitemap https://example.com/sitemap.xml

# Staging/dev lockdown
python src/robots_txt_generator.py --preset lockdown --output robots.txt

# List available presets
python src/robots_txt_generator.py --list-presets

# List known crawlers
python src/robots_txt_generator.py --list-crawlers
Chapter 2

Presets

Follow this guide to get Robots.txt Generator up and running in your environment.

Presets

PresetDescription
standardAllow everything, block admin paths
strictBlock AI crawlers and SEO tools
openAllow all crawlers everywhere
lockdownBlock everything (staging/dev)

Configuration Reference

See examples/robots_config.json for a full example.

Chapter 3
🔒 Available in full product

CLI Flags

You’ve reached the end of the free preview

Get the full Robots.txt Generator and unlock everything.

All Chapters

Get the complete guide with every chapter unlocked, including code samples, diagrams, and best practices.

Full Tool Suite

Access all interactive tools with complete data, all workload profiles, and the full scenario library.

Source Files

Downloadable source code, configuration files, and working examples from every chapter.

Lifetime Updates

Free updates for life. Every new chapter, tool, and improvement included.

Buy Now — $19 →
📦 Free sample included — download another copy for the full product.
Robots.txt Generator v1.0.0 — Free Preview