This chapter covers the core features and capabilities of ETL Pipeline.
Follow this guide to get ETL Pipeline up and running in your environment.
# CSV to JSON conversion
python src/etl_pipeline.py --source data.csv --dest output.json
# JSON API to SQLite
python src/etl_pipeline.py --source https://api.example.com/v1/users --dest users.db --table users
# Full pipeline from config
python src/etl_pipeline.py --config examples/pipeline_config.json
# Nested JSON extraction
python src/etl_pipeline.py --source response.json --dest items.csv --records-path "data.items"Create a JSON config for complex ETL jobs:
{
"source": "https://api.example.com/v1/products",
"destination": "products.db",
"table": "products",
"records_path": "data.results",
"transforms": [
{"type": "filter", "field": "status", "value": "active"},
{"type": "rename", "mapping": {"product_name": "name", "product_id": "id"}},
{"type": "map", "field": "price", "operation": "float"},
{"type": "add_field", "field": "imported_at", "value": "$NOW"}
],
"stats_file": "pipeline_stats.json"
}| Flag | Default | Description |
|---|---|---|
--source, -s | — | Source file path or URL |
--dest, -d | — | Destination file path |
--config, -c | — | Pipeline config file (JSON) |
--table, -t | — | SQLite table name |
--records-path | — | Dot-path to records array in JSON (e.g. data.items) |
--stats-file | — | Write pipeline stats to this JSON file |
--log-level | INFO | Logging level (DEBUG, INFO, WARNING, ERROR) |
| Type | Description | Config Fields |
|---|---|---|
filter | Keep records matching a condition | field, value, operator |
rename | Rename fields | mapping (dict) |
map | Transform a field value | field, operation |
add_field | Add a new field | field, value ($NOW for timestamp) |
remove_fields | Drop fields | fields (list) |
Get the full ETL Pipeline and unlock everything.
Get the complete guide with every chapter unlocked, including code samples, diagrams, and best practices.
Access all interactive tools with complete data, all workload profiles, and the full scenario library.
Downloadable source code, configuration files, and working examples from every chapter.
Free updates for life. Every new chapter, tool, and improvement included.