Features

This chapter covers the core features and capabilities of Data Sync.

Features

• Hash-based diff engine — SHA-256 content hashing detects added, changed, deleted, and unchanged records
• Multi-format endpoints — Read and write JSON files, CSV files, and SQLite tables
• Conflict resolution — Three strategies: source-wins, dest-wins, newest-wins (by timestamp field)
• Incremental sync — Persist state between runs to detect true conflicts (both sides changed)
• Dry-run mode — Preview the sync plan without writing any changes
• Bidirectional sync — Propagate dest-only records back to source instead of deleting them
• Detailed reporting — Diff report with per-record change breakdown and summary statistics
• Config file support — Define sync jobs in JSON for repeatable, scriptable execution

Requirements

• Python 3.10+
• No external dependencies (stdlib only)

Quick Start

Follow this guide to get Data Sync up and running in your environment.

Quick Start

# Sync two JSON files by a key field
python src/data_sync.py --source customers_a.json --dest customers_b.json --key customer_id

# Sync CSV to SQLite with conflict strategy
python src/data_sync.py --source orders.csv --dest warehouse.db --key order_id --strategy source-wins

# Preview changes without writing (dry-run)
python src/data_sync.py --source new_data.json --dest master.json --key id --dry-run

# Incremental sync with state tracking
python src/data_sync.py --source export.json --dest mirror.json --key id --state sync_state.json

# Full pipeline from a config file
python src/data_sync.py --config examples/data_sync_config.json

Configuration Reference

Define a sync job in JSON:

{
    "source": "customers_export.json",
    "dest": "customers_master.json",
    "key": "customer_id",
    "strategy": "newest-wins",
    "timestamp_field": "updated_at",
    "state": "sync_state.json",
    "dry_run": false,
    "bidirectional": false,
    "source_table": "customers",
    "dest_table": "customers"
}

CLI Flags

Config Schema