Migration Methodology Guide

Data Migration Factory | Datanest Digital | https://datanest.dev

Overview

This guide describes the end-to-end methodology for migrating data from

heterogeneous source systems into a Databricks Lakehouse using the Data

Migration Factory toolkit. The methodology is structured in six phases,

each with clear inputs, activities, deliverables, and exit criteria.

Methodology Phases

Phase 1        Phase 2       Phase 3        Phase 4       Phase 5         Phase 6
Discovery  --> Assessment --> Development --> Execution --> Cutover --> Decommission
(1-2 wk)      (1-2 wk)      (2-4 wk)       (2-6 wk)     (1 day)      (1-4 wk)

Phase 1: Discovery

Objective

Establish the scope, identify all source systems, and align stakeholders on

migration goals, constraints, and success criteria.

Activities

1. Stakeholder interviews — Meet data owners, application teams, and

operations to understand current data flows and pain points.

2. Source system inventory — Document all databases, file stores, and

APIs that hold data to be migrated.

3. Dependency mapping — Identify upstream producers and downstream

consumers of each data source.

4. Constraint identification — Document maintenance windows, compliance

requirements, data residency rules, and SLAs.

5. Success criteria definition — Agree on measurable criteria: zero data

loss, max acceptable downtime, performance thresholds.

Deliverables

• Source system inventory spreadsheet
• Data flow diagrams
• Migration scope document with in/out-of-scope tables
• Success criteria matrix

Exit Criteria

• [ ] All source systems identified and documented
• [ ] Stakeholder alignment on scope and success criteria
• [ ] Migration timeline agreed

Phase 2: Assessment

Objective

Catalog every table, measure volumes, evaluate complexity, and produce a

prioritised wave plan.

Toolkit Components

Activities

1. Automated assessment — Run source_assessment.py against each source

system. The notebook catalogs tables, column metadata, row counts, primary

keys, foreign keys, and timestamp column availability.

2. Complexity scoring — Each table receives a complexity score based on:

• Row volume (1-5 scale)
• Column width (1-4 scale)
• Incremental readiness (PK + timestamp availability)
• Dependency depth (foreign key chain length)

3. Wave planning — Run migration_wave_planner.py to group tables into

ordered waves. Parameters:

• --max-tables-per-wave: Limit tables per batch (default: 20)
• --max-rows-per-wave: Limit total rows per batch (default: 500M)

4. Manual review — Adjust wave assignments for business priority, testing

dependencies, or regulatory requirements.

Wave Strategy Guidelines

Deliverables

• Assessment catalog in Unity Catalog (migration.assessment.source_tables)
• Dependency graph (migration.assessment.table_dependencies)
• Wave plan JSON file
• Reviewed and approved wave schedule

Exit Criteria

• [ ] All source tables cataloged with row counts and metadata
• [ ] Wave plan generated and reviewed by technical lead
• [ ] Estimated timeline per wave documented

Phase 3: Development

Objective

Configure extractors, build target schemas, implement validation, and establish

operational tooling.

Activities

1. Target schema design

• Create Unity Catalog catalogs and schemas for each layer (raw, curated)
• Define naming conventions
• Configure Delta table properties (partitioning, Z-ORDER columns, liquid clustering)

2. Extractor configuration

• Select the appropriate extractor for each source:

• Configure JDBC connection details, secret scopes, primary keys, and

partition columns for each table.

3. Validation setup

• Configure validation/reconciliation.py with source JDBC details and

target table references.

• Configure validation/data_sampling.py for large-table spot checks.
• Define tolerance thresholds per table (default: 0% for row counts,

configurable for aggregates).

4. Dashboard deployment

• Import dashboards/migration_progress.sql into Databricks SQL.
• Create visualisations for wave progress, reconciliation trends, and

pipeline health.

5. Testing

• Run each extractor against a dev/staging environment.
• Validate full load, then test incremental with known changes.
• Confirm reconciliation detects intentionally introduced discrepancies.

Deliverables

• Configured extractor notebooks (one per source table or reusable parameterised)
• Target schema DDL or Unity Catalog configuration
• Validation notebooks with per-table parameters
• Dashboard with all 9 query panels
• Test results document

Exit Criteria

• [ ] All extractors tested in dev with full + incremental cycles
• [ ] Reconciliation validated (catches known diffs, passes clean runs)
• [ ] Dashboard operational with live data
• [ ] Code reviewed and merged to main branch

Phase 4: Execution

Objective

Execute migration waves sequentially, validate each wave before proceeding,

and establish incremental pipelines for ongoing sync.

Wave Execution Procedure

For each wave:

1. Full load — Run the extractor with load_type = full for all tables

in the wave.

2. Row count validation — Run reconciliation with check_types = row_count.

3. Hash validation — Run reconciliation with check_types = hash on

tables with primary keys.

4. Aggregate validation — Run reconciliation with check_types = aggregate

on tables with numeric columns.

5. Sampling — Run data_sampling.py for large tables (>10M rows).

6. Review results — All checks must PASS before enabling incremental mode.

7. Enable incremental — Switch extractors to load_type = incremental_*

and schedule them (every 15-60 minutes depending on SLA).

8. Monitor — Watch the migration progress dashboard for 24 hours.

9. Sign off wave — Get stakeholder confirmation before starting the next wave.

Common Issues and Remediation

Deliverables

• Per-wave execution log
• Reconciliation results in migration.reconciliation.reconciliation_results
• Incremental pipelines running on schedule
• Wave sign-off records

Exit Criteria

• [ ] All waves executed and validated
• [ ] Incremental pipelines operational with <15 min latency
• [ ] Zero FAIL results in latest reconciliation run
• [ ] All waves signed off by data owners

Phase 5: Cutover

Objective

Transition production workloads from source to target with minimal downtime

and zero data loss.

Reference

See runbooks/cutover_runbook.md for the full step-by-step procedure.

Summary

1. Parallel run (48 hours) — Both systems active, continuous reconciliation.

2. Freeze source — Stop writes, run final sync.

3. Final validation — Full reconciliation, all checks PASS.

4. Switchover — Update connection strings, redirect consumers.

5. Confirmation — Monitor for 30 minutes, run smoke tests.

6. Rollback (if needed) — Revert connections, re-enable source.

Deliverables

• Parallel run metrics report
• Final reconciliation results
• Cutover sign-off form
• Rollback documentation (if invoked)

Exit Criteria

• [ ] Production traffic flowing through target system
• [ ] Source set to read-only
• [ ] No P1/P2 issues in first 4 hours post-cutover

Phase 6: Decommissioning

Objective

Safely retire source systems after confirming the target is stable and complete.

Reference

See runbooks/decommissioning_checklist.md for the detailed checklist.

Summary

1. Complete hypercare period (minimum 7 days).

2. Disable source write access.

3. Take final backup and archive.

4. Disable source read access.

5. Decommission infrastructure.

6. Update documentation and close project.

Deliverables

• Final backup with verified integrity
• Decommissioning sign-off
• Updated architecture diagrams
• Lessons learned document

Exit Criteria

• [ ] Source system offline
• [ ] Backup archived per retention policy
• [ ] All documentation updated
• [ ] Project closed

Key Principles

1. Validate early and often. Run reconciliation after every extraction,

not just at the end.

2. Start simple, build confidence. Wave 1 should be small reference tables.

Prove the pattern before tackling large, complex tables.

3. Automate everything. Manual data movement introduces errors. Use the

extractor notebooks with parameterised widgets and schedule them via

Databricks Workflows.

4. Keep the rollback path open. Never decommission the source until the

hypercare period is complete and all stakeholders have signed off.

5. Document decisions. Record why tables were grouped into specific waves,

why certain extractors were chosen, and any deviations from the plan.

6. Monitor continuously. The migration progress dashboard should be the

team's primary view throughout the project. Stale pipelines and failed

reconciliation checks should trigger immediate investigation.

Glossary

*Data Migration Factory | Datanest Digital | https://datanest.dev*