Overview

This chapter covers the core features and capabilities of Data Mesh Starter Kit.

Overview

The Data Mesh Starter Kit is a comprehensive, practitioner-focused toolkit for organizations transitioning from centralized data architectures to a domain-oriented data mesh. It provides assessments, workshop templates, Terraform infrastructure modules, governance frameworks, and communication templates — everything a data engineering team needs to plan, execute, and sustain a data mesh adoption.

This kit is designed for:

• Data Platform Engineers building self-serve infrastructure for domain teams
• Data Architects designing federated ownership models
• Engineering Managers aligning stakeholders on data mesh principles
• CTOs and VPs of Engineering evaluating data mesh readiness

What's Included

Assessment Tools

Workshop Templates

Infrastructure as Code

Data Product Templates

Governance Framework

Communication Templates

Case Study

Getting Started

A data mesh distributes data ownership to the teams that understand the business while retaining shared standards. A domain is a durable business capability such as orders, fulfillment, or customer support. A data product is a domain-owned, discoverable dataset or interface with an explicit contract, owner, quality controls, and service expectations. Federated governance means a central group defines interoperable policies—security, naming, metadata, and minimum quality—while domain teams decide how to implement and operate their products.

Start with one valuable, bounded domain rather than reorganizing the entire company. The kit’s assessment, workshops, infrastructure module, product template, scorecard, and agreements form a practical path from discovery to a working pilot.

1. Define the First Domain

Run assessment/maturity_assessment.md and use assessment/scoring_calculator.py to identify readiness gaps. Then facilitate workshops/domain_decomposition.md with a domain lead, data producer, likely consumers, platform engineer, and governance representative. Select a domain that has clear ownership, recurring demand, accessible source data, and a measurable pain point.

Capture the result in a small domain definition:

domain:
  id: orders
  display_name: Order Management
  purpose: Own the lifecycle from accepted order to fulfillment handoff
  owner:
    team: commerce-operations
    contact: commerce-ops@example.org
  source_systems:
    - order-service
    - payment-events
  environments: [dev, staging, prod]
  governance:
    classification: confidential
    retention_days: 2555

Review boundaries explicitly. For example, Orders may own order status, while Payments owns authorization outcomes and Customer owns identity. Record those producer-consumer expectations with communication/domain_agreements.md; this prevents shared tables from becoming ownerless.

2. Provision a Domain Workspace

Copy terraform/domain-workspace/ into your infrastructure repository and supply organization-specific networking, identity, state backend, tags, and naming conventions. Review the plan before applying:

cd terraform/domain-workspace
terraform init
terraform plan \
  -var="domain_name=orders" \
  -var="environment=dev" \
  -out=orders-dev.tfplan
terraform apply orders-dev.tfplan

Use separate state and credentials for each environment. The central platform team owns the reusable module and guardrails; the domain team owns its declared resources and products. Do not use the example defaults as production security policy.

3. Create a Data Product

Choose one consumer outcome, such as “daily fulfilled orders for finance reconciliation.” Fork templates/data_product_template.py and fill in its schema contract, SLA, ownership, and quality checks. Keep transformations version-controlled and make failures visible to the owning team.

Create a product descriptor beside the implementation:

data_product:
  id: orders.fulfilled_orders
  name: Fulfilled Orders
  owner: commerce-operations
  interface:
    type: delta-table
    identifier: prod_orders.gold.fulfilled_orders
  contract:
    primary_key: [order_id]
    refresh: "0 6 * * *"
    freshness_hours: 8
    schema_version: 1.0.0
  quality:
    - rule: order_id IS NOT NULL
    - rule: fulfilled_at <= current_timestamp()
  consumers: [finance, customer-support]

Run the product against representative dev data, validate access as a consumer, and score it with templates/data_product_scorecard.md. A product is not ready merely because its table exists: consumers must be able to find, understand, request, and safely use it.

4. Run Catalog Discovery

Use communication/data_product_catalog.md as the catalog entry template. If your catalog supports repository ingestion, commit one descriptor per product and configure its discovery job to scan for those files. Otherwise, copy the same metadata into Unity Catalog, your enterprise catalog, or an internal portal.

catalog_entry:
  product_id: orders.fulfilled_orders
  description: One row per fulfilled customer order
  documentation: https://docs.example.org/data/orders/fulfilled-orders
  tags: [orders, finance, certified]
  access_request: https://access.example.org/products/orders.fulfilled_orders
  support_channel: "#data-orders"
  last_reviewed: 2026-07-30

Run discovery and confirm the product is searchable by name and tag, its owner and SLA appear, lineage points to real sources, and the access link works for a test consumer. Treat missing metadata as a release failure.

5. Next Steps

Operate the pilot for several refresh cycles. Track freshness, quality failures, access-request lead time, adoption, and support load. Use governance/federated_governance_model.md to agree which policies are global and governance/platform_team_responsibilities.md to establish a RACI. After the first product meets its SLA and has active consumers, add a second product in the same domain before expanding to another domain. Feed lessons back into the Terraform module, templates, scorecard, and agreements so each new domain starts with a stronger paved road.