Data Catalog Builder

Data Catalog Builder

Automated metadata discovery, data dictionary generation, quality scoring, and searchable catalog for Databricks Unity Catalog.

By Datanest Digital | Version 1.0.0 | $39

What You Get

• Catalog Scanner — Discover tables, columns, and metadata across Unity Catalog schemas
• Metadata Enricher — Add business descriptions, owners, tags, and classification labels
• Lineage Mapper — Trace column-level lineage from bronze to gold layers
• Search Index — Full-text search across table/column names, descriptions, and tags
• Quality Scorer — Score tables on completeness, freshness, documentation, and conformance
• Catalog Reporter — Generate Markdown data dictionaries from Jinja2 templates

File Tree

data-catalog-builder/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│  ├── catalog_scanner.py      # Unity Catalog metadata discovery
│  ├── metadata_enricher.py     # Business metadata enrichment
│  ├── lineage_mapper.py      # Column-level lineage tracing
│  ├── search_index.py       # Full-text catalog search
│  ├── quality_scorer.py      # Data quality scoring engine
│  └── catalog_reporter.py     # Markdown report generator
├── configs/
│  ├── catalog_config.yaml     # Scanner and enrichment settings
│  ├── classification_rules.yaml  # PII / sensitivity classification
│  └── templates/
│    └── table_doc.md.j2     # Jinja2 table documentation template
├── notebooks/
│  ├── scan_catalog.py       # Run catalog discovery scan
│  └── catalog_dashboard.py     # Catalog health dashboard
├── tests/
│  ├── conftest.py         # Shared test fixtures
│  ├── test_catalog_scanner.py   # Scanner tests
│  └── test_quality_scorer.py    # Quality scoring tests
└── guides/
  └── data-catalog-guide.md    # Setup and usage guide

Getting Started

Follow this guide to get data catalog builder up and running in your environment.

Getting Started

1. Configure the Scanner

Edit configs/catalog_config.yaml with your Unity Catalog settings:

scanner:
 catalogs: ["main"]
 schemas: ["bronze", "silver", "gold"]
 exclude_patterns: ["_tmp_*", "_staging_*"]

2. Scan Your Catalog

from src.catalog_scanner import CatalogScanner

scanner = CatalogScanner(config_path="configs/catalog_config.yaml")
catalog = scanner.scan_all()
print(f"Discovered {catalog.table_count} tables across {catalog.schema_count} schemas")

3. Enrich with Business Metadata

from src.metadata_enricher import MetadataEnricher

enricher = MetadataEnricher(config_path="configs/catalog_config.yaml")
enricher.apply_tags(catalog)
enricher.classify_columns(rules_path="configs/classification_rules.yaml")

4. Score Data Quality

from src.quality_scorer import QualityScorer

scorer = QualityScorer()
scores = scorer.score_catalog(catalog)
for table, score in scores.items():
  print(f"{table}: {score.overall:.0f}/100")

5. Generate Documentation

from src.catalog_reporter import CatalogReporter

reporter = CatalogReporter(template_path="configs/templates/table_doc.md.j2")
reporter.generate_all(catalog, output_dir="/Workspace/docs/catalog")

Architecture

┌────────────────────────────────────────────────────────┐
│ Unity Catalog                     │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐      │
│ │ Bronze  │ │ Silver │ │ Gold  │      │
│ │ tables  │ │ tables │ │ tables │      │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘      │
└───────┼──────────────┼─────────────┼──────────────────┘
    │       │       │
    ▼       ▼       ▼
┌──────────────────────────────────────────┐
│ Catalog Scanner             │
│ (discover tables, columns, stats)    │
└─────────────────┬────────────────────────┘
         │
  ┌─────────────┼─────────────┐
  ▼       ▼       ▼
┌────────┐ ┌──────────┐ ┌──────────────┐
│Metadata│ │ Quality │ │  Lineage  │
│Enricher│ │ Scorer  │ │  Mapper   │
└───┬────┘ └────┬─────┘ └──────┬───────┘
  │       │        │
  ▼       ▼        ▼
┌──────────────────────────────────────────┐
│ Search Index + Catalog Reporter    │
│ (search, browse, export documentation) │
└──────────────────────────────────────────┘