ADF-to-Databricks Integration Patterns

Datanest Digital — datanest.dev

Overview

This guide covers production patterns for orchestrating Databricks notebooks from Azure Data Factory. It addresses cluster strategy, parameter passing, error handling, medallion architecture orchestration, and operational considerations.

1. Linked Service Authentication Patterns

Managed Identity (Recommended)

Managed identity is the most secure approach — no tokens or credentials to manage.

{
 "type": "AzureDatabricks",
 "typeProperties": {
  "domain": "https://adb-<id>.azuredatabricks.net",
  "authentication": "MSI",
  "workspaceResourceId": "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Databricks/workspaces/<ws>"
 }
}

Requirements:

• ADF managed identity must have Contributor role on the Databricks workspace
• The workspace must have Unity Catalog or legacy ACLs configured for the service principal

Access Token via Key Vault

For environments where MSI is not supported (e.g., cross-tenant):

{
 "type": "AzureDatabricks",
 "typeProperties": {
  "domain": "https://adb-<id>.azuredatabricks.net",
  "accessToken": {
   "type": "AzureKeyVaultSecret",
   "store": { "referenceName": "LS_AzureKeyVault" },
   "secretName": "databricks-access-token"
  }
 }
}

2. Cluster Strategy

Existing Interactive Cluster

Best for development and small-to-medium workloads with predictable scheduling.

{
 "existingClusterId": "@{pipeline().parameters.p_cluster_id}"
}

Pros: Fast startup, cost-effective for frequent short runs

Cons: Shared resource contention, single point of failure

New Job Cluster (Recommended for Production)

Spins up a dedicated cluster per pipeline run — full isolation.

{
 "newClusterVersion": "14.3.x-scala2.12",
 "newClusterNumOfWorker": "2",
 "newClusterSparkConf": {
  "spark.databricks.delta.preview.enabled": "true",
  "spark.sql.adaptive.enabled": "true"
 },
 "newClusterNodeType": "Standard_DS3_v2",
 "newClusterInitScripts": [],
 "newClusterDriverNodeType": "Standard_DS3_v2"
}

Sizing guidance:

Instance Pools

Pre-provision VMs to reduce cluster startup time from ~5 minutes to ~30 seconds.

Configure instance pools in Databricks, then reference via:

{
 "newClusterVersion": "14.3.x-scala2.12",
 "instancePoolId": "<pool-id>",
 "newClusterNumOfWorker": "2"
}

3. Parameter Passing Patterns

Direct Parameters

Pass scalar values directly from ADF to notebooks:

{
 "type": "DatabricksNotebook",
 "typeProperties": {
  "notebookPath": "/Shared/pipelines/bronze/ingest",
  "baseParameters": {
   "source_path": "@pipeline().parameters.p_source_path",
   "load_date": "@formatDateTime(utcNow(), 'yyyy-MM-dd')",
   "pipeline_run_id": "@pipeline().RunId"
  }
 }
}

In the Databricks notebook:

source_path = dbutils.widgets.get("source_path")
load_date = dbutils.widgets.get("load_date")
pipeline_run_id = dbutils.widgets.get("pipeline_run_id")

Passing Complex Objects

ADF base parameters only support strings. For complex objects, serialize to JSON:

{
 "config_json": "@string(json(concat('{\"tables\": [\"orders\", \"customers\"], \"mode\": \"incremental\"}')))"
}

In the notebook:

import json
config = json.loads(dbutils.widgets.get("config_json"))
tables = config["tables"]

Returning Values from Notebooks

Use dbutils.notebook.exit() to return values to ADF:

import json
result = {"status": "SUCCESS", "rows_processed": row_count, "output_path": output_path}
dbutils.notebook.exit(json.dumps(result))

Access in subsequent ADF activities:

@activity('RunBronzeNotebook').output.runOutput.status
@activity('RunBronzeNotebook').output.runOutput.rows_processed

4. Medallion Architecture Orchestration

Sequential Bronze → Silver → Gold

The included databricks_notebook_activity.json pipeline implements this pattern:

[Bronze Notebook] → [Check Output] → [Silver Notebook] → [Check Output] → [Gold Notebook]

Each stage validates the output of the previous stage before proceeding. This prevents cascading failures where corrupted bronze data propagates through silver and gold layers.

Parallel Fan-Out Pattern

For independent tables that can be processed simultaneously:

          ┌─ [Bronze: Orders]  ─┐
[Get Table List] ─→ ├─ [Bronze: Customers] ├─→ [Silver: Join & Transform]
          └─ [Bronze: Products]  ─┘

Implementation using ForEach with isSequential: false:

{
 "type": "ForEach",
 "typeProperties": {
  "items": "@pipeline().parameters.p_table_list",
  "isSequential": false,
  "batchCount": 5,
  "activities": [
   {
    "type": "DatabricksNotebook",
    "typeProperties": {
     "notebookPath": "/Shared/pipelines/bronze/generic_ingest",
     "baseParameters": {
      "table_name": "@item().table_name",
      "source_schema": "@item().source_schema"
     }
    }
   }
  ]
 }
}

Dependency-Aware DAG

For complex dependency graphs, use nested pipelines:

[Master Pipeline]
 ├─ Execute Pipeline: bronze_ingestion (parallel ForEach)
 ├─ Execute Pipeline: silver_transforms (sequential, waits for bronze)
 └─ Execute Pipeline: gold_aggregations (sequential, waits for silver)

5. Error Handling

Retry Strategy

Configure retries at the activity level:

{
 "policy": {
  "timeout": "0.02:00:00",
  "retry": 2,
  "retryIntervalInSeconds": 120
 }
}

Recommended settings by scenario:

Notebook-Level Error Handling

Structure notebooks to return structured error information:

try:
  # Processing logic
  result = {"status": "SUCCESS", "rows_processed": count}
except Exception as e:
  result = {"status": "FAILED", "error": str(e), "error_type": type(e).__name__}
finally:
  dbutils.notebook.exit(json.dumps(result))

ADF Conditional Branching on Notebook Output

{
 "type": "IfCondition",
 "typeProperties": {
  "expression": {
   "value": "@equals(activity('RunNotebook').output.runOutput.status, 'SUCCESS')",
   "type": "Expression"
  },
  "ifTrueActivities": [...],
  "ifFalseActivities": [...]
 }
}

6. Cost Optimization

• Use job clusters in production — they auto-terminate after the notebook completes
• Set newClusterSparkConf["spark.databricks.cluster.profile"] to "singleNode" for light workloads
• Use spot instances for non-time-critical batch jobs (up to 80% cost reduction)
• Configure instance pools to reduce startup latency without paying for idle clusters
• Set auto-termination on interactive clusters (minimum 10 minutes)

7. Monitoring & Observability

Pipeline-Level Monitoring

ADF provides built-in monitoring for Databricks activities:

• Cluster startup duration
• Notebook execution duration
• Output size

Cross-Platform Correlation

Pass pipeline().RunId to every notebook as a parameter. Use this to correlate:

• ADF pipeline runs (Azure Monitor)
• Databricks job runs (Databricks UI / API)
• Application Insights traces

Log Analytics KQL Queries

ADFPipelineRun
| where PipelineName contains "Databricks"
| where Status == "Failed"
| project TimeGenerated, PipelineName, RunId, ErrorMessage = Parameters
| order by TimeGenerated desc

© 2026 Datanest Digital. All rights reserved.

ADF Cost Optimization Guide

Datanest Digital — datanest.dev

Overview

Azure Data Factory billing is based on four cost dimensions: pipeline orchestration (activity runs), data movement (DIU-hours), data flow execution (vCore-hours), and integration runtime (SHIR uptime). This guide provides actionable strategies to reduce costs across all dimensions.

1. Understanding ADF Pricing Dimensions

*Prices are approximate and region-dependent. Check Azure pricing calculator for current rates.*

2. Pipeline Activity Run Optimization

Reduce Activity Count

Every activity execution incurs a charge. Consolidate where possible:

• Combine multiple Copy activities into a single parameterized Copy with wildcard paths
• Use ForEach instead of duplicating activities for similar operations
• Avoid unnecessary Lookup activities — pass values as parameters instead of querying every run
• Remove diagnostic SetVariable activities from production pipelines

Batch Metadata Lookups

Instead of one Lookup per table in a ForEach loop:

BAD: ForEach → [Lookup Config] → [Copy Data]  (2 activities × N tables)
GOOD: [Lookup All Configs] → ForEach → [Copy Data]  (1 + N activities)

Schedule Consolidation

• Consolidate pipelines that run at the same frequency into a single master pipeline
• Use tumbling window triggers instead of schedule triggers for backfill-heavy workloads (tumbling windows handle reruns efficiently)

3. Data Integration Unit (DIU) Optimization

DIUs determine the compute power allocated to Copy activities. More DIUs = faster copies but higher cost.

Right-Sizing DIU Allocation

Auto DIU vs. Fixed DIU

ADF defaults to "Auto" (up to 256 DIUs). For cost control:

{
 "dataIntegrationUnits": 4
}

Set this explicitly in your Copy activity. Monitor the actual DIU usage in pipeline run output:

activity('CopyData').output.usedDataIntegrationUnits

If actual usage is consistently below your setting, reduce it.

Parallel Copies

The parallelCopies property controls concurrent read/write threads within a single Copy activity:

{
 "parallelCopies": 4
}

• Default is null (ADF auto-determines)
• For database sources, keep at 4-8 to avoid overwhelming the source
• For blob/ADLS, higher values (16-32) can improve throughput

4. Self-Hosted Integration Runtime Sizing

SHIR is billed per node, per hour — whether active or idle.

Sizing Guidelines

Cost Reduction Strategies

1. Auto-shutdown for non-production — Use Azure Automation to stop SHIR VMs outside business hours:

  # Azure Automation runbook
  $vms = Get-AzVM -ResourceGroupName "rg-shir" -Status
  foreach ($vm in $vms) {
    if ($vm.PowerState -eq "VM running") {
      Stop-AzVM -Name $vm.Name -ResourceGroupName "rg-shir" -Force
    }
  }

2. Shared SHIR — A single SHIR node can be shared across multiple ADF instances (up to 4). Use this for dev/test environments.

3. Right-size VMs — Start with Standard_D2s_v3 and scale up only if you see queue times in SHIR monitoring.

4. Use Azure IR where possible — If your source is publicly accessible or in Azure, prefer Azure IR over SHIR.

5. Concurrency Tuning

Pipeline Concurrency

Limit concurrent pipeline runs to prevent resource contention and cost spikes:

{
 "policy": {
  "pipelineConcurrency": 5
 }
}

ForEach Batch Size

Default batchCount is 20 (max 50). Lower this to reduce peak resource consumption:

{
 "type": "ForEach",
 "typeProperties": {
  "batchCount": 5,
  "isSequential": false
 }
}

Trigger Staggering

If multiple pipelines trigger at the same time (e.g., hourly on the hour), stagger them:

This flattens the SHIR load curve and avoids queueing.

6. Data Flow Cost Optimization

Data flows are the most expensive ADF component. Optimize aggressively.

Use Copy Activity Instead of Data Flow

If your transformation is simple (column mapping, type conversion, filtering), a Copy activity with column mapping is 10-50x cheaper than a data flow.

Use Data Flows for:

• Complex joins across multiple sources
• Window functions and aggregations
• Slowly changing dimensions
• Schema drift handling

Use Copy Activity for:

• Direct source-to-sink copies
• Simple column selection/renaming
• File format conversion (CSV → Parquet)

TTL (Time to Live)

Data flow clusters take 3-5 minutes to start. TTL keeps the cluster warm:

{
 "timeToLive": 10
}

• Set TTL to 10 minutes if data flows run every 15 minutes (avoids repeated startup)
• Set TTL to 0 for infrequent runs (don't pay for idle compute)

Right-Size Compute

Start with General Purpose, 8 cores and scale based on actual execution metrics:

activity('DataFlowActivity').output.runStatus.metrics.sink1.rowsWritten
activity('DataFlowActivity').output.runStatus.profile.compute.coreCount

7. Monitoring Costs

ADF Metrics to Track

Set up Azure Monitor dashboards for:

• PipelineSucceededRuns / PipelineFailedRuns — track failure-driven reruns
• ActivitySucceededRuns — total billable activity executions
• IntegrationRuntimeCpuPercentage — SHIR utilization (low = over-provisioned)
• IntegrationRuntimeAvailableMemory — SHIR memory headroom

Cost Analysis Queries

Use Log Analytics to identify expensive pipelines:

ADFActivityRun
| where TimeGenerated > ago(30d)
| summarize RunCount = count(), TotalDurationMin = sum(Duration) / 60000 by PipelineName
| order by RunCount desc
| take 20

Monthly Cost Estimation Formula

Monthly Cost ≈
 (Activity Runs / 1000 × $1.00)
 + (DIU-Hours × $0.25)
 + (SHIR Node Hours × $0.10)
 + (Data Flow vCore-Hours × $0.268)

8. Quick Wins Checklist

• [ ] Set explicit DIU values on all Copy activities (don't rely on Auto)
• [ ] Consolidate small Lookup activities into batch queries
• [ ] Shut down SHIR VMs in dev/test outside business hours
• [ ] Replace simple Data Flows with Copy + column mapping
• [ ] Set pipeline concurrency limits on resource-intensive pipelines
• [ ] Stagger trigger schedules to flatten peak load
• [ ] Enable data flow TTL only for frequently-triggered flows
• [ ] Review failed pipeline runs — reruns double your cost
• [ ] Use parallelCopies to optimize throughput per DIU
• [ ] Monitor SHIR CPU — if consistently < 30%, downsize the VM

© 2026 Datanest Digital. All rights reserved.