Agent Pool Sizing Guide

Self-hosted vs Microsoft-hosted agents for data platform CI/CD workloads.

Datanest Digital — https://datanest.dev

Overview

Choosing the right agent pool strategy directly impacts pipeline execution time,

cost, and security. This guide provides sizing recommendations for the four pipeline

types included in this product, with concrete guidance on when to use Microsoft-hosted

agents vs self-hosted agents.

Decision Matrix

Recommendation by Pipeline

Terraform Infrastructure Pipeline

Recommended: Self-hosted (for production), Microsoft-hosted (for dev)

Agent Sizing:

• vCPUs: 2
• RAM: 4 GB
• Disk: 50 GB (SSD)
• Network: VNET with access to state storage and target subscriptions

Databricks DABs Deploy Pipeline

Recommended: Microsoft-hosted (all environments)

If using private-link Databricks workspaces: Switch to self-hosted agents

with VNET connectivity to the workspace's private endpoint.

Agent Sizing (if self-hosted):

• vCPUs: 2
• RAM: 4 GB
• Disk: 30 GB (SSD)

ADF ARM Deploy Pipeline

Recommended: Microsoft-hosted (all environments)

Agent Sizing (if self-hosted):

• vCPUs: 2
• RAM: 2 GB
• Disk: 20 GB

SQL Database Migration Pipeline

Recommended: Self-hosted (for staging and production)

Agent Sizing:

• vCPUs: 4
• RAM: 8 GB
• Disk: 50 GB (SSD)
• Network: VNET with SQL Server private endpoint access

Self-Hosted Agent Setup

Infrastructure Options

Recommended: VMSS-Based Agent Pool

Azure VM Scale Sets provide the best balance of auto-scaling, cost control, and

maintenance simplicity for self-hosted agents.

# Create a VMSS for pipeline agents
az vmss create \
  --name "vmss-azdo-agents" \
  --resource-group "rg-azdo-agents" \
  --image "Ubuntu2204" \
  --vm-sku "Standard_D2s_v5" \
  --instance-count 0 \
  --upgrade-policy-mode manual \
  --vnet-name "vnet-data-platform" \
  --subnet "snet-azdo-agents" \
  --admin-username "azureagent" \
  --generate-ssh-keys

Register as an Azure DevOps agent pool:

1. Go to Organization Settings > Agent pools > Add pool.

2. Select Azure virtual machine scale set.

3. Select the VMSS resource and configure:

Agent Software Requirements

Install these tools on the self-hosted agent image:

Cost Optimization

Microsoft-Hosted

• Free tier: 1 free parallel job with 1,800 minutes/month.
• Paid: Each additional parallel job is approximately $40/month.
• Optimize by reducing pipeline duration (parallel stages, caching).

Self-Hosted

• VMSS with auto-scale to zero: Pay only when pipelines are running.
• Spot instances: Use Azure Spot VMs for dev/test agent pools (up to 90% savings).

Not recommended for production deployments due to eviction risk.

• Reserved instances: For always-on standby agents, 1-year reservations save ~35%.

Cost Estimation (Monthly)

Security Considerations

1. Network isolation: Place self-hosted agents in a dedicated subnet with NSG rules

that only allow outbound access to required endpoints.

2. Ephemeral agents: Configure VMSS agents to reimage after every job to prevent

credential leakage between pipeline runs.

3. Managed identity: Assign a system-assigned managed identity to the VMSS and use

it for Azure authentication instead of service principal secrets.

4. Agent updates: Enable automatic agent software updates in the pool configuration.

5. Firewall rules: Microsoft-hosted agents use dynamic IP ranges. If your resources

require IP allowlisting, self-hosted agents with a static outbound IP (via NAT Gateway)

are the only reliable option.

Copyright 2024-2026 Datanest Digital. All rights reserved.

Unified Data Platform CI/CD Architecture Guide

End-to-end CI/CD architecture for deploying infrastructure, compute, orchestration, and storage layers as a single platform.

Datanest Digital — https://datanest.dev

Overview

A modern data platform on Azure consists of multiple layers — infrastructure (Terraform),

compute (Databricks), orchestration (Azure Data Factory), and storage (SQL databases,

Delta Lake). Each layer has distinct deployment concerns, but they share common

dependencies and must be deployed in the correct order.

This guide describes how to compose the four pipelines in this product into a unified

CI/CD architecture that deploys the entire platform end-to-end with proper dependency

ordering, environment isolation, and promotion gates.

Architecture Diagram

┌──────────────────────────────────────────────────────────────────┐
│                        Source Repository                         │
│                                                                  │
│  infrastructure/    databricks/    adf/         database/        │
│  └─ terraform/      └─ bundles/    └─ arm/      └─ dacpac/      │
│     ├─ modules/     └─ src/        └─ params/   └─ schemas/     │
│     └─ envs/        └─ tests/                   └─ migrations/  │
└──────────┬──────────────┬─────────────┬──────────────┬──────────┘
           │              │             │              │
           ▼              ▼             ▼              ▼
┌──────────────┐  ┌────────────┐  ┌──────────┐  ┌──────────────┐
│  Terraform   │  │  DABs      │  │  ADF ARM │  │  SQL DACPAC  │
│  Pipeline    │  │  Pipeline  │  │  Pipeline │  │  Pipeline    │
└──────┬───────┘  └─────┬──────┘  └────┬─────┘  └──────┬───────┘
       │                │              │                │
       ▼                ▼              ▼                ▼
┌──────────────────────────────────────────────────────────────────┐
│                     Environment: dev                             │
│  VNET, Storage, Key Vault │ Databricks │ ADF │ SQL Database     │
└──────────────────────────────┬───────────────────────────────────┘
                               │ (automated)
                               ▼
┌──────────────────────────────────────────────────────────────────┐
│                     Environment: staging                         │
│  VNET, Storage, Key Vault │ Databricks │ ADF │ SQL Database     │
└──────────────────────────────┬───────────────────────────────────┘
                               │ (approval gate)
                               ▼
┌──────────────────────────────────────────────────────────────────┐
│                     Environment: prod                            │
│  VNET, Storage, Key Vault │ Databricks │ ADF │ SQL Database     │
└──────────────────────────────────────────────────────────────────┘

Deployment Order

Infrastructure must exist before application-layer resources can be deployed.

The correct dependency chain is:

1. Terraform Infrastructure (VNET, Storage, Key Vault, Databricks workspace, ADF, SQL Server)
       │
       ├── 2a. Databricks DABs Deploy (notebooks, jobs, DLT pipelines)
       │
       ├── 2b. ADF ARM Deploy (pipelines, datasets, linked services)
       │
       └── 2c. SQL Database Migration (schemas, stored procedures, views)

Steps 2a, 2b, and 2c can run in parallel after Terraform completes, because

they target independent resources.

Orchestrator Pipeline

Create a top-level orchestrator pipeline that invokes the four component pipelines

in the correct order:

# pipelines/orchestrator.yml
trigger:
  branches:
    include:
      - main
  paths:
    include:
      - infrastructure/**
      - databricks/**
      - adf/**
      - database/**

parameters:
  - name: environment
    type: string
    default: dev
    values: [dev, staging, prod]

stages:
  # --- Stage 1: Infrastructure ---
  - stage: Infrastructure
    displayName: "Deploy Infrastructure"
    jobs:
      - template: terraform-infrastructure.yml
        parameters:
          environment: ${{ parameters.environment }}

  # --- Stage 2: Application Layer (parallel) ---
  - stage: ApplicationLayer
    displayName: "Deploy Application Layer"
    dependsOn: Infrastructure
    jobs:
      - template: databricks-dabs-deploy.yml
        parameters:
          environment: ${{ parameters.environment }}

      - template: adf-arm-deploy.yml
        parameters:
          environment: ${{ parameters.environment }}

      - template: sql-database-migration.yml
        parameters:
          environment: ${{ parameters.environment }}

  # --- Stage 3: Validation ---
  - stage: Validation
    displayName: "End-to-End Validation"
    dependsOn: ApplicationLayer
    jobs:
      - job: IntegrationTests
        steps:
          - script: echo "Run integration test suite"

Selective Deployment

Not every commit touches all four layers. Use path-based triggers to avoid

unnecessary deployments:

Each pipeline has its own trigger.paths.include filter. The orchestrator pipeline

triggers on any path change and can skip unchanged components using conditions.

Conditional Stage Execution

stages:
  - stage: Infrastructure
    condition: |
      or(
        contains(variables['Build.SourceVersionMessage'], '[infra]'),
        eq(variables['Build.Reason'], 'Manual'),
        contains(join(',', split(variables['System.PullRequest.TargetBranch'], '/')), 'main')
      )

Environment Isolation

Resource Naming Convention

<resource-type>-<project>-<environment>

Subscription Strategy

The multi-subscription model is recommended for production data platforms because

it provides the strongest isolation boundary for RBAC, cost tracking, and quota

management.

Artifact Flow

Each pipeline stage publishes artifacts that downstream stages can consume:

Terraform Plan → Plan Artifact (.tfplan)
                      │
Terraform Apply → Output Artifact (terraform-outputs.json)
                      │
                      ├── DABs Deploy reads workspace URL from outputs
                      ├── ADF Deploy reads linked service endpoints from outputs
                      └── SQL Migration reads server FQDN from outputs

Passing Terraform Outputs to Downstream Pipelines

# In the Terraform Apply stage:
- script: |
    terraform output -json > $(Build.ArtifactStagingDirectory)/outputs.json
  displayName: "Export Outputs"

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: "$(Build.ArtifactStagingDirectory)/outputs.json"
    ArtifactName: "terraform-outputs"

# In a downstream stage:
- task: DownloadBuildArtifacts@1
  inputs:
    artifactName: "terraform-outputs"

- script: |
    DATABRICKS_HOST=$(jq -r '.databricks_host.value' terraform-outputs/outputs.json)
    echo "##vso[task.setvariable variable=DATABRICKS_HOST]$DATABRICKS_HOST"
  displayName: "Parse Terraform Outputs"

Rollback Strategy

Infrastructure (Terraform)

Terraform state tracks the desired state. Rollback options:

1. Revert the commit and re-run the pipeline (preferred).

2. Target-specific resource with terraform apply -target=.

3. Import previous state if state was corrupted.

Avoid terraform destroy as a rollback mechanism in production.

Databricks (DABs)

1. Redeploy previous bundle version from a tagged Git commit.

2. DABs tracks deployment state; redeploying an older commit restores previous

notebook and job definitions.

ADF (ARM Templates)

1. Incremental mode (default) only adds/updates resources; it does not delete.

2. Rollback by redeploying the previous ARM template version from Git history.

3. ADF also supports the live/publish model with adf_publish branch as a fallback.

SQL (DACPAC)

1. DACPAC is declarative — it converges schema to the desired state.

2. Rollback by deploying the previous DACPAC artifact.

3. For data-destructive changes, take a database snapshot before deployment.

Monitoring and Observability

DORA Metrics

Track these four metrics to measure CI/CD effectiveness:

Use the monitoring/pipeline-dashboard.yml definition to set up dashboards that

track these metrics automatically.

Alerting

Configure alerts for:

• Pipeline failure (immediate notification via Teams/Slack)
• Pipeline duration exceeding 2x the 30-day average
• Approval requests pending longer than 4 hours
• Consecutive failures on the same pipeline (3+ in a row)

Repository Structure

Recommended repository layout for the unified data platform:

repo-root/
├── .azuredevops/
│   └── pipelines/
│       ├── orchestrator.yml
│       ├── terraform-infrastructure.yml
│       ├── databricks-dabs-deploy.yml
│       ├── adf-arm-deploy.yml
│       ├── sql-database-migration.yml
│       └── pr-validation.yml
├── templates/
│   ├── shared-steps.yml
│   └── notification-template.yml
├── infrastructure/
│   └── terraform/
│       ├── main.tf
│       ├── variables.tf
│       ├── outputs.tf
│       ├── modules/
│       │   ├── networking/
│       │   ├── storage/
│       │   ├── databricks/
│       │   ├── data-factory/
│       │   └── sql-server/
│       └── environments/
│           ├── dev.tfvars
│           ├── staging.tfvars
│           └── prod.tfvars
├── databricks/
│   ├── databricks.yml          # DABs bundle config
│   ├── resources/
│   │   ├── jobs.yml
│   │   └── dlt_pipelines.yml
│   └── src/
│       ├── notebooks/
│       └── libraries/
├── adf/
│   └── arm-templates/
│       ├── ARMTemplateForFactory.json
│       └── ARMTemplateParametersForFactory.json
├── database/
│   └── DataPlatformDB/
│       ├── DataPlatformDB.sqlproj
│       ├── Tables/
│       ├── Views/
│       ├── StoredProcedures/
│       └── PostDeployment/
├── tests/
│   ├── unit/
│   ├── integration/
│   └── smoke/
└── docs/
    └── architecture/

Implementation Checklist

Use this checklist when setting up the unified CI/CD architecture:

• [ ] Create Azure subscriptions (or resource groups) for dev, staging, prod
• [ ] Provision Terraform state storage account per environment
• [ ] Create service principals with scoped RBAC per environment
• [ ] Create service connections in Azure DevOps (see config/service-connections.md)
• [ ] Create variable groups with Key Vault linking (see config/variable-groups.md)
• [ ] Configure branch policies (see config/branch-policies.md)
• [ ] Import pipeline YAML files into Azure DevOps
• [ ] Configure environment approval gates (see config/release-gates.json)
• [ ] Create Azure DevOps environments: dev, staging, prod (per pipeline)
• [ ] Set up notification webhooks (Teams, Slack)
• [ ] Import the monitoring dashboard definition
• [ ] Run an end-to-end deployment to dev to validate the full chain
• [ ] Promote to staging with approval gate verification
• [ ] Document any customizations in the repository wiki

Copyright 2024-2026 Datanest Digital. All rights reserved.