Kubernetes Deployment Patterns & Best Practices

A comprehensive guide to deploying, scaling, and operating applications on Kubernetes. Covers deployment strategies, resource management, security, observability, and GitOps workflows.

Table of Contents

1. Deployment Strategies

2. Resource Management

3. Pod Security

4. Health Checks & Probes

5. Autoscaling

6. Network Policies

7. Secrets Management

8. Kustomize Patterns

9. Helm Best Practices

10. Observability

11. GitOps Workflows

12. Production Checklist

Deployment Strategies

Rolling Update (Default)

The default Kubernetes deployment strategy. Gradually replaces old pods with new ones, maintaining availability throughout the update.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # Allow 1 extra pod during update
    maxUnavailable: 0  # Never drop below desired count

When to use: Most applications. Zero-downtime by default. Simple and well-understood.

Key settings:

• maxSurge: 1, maxUnavailable: 0 — safest option, but slowest. Each new pod must be ready before an old pod is terminated.
• maxSurge: 25%, maxUnavailable: 25% — faster, trades some capacity for speed. Good for stateless services with many replicas.

Blue-Green Deployment

Run two identical environments (blue and green). Switch traffic from blue to green atomically by updating the Service selector.

# Blue deployment (current production)
metadata:
  name: app-blue
  labels:
    app: myapp
    version: blue

# Green deployment (new version)
metadata:
  name: app-green
  labels:
    app: myapp
    version: green

# Service — switch traffic by changing the selector
spec:
  selector:
    app: myapp
    version: green  # Flip between "blue" and "green"

When to use: When you need instant rollback capability or must validate the full deployment before routing traffic.

Trade-off: Requires 2x resources during the transition period.

Canary Deployment

Route a small percentage of traffic to the new version. Gradually increase if metrics are healthy.

The simplest approach uses replica ratios: if you have 10 replicas total, set 9 to the old version and 1 to the new version (10% canary). More sophisticated approaches use Istio, Linkerd, or Argo Rollouts for weighted traffic splitting.

# With Argo Rollouts (recommended)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10     # 10% traffic to canary
        - pause: { duration: 5m }
        - setWeight: 30
        - pause: { duration: 5m }
        - setWeight: 60
        - pause: { duration: 5m }
      canaryService: app-canary
      stableService: app-stable

When to use: High-risk deployments, user-facing services where you need to validate with real traffic before full rollout.

Resource Management

Requests vs Limits

Every container should define both requests and limits:

• Requests — guaranteed minimum resources. The scheduler uses these to place pods on nodes.
• Limits — maximum resources. The kubelet enforces these. CPU is throttled; memory causes OOMKill.

resources:
  requests:
    cpu: 250m      # 0.25 CPU cores guaranteed
    memory: 256Mi  # 256 MiB guaranteed
  limits:
    cpu: "1"       # Throttle at 1 core
    memory: 512Mi  # OOMKill above 512 MiB

Rules of thumb:

• Set CPU limit to 2-4x the request. CPU is compressible — throttling is better than wasted capacity.
• Set memory limit to 1.5-2x the request. Memory is not compressible — set this based on actual peak usage from load testing.
• Never set requests without limits, or pods can consume all node resources.

Quality of Service Classes

Kubernetes assigns QoS classes based on resource configuration:

For production workloads, always aim for Guaranteed or Burstable.

LimitRange

Set default resource constraints for a namespace to prevent unbounded pods:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: app
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 256Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      max:
        cpu: "4"
        memory: 4Gi

Pod Security

Security Context

Run containers as non-root with minimal privileges. This prevents container breakout attacks and limits the blast radius of compromised pods.

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault

containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]

Key settings:

• runAsNonRoot: true — prevents running as UID 0 even if the image defaults to root.
• readOnlyRootFilesystem: true — prevents writing to the container filesystem. Use emptyDir volumes for /tmp.
• capabilities.drop: ["ALL"] — removes all Linux capabilities. Add specific ones back if needed (rare).
• seccompProfile.type: RuntimeDefault — applies the container runtime's default seccomp profile.

Pod Security Standards

Kubernetes 1.25+ includes built-in pod security admission. Apply at the namespace level:

apiVersion: v1
kind: Namespace
metadata:
  name: app
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted

The three levels are:

• Privileged — no restrictions (use only for system namespaces)
• Baseline — prevents known privilege escalations
• Restricted — heavily restricted, follows current hardening best practices

Health Checks & Probes

Kubernetes provides three types of probes. Use all three for production workloads.

Startup Probe

Protects slow-starting containers. The kubelet won't run liveness or readiness probes until the startup probe succeeds. This prevents aggressive liveness probes from killing pods that are still initializing.

startupProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 30    # 30 × 2s = 60 seconds to start
  periodSeconds: 2

Liveness Probe

Detects deadlocks and hung processes. If the liveness probe fails, the kubelet restarts the pod.

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 15
  timeoutSeconds: 3
  failureThreshold: 3

Warning: Don't make the liveness probe depend on external services (database, cache). If the database is down, restarting the pod won't help — it will create a cascade of restarts.

Readiness Probe

Controls whether the pod receives traffic from the Service. If readiness fails, the pod is removed from Service endpoints but not restarted.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

Design pattern: The /ready endpoint should check that the application can serve requests (database connected, cache warmed, etc.), while /health only checks that the process is alive.

Autoscaling

Horizontal Pod Autoscaler (HPA)

Scale the number of pods based on CPU, memory, or custom metrics.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down

Key principles:

• Set minReplicas >= 2 for production (HA).
• Use the behavior block to prevent flapping. Scale up fast, scale down slowly.
• Target 60-80% utilization. Below 60% wastes resources; above 80% risks latency.

Vertical Pod Autoscaler (VPA)

Automatically adjusts resource requests/limits based on actual usage. Useful for right-sizing but can conflict with HPA.

Recommendation: Use VPA in "Off" mode to get sizing recommendations, then apply them manually.

Pod Disruption Budgets (PDB)

Prevent voluntary disruptions (node drain, rolling update) from removing too many pods at once.

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: app

Network Policies

Default Deny

Always start with a default deny policy, then create allow-list policies for specific traffic. This follows the principle of least privilege.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Allow Specific Traffic

spec:
  podSelector:
    matchLabels:
      app: myapp
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - port: 8080
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: kube-system
      ports:
        - port: 53
          protocol: UDP

Important: NetworkPolicies require a CNI that supports them (Calico, Cilium, Weave Net). The default kubenet does not enforce them.

Secrets Management

Don't Store Secrets in Git

Kubernetes Secrets are base64-encoded, not encrypted. Never commit them to version control.

Recommended approaches:

1. External Secrets Operator — syncs secrets from AWS Secrets Manager, Vault, GCP Secret Manager.

2. Sealed Secrets — encrypt secrets with a cluster-specific key. Safe to commit the encrypted version.

3. SOPS — encrypts YAML files with age, PGP, or cloud KMS. Works with any GitOps tool.

External Secrets Operator Example

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: app-secrets
  data:
    - secretKey: database-url
      remoteRef:
        key: prod/app/database-url

Kustomize Patterns

Base + Overlays

Organize manifests as a base configuration with environment-specific overlays:

├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/
│   │   └── kustomization.yaml
│   └── prod/
│       └── kustomization.yaml

Strategic Merge Patches

Use JSON patches for precise field modifications:

patches:
  - target:
      kind: Deployment
      name: app
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 5

Component Pattern

For optional features (monitoring, RBAC) that may or may not be included:

# kustomization.yaml
components:
  - ../../components/monitoring
  - ../../components/rbac

Helm Best Practices

1. Pin chart versions in CI/CD. Never use helm install without --version.

2. Use values files per environment (values-dev.yaml, values-prod.yaml) instead of long --set chains.

3. Validate templates before deploying: helm template . -f values-prod.yaml | kubectl apply --dry-run=server -f -

4. Use named templates (_helpers.tpl) for reusable label blocks and selectors.

5. Set revisionHistoryLimit to avoid accumulating old ReplicaSets.

Observability

The Three Pillars

1. Metrics — Prometheus + Grafana. Use ServiceMonitor for auto-discovery.

2. Logs — Structured JSON logging shipped via Promtail/Fluentbit to Loki or Elasticsearch.

3. Traces — OpenTelemetry SDK → Jaeger or Tempo.

Essential Alerts

At minimum, configure alerts for:

• High error rate (5xx > 5% for 5 minutes)
• Pod crash loops (restarts > 3 in 15 minutes)
• High memory usage (> 90% of limit for 10 minutes)
• Pod not ready for 5+ minutes
• HPA at maximum capacity for 15+ minutes
• Certificate expiry < 14 days
• PVC usage > 85%

GitOps Workflows

ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/org/manifests
    targetRevision: main
    path: overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Flux

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: manifests
  path: ./overlays/prod
  prune: true

Key principle: The Git repository is the source of truth. All changes go through pull requests, are reviewed, and are applied automatically by the GitOps controller.

Production Checklist

Before deploying to production, verify each item:

• [ ] All containers have resource requests AND limits
• [ ] Liveness, readiness, and startup probes are configured
• [ ] Security context: non-root, read-only filesystem, drop all capabilities
• [ ] PodDisruptionBudget is configured (minAvailable >= 1)
• [ ] HPA is configured with appropriate min/max replicas
• [ ] NetworkPolicies enforce least-privilege network access
• [ ] Default-deny network policy is applied to the namespace
• [ ] Secrets are not stored in Git (use External Secrets, Sealed Secrets, or SOPS)
• [ ] RBAC follows least-privilege principle
• [ ] Images use specific tags, not :latest
• [ ] Topology spread constraints distribute pods across zones
• [ ] Monitoring: ServiceMonitor and alerting rules are configured
• [ ] Pod security standards are enforced at the namespace level
• [ ] terminationGracePeriodSeconds allows graceful shutdown
• [ ] revisionHistoryLimit is set to avoid unbounded ReplicaSet history

*Part of Kubernetes Manifests Pack — (c) 2026 Datanest Digital (datanest.dev)*

Kubernetes Manifests Pack

Production-ready Kubernetes manifests with Kustomize overlays, Helm values, batch jobs, and Prometheus monitoring.

Deploy a complete application stack to any Kubernetes cluster. Includes base manifests following the app.kubernetes.io labeling convention, Kustomize overlays for dev/prod, Helm value files, database migration and cleanup jobs, and Prometheus ServiceMonitor + alerting rules.