Event-log diagnosis

Watch the diagnostic find why this job is slow

This page bundles a synthetic Spark workload pattern using the playbook's metric schema — the same metrics your cluster's Spark listeners report. It then runs the playbook's actual diagnostic checks over it (the same thresholds and severity rules as notebooks/spark_diagnostic.py). Open any finding to see the evidence and the exact fix from the guides — then compare the before/after benchmark.

🔍 Diagnostic report

Checks run: SKEW · SPILL · SMALL_FILES · GC · CONFIG · SHUFFLE — over

Deterministic — same log, same findings, every time.

📊 Before / after benchmark

The same job after applying the three fixes above — illustrative comparison using the playbook's benchmarking_framework.py method (wall clock, shuffle, spill, GC, stage/task counts).

🔒 This demo diagnoses one bundled log. The full playbook diagnoses yours.

notebooks/spark_diagnostic.py — 5 check families against your live cluster (spill, GC, shuffle, small files, config), plus an optional row-level DataFrame skew analyzer and Delta DESCRIBE DETAIL small-file inspector
notebooks/benchmarking_framework.py — register before/after runs, capture stage metrics (wall clock, shuffle, spill, GC, executor time), compare across runs
notebooks/cost_per_query_estimator.py — DBU cost per query from your cluster config and pricing tier
guides/ — 25 optimization patterns with before/after code, cluster sizing, AQE tuning, Photon, memory management, Spark UI reading
tools/ — partition_decision_tree.py and cluster_calculator.py CLIs

Watch the diagnostic find why this job is slow

🔍 Diagnostic report

📊 Before / after benchmark

🔒 This demo diagnoses one bundled log. The full playbook diagnoses yours.

Ready for the complete Spark Optimization Playbook?

Included with Spark Optimization Playbook

Ready for the complete Spark Optimization Playbook?