Job Success Rate (24h), Databricks

​

At a glance

The percentage of Databricks job runs that completed successfully over the last 24 hours, shown as a gauge. This is the Databricks-distinctive defining metric for pipeline health: if scheduled runs are failing, your data pipelines are broken and every downstream table, dashboard, and feature store is at risk of going stale. A single number gives the platform owner an instant read on whether the lakehouse is delivering its contracts. The alert fires below 95%, the floor at which the team should assume something systemic is wrong rather than a one-off flake.

​

Calculation

SuccessRate(24h) = SUCCESS
                   ----------------------------------  × 100
                   SUCCESS + FAILED + TIMEDOUT

(runs reaching a terminal state in the last 24h; CANCELED excluded)
Alert fires when SuccessRate < 95%

1. Cancellations are excluded so the rate measures quality, not activity. A run that a human or an upstream task cancelled is neither a success nor a failure of the pipeline; it is a deliberate act. Including cancellations would let a busy operations day drag the rate down even when nothing is genuinely broken. The denominator is only runs that actually attempted to complete.
2. The rate is run-weighted, not job-weighted. A job that runs hourly contributes 24 runs to the window; a daily job contributes one. This is intentional: a flaky high-frequency job has more downstream impact (more stale refreshes) and should move the gauge more than a single daily job. To see per-job patterns instead, use Top 10 Failing Workflows (7d).
3. Small denominators are volatile. With only 8 runs in the window, one failure is a 12.5-point drop. The gauge shows the absolute rate, but the card annotates the run count so a 87.5% from 7-of-8 is read differently from 87.5% across 400 runs. Low-volume workspaces should weight the failed-run worklist over the headline percentage.

​

Worked example

1. 94.4% is below the floor, so this is not a normal day. In a healthy steady state this workspace sits at 98 to 99%. The drop of four to five points means a cluster of failures, not background flakiness. The first move is to open Failed Jobs (24h) and see whether the 11 failures share a root cause.
2. The worklist shows 7 of the 11 failures are the same hourly job. prod_orders_hourly failed seven times overnight on the same schema-mismatch error. That is one broken pipeline expressing itself as seven failed runs, which is why a single bug dropped the rate so far: the high-frequency job dominates the denominator. Fixing that one job recovers most of the gap. Cross-check Failed Job Burst (>5 failures in 1h) to confirm whether the seven clustered in one hour (a cascade) or spread evenly (a deterministic per-run bug).
3. The two timed-out runs are a separate, slower problem. A nightly feature build hit its timeout twice. That is a duration / data-volume issue, not a code error; it is worth a same-day look via Long-Running Jobs (>1h) but is not what dragged the gauge down.

Reading the 94.4%:
  Dominant cause:  prod_orders_hourly × 7 (schema mismatch)  → fix recovers ~3.6 pts
  Secondary:       feature_build × 2 timeouts                → duration tuning
  Residual:        2 unrelated one-off failures              → background flake, ignore
  -----------------------------------------------------------------
  After fixing the hourly job, projected rate: ~98.5% (back above floor)

​

Sibling cards to read alongside

​

Reconciling against the source

Workflows → Job runs with the 24-hour filter shows every run and its result state; counting Succeeded against the total of Succeeded + Failed + Timed out reproduces the rate. System tables: system.lakeflow.job_run_timeline holds run-level terminal states for an exact SQL reconcile. Each job’s Runs tab shows the per-job success history, useful for confirming which job is dragging the rate.

SELECT
  ROUND(100.0 *
    SUM(CASE WHEN result_state = 'SUCCEEDED' THEN 1 ELSE 0 END) /
    SUM(CASE WHEN result_state IN ('SUCCEEDED','FAILED','TIMED_OUT') THEN 1 ELSE 0 END)
  , 1) AS success_rate_pct
FROM system.lakeflow.job_run_timeline
WHERE period_end_time >= now() - INTERVAL 24 HOURS;

​

Known limitations / FAQs

​

Tracked live in Vortex IQ Nerve Centre