Failed Jobs (24h), Databricks

​

At a glance

The count of Databricks job runs that finished in a failure state over the last 24 hours, surfaced as a triage queue. A “failed job” here means a scheduled or triggered run whose terminal result_state is FAILED or TIMEDOUT. For a data platform team this is the single most operationally urgent number on the Databricks board: every failed run is a table that did not refresh, a feature that did not build, or a report that will show stale data this morning. The card is both the count and the worklist of exactly which runs to investigate first.

​

Calculation

FailedJobs(24h) = COUNT(run)
                  WHERE state.result_state IN ('FAILED', 'TIMEDOUT')
                  AND end_time >= now - 24h

1. Runs, not jobs, are counted. If one nightly job retries and fails three times, that is three failed runs but one broken pipeline. The count reflects runs; the worklist groups by job so a flapping job is visible as one entry with a retry count, not three separate alarms.
2. TIMEDOUT is treated as a failure on purpose. A run that hits its timeout produced no usable output and usually signals either a data-volume spike or a stuck stage. Folding it into the same count keeps the triage queue honest: from a downstream consumer’s point of view, a timed-out table is just as missing as an errored one.
3. CANCELED is excluded deliberately. A cancelled run is an intentional act (a human killed it, or an upstream task short-circuited the workflow). Counting cancellations as failures would inflate the queue with non-incidents and erode trust in the alert.

​

Worked example

1. prod_orders_ingest failed and its retry failed too (critical). This is the page-worthy event: the table feeding every downstream report did not refresh, and the automatic retry did not save it, so the failure is deterministic (bad input or a code regression), not transient. The run detail shows a schema-mismatch error: an upstream source added a column. The fix is a quick schema evolution change. While it is broken, Pipeline Lag (since last success) on the orders table is climbing and any morning report built on it will be stale.
2. feature_store_build timed out at the 120-minute limit (critical). Not a code error but a duration blowout. The likely cause is a data-volume spike or a skewed join. The engineer checks Long-Running Jobs (>1h) to confirm it was genuinely stuck rather than slow, then either raises the timeout for tonight and investigates skew, or fixes the join before the next scheduled run.
3. marketing_attribution and adhoc_export_csv are standard / low tier. These did not page anyone and can wait until the two criticals are resolved. The attribution job is worth a same-day fix because a stakeholder relies on it; the ad-hoc export is genuinely low priority.

Triage decision in plain terms:
  CRITICAL, fix now:   prod_orders_ingest (schema), feature_store_build (timeout)
  STANDARD, fix today: marketing_attribution
  LOW, fix when free:  adhoc_export_csv
  -----------------------------------------------------------------
  Blast radius of the criticals: all overnight reporting + the recs feature store

​

Sibling cards to read alongside

​

Reconciling against the source

Workflows → Job runs lists every run with its result state and a 24-hour filter; set the status filter to “Failed” to match the card’s core count (then add timed-out runs). System tables: system.lakeflow.job_run_timeline (and system.lakeflow.jobs) hold run-level history you can query in SQL for an exact reconcile. Each run page shows the failing task, the stack trace, the cluster, and the retry chain for root-cause work.

SELECT result_state, COUNT(*) AS runs
FROM   system.lakeflow.job_run_timeline
WHERE  period_end_time >= now() - INTERVAL 24 HOURS
AND    result_state IN ('FAILED', 'TIMED_OUT')
GROUP  BY result_state;

​

Known limitations / FAQs

​

Tracked live in Vortex IQ Nerve Centre