Idle Cluster DBU Wasted (24h), Databricks

​

At a glance

The DBU burned over the last 24 hours by clusters that were running but had no active job or query on them. This is pure waste: compute you paid for that did no work. The most common cause is auto-termination set too generously (or off), so an all-purpose cluster sits warm for an hour after the last analyst logs off, or a job cluster lingers after its run completes. The card is the single best signal for auto-termination tuning, and the alert fires when idle waste crosses a tenth of total spend, the point at which it stops being rounding error and starts being a line item.

​

Calculation

IdleDBU(24h) = Σ usage_quantity
               WHERE cluster_state = 'RUNNING'
               AND active_job_runs = 0
               AND active_commands  = 0
               AND interval within last 24h

IdleShare = IdleDBU(24h) / TotalDBU(24h)
Alert fires when IdleShare > 0.10

1. Idle is the absence of work, not low utilisation. A cluster running a small query at 5% CPU is busy, not idle, and its DBU is productive spend (right-sizing is a different conversation, handled by Avg Cluster CPU Utilisation %). This card counts only intervals with literally no workload.
2. Start-up and shutdown overhead is excluded. Every cluster pays a few minutes of DBU to acquire instances and initialise Spark, and a moment to tear down. That is unavoidable and is not waste; the card carves it out so a workload that spins clusters up and down frequently is not unfairly penalised.
3. The alert is a ratio, not an absolute. Ten DBU of idle on a 2,000 DBU day is noise; ten DBU on an 80 DBU day is 12.5% and worth acting on. Expressing waste as a share of total keeps the signal meaningful for both large and small workspaces.

​

Worked example

1. ml-sandbox-ap is the worst offender at 69% idle with auto-termination off. A data scientist spun it up to prototype, ran a few cells, and left it running all night. Two thirds of its spend was paid for nothing. The fix is immediate and high-ROI: turn on auto-termination at 30 minutes via a cluster policy so sandbox clusters cannot be left running indefinitely. Estimated saving: roughly 90 DBU/day, every day.
2. analyst-shared-ap is the biggest absolute waste at 188 DBU, driven by a 120-minute timeout. Fourteen analysts share it, so it is genuinely useful during the day, but the two-hour idle window means it sits warm long after the last person leaves. Dropping the timeout to 30 minutes would recover most of the 188 DBU without affecting working hours, because nobody needs a cluster to stay warm for two hours of inactivity.
3. The two production ETL clusters are healthy (4% and 9% idle). Their tight 10-minute auto-termination keeps idle near the unavoidable start-up overhead. They are the template: apply the same policy to the interactive clusters.

Recoverable waste, ranked by ease of fix:
  ml-sandbox-ap:     ~90 DBU/day   (turn auto-term ON, 30 min)   ← do first
  analyst-shared-ap: ~150 DBU/day  (cut timeout 120 → 30 min)    ← do second
  ----------------------------------------------------------------
  Combined: ~240 DBU/day ≈ 16% of total spend, recovered with two policy changes

​

Sibling cards to read alongside

​

Reconciling against the source

Compute → cluster → Event log shows TERMINATING reasons (INACTIVITY is the auto-termination event) and lets you see how long a cluster idled before it shut down. System tables: join system.billing.usage (DBU per interval) against system.compute.clusters / the cluster event timeline to attribute DBU to intervals with no active run. This is the source the card reconstructs. Cluster settings: the autotermination_minutes field per cluster is the lever that drives this number.

-- Total and idle DBU sketch over 24h; idle = usage with no concurrent run
SELECT u.usage_metadata.cluster_id,
       SUM(u.usage_quantity) AS total_dbu
FROM   system.billing.usage u
WHERE  u.usage_date >= current_date() - INTERVAL 1 DAYS
GROUP  BY u.usage_metadata.cluster_id;
-- Cross-reference against system.lakeflow.job_run_timeline and query history
-- to subtract intervals where a run/query was active.

​

Known limitations / FAQs

​

Tracked live in Vortex IQ Nerve Centre