Skip to main content
Nerve Centre KPIs · Audit Profile · Sentiment Settings The questions every BigQuery operator asks at 9am on Monday: why did bytes-scanned (on-demand dollars) jump 50% week-over-week, are reservation slots saturated and queuing jobs, which queries are scanning the most data and should be partitioned or pruned, and is the streaming-ingest pipeline keeping up. Cross-references ecommerce platforms for data-pipeline insight - Dataflow / Fivetran ingest stalled while orders are flowing indicates an analytics blind spot, and runaway scan cost while order volume is flat indicates inefficient queries burning real money.

​
What this audit checks

​
Authentication & access

  • Service-account key valid and not expired / rotated (datasets.list returns 200)
  • Principal holds roles/bigquery.jobUser - required to read INFORMATION_SCHEMA.JOBS
  • Principal holds roles/monitoring.viewer - required for slot-utilisation time series (degrades to job-derived slot-ms if absent)
  • Default location matches dataset location (region-us / region-eu INFORMATION_SCHEMA view resolves)

​
Bytes-Scanned Cost & Efficiency

  • Bytes scanned +50% week-over-week without proportional query-count growth (runaway / un-pruned query)
  • Single query scanning >1 TiB on an un-partitioned or un-clustered table (partitioning candidate)
  • Estimated on-demand cost per query rising >25% week-over-week (query inefficiency or result-set bloat)
  • Query cache hit rate below 50% (cached results are free - low rate means a cost lever is unused)

​
Slots, Capacity & Concurrency

  • Reservation slot utilisation >90% sustained 10m with jobs in PENDING state (reservation undersized)
  • Concurrent query jobs approaching the per-project interactive quota (default ~100; queries about to queue or fail)
  • Pending-jobs queue depth >5 sustained (slot starvation or concurrency cap)
  • Slot-ms consumed climbing faster than query count (capacity-priced inefficiency)

​
Performance & Errors

  • Query latency p95 >5s sustained 15m (INFORMATION_SCHEMA.JOBS duration percentiles)
  • Slow-query rate >5% of total query jobs (duration > 5000ms)
  • Job error rate spike >1% in 1h window (quota exceeded, permission, syntax, resources exhausted)
  • Quota-exceeded errors detected (>0 in 24h - bytes-billed / concurrency / API quota hit)

​
Storage & Backup

  • Total logical storage +25% month-over-month (table growth outpacing cleanup)
  • Largest table growing faster than expected (top storage-cost driver)
  • Streaming buffer lag >10s (freshly streamed rows not yet queryable; ingest backpressure)
  • Newest table snapshot / scheduled export older than 72h (recovery window shrinking beyond 7-day time travel)

​
Cross-channel: analytics spend vs business volume

  • Bytes-scanned cost +50% WoW while ecom order volume flat (sibling = bigcommerce.order / shopify.order / adobe_commerce.order; inefficient queries burning money)
  • Streaming / scheduled ingest stalled while ecom orders flowing (sibling = ecom order rate; Dataflow / Fivetran pipeline failure = analytics blind spot)
  • Expensive query cluster co-occurs with checkout-completion drop (sibling = ecom checkout; dashboard load starving interactive queries)

​
Data sources

  • GET https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/datasets - Auth probe + active dataset count
  • GET https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/jobs - Running / pending / done jobs with state + statistics (bytesProcessed, slotMs, errorResult)
  • POST https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/queries - Read INFORMATION_SCHEMA.JOBS / JOBS_TIMELINE / TABLE_STORAGE for cost, latency, slot-ms, storage trends
  • GET https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/datasets/{datasetId}/tables/{tableId} - Per-table active / long-term logical bytes + row count for storage growth tracking
  • GET https://monitoring.googleapis.com/v3/projects/{projectId}/timeSeries - Slot allocation / utilisation time series (bigquery.googleapis.com/slots/allocated)