ClickHouse Event Ingest vs Ecom Orders, ClickHouse

​

At a glance

A dual-axis view that plots ClickHouse event-ingest rate against storefront order (and click) rate over the same period. In a healthy analytics pipeline the two move together: when orders and clicks rise, the events describing them flow into ClickHouse at a matching rate. The diagnostic power is in the divergence. If orders keep flowing on the storefront but ingest into ClickHouse flattens or stalls, the pipeline between the storefront and the database has broken: a producer crashed, a Kafka or queue consumer is stuck, or inserts are being rejected. That gap is invisible if you watch ingest alone (it just looks quiet) and invisible if you watch orders alone (they look fine). Seen together, a stalled-ingest-while-orders-flowing pattern is an unmistakable “pipeline broken” signal, and because the lost events are analytics data, the damage compounds silently until someone reconnects the feed.

​

Calculation

-- Ingest rate from the InsertedRows event counter delta
SELECT
    toStartOfInterval(event_time, INTERVAL 1 MINUTE) AS bucket,
    max(ProfileEvent_InsertedRows) - min(ProfileEvent_InsertedRows) AS rows_ingested,
    rows_ingested / 60 AS rows_per_sec
FROM system.metric_log
WHERE event_time > now() - INTERVAL 24 HOUR
GROUP BY bucket
ORDER BY bucket

​

Worked example

1. Orders are healthy, so the storefront is fine. The order series is steady at 31 to 36 per minute throughout. The business is operating normally; shoppers are buying.
2. Ingest has collapsed independently. Rows per second fell from ~52,000 to under 200, a near-total stall, with no corresponding drop in orders. The two series have decoupled, which is the signature of a broken pipeline rather than a quiet period.
3. Data is being lost right now. Every order and click happening since 13:45 should have produced events that are not arriving. Dashboards, attribution, and any storefront feature reading ClickHouse are silently going stale, and the gap grows every minute the feed stays down.

Why ingest stalled while orders kept flowing:
  - Orders/min: steady ~33 (storefront healthy)
  - Ingest: 52k rows/sec -> <200 rows/sec (near-total stall)
  - Decoupling started 13:45, total by 13:50
  - Likely causes, in order of probability:
      1. Kafka consumer stuck / crashed (lag climbing, no commits)
      2. Inserts rejected at ClickHouse (e.g. TOO_MANY_PARTS code 252 on the target table)
      3. Producer healthy, consumer healthy, but a schema / auth change is bouncing inserts
  - First checks:
      1. Consumer lag on the topic + consumer process health
      2. Too Many Parts Errors (24h) and Failed Queries (24h) on the target table
      3. ClickHouse error log for rejected INSERTs

1. The divergence is the signal, not either series alone. Quiet ingest looks fine in isolation and steady orders look fine in isolation; only the two together expose a broken feed.
2. Lost analytics data is unrecoverable in real time. Unlike a slow query you can re-run, events not ingested during a stall are gone unless the queue retains and replays them. Every minute of stall is permanent data loss to your reporting, so this pages.
3. The error counters tell you which side broke. Errors climbing means ClickHouse is rejecting inserts (fix the database); errors flat at zero with ingest at zero means events are not arriving (fix the consumer or queue).

​

Sibling cards

​

Reconciling against the source

Read the ingest counter in clickhouse-client:

SELECT value FROM system.events WHERE event = 'InsertedRows'

Snapshot it, wait, snapshot again, and divide the delta by the elapsed seconds to get the live rate the card plots. Confirm inserts are actually landing (not being rejected) with SELECT count(), max(event_time) FROM system.query_log WHERE type = 'QueryFinish' AND query_kind = 'Insert' AND event_time > now() - INTERVAL 15 MINUTE, and check for rejected inserts with ... WHERE type = 'ExceptionWhileProcessing'. On ClickHouse Cloud, the same system.events and system.query_log reads run in the SQL console; the order and click side of this card comes from your storefront connector, so reconcile that half against the storefront’s own order reporting, not against ClickHouse.

​

Known limitations / FAQs

​

Tracked live in Vortex IQ Nerve Centre