MEMORY_LIMIT_EXCEEDED (24h), ClickHouse

​

At a glance

The count of queries killed in the last 24 hours because they tried to use more memory than their limit allowed. ClickHouse aborts a query the instant its tracked memory crosses max_memory_usage (per-query) or max_server_memory_usage (whole-server), raising error code 241, MEMORY_LIMIT_EXCEEDED. A non-zero reading is rarely “we are short on RAM”; it is almost always one heavy, unbounded query (a GROUP BY over high-cardinality keys, an under-filtered JOIN, or a DISTINCT on a huge column) trying to swallow the instance. The card is your early-warning that a single bad query is putting the whole server at risk.

​

Calculation

SELECT count() AS memory_limit_kills_24h
FROM system.query_log
WHERE type = 'ExceptionWhileProcessing'
  AND exception_code = 241          -- MEMORY_LIMIT_EXCEEDED
  AND event_time >= now() - INTERVAL 24 HOUR;

SELECT
    event_time,
    user,
    query_duration_ms,
    formatReadableSize(memory_usage) AS peak_mem,
    substring(query, 1, 200) AS query_head
FROM system.query_log
WHERE exception_code = 241
  AND event_time >= now() - INTERVAL 24 HOUR
ORDER BY memory_usage DESC
LIMIT 10;

​

Worked example

1. This is one bad query, not a capacity problem. Server memory was never the constraint; the per-query max_memory_usage was. The 14 kills are 14 reruns of the same dashboard tile auto-refreshing every ~7 minutes. Throwing more RAM at the server would not help; the per-query ceiling would still bite.
2. The fix is the query, not the limit. groupArray(event_name) over every distinct user_id materialises an unbounded array per user. The correct fix is to bound it (groupArrayLast(50)(...), a LIMIT, or a pre-aggregated materialised view), or to enable spilling with max_bytes_before_external_group_by so the aggregation goes to disk instead of dying.
3. Sensitivity caught it at the first occurrence. Because the threshold is > 0, the card went red at 16:02 on the very first kill, before the dashboard refreshed 13 more times. The team could have killed the saved tile or capped the user before the pattern repeated.

Why "one heavy query" not "low on RAM":
  - Server RAM ceiling (max_server_memory_usage): ~52 GB, never breached
  - Per-query ceiling (max_memory_usage):          10 GB, hit on every run
  - Other queries during the window:               unaffected, completing normally
  => The kills are isolated to one statement hitting the per-query ceiling.
     Raising the server ceiling fixes nothing; raising the per-query ceiling
     just lets one tile starve everyone else. Fix the query.

​

Sibling cards platform teams should reference together

​

Reconciling against the source

system.query_log for the authoritative per-execution record: filter exception_code = 241 over your window and read query, user, memory_usage, and exception for the full failure text and the offending SQL. system.metrics (MemoryTracking) and system.asynchronous_metrics for the live server-memory picture at the time of the kills, to distinguish per-query from server-wide pressure. SELECT * FROM system.settings WHERE name LIKE '%max_memory%' to read the effective per-query, per-user, and server ceilings that the killed queries were measured against. ClickHouse Cloud: the service’s query insights / monitoring view exposes the same memory-limit errors against the managed memory ceiling for the service tier.

​

Known limitations / FAQs

​

Tracked live in Vortex IQ Nerve Centre