Replication Queue Size (Top Tables), ClickHouse

​

At a glance

The number of pending replication tasks queued on each ReplicatedMergeTree table, ranked highest first. In a replicated ClickHouse cluster, every write (insert, merge, mutation) is recorded as an entry in a shared log held in ClickHouse Keeper or ZooKeeper, and each replica works through that log to stay in sync. system.replicas.queue_size is the count of log entries a replica still has to apply. A healthy queue drains continuously and sits near zero. A queue that climbs and stays high means a replica is falling behind: it is accepting work faster than it can replay, which leads to stale reads on that replica and, if it persists, growing Replication Lag (absolute_delay). This card is the per-table breakdown so a DBA sees exactly which table is stuck.

​

Calculation

SELECT
    database,
    table,
    queue_size,
    inserts_in_queue,
    merges_in_queue,
    is_session_expired,
    future_parts
FROM system.replicas
ORDER BY queue_size DESC
LIMIT 10

​

Worked example

1. The queue is insert-dominated (318 of 342). This replica is behind on fetching new parts from its peers, not on applying merges. The cause is on the fetch path: either this replica’s network or disk cannot keep up with the inflow, or the peer it fetches from is itself overloaded.
2. Only one table is affected. order_events_repl at 41 is normal merge churn; the rest are near zero. A single insert-heavy table behind on one replica points at that table’s write rate outpacing this node’s fetch throughput, not a cluster-wide Keeper problem (which would back up every table at once).
3. Stale reads are already possible. Any query routed to replica-02 for clickstream_repl is missing up to 318 parts of recent data. If the load balancer does not prefer in-sync replicas, dashboards reading this table may show data that is minutes behind the other replicas.

Diagnosing an insert-dominated queue:
  1. Confirm it is not Keeper:
     SELECT is_session_expired, zookeeper_exception FROM system.replicas
     WHERE table = 'clickstream_repl';
     (expired session or a ZK exception = fix Keeper first)
  2. Confirm the source replica is healthy:
     a peer that is itself overloaded cannot serve fetches fast enough.
  3. Check this node's fetch capacity:
     network saturation or slow disk on replica-02 throttles part fetches.
  4. If a single entry is stuck (poison), inspect system.replication_queue:
     SELECT type, last_exception, num_tries FROM system.replication_queue
     WHERE table = 'clickstream_repl' ORDER BY num_tries DESC;
     a high num_tries with a repeating last_exception is the blocker.

1. Read the insert-vs-merge split, not just the total. An insert-heavy queue is a fetch/capacity problem; a merge-heavy queue is a merge-throughput problem. The breakdown tells you where to look.
2. One table behind on one replica is a local problem; every table behind is a Keeper problem. Cluster-wide queue growth points at the coordination layer (Keeper/ZooKeeper session or latency), not at any single table.
3. A growing queue means stale reads now, lag later. Route reads to in-sync replicas while you drain it, and treat sustained depth as the early warning for Replication Lag (absolute_delay).

​

Sibling cards

​

Reconciling against the source

Run the same query the card uses against system.replicas from clickhouse-client:

SELECT database, table, queue_size, inserts_in_queue, merges_in_queue
FROM system.replicas ORDER BY queue_size DESC LIMIT 10

Drill into individual stuck entries with SELECT type, num_tries, last_exception FROM system.replication_queue ORDER BY num_tries DESC. Check the coordination layer with SELECT is_session_expired, zookeeper_exception, log_pointer, log_max_index FROM system.replicas and confirm Keeper/ZooKeeper health. On ClickHouse Cloud, replication is managed by the service, so this card and these tables apply to self-managed ReplicatedMergeTree clusters; Cloud’s own monitoring view surfaces replica sync state.

​

Known limitations / FAQs

​

Tracked live in Vortex IQ Nerve Centre