Connection Pool at >90% Saturation, ClickHouse

​

At a glance

The alert feed for moments when the ClickHouse connection pool runs hot: concurrent connections sit above 90% of the configured max_connections ceiling for a sustained minute. When the pool saturates, new client connections are refused (you see Too many simultaneous queries or connection timeouts), so dashboards stall and ingest jobs back up. This card lists each saturation event with its start time, peak utilisation, and duration, so the on-call DBA can see whether the cause was a query storm, a connection leak, or genuine capacity demand.

​

Calculation

saturation_pct = (TCPConnection + HTTPConnection + MySQLConnection + PostgreSQLConnection)
                 / max_connections * 100

SELECT metric, value
FROM system.metrics
WHERE metric IN ('TCPConnection','HTTPConnection','MySQLConnection','PostgreSQLConnection')

SELECT value FROM system.server_settings WHERE name = 'max_connections'

​

Worked example

1. The 08:31 event is a recurring pattern. It lands every weekday at the same minute. This is the classic “everyone opens their dashboard at the start of the day” storm: dozens of BI panels each opening a connection simultaneously. It self-resolves in minutes but each occurrence risks refused connections at the peak.
2. The 19 Apr event is the dangerous one. It ran for 11.5 minutes and peaked at 99%. A duration that long with no obvious traffic trigger points to a connection leak: a service opening connections and not closing them. Left unchecked, the next one hits 100% and starts refusing connections.
3. None of these show up in query latency. A refused connection never becomes a query, so Query Latency p95 (ms) can look perfectly healthy while clients are being turned away. This card is the only place the impact is visible.

Triaging the 19 Apr leak:
  - Confirm live connection sources:
      SELECT client_name, count() FROM system.processes GROUP BY client_name ORDER BY count() DESC
  - A single client holding hundreds of idle connections = leak
  - Short-term: raise max_connections to add headroom (buys time, not a fix)
  - Real fix: enable connection pooling / reuse in the offending client,
              or set an idle-connection timeout so leaked connections drop

1. Saturation is invisible in query metrics. Refused connections never become queries, so latency and error-rate cards can look green while clients are locked out. Trust this card as the canonical view of pool starvation.
2. Duration distinguishes traffic from leaks. A short spike at a predictable time is a refresh storm; a long, slowly climbing event with no traffic trigger is a leak. Read the duration column, not just the peak.
3. Headroom is the real metric. A pool living at 88% is one burst away from refusing connections. Pair this alert with Connection Pool Saturation % and keep steady-state utilisation well below 70%.

​

Sibling cards

​

Reconciling against the source

Check live connection counts against the ceiling in clickhouse-client:

SELECT metric, value FROM system.metrics WHERE metric LIKE '%Connection';
SELECT value FROM system.server_settings WHERE name = 'max_connections';

See who is connected and what they are running in system.processes (one row per active query/connection), grouped by client_name or user to find a leak. On ClickHouse Cloud, max_connections is managed per service tier; the same system.metrics query works in the SQL console, and the managed monitoring view shows connection utilisation over time.

​

Known limitations / FAQs

​

Tracked live in Vortex IQ Nerve Centre