Instance logs: ClickHouse first, Kafka only as a buffer

Every apply VM writes logs that must not disappear. The studio shows the latest 20 KB. Errors go to Sentry. Prometheus is for metrics, not log text. Kafka is optional glue — not the system of record.

Solid arrows are live. Dashed is optional. ClickHouse holds every line; the UI never asks Kafka or Prometheus for log text.

What is live now

VMs and the studio POST lines to /api/logs and the apply ingest.
graphai.jobs_log_inbox is the durable queue (ClickHouse MergeTree). Vercel cannot host Kafka.
graphai.jobs_instance_logs is the system of record, ordered by (instance, at). Logical stream per VM; one physical table so we do not CREATE TABLE per name.
The Progress log view loads the latest 20 KB for the selected instance, including previous sessions. Clear view does not delete ClickHouse.
Error lines POST to Sentry with an instance fingerprint.

How to glue them

ClickHouse ← VMs. Direct HTTP insert is enough at our volume. The inbox table is the WAL if a second write fails.

Sentry ← errors only. Fingerprint by instance + message shape. Never send the 20 KB window as a Sentry breadcrumb dump.

Prometheus ← metrics. Scrape GCP Cloud Monitoring / node exporters: CPU, apply success rate, ingest lag. A CH query can export log_lines_total as a gauge if we add a recording rule later.

Kafka / Pub/Sub ← only under burst. If ingest 429s, put GCP Managed Kafka or Pub/Sub in front: topic key = instance id, ClickHouse ClickPipes or Kafka engine as consumer. UI still reads ClickHouse.

How not to glue them

Do not store log bodies in Prometheus. Cardinality will explode on instance names and messages.
Do not use Sentry as a log archive. It is for exceptions you will triage.
Do not query Kafka from the studio. Kafka is a buffer, not a database.
Do not run Kafka on Vercel. There is no durable broker there.
Do not CREATE a ClickHouse table per VM from user input. Isolate with instance in the sort key.

ClickHouse vs Kafka subscribe

ClickHouse Cloud does not give the studio a Kafka-style consumer group over HTTP. It can read from Kafka if we provision a cluster. Until then the inbox MergeTree is the queue: append-only, ordered by time, never truncated by the UI.