Scaling jobs.graphai.eu on GCP

This public, non-security overview explains how the current platform can scale from request-bound crawls into an event-driven data platform while keeping ClickHouse as the fast analytical serving layer.

The architecture in one diagram

Public pages remain separate from private applications, mailbox content, billing, and administration. Crawlers are interchangeable: native ATS, HTML, Tavily, Cloud Run, and GCE Spot workers all use the same job-observation contract.

TOGAF-style architecture diagram for jobs.graphai.eu

What changes at scale

Short APIs enqueue crawl work and return a crawl ID instead of waiting for a full crawl.
Pub/Sub carries crawl, job, mail, and application events with replay and dead-letter handling.
Cloud Storage keeps immutable Bronze data; Parquet/Iceberg-compatible Silver data is normalized; ClickHouse serves Gold marts.
Cloud Run handles ordinary workers, while GCE Spot is plugged in for Playwright and long-running jobs.
OpenTelemetry, rate limits, robots policy, idempotency keys, and cost caps protect reliability and budget.

Decision package

The deployable Terraform foundation and full architecture document are maintained with the application. Production worker activation remains a decision point because it requires image selection, quotas, monthly limits, and the desired crawl-provider mix.