How job listings are extracted

Career pages and ATS boards become rows in ClickHouse. Playwright (and later Vertex Gemini / Bedrock Claude) add duplicate clusters, URL scores, skills beyond the 20-term catalog, and a demand forecast. LinkedIn Easy Apply stays out of scope.

Pipeline

1. Discover listings

Nothing here hits LinkedIn Easy Apply. Sources, in order:

Job-alert mail (Stepstone, eFinancialCareers, Indeed, Gmail JOB_ALERTS) → URI + company + title.
ATS APIs from a known career URL: Greenhouse, Lever, Ashby, SmartRecruiters, Workable, Personio, Recruitee (lib/crawl-jobs.js detectBoard).
JSON-LD JobPosting on the company careers HTML.
Anchor harvest on paths matching /jobs, /career, /stellen (skip privacy, login, imprint).
Apply ledger after a Playwright worker submits or emails — this is what fills apply_jobs.applications_wave.

The crawler User-Agent is GraphaiJobs/1.0 (https://jobs.graphai.eu; hello@graphai.eu). Failures stay on the company profile (“only company information was updated”) rather than inventing jobs.

2. Career-URL scoring

Each listing URI gets a 0–100 score and a class. Direct ATS beats aggregators. Easy Apply is scored to stay out of the apply queue.

Class	Examples	Score
`ats_direct`	Greenhouse, Lever, Ashby, Workable, Personio	85–95
`career_hub`	Company `/jobs` or `/careers` HTML	~78
`ats_login`	Workday, SuccessFactors	65–70
`mailto`	`mailto:career@…`	35
`aggregator`	Indeed, Stepstone, Glassdoor	20
`linkedin`	LinkedIn jobs / Easy Apply	5 (skipped for apply)

3. Skills: the 20-term list, then beyond

Studio skill chips still use the original catalog in lib/skills-catalog.js (python, javascript, typescript, sql, cloud, ml, systems, embedded, networking, security, dataeng, product, ux, project, comms, leadership, stakeholder, problem — plus java and devops in the apply twin). That list is too short for architect / platform roles.

skills_extended matches tokens the 20-term list never had: Playwright, Kafka, ClickHouse, MLflow, Vertex AI, Bedrock, OpenTelemetry, GDPR, AI Act, staff/principal titles, and the rest of the extended lexicon. When APPLY_VERTEX_ENRICH=1 or APPLY_BEDROCK_ENRICH=1, Gemini 2.5 Flash or Claude can add skills_llm (burst only, not 24/7).

4. Duplicate clustering (MLflow + PyTorch)

Default encoder is a hashed bag-of-tokens embedding (64-d, 4096 buckets) in PyTorch — no 400 MB sentence-transformer download on the Mac or Spot VMs. Cosine ≥ 0.88 unions rows into cluster_id. The earliest applied_on member is canonical; others set is_duplicate=1 and duplicate_of.

Each load is an MLflow run under gcp_spot/mlruns (dup rate, mean URL score, mean extended-skill count). Vertex or Bedrock can replace the hash encoder later without changing the ClickHouse schema — only embedding_provider and ai_model change.

5. Demand forecast

Weekly apply counts per canonical title. A one-line PyTorch linear trend (fallback: exponential smoothing) writes demand_obs_7d, demand_obs_30d, demand_forecast_7d, demand_forecast_30d. This is demand in this apply corpus, not a public job-board index.

6. ClickHouse

Company name, slug, host, listing URI, and the AI columns live together in apply_jobs.job_offers_ai (same Cloud service as applications_wave). Studio crawls still write graphai.jobs_listings / graphai.jobs_catalog for the signed-in gallery.

AI column	Meaning
`cluster_id` / `is_duplicate` / `duplicate_of`	Same offer posted on several URLs
`career_url_score` / `career_url_class` / `ats_kind`	How apply-able the URL is
`skills_base`	Hits on the original 20-term catalog
`skills_extended`	Skills outside that catalog
`skills_llm`	Vertex / Bedrock list (optional)
`demand_obs_` / `demand_forecast_`	Observed and forecasted apply volume
`mlflow_run_id`	Trace back to the local experiment

Reload:

cd ~/deepline/data/karlsruhe-public-co-job-apps/gcp_spot
python3 -u job_ai_enrichment.py

Caps

GCP apply fleet €200 / 200 VMs. AWS envelope €100 / month, one Spot apply-aws-1.
Vertex and Bedrock are burst (failing screens, optional skill JSON). AgentCore Browser is not 24/7. AWS Transform is not used.
Proton Bridge SMTP stays on the Mac. No SES.