feat(webapp,run-engine): queue metrics and health dashboard - #4131
feat(webapp,run-engine): queue metrics and health dashboard#4131ericallam wants to merge 58 commits into
Conversation
🦋 Changeset detectedLatest commit: 5ec0997 The changes in this PR will be included in the next version bump. This PR includes changesets to release 27 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR adds queue-metrics ingestion, storage, query, and UI support. It introduces a Redis/ClickHouse metrics pipeline package, ClickHouse queue-metrics tables and query helpers, run-queue emission hooks, gap-filling support in TSQL, and new webapp admin, dashboard, list, and detail routes. It also adds environment and feature-flag gating, seed tooling, and tests across the pipeline and query layers. Related PRs: None found. Suggested labels: enhancement, area: webapp, area: run-engine, area: internal-packages Suggested reviewers: ericallam, matt-aitken 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
a892684 to
9412bf5
Compare
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
6432d9f to
3c67a0c
Compare
1416ee6 to
cbe5444
Compare
…signals Gauges are read inside the enqueue/dequeue Lua and returned on the script reply as a 2-tuple; counters are cumulative odometers. The run-queue Redis carries no metrics stream of its own.
…d live per-key breakdown CK queues now emit two extra gauge fields from the CK-path Lua scripts: the number of concurrency keys with queued runs (ZCARD of the ckIndex) and the head-of-line wait of the most-starved key (now minus the oldest ckIndex score). Both flow through the existing stream into new max-aggregated columns on the 10s and 5m tiers, and non-CK scripts keep the 7-field gauge shape. The queue detail page grows a concurrency-keys section for queues with CK activity: charts for backlogged keys and most-starved wait, plus a live per-key table (queued, running, oldest wait) read from the ckIndex zset, most starved first. Per-key history is intentionally not stored: key values are user-controlled, so the per-key dimension stays in Redis where it is bounded by the live backlog. The queue simulator gains a tenant-hotspot scenario that stages the CK gauge columns and a live ckIndex so the per-key table and charts render with data.
…trics history Queues that shard work with concurrencyKey get a per-key history tier. Counter events for CK runs advance a second per-key odometer and carry both readings on ONE stream entry (cum + ck/ckcum), so per-key attribution adds no stream volume; the consumer expands the entry into a base row and a per-key row. A new 10s AggregatingMergeTree tier keyed by (queue, concurrency_key) holds per-key enqueue/started/ack deltas, backlog/running maxes, and wait sums. Rows are activity-bound: a (queue, key, bucket) row exists only when that key had events, so user-controlled key cardinality cannot inflate the table (benchmarked at ~19 bytes per event with reads under 100ms in the worst shapes). The per-queue tiers stay exact: their counter and wait aggregations now consume only base rows, so per-key odometers never merge under one queue_name and waits are never double counted. A per-queue key limiter (default 10k) acts as a safety valve; on overflow the per-key row is dropped while the base row keeps per-queue counts exact. Per-key odometers use a short TTL, which cumulative counters make loss-free. The queue detail page gains a top-keys-by-backlog chart, a key table merging live state (queued, running, oldest wait) with range stats (started, peak backlog, mean delay), and click-through per-key drill-down charts. The new queue_metrics_by_key table is also queryable directly; its delta columns require grouping or pinning BOTH queue and concurrency_key, enforced by the compile-time merge guard which now supports compound keys.
…il page The queue detail page splits into Overview (the existing concurrency, backlog, delay, and throttle charts) and a Concurrency keys tab that holds all per-key content. The tab only appears for queues with key activity. Inside it, the grouped per-key charts now lead (backlog by key plus a new throughput-by-key chart that makes fair-share visible), followed by the two whole-queue health charts retitled to say what they aggregate (keys with queued runs count, most-starved key wait), then the key table and drill-down.
…ery surfaces TableSchema gains a hidden flag: hidden tables stay fully queryable through the engine (the concurrency-keys tab keeps working, tenancy and the merge guard still apply) but are excluded from the query editor autocomplete, the Query page schema and examples panels, the AI query generator context, and the schema API. queue_metrics_by_key is hidden until the per-key surface is settled.
…n re-enqueue The wait metric measured dequeue time minus the original trigger time even when a run re-entered the queue after a waitpoint, checkpoint, or pending version, so the whole wait or checkpoint duration showed up as scheduling delay. Re-enqueues now anchor to the re-enqueue time while first enqueues keep the trigger or delay anchor; queue ordering is unchanged, so re-enqueued runs keep their original position. Nacked runs never left the queue stint and keep the original anchor. Also replaces the :ck: suffix regexes on user-controlled queue names with indexOf slicing (identical semantics) to remove a polynomial regex flagged by code scanning.
Behind the per-org queue metrics UI flag: the task detail page links its queue to the queue detail view and shows live queued/running counts, delay p95, peak backlog, and a queue backlog chart; the run inspector links the queue and concurrency key, and queued, delayed, and pending-version runs get a "Waiting in queue" block with an at-limit callout and per-key counts. Adds optional concurrency-key params to the run engine queue reads.
…ask detail queue chart The run inspector's "Waiting in queue" section now leads with two borderless stat + last-30-minutes sparklines (backlog and scheduling delay), matching the inspector's divider-and-rows layout instead of nested cards. On the task detail page the queue backlog moves behind a tab in the activity card rather than sitting beside the runs-by-status chart, so it does not crowd the agent-task hero charts.
6c0d3da to
ddae31f
Compare
…ersion collision Main added 035_fix_error_display_derivation.sql, which collides with the queue-metrics migration that also landed as 035. goose versions by numeric prefix, so two 035 files break the migrator. Renumber the queue-metrics migration to 036 (it sorts after the error-display fix on a fresh database).
## Queue pages design overhaul (frontend for the queue-metrics feature) Builds the dashboards on top of the queue-metrics pipeline (base branch). ### Queues list <img width="1440" height="788" alt="Screenshot 2026-07-20 at 01 02 55" src="https://github.com/user-attachments/assets/0b97dd6c-d05b-4dba-99c9-04266ec22cb3" /> ### Queue detail <img width="1440" height="1272" alt="tab-overview" src="https://github.com/user-attachments/assets/22b33f82-522d-4410-a477-fe5d2c2f7adf" /> <img width="1440" height="1532" alt="tab-keys" src="https://github.com/user-attachments/assets/b05619fa-93a8-43ef-8838-a6d7fc5d789b" /> ### Under the hood - Percent-based concurrency overrides (schema + migration + recalc on env limit change, capped at the env limit) — covered by unit tests - All recurring refresh hits ClickHouse only (15s blocks / 60s charts, paused in hidden tabs); Redis/PG serve first paint - Added `MetricsLayout` that is currently used for `/queues + $id`, `/agents + $id`, `/settings/usage` <img width="500" height="auto" alt="Screenshot 2026-07-20 at 01 07 31" src="https://github.com/user-attachments/assets/e8a798e7-deee-455d-be96-414f7d09a06c" />
…d-health # Conflicts: # apps/webapp/app/routes/_app.orgs.$organizationSlug.settings.usage/route.tsx
The run inspector now shows a mini queue view while a run is waiting: Status (with a paused / at-limit warning), Concurrency and how long the run has waited, a p50/p95 scheduling-delay chart, and a link to the queue.
The four header chart tiles rendered even in the not-success states (engine-version upgrade, no tasks) and when the list filtered to empty, showing four empty 'No activity' cards above the blank state. Gate them on the same queue set the table shows.
**Stacked on #4131** — targets `feat/queue-metrics-and-health`, not `main`. GitHub will auto-retarget this to `main` once #4131 merges. (Targeting `main` now would wrongly include all of #4131's own commits.) UI polish for the Queue metrics & health pages (TRI-12068). 49 commits, all under `apps/webapp/app/` — no engine/ClickHouse/package changes. **Highlights** - **Layout** — consolidated queue-detail header into one filter bar; shared `MetricsLayout` spacing (2.5 scale); paused-queue banner mirroring the environment-paused banner. - **Charts** — inline legends under titles; threshold gradient coloring (Env saturation orange only >100%; Concurrency orange only at/over the limit); fullscreen legend spacing; result caching so tab switches don't reload; brighter hover cursor + full x-axis labels. - **Tables** — whole-row click via stretched link; custom Queues icon; removed column sorting; Limit-cell spacing + hover-bright; "Limited by" tooltip sizing. - **Controls** — Pause moved into the filter bar (orange label + accurate tooltip copy); restyled override-concurrency modal. - **Copy** — plain-language pass; inline colour swatches; Throttled reads "% of current period". Also consolidated the three queue-metrics `.server-changes` notes into one. **Known follow-up (not in this PR):** the Concurrency-keys table is capped at 50 keys (union of live Redis backlog + ClickHouse history, no single cursor across both) and truncates silently — fine for most queues, a real limit for high-cardinality per-tenant keys. Verified locally against the `queue-metrics-demo` seed; `pnpm typecheck --filter webapp` passes.
…d-health # Conflicts: # apps/webapp/app/v3/services/worker/workerGroupTokenService.server.ts
…#4327) ## Overview Adds the first **Report** — `health`. The server computes a deterministic verdict and renders it as text + unicode sparklines. `health` separates **flow** (is work starting?), **execution** (are the runs that start succeeding?), and **liveness** (is the telemetry fresh?), each with a headline verdict and a suggested next action — no LLM in the verdict, so there's nothing to hallucinate. Surfaces: - **`get_report` MCP tool** — markdown with sparklines + 🟢/🟡/🔴 status markers (or ANSI via `color`). - **`report` MCP prompt** — a slash command in hosts that support prompts. - **`trigger report [key]` CLI** — colourised in a real terminal, plain markdown when piped. - **`GET /api/v1/reports/:key`** — `format=markdown|ansi|json`. Flow is diagnosed by a cause tree (env-limit saturation, queue throttling, key starvation, trigger spike, dequeue stall) off measured queue metrics, with a runs-snapshot fallback. One `ReportViewModel`, rendered to markdown / ANSI / JSON. Also: `trigger mcp` now always starts the server — the interactive install wizard is gated behind `trigger mcp --install`. Previously a TTY dropped into the wizard, so MCP hosts that spawn the server over a PTY never got a server and timed out. --- ## Preview **Degraded — env concurrency-limit saturation:** ``` /report health prod · last 1h · vs your 7d normal 🟡 Flow slowing · 🟢 Execution healthy · 🟢 data fresh FLOW 🟡 at your env concurrency limit (last 40 min) concurrency 100/100 ▁▅▆█████ pinned 40 of last 60 min pending 1,910 ↑ 16× ▁▁▂▃▅▅▇█ (normal ~120) start latency p95 42s ↑ 6× ▁▁▂▄▆▆▇█ (normal ~7s) worst queue email-sends — 82% of pending read: limit saturated → starts lag → backlog grows not workers, not platform — dequeue keeps pace at ~820/min nothing dead-lettered EXECUTION 🟢 the runs that DO start are fine failures 1.3% (normal ~1.1%) · durations normal read: NOT a code problem LIVENESS 🟢 fresh — last completion 4s ago → Raise the env concurrency limit or do nothing — backlog drains in ~2.3 min once triggers ease ``` **Healthy:** ``` /report health prod · last 1h · vs your 7d normal 🟢 Flow healthy · 🟢 Execution healthy · 🟢 data fresh FLOW 🟢 starting normally — pending 84 (normal ~120) · starts p95 6s EXECUTION 🟢 completing normally — failures 0.9% (normal ~1.1%) · durations normal LIVENESS 🟢 fresh — last completion 2s ago → nothing to do ```
…d-health # Conflicts: # apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.tasks.standard.$taskParam/route.tsx
…ment override cap Address review findings on the queue metrics PR: - useMetricResourceQuery kept the previous query's rows when the signature changed with no cache entry for the new key, so showLoading (isLoading && !rows) stayed false and a chart painted the old time range as if it were the new one. A stale `failed` also survived, rendering an "invalid" chart state over a healthy new query. Reset both when the signature actually changes; interval and on-focus refreshes reuse the signature and are unaffected. - Document in the OpenAPI spec that the queue concurrency override rejects limits above the environment maximum with a 400 instead of capping them. - Correct the enqueueSystem TTL comment: the pending-version promotion does pass includeTtl, so it is not in the "must not add TTL" set.
Follow-up:
|
| const running = fresh ? toNumber(fresh.running) : waiting.running; | ||
| const limit = waiting.concurrencyLimit ?? (fresh ? toNumber(fresh.q_limit) || null : null); | ||
|
|
||
| // The concurrency limit applies per key on keyed queues. PENDING renders as "Queued". | ||
| const atLimit = limit !== null && (key ? key.running >= limit : running >= limit); | ||
| const showAtLimit = status === "PENDING" && atLimit && !paused; | ||
| const pct = limit && limit > 0 ? Math.min(100, Math.round((running / limit) * 100)) : null; |
There was a problem hiding this comment.
🟡 Concurrency figure for a waiting run on a per-key queue mixes whole-queue and per-key numbers
In the run inspector's waiting-queue block, the concurrency tile shows the whole queue's running count against a per-key limit (running at apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.runs.$runParam.spans.$spanParam/route.tsx:1281), while the at-limit check uses the key's own running count, so the displayed ratio and percentage don't line up with reality.
Impact: For a run on a concurrency-keyed queue, the shown number and percentage can look maxed out (or wildly over 100% before clamping) even when that run's key is nowhere near its limit, misleading the reader about why the run is waiting.
Whole-queue running vs per-key limit mismatch
The concurrency limit on a keyed queue applies per concurrency key. waiting.concurrencyLimit is that per-key limit. The code computes atLimit correctly using key.running >= limit for keyed runs, but then the Concurrency MiniStat renders value={running.toLocaleString()} where running is the whole-queue running count (from max_running / currentConcurrencyOfQueues[queueName]), and pct = Math.min(100, Math.round((running / limit) * 100)) also uses whole-queue running against the per-key limit. So a keyed queue with, say, 8 running queue-wide and a per-key limit of 2 renders "8 / 2 · 100%" even when the run's own key has only 1 running of 2. The asymmetry (per-key for atLimit, whole-queue for the displayed value/percent) indicates the display should use the key's running count for keyed runs.
Prompt for agents
In WaitingInQueueBlock (resources...spans.$spanParam/route.tsx), the Concurrency MiniStat displays the whole-queue running count (`running`) with a per-key `limit` and computes `pct` from whole-queue running, but `atLimit` uses the per-key `key.running`. For concurrency-keyed runs the limit is per key, so the displayed value and percentage should be based on the key's running count too. Consider deriving a display running value that uses `key.running` when `key` is present (mirroring the `atLimit` logic) so the number, the `/ limit` suffix, and the percentage are all consistent for keyed queues. Verify the non-keyed path is unchanged.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Adds per-queue observability to the Queues page: depth (backlog), throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and the scheduling delay (how long runs wait between becoming eligible and actually starting). Each queue shows health at a glance in the list, plus a per-queue detail page with charts, so you can answer "does this queue have enough concurrency to keep up?".
Both the data collection and the dashboard are off by default and gated independently: metric emission is a global switch, and the dashboard is turned on per organization. With both off, the Queues page is unchanged.
Design
Queue operations emit two kinds of signal. Gauges (depth, running, limit, throttled) are read inside the same Redis script that performs the enqueue or dequeue, so the reading is atomic, and returned on the script's reply for the app to forward. Counters (enqueued, started, completed) are cumulative odometers, so a dropped reading self-heals: the next one restates the running total. Both land on one Redis stream on a dedicated metrics instance (falling back to the run queue's Redis when self-hosting), drain through a consumer into ClickHouse (raw, a 10-second-bucket materialized view, and a 30-day aggregate), and the dashboards read the aggregate. The run queue's own Redis carries no metrics stream.
The one change that is live the moment this deploys, independent of both flags, is the enqueue/dequeue script reply shape: those scripts now return a 2-tuple so the gauge reading can ride back to the app. That path is exercised on every queue op, so it is the part of
run-engineworth the closest review.