Skip to content

feat(webapp,run-engine): queue metrics and health dashboard - #4131

Open
ericallam wants to merge 58 commits into
mainfrom
feat/queue-metrics-and-health
Open

feat(webapp,run-engine): queue metrics and health dashboard#4131
ericallam wants to merge 58 commits into
mainfrom
feat/queue-metrics-and-health

Conversation

@ericallam

Copy link
Copy Markdown
Member

Summary

Adds per-queue observability to the Queues page: depth (backlog), throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and the scheduling delay (how long runs wait between becoming eligible and actually starting). Each queue shows health at a glance in the list, plus a per-queue detail page with charts, so you can answer "does this queue have enough concurrency to keep up?".

Both the data collection and the dashboard are off by default and gated independently: metric emission is a global switch, and the dashboard is turned on per organization. With both off, the Queues page is unchanged.

Design

Queue operations emit two kinds of signal. Gauges (depth, running, limit, throttled) are read inside the same Redis script that performs the enqueue or dequeue, so the reading is atomic, and returned on the script's reply for the app to forward. Counters (enqueued, started, completed) are cumulative odometers, so a dropped reading self-heals: the next one restates the running total. Both land on one Redis stream on a dedicated metrics instance (falling back to the run queue's Redis when self-hosting), drain through a consumer into ClickHouse (raw, a 10-second-bucket materialized view, and a 30-day aggregate), and the dashboards read the aggregate. The run queue's own Redis carries no metrics stream.

The one change that is live the moment this deploys, independent of both flags, is the enqueue/dequeue script reply shape: those scripts now return a 2-tuple so the gauge reading can ride back to the app. That path is exercised on every queue op, so it is the part of run-engine worth the closest review.

@changeset-bot

changeset-bot Bot commented Jul 3, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 5ec0997

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 27 packages
Name Type
@trigger.dev/core Patch
trigger.dev Patch
@trigger.dev/build Patch
@trigger.dev/python Patch
@trigger.dev/redis-worker Patch
@trigger.dev/schema-to-json Patch
@trigger.dev/sdk Patch
@internal/cache Patch
@internal/clickhouse Patch
@internal/llm-model-catalog Patch
@internal/metrics-pipeline Patch
@trigger.dev/rbac Patch
@internal/redis Patch
@internal/replication Patch
@internal/run-engine Patch
@internal/run-store Patch
@internal/schedule-engine Patch
@trigger.dev/sso Patch
@internal/testcontainers Patch
@internal/tracing Patch
@internal/tsql Patch
@internal/dashboard-agent Patch
@internal/sdk-compat-tests Patch
@trigger.dev/react-hooks Patch
@trigger.dev/rsc Patch
@trigger.dev/database Patch
@trigger.dev/otlp-importer Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

This PR adds queue-metrics ingestion, storage, query, and UI support. It introduces a Redis/ClickHouse metrics pipeline package, ClickHouse queue-metrics tables and query helpers, run-queue emission hooks, gap-filling support in TSQL, and new webapp admin, dashboard, list, and detail routes. It also adds environment and feature-flag gating, seed tooling, and tests across the pipeline and query layers.

Related PRs: None found.

Suggested labels: enhancement, area: webapp, area: run-engine, area: internal-packages

Suggested reviewers: ericallam, matt-aitken

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description is missing required template sections, including Closes #issue, checklist, Testing, Changelog, and Screenshots. Add the template sections and fill them out, starting with Closes #, the checklist, testing steps, changelog, and screenshots.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and accurately summarizes the main change: queue metrics and health dashboard for webapp/run-engine.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/queue-metrics-and-health

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

github-advanced-security[bot]

This comment was marked as resolved.

github-advanced-security[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

@ericallam
ericallam marked this pull request as ready for review July 3, 2026 10:26
devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

@ericallam
ericallam force-pushed the feat/queue-metrics-and-health branch from a892684 to 9412bf5 Compare July 4, 2026 08:30
@pkg-pr-new

pkg-pr-new Bot commented Jul 4, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@5ec0997

trigger.dev

npm i https://pkg.pr.new/trigger.dev@5ec0997

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@5ec0997

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@5ec0997

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@5ec0997

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@5ec0997

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@5ec0997

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@5ec0997

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@5ec0997

commit: 5ec0997

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ericallam
ericallam force-pushed the feat/queue-metrics-and-health branch from 6432d9f to 3c67a0c Compare July 4, 2026 22:16
devin-ai-integration[bot]

This comment was marked as resolved.

@ericallam ericallam closed this Jul 5, 2026
@ericallam ericallam reopened this Jul 5, 2026
@ericallam
ericallam force-pushed the feat/queue-metrics-and-health branch from 1416ee6 to cbe5444 Compare July 5, 2026 12:44
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

View 1 additional finding in Devin Review.

Open in Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

Open in Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

…d live per-key breakdown

CK queues now emit two extra gauge fields from the CK-path Lua scripts:
the number of concurrency keys with queued runs (ZCARD of the ckIndex)
and the head-of-line wait of the most-starved key (now minus the oldest
ckIndex score). Both flow through the existing stream into new
max-aggregated columns on the 10s and 5m tiers, and non-CK scripts keep
the 7-field gauge shape.

The queue detail page grows a concurrency-keys section for queues with
CK activity: charts for backlogged keys and most-starved wait, plus a
live per-key table (queued, running, oldest wait) read from the ckIndex
zset, most starved first. Per-key history is intentionally not stored:
key values are user-controlled, so the per-key dimension stays in Redis
where it is bounded by the live backlog.

The queue simulator gains a tenant-hotspot scenario that stages the CK
gauge columns and a live ckIndex so the per-key table and charts render
with data.
…trics history

Queues that shard work with concurrencyKey get a per-key history tier.
Counter events for CK runs advance a second per-key odometer and carry
both readings on ONE stream entry (cum + ck/ckcum), so per-key
attribution adds no stream volume; the consumer expands the entry into a
base row and a per-key row. A new 10s AggregatingMergeTree tier keyed by
(queue, concurrency_key) holds per-key enqueue/started/ack deltas,
backlog/running maxes, and wait sums. Rows are activity-bound: a
(queue, key, bucket) row exists only when that key had events, so
user-controlled key cardinality cannot inflate the table (benchmarked at
~19 bytes per event with reads under 100ms in the worst shapes).

The per-queue tiers stay exact: their counter and wait aggregations now
consume only base rows, so per-key odometers never merge under one
queue_name and waits are never double counted. A per-queue key limiter
(default 10k) acts as a safety valve; on overflow the per-key row is
dropped while the base row keeps per-queue counts exact. Per-key
odometers use a short TTL, which cumulative counters make loss-free.

The queue detail page gains a top-keys-by-backlog chart, a key table
merging live state (queued, running, oldest wait) with range stats
(started, peak backlog, mean delay), and click-through per-key
drill-down charts. The new queue_metrics_by_key table is also queryable
directly; its delta columns require grouping or pinning BOTH queue and
concurrency_key, enforced by the compile-time merge guard which now
supports compound keys.
…il page

The queue detail page splits into Overview (the existing concurrency,
backlog, delay, and throttle charts) and a Concurrency keys tab that
holds all per-key content. The tab only appears for queues with key
activity. Inside it, the grouped per-key charts now lead (backlog by
key plus a new throughput-by-key chart that makes fair-share visible),
followed by the two whole-queue health charts retitled to say what they
aggregate (keys with queued runs count, most-starved key wait), then
the key table and drill-down.
…ery surfaces

TableSchema gains a hidden flag: hidden tables stay fully queryable
through the engine (the concurrency-keys tab keeps working, tenancy and
the merge guard still apply) but are excluded from the query editor
autocomplete, the Query page schema and examples panels, the AI query
generator context, and the schema API. queue_metrics_by_key is hidden
until the per-key surface is settled.
…n re-enqueue

The wait metric measured dequeue time minus the original trigger time
even when a run re-entered the queue after a waitpoint, checkpoint, or
pending version, so the whole wait or checkpoint duration showed up as
scheduling delay. Re-enqueues now anchor to the re-enqueue time while
first enqueues keep the trigger or delay anchor; queue ordering is
unchanged, so re-enqueued runs keep their original position. Nacked
runs never left the queue stint and keep the original anchor.

Also replaces the :ck: suffix regexes on user-controlled queue names
with indexOf slicing (identical semantics) to remove a polynomial
regex flagged by code scanning.
Behind the per-org queue metrics UI flag: the task detail page links its
queue to the queue detail view and shows live queued/running counts, delay
p95, peak backlog, and a queue backlog chart; the run inspector links the
queue and concurrency key, and queued, delayed, and pending-version runs
get a "Waiting in queue" block with an at-limit callout and per-key counts.
Adds optional concurrency-key params to the run engine queue reads.
…ask detail queue chart

The run inspector's "Waiting in queue" section now leads with two borderless
stat + last-30-minutes sparklines (backlog and scheduling delay), matching the
inspector's divider-and-rows layout instead of nested cards. On the task detail
page the queue backlog moves behind a tab in the activity card rather than
sitting beside the runs-by-status chart, so it does not crowd the agent-task
hero charts.
@ericallam
ericallam force-pushed the feat/queue-metrics-and-health branch from 6c0d3da to ddae31f Compare July 13, 2026 19:44
…ersion collision

Main added 035_fix_error_display_derivation.sql, which collides with the
queue-metrics migration that also landed as 035. goose versions by numeric
prefix, so two 035 files break the migrator. Renumber the queue-metrics
migration to 036 (it sorts after the error-display fix on a fresh database).

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

Open in Devin Review

kathiekiwi and others added 2 commits July 21, 2026 19:29
## Queue pages design overhaul (frontend for the queue-metrics feature)

Builds the dashboards on top of the queue-metrics pipeline (base
branch).

### Queues list

<img width="1440" height="788" alt="Screenshot 2026-07-20 at 01 02 55"
src="https://github.com/user-attachments/assets/0b97dd6c-d05b-4dba-99c9-04266ec22cb3"
/>

### Queue detail

<img width="1440" height="1272" alt="tab-overview"
src="https://github.com/user-attachments/assets/22b33f82-522d-4410-a477-fe5d2c2f7adf"
/>

<img width="1440" height="1532" alt="tab-keys"
src="https://github.com/user-attachments/assets/b05619fa-93a8-43ef-8838-a6d7fc5d789b"
/>


### Under the hood
- Percent-based concurrency overrides (schema + migration + recalc on
env limit change, capped at the env limit) — covered by unit tests
- All recurring refresh hits ClickHouse only (15s blocks / 60s charts,
paused in hidden tabs); Redis/PG serve first paint
- Added `MetricsLayout` that is currently used for `/queues + $id`,
`/agents + $id`, `/settings/usage`

<img width="500" height="auto" alt="Screenshot 2026-07-20 at 01 07 31"
src="https://github.com/user-attachments/assets/e8a798e7-deee-455d-be96-414f7d09a06c"
/>
…d-health

# Conflicts:
#	apps/webapp/app/routes/_app.orgs.$organizationSlug.settings.usage/route.tsx

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

Open in Devin Review

The run inspector now shows a mini queue view while a run is waiting:
Status (with a paused / at-limit warning), Concurrency and how long the
run has waited, a p50/p95 scheduling-delay chart, and a link to the queue.
devin-ai-integration[bot]

This comment was marked as resolved.

The four header chart tiles rendered even in the not-success states
(engine-version upgrade, no tasks) and when the list filtered to empty,
showing four empty 'No activity' cards above the blank state. Gate them
on the same queue set the table shows.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

Open in Devin Review

**Stacked on #4131** — targets `feat/queue-metrics-and-health`, not
`main`. GitHub will auto-retarget this to `main` once #4131 merges.
(Targeting `main` now would wrongly include all of #4131's own commits.)

UI polish for the Queue metrics & health pages (TRI-12068). 49 commits,
all under `apps/webapp/app/` — no engine/ClickHouse/package changes.

**Highlights**
- **Layout** — consolidated queue-detail header into one filter bar;
shared `MetricsLayout` spacing (2.5 scale); paused-queue banner
mirroring the environment-paused banner.
- **Charts** — inline legends under titles; threshold gradient coloring
(Env saturation orange only >100%; Concurrency orange only at/over the
limit); fullscreen legend spacing; result caching so tab switches don't
reload; brighter hover cursor + full x-axis labels.
- **Tables** — whole-row click via stretched link; custom Queues icon;
removed column sorting; Limit-cell spacing + hover-bright; "Limited by"
tooltip sizing.
- **Controls** — Pause moved into the filter bar (orange label +
accurate tooltip copy); restyled override-concurrency modal.
- **Copy** — plain-language pass; inline colour swatches; Throttled
reads "% of current period".

Also consolidated the three queue-metrics `.server-changes` notes into
one.

**Known follow-up (not in this PR):** the Concurrency-keys table is
capped at 50 keys (union of live Redis backlog + ClickHouse history, no
single cursor across both) and truncates silently — fine for most
queues, a real limit for high-cardinality per-tenant keys.

Verified locally against the `queue-metrics-demo` seed; `pnpm typecheck
--filter webapp` passes.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

View 1 additional finding in Devin Review.

Open in Devin Review

…d-health

# Conflicts:
#	apps/webapp/app/v3/services/worker/workerGroupTokenService.server.ts
devin-ai-integration[bot]

This comment was marked as resolved.

…#4327)

## Overview

Adds the first **Report** — `health`. The server computes a
deterministic verdict and renders it as text + unicode sparklines.
`health` separates **flow** (is work starting?), **execution** (are the
runs that start succeeding?), and **liveness** (is the telemetry
fresh?), each with a headline verdict and a suggested next action — no
LLM in the verdict, so there's nothing to hallucinate.

Surfaces:
- **`get_report` MCP tool** — markdown with sparklines + 🟢/🟡/🔴 status
markers (or ANSI via `color`).
- **`report` MCP prompt** — a slash command in hosts that support
prompts.
- **`trigger report [key]` CLI** — colourised in a real terminal, plain
markdown when piped.
- **`GET /api/v1/reports/:key`** — `format=markdown|ansi|json`.

Flow is diagnosed by a cause tree (env-limit saturation, queue
throttling, key starvation, trigger spike, dequeue stall) off measured
queue metrics, with a runs-snapshot fallback. One `ReportViewModel`,
rendered to markdown / ANSI / JSON.

Also: `trigger mcp` now always starts the server — the interactive
install wizard is gated behind `trigger mcp --install`. Previously a TTY
dropped into the wizard, so MCP hosts that spawn the server over a PTY
never got a server and timed out.

---

## Preview

**Degraded — env concurrency-limit saturation:**
```
/report health        prod · last 1h · vs your 7d normal

🟡 Flow slowing  ·  🟢 Execution healthy  ·  🟢 data fresh

FLOW        🟡 at your env concurrency limit (last 40 min)
  concurrency     100/100           ▁▅▆█████   pinned 40 of last 60 min

  pending         1,910     ↑ 16×   ▁▁▂▃▅▅▇█   (normal ~120)

  start latency   p95 42s   ↑ 6×    ▁▁▂▄▆▆▇█   (normal ~7s)

  worst queue     email-sends — 82% of pending

  read: limit saturated → starts lag → backlog grows
        not workers, not platform — dequeue keeps pace at ~820/min
        nothing dead-lettered

EXECUTION   🟢 the runs that DO start are fine
  failures 1.3% (normal ~1.1%) · durations normal
  read: NOT a code problem

LIVENESS    🟢 fresh — last completion 4s ago

→ Raise the env concurrency limit
  or do nothing — backlog drains in ~2.3 min once triggers ease
```

**Healthy:**
```
/report health        prod · last 1h · vs your 7d normal

🟢 Flow healthy  ·  🟢 Execution healthy  ·  🟢 data fresh

FLOW        🟢 starting normally — pending 84 (normal ~120) · starts p95 6s

EXECUTION   🟢 completing normally — failures 0.9% (normal ~1.1%) · durations normal

LIVENESS    🟢 fresh — last completion 2s ago

→ nothing to do
```
devin-ai-integration[bot]

This comment was marked as resolved.

samejr added 2 commits July 28, 2026 14:35
…d-health

# Conflicts:
#	apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.tasks.standard.$taskParam/route.tsx
…ment override cap

Address review findings on the queue metrics PR:

- useMetricResourceQuery kept the previous query's rows when the signature
  changed with no cache entry for the new key, so showLoading (isLoading &&
  !rows) stayed false and a chart painted the old time range as if it were the
  new one. A stale `failed` also survived, rendering an "invalid" chart state
  over a healthy new query. Reset both when the signature actually changes;
  interval and on-focus refreshes reuse the signature and are unaffected.
- Document in the OpenAPI spec that the queue concurrency override rejects
  limits above the environment maximum with a 400 instead of capping them.
- Correct the enqueueSystem TTL comment: the pending-version promotion does
  pass includeTtl, so it is not in the "must not add TTL" set.
@samejr

samejr commented Jul 28, 2026

Copy link
Copy Markdown
Member

Follow-up: eligibleAtMs anchoring for PENDING_VERSION promotions

Came out of Devin's eligibleAtMs thread (now resolved). Flagging rather than fixing, because it's a call on what the metric should mean.

eligibleAtMs = includeTtl ? queuePositionMs : Date.now() uses includeTtl as a proxy for "first enqueue, so anchor at the queue position". That holds for the trigger path (engine/index.ts:1171) and the delayed-run system (delayedRunSystem.ts:166). But pendingVersionSystem.ts:158 also passes includeTtl: true — deliberately, and arming the TTL there is right ("the first time this run is actually entering the run queue").

The knock-on is that a PENDING_VERSION run now anchors eligibleAtMs at its original queueTimestamp. promotePendingVersionRuns only flips the status, it doesn't touch queueTimestamp, so wait = Date.now() - eligibleAtMs (run-queue/index.ts:1019) ends up including the entire period the run sat waiting for a worker version to deploy — minutes to hours. That's the same distortion the comment above says the Date.now() branch exists to prevent for waitpoint/checkpoint re-enqueues, and it feeds the scheduling-delay charts this PR adds.

Both comments read as though the run enters the queue now on this path (the TTL is armed from now), so anchoring the wait metric at now looks like the consistent choice. Since the two concerns are conflated behind one flag it isn't a one-liner — probably a separate opt-in, e.g. an anchorEligibilityAtQueuePosition set by the trigger and delayed-run call sites and defaulted off, which leaves pending-version at now().

ttl.test.ts covers first-enqueue and re-enqueue anchoring but not this path, so nothing pins the current behaviour either way. Happy to implement + test it if you agree it should anchor at now.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

Comment on lines +1281 to +1287
const running = fresh ? toNumber(fresh.running) : waiting.running;
const limit = waiting.concurrencyLimit ?? (fresh ? toNumber(fresh.q_limit) || null : null);

// The concurrency limit applies per key on keyed queues. PENDING renders as "Queued".
const atLimit = limit !== null && (key ? key.running >= limit : running >= limit);
const showAtLimit = status === "PENDING" && atLimit && !paused;
const pct = limit && limit > 0 ? Math.min(100, Math.round((running / limit) * 100)) : null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Concurrency figure for a waiting run on a per-key queue mixes whole-queue and per-key numbers

In the run inspector's waiting-queue block, the concurrency tile shows the whole queue's running count against a per-key limit (running at apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.runs.$runParam.spans.$spanParam/route.tsx:1281), while the at-limit check uses the key's own running count, so the displayed ratio and percentage don't line up with reality.
Impact: For a run on a concurrency-keyed queue, the shown number and percentage can look maxed out (or wildly over 100% before clamping) even when that run's key is nowhere near its limit, misleading the reader about why the run is waiting.

Whole-queue running vs per-key limit mismatch

The concurrency limit on a keyed queue applies per concurrency key. waiting.concurrencyLimit is that per-key limit. The code computes atLimit correctly using key.running >= limit for keyed runs, but then the Concurrency MiniStat renders value={running.toLocaleString()} where running is the whole-queue running count (from max_running / currentConcurrencyOfQueues[queueName]), and pct = Math.min(100, Math.round((running / limit) * 100)) also uses whole-queue running against the per-key limit. So a keyed queue with, say, 8 running queue-wide and a per-key limit of 2 renders "8 / 2 · 100%" even when the run's own key has only 1 running of 2. The asymmetry (per-key for atLimit, whole-queue for the displayed value/percent) indicates the display should use the key's running count for keyed runs.

Prompt for agents
In WaitingInQueueBlock (resources...spans.$spanParam/route.tsx), the Concurrency MiniStat displays the whole-queue running count (`running`) with a per-key `limit` and computes `pct` from whole-queue running, but `atLimit` uses the per-key `key.running`. For concurrency-keyed runs the limit is per key, so the displayed value and percentage should be based on the key's running count too. Consider deriving a display running value that uses `key.running` when `key` is present (mirroring the `atLimit` logic) so the number, the `/ limit` suffix, and the percentage are all consistent for keyed queues. Verify the non-keyed path is unchanged.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants