185: Cost & token accounting for every run: per-role, per-model ledger surfaced on completion

[agent-relay-code]

Motivation

We can turn issues into reviewed PRs, but we cannot answer "what did this PR cost?" Cursor's agent-swarm model-economics work makes dollar-per-task the optimization target: the same job ran $10,565 (frontier model in every role) vs $1,339 (smart planner + cheap workers), and workers burn 69–90% of tokens while being a minority of cost. NightCTO already has a metered cost ladder (packages/openrouter-client), but the Factory — the actual code-writing swarm — flies blind. This gives Factory a first-class cost/usage ledger so every run reports its own spend, per role and per model. It is the foundation for budget enforcement and cost-aware model routing (filed as follow-ups, blocked on this).

Goal

Record token + USD usage for every dispatched run, attributed by role (implementer / reviewer / babysitter / triage) and model, and surface a bounded per-run cost summary on run completion — riding the existing observability event pipeline, not a new side channel.

What to build

• A CostLedger accounting seam (src/cost/ledger.ts) that accumulates { runId, role, model, inputTokens, outputTokens, usd } records and exposes getRunTotal(runId) + getRunByRole(runId).
• A pricing table (src/cost/pricing.ts) mirroring the shape of nightcto's packages/openrouter-client/src/models.ts (costPer1MInput / costPer1MOutput keyed by model id) with estimateCostUsd(model, inTok, outTok). An unknown model yields usd: null and a bounded cost.model.unpriced event — never a throw.
• A usage-reporting seam on the fleet/spawn path: when the fleet/agent runtime reports token usage for a spawned agent, record it against the run + role; where the runtime reports nothing, record tokens: null (do not fabricate).
• Emit a bounded run.cost.v1 event on run completion via src/observability/events.ts (respect the closed attribute allowlist — model id, role, token counts, usd only; never task text / paths / prompts). Attach the aggregate to the run record so cloud-reporter.ts and callers can read per-run + per-role spend.

Constraints

• Preserve the bounded-allowlist and non-fatal posture of the observability layer (src/observability/cloud-reporter.ts:63-67) — cost accounting must never alter control flow or become a content side channel.
• Deterministic: no wall-clock / random in ledger identity; keep the per-run traceId correlation from events.ts.

Acceptance

• CostLedger records per-role, per-model token + USD; getRunTotal / getRunByRole return the breakdown.
• src/cost/pricing.ts prices known models; unknown → null + bounded event, no throw.
• A single run.cost.v1 bounded event is emitted on completion and the aggregate is on the run record.
• Manifest updated: add the new feature(s) to .agentworkforce/features/manifest.yaml (id, name, cli/api, description, location → the real new files, verify_tier) and bump catalog.feature_count + the matching tier_counts so featuremap:check stays green.
• Public surface exported from src/index.ts.

✅ End-to-end verification — Factory success criteria (REQUIRED)

"Compiles" is not done. Drive a full run through the in-process fake fleet, feed synthetic per-agent usage, and observe a correct per-role / per-model cost summary emitted — then check the harness in.

Run it

npm run build
npm run featuremap:check
npm test
npm run verify:e2e

The E2E (test/e2e/run-cost-accounting.test.ts, runnable via npm test / verify:e2e) must:

1. Run a dispatch → spawn → writeback lifecycle on the in-process fake fleet (src/fleet/internal-fleet-client.ts) with two roles on two different models, each reporting synthetic token usage.
2. Assert the CostLedger has per-role, per-model records and getRunTotal equals the sum of the priced parts.
3. Assert exactly one bounded run.cost.v1 event is emitted and it carries only allowlisted attributes — a raw task string / path must not be reachable (assert the negative).
4. Assert an unpriced model yields usd: null + the bounded cost.model.unpriced event and does not throw.

Deliverable

• test/e2e/run-cost-accounting.test.ts proving per-run / per-role cost is captured and emitted, one command.
• Fails loudly if usage isn't attributed, if the event carries non-allowlisted content, or if an unpriced model throws (no false green).

Anchor files

• src/orchestrator/factory.ts (dispatch / spawn / writeback / reconcile seams) → new src/cost/{ledger,pricing}.ts
• src/observability/events.ts (bounded event contract; add run.cost.v1), src/observability/cloud-reporter.ts:63-67 (non-fatal posture)
• src/ports/fleet.ts / src/fleet/internal-fleet-client.ts (usage-reporting seam; fake fleet for the E2E)
• src/config/schema.ts (any config toggle), src/index.ts (public barrel)
• .agentworkforce/features/manifest.yaml + .agentworkforce/features/verify/procedures.md
• Pricing-table precedent: nightcto packages/openrouter-client/src/models.ts

Out of scope

• Budget enforcement and cost-aware model routing (separate follow-up issues, blocked on this one).
• Capturing usage from CLIs that report none — record null, don't estimate token counts you don't have.

Related

• Observability follow-up Export OpenTelemetry spans over OTLP (observability v1 follow-up) #120 (OTLP spans around dispatch/spawn/writeback) — same seams.
• Motivation: Cursor "agent swarm model economics".

Fixes #185

---

Summary by cubic

Adds per-run cost and token accounting with a per-role, per-model ledger and emits a privacy-bounded run.cost.v1 summary at completion. Addresses Linear #185 by surfacing USD and token totals on the durable run, keeping unknown model pricing non-fatal, and bounding the reported breakdown.

• New Features
  • CostLedger records { runId, role, model, inputTokens, outputTokens, usd }, supports deterministic upserts (entryId), and exposes getRunTotal()/getRunByRole(); boundedRunCostTotal() enforces a max of 32 models per role and folds overflow into factory/other-models.
  • estimateCostUsd() with MODEL_PRICING; unknown models return usd: null and emit cost.model.unpriced (no throw).
  • Fleet usage seam: AgentUsage and onAgentUsage() on FleetClient; internal-fleet-client normalizes legacy runtime events, and relay-fleet-client forwards lifecycle usage.
  • Orchestrator records per-agent usage, drains async usage before terminal save, finalizes missing agents with null tokens, attaches a bounded aggregate to DispatchLifecycle.cost, and emits exactly one run.cost.v1 event (allowlist-only).
  • Observability: added FactoryCloudRunCostV1Schema; run.cost.v1 requires a cost payload, cost is forbidden on other events, and only bounded fields are admitted (role/model/token/usd).
  • Manifest updated with cloud-run-cost-accounting; public exports added in src/index.ts.
  • Tests: unit tests for ledger/pricing (including overflow folding) and an E2E run-cost-accounting proving per-role/model totals, async drain, single bounded event, and unpriced handling.

^{Written for commit 9ed8c50. Summary will update on new commits.}

[kjgbot]

This should be displayed on the cloud dashboard so each run is cost tracked

[kjgbot]

@kjgbot Acknowledged. The Factory producer side is implemented in commit 9ed8c50: terminal runs persist the bounded per-role/per-model aggregate and enqueue run.cost.v1 through the authenticated Cloud reporter. I validated npm run build, npm run featuremap:check, all 1,324 tests, and npm run verify:e2e.

I also checked current AgentWorkforce/cloud main before calling this displayed: its strict factory.telemetry.v1 consumer does not yet admit the new cost payload or the model/token/USD attributes, so it would reject this event today. A coordinated Cloud consumer/dashboard change is still required; I have not encoded cost into unrelated legacy fields because that would violate the bounded event contract. Happy to discuss the rollout/scope.