146: Stand up factory-test-infra: Cloudflare-based ephemeral verification environments (Workers for Platforms + Containers) + factory provider wiring

[agent-relay-code]

Part of Epic: Factory Verification & Environments (#141). This is the tier-5/6 data-plane home. The provider (#142) and gate (#145) run on what this stands up. Best done first in Track C.

Decision: Cloudflare-first, in a dedicated factory-test-infra repo

factory is a library (@agent-relay/factory) — it must not own or operate the environment substrate. The org is already all-in on Cloudflare + SST + wrangler + workerd/miniflare, so tier-5/6 verification environments run on Cloudflare, not a bespoke Kubernetes cluster:

• Ephemeral per-feature environment → a Workers for Platforms dispatch namespace (isolated worker deployments, created and torn down as a unit — the role a K8s namespace would have played).
• Non-Worker services (Postgres, Redis, a non-JS API) → Cloudflare Containers, with Hyperdrive / D1 / KV / R2 / Queues for state.
• CI tier → @cloudflare/vitest-pool-workers / miniflare (cloud already runs this — test:integration, miniflare-sage).
• Load → k6 run from a Container/Worker against the environment URL (see Load-simulation harness: in-cluster load generation with SLO gates and evidence #144).

Kept in its own repo (not merged into cloud's infra/) so the spiky, ephemeral verification account/config is isolated from product infra for blast-radius + cost accounting. K8s/Daytona remain optional escape-hatch providers behind the same EnvironmentProvider port (#142) for the minority of target stacks that genuinely can't run on Cloudflare — Cloudflare Containers is not a general Kubernetes replacement for arbitrary heavy container stacks.

⚠️ Repo creation is an owner action — factory-test-infra does not exist yet (the GitHub App is denied org repo creation: 403 Resource not accessible by integration). First step: an owner creates agentworkforce/factory-test-infra (private), or grants the app repo-creation. Everything below is its intended contents.

What (in factory-test-infra)

1. Cloudflare account/config as code — SST (or wrangler + Terraform Cloudflare provider) for: the Workers for Platforms dispatch namespace(s) used for verification, Containers definitions for the non-Worker services, and the D1/KV/R2/Queues/Hyperdrive bindings a verification environment can request.
2. Isolation — one dispatch namespace per environment; per-environment bindings/secrets scoped so environments can't read each other or prod. (Dispatch-namespace isolation replaces NetworkPolicies.)
3. Quotas & cost caps — per-environment resource limits (Container instance sizes/counts, request caps), a max concurrent environments cap, and a per-run cost/time budget that aborts + reaps runaway runs. Use Cloudflare usage/analytics + account budgets.
4. Reaper ops — a scheduled sweep (Cron Trigger) that deletes dispatch namespaces / Container instances by ownership/age labels (belt-and-braces with the provider TTL reaper in EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142); leak alerting.
5. Base templates — wrangler/Container scaffolding that target repos' verification-stack descriptors (Verification-stack descriptor contract + deployer: repos declare their full stack, Factory brings it up #143) render into.

What (in factory)

6. Provider wiring — CloudflareEnvironmentProvider (EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142) reads the factory-test-infra account/API-token config (via a Resource.*-style secret, not process.env); a reusable CI workflow that runs the Track-C E2E suites (EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142–Verification gate: provision → deploy → E2E → load → evaluate → teardown, wired into the merge gate #145) against miniflare + a real dispatch namespace, and tears down.

Acceptance

• factory-test-infra repo exists with Cloudflare account/config as code (dispatch namespace(s), Containers, bindings) documented + scripted; miniflare path for CI.
• One dispatch namespace per environment; per-env bindings/secrets isolate environments from each other and prod.
• Per-env limits + max-concurrency + cost/time budget enforced; runaway runs aborted and reaped.
• Scheduled reaper (Cron Trigger) deletes orphaned namespaces/Container instances; leak alerting.
• factory's CloudflareEnvironmentProvider connects; reusable CI workflow runs EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142–Verification gate: provision → deploy → E2E → load → evaluate → teardown, wired into the merge gate #145 suites and tears down.

✅ End-to-end verification — Factory success criteria (REQUIRED)

"Compiles" is not done. Provision a real dispatch namespace + Container, prove the guardrails bind, and prove CI runs the suite end to end.

Run it

# in factory-test-infra
bash ci/verification-env/up.sh        # creates a dispatch namespace + bindings + a Container service
bash ci/verification-env/run-suite.sh # runs factory #142–#145 E2E suites against it
bash ci/verification-env/down.sh      # deletes the namespace + Container

The E2E must prove the guardrails bind (not just exist):

1. Isolation: a worker/Container in env-A cannot read env-B's bindings/secrets or reach its private service (assert the failure).
2. Quota/cost cap: requesting Container instances beyond the per-env limit is refused; a run past its time/cost budget is aborted and its namespace reaped.
3. Concurrency cap: provisioning beyond max-concurrent environments is refused.
4. Reaper: an orphaned dispatch namespace (owner label, past age) is deleted by the Cron sweep.
5. Load scale: a k6 load run (Load-simulation harness: in-cluster load generation with SLO gates and evidence #144) against a deployed Worker/Container completes and the environment absorbs it (Workers autoscale; Container instance scaling within the cap).
6. CI: the workflow runs factory EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142–Verification gate: provision → deploy → E2E → load → evaluate → teardown, wired into the merge gate #145 suites green against a real dispatch namespace and tears down.

Deliverable

• factory-test-infra ci/verification-env/{up,run-suite,down}.sh + reusable CI workflow, one-command.
• Guardrail assertions (isolation, quota/cost refuse, concurrency refuse, reaper delete, load absorb) each fail loudly if the guardrail is removed.

Anchor points

• New repo factory-test-infra: infra/ (SST or wrangler + Terraform Cloudflare provider), containers/ (Container service defs), ci/verification-env/ (up/run/down + miniflare), Cron-Trigger reaper worker.
• In factory: src/environments/cloudflare-provider.ts (from EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142) reads the account/token; src/config/schema.ts env config; .github/workflows/ reusable verification workflow.
• src/orchestrator/reaper.ts (identity-checked reaper pattern to mirror).
• Precedent for the Cloudflare/SST/miniflare posture: relaycast-cloud (workerd + @cloudflare/vitest-pool-workers), cloud's miniflare-sage service + test:integration.

Fixes #146

---

Summary by cubic

Adds Cloudflare-based ephemeral verification environments and a reusable CI workflow. Each run gets an untrusted Workers for Platforms dispatch namespace with a sentinel metadata Worker, strict guardrails, and an identity-checked reaper, meeting #146.

• New Features
  • CloudflareEnvironmentProvider: provisions one dispatch namespace per run; attaches a metadata Worker; enforces TTL, max-concurrency, run-cost, and container instance caps; adds a serializing + cross-instance reconciliation fix to close concurrency races; identity-checked destroy and a TTL/owner reaper.
  • Credential boundary: config holds only policy and a resource name; credentials come from a linked Resource object at runtime (no process.env), with a narrow token used by an HTTP client.
  • Reusable CI: .github/workflows/cloudflare-verification.yml runs factory-test-infra’s ci/verification-env/{up,run-suite,down}.sh in Miniflare and real modes, materializes a JSON resource file for the real run, always tears down, and uploads artifacts.
  • Docs and wiring: docs/cloudflare-environment-provider.md (lifecycle, guardrails, workflow usage); factory-test-infra/README.md handoff updated for Cloudflare; config schema, exports, and tests added.

^{Written for commit 52bf6b4. Summary will update on new commits.}

[kjgbot]

End-to-end merge blocker from the Factory-wide verification pass:

• The issue's required AgentWorkforce/factory-test-infra repository does not exist yet; this PR explicitly documents that owner action as pending.
• The diff adds a contract/README under this repository, but does not deliver the required factory-test-infra/ci/verification-env/{up,run-suite,down}.sh implementation in the dedicated infrastructure repository.
• The hosted Factory checks are green, but they do not provision a real Workers for Platforms dispatch namespace plus Container or prove the required negative guardrails: cross-environment isolation, quota/cost refusal, concurrency refusal, orphan reaping, bounded load scaling, and teardown.
• The branch must also integrate current main before final validation.

I am marking this draft so the current provider/unit-test green cannot be read as satisfying the issue's explicit real-environment acceptance criteria. Ready-to-merge evidence needs the infrastructure repository and scripts, a trusted real-Cloudflare workflow run with teardown artifacts, every guardrail assertion demonstrated (including failure when removed), current-main local/hosted suites green, and zero unresolved review threads.

[kjgbot]

Addressed the blocker in the coupled implementation now published as AgentWorkforce/factory-test-infra#2.

Concrete fixes:

• factory-test-infra now contains Terraform account/per-environment resources, bounded Containers, isolated dispatch/user-worker templates, analytics-backed Cron reaper and leak alerting, exact ci/verification-env/{up,run-suite,down}.sh scripts, mutation-tested guardrail assertions, k6 load, and protected real-Cloudflare CI (commit f0a6874).
• Factory now reconciles max concurrency after namespace creation across provider instances, points its handoff at the live companion implementation, defaults the reusable cross-repository workflow to Factory main unless an explicit ref is supplied, and is merged with current main (fix commit 419a0b8; merged head bc6ef58).

Validation on these heads: Factory build + 71 files/1,332 tests + CLI/featuremap + tier-2 dry-run/status; infra typecheck + 14 tests + Wrangler Container/reaper dry-runs + Terraform fmt/validate + actionlint; exact Miniflare up/run-suite/down with 403 cross-environment isolation, quota/concurrency refusals, cost/time aborts, orphan deletion, 200 requests/0 errors, and idempotent teardown.

The remaining acceptance item is intentionally not overstated: a trusted owner must supply the protected Cloudflare Resource secret and run the real job so the dispatch namespace, Container, k6, and teardown artifacts exist.

[kjgbot]

Final safe-work update before the owner-gated real run:

• Companion PR AgentWorkforce/factory-test-infra#2 is green on head 1546978; its automatic main-compatibility workflow and the explicitly pinned coupled run against Factory head bc6ef58 both passed Terraform, Miniflare guardrails, Track-C tests, identity-checked teardown, and evidence upload. Coupled run: https://github.com/AgentWorkforce/factory-test-infra/actions/runs/29852473802
• Five companion review findings were addressed in 1546978 (full Request forwarding, valid GraphQL scalar casing, finite bounded Container delays, defensive teardown state validation) or verified against the official Cloudflare type (DynamicDispatchLimits.subRequests); every original thread has a commit/validation reply and is resolved.
• Factory PR 146: Stand up factory-test-infra: Cloudflare-based ephemeral verification environments (Workers for Platforms + Containers) + factory provider wiring #182 remains fresh-head green, mergeable/CLEAN, and integrated with current main.

The real Cloudflare job is the sole remaining acceptance proof. The dedicated repository currently has no factory-verification environment or cloudflare_resource_json secret, and this runner has no Cloudflare Resource credentials, so I have not represented the skipped real job as success. Happy to walk an owner through the Resource shape, protection settings, expected artifacts, or any implementation trade-off before that run.

[kjgbot]

Current-base verification update:

• Merged the latest main through 2079ae0 (release 0.1.55); PR head is now 52bf6b4 and GitHub reports MERGEABLE/CLEAN.
• All six fresh-head Factory checks pass on 52bf6b4.
• Re-ran the coupled companion workflow after that push; run 29853356052 passed and its checkout log confirms the Factory commit was exactly 52bf6b4.
• Both repositories are clean and synchronized with their remote PR heads, and both PRs have zero unresolved review threads.

The protected real-Cloudflare job remains the only unmet issue acceptance item; neither repository secrets nor this runner currently provide the required Resource credentials.