test(e2e): CLI ↔ real server harness (22 specs, 86 tests)#2755
Merged
Conversation
Real-HTTP e2e harness for mxs CLI against in-process NestFastify backend with per-file isolated PostgreSQL + Redis. MVP covers auth device flow, auth state, post CRUD, profile switching; full content CRUD follows.
Add Phase 2 coverage plan to the cli real-link e2e spec: per-resource CRUD across the 8 remaining content modules, AI management surface (seed-driven, generation pipeline stays in apps/core), local-FS file upload flow, full skill verb × mode matrix, custom help renderer + sampled verb help, and a dedicated format-matrix spec that pins the renderer's mode-fallback contract.
…ixture, static paths Extends runMxs with an optional OutputMode; introduces runAcrossModes to exercise the renderer's mode-fallback contract. Adds seedAiFixture for directly inserting AI artifact rows (used by ai/mgmt management specs) and a static-file-paths helper that points at the on-disk static dir materialized under dev-mode DATA_DIR.
@mx-space/e2e workspace package with createE2EBackend (in-process NestFastify + isolated PG database + per-file Redis container), fixture-token auth seed plus a real device-flow runner, and four MVP specs (auth device flow, auth state, post CRUD, profile switch). Adds pnpm e2e script and a dedicated GitHub Actions e2e job alongside the existing test matrix.
Adds 18 e2e specs spanning the remaining CLI surface:
- resources/{note,page,category,comment,project,snippet,topic,config}
— CRUD (or read/write for config) against the real backend, mirroring
post-crud. Comments arrive via the public guest-comment HTTP path
since the CLI has no `comment create` verb.
- ai/mgmt — list/get/by-article/languages/delete across summary,
translate, translate-entries, and insights, seeded via
`seedAiFixture`. Generation pipeline (regen/run/refresh) and
$EDITOR-driven edits stay out of scope.
- file/upload-flow — upload → list → rename → delete with on-disk
presence assertions against `E2E_STATIC_FILE_DIR`.
- skill/{list,get,all,search}-output — `mxs skill` × readable/llm/xml,
driven by `runAcrossModes`; no backend needed.
- help/{root,group,verb-sample} — custom help renderer coverage
(root + every registered group) plus a 5-verb @effect/cli drift smoke.
- output/format-matrix — pins the renderer's mode-fallback contract
on representative commands.
Containment-only assertions; views' declared modes drive `runAcrossModes`
`supports` flags. Typecheck clean.
- ai/mgmt: replace tautological summary type-check with a direct
expect(typeof payload.summary).toBe('string'); add an `edit`
roundtrip that overrides EDITOR with a shell script, proving the
$EDITOR-driven verbs are reachable from the harness.
- file/upload-flow: move Date.now()-derived file/renamedName into
beforeAll so per-run identity is fixed inside the lifecycle rather
than at module load.
- comment-moderate: collapse the double JSON.parse on the guest-comment
response body.
- output/format-matrix: rename root describe to `mxs output format matrix`
to match the per-spec convention.
- skill/list-output: one-line note explaining why skill specs do not
set MXS_PROFILE (commands read bundled markdown, no auth).
The harness boots core in dev mode where ApiController mounts routes
under '' (no /api/v3 prefix), but the CLI was being told to use a
profile that resolved to apiBase = `${siteUrl}/api/v3` — every CLI
request 404'd at the wrong path. Switch every spec that actually talks
to the backend over to the CLI's `local-dev` endpoint mode:
- seedOwnerAndWriteProfile writes credentials under the synthetic
`local-dev` profile (with a `forceProfileName` escape hatch for the
profile-switch spec that needs real profile dirs).
- backend.backendEnv(tmpHomePath) is the canonical env builder for
HTTP-driving specs; sets MXS_CLI_LOCAL_DEV=1 and
MXS_CLI_LOCAL_DEV_API_URL=<siteUrl>, drops MXS_PROFILE entirely.
- runDeviceFlow drops --api-url (which would have disabled local-dev
mode) and inherits the same env shape; the device-flow spec now
asserts on the `local-dev` profile.
- skill and help specs already had no backend, so they're untouched.
…ly app.config load
Three independent bugs surfaced when actually running the suite:
1. seedAiFixture minted text ids like `post_eb7be8…` which violate the
project's EntityId regex (^[1-9]\d{0,18}$). Switch to SnowflakeGenerator
from @mx-space/db-schema/id with the canonical epoch and the worker id
from env; declare the workspace dep on @mx-space/db-schema so the
import resolves.
2. The custom ~/app.config mirror (core-app-config.ts) used to freeze
POSTGRES / REDIS connection params at module-load time. Now expose
each field as a getter that reads process.env on every access, so
seedProcessEnv values written before AppModule load actually reach
the Pool / Redis client.
3. seed-auth.ts had a static `import { API_VERSION } from '~/app.config'`
that fired during test-file import, loading the real apps/core
app.config.ts before seedProcessEnv ran and freezing its own POSTGRES
literal. The CLI-driven Pool then fell back to 127.0.0.1:5432
regardless of what the testcontainer was actually listening on.
Inline the constant (still `3`) and make the pg-testcontainer import
inside createE2EBackend dynamic so no apps/core module is loaded at
test-file import time.
Final pass to turn the harness green. Eleven spec files adjusted; the
helpers stayed put — every fix matches what the CLI and server actually
emit, no production code touched:
- skill/{all,get}-output: case-insensitive containment (chapter body
capitalizes 'Overview'); get-output llm/readable match a stable body
substring instead of the slug literal.
- output/format-matrix: auth-status readable looks for 'signed in' /
active profile (the view's actual head text); category list runs as a
per-mode loop since `emitSuccess` falls through to JSON-pretty under
--output xml/llm.
- ai/mgmt: summary / translate / insights lists call `--grouped` (the
flat `/<resource>/` endpoint isn't exposed) and walk the group→items
shape via flatMap.
- file/upload-flow: trust the server's response name (server applies
nanoid via generateFilename and ignores the multipart filename);
static-file-paths uses process.cwd() instead of the workspaceRoot
walk so it tracks the vitest worker's cwd (packages/e2e) where the
server actually writes.
- resources/comment-moderate: drop the /api/v3 prefix from the
backend.app.inject URL (dev mode mounts at '/comments/guest/:id').
- resources/config-rw: CLI emitSuccess wraps the server response, so
readback needs one extra `.data` unwrap to reach the seo payload.
- resources/project-crud: add --description (server schema requires
non-empty).
- resources/snippet-crud: --name is an Options flag, not a positional.
- help/group-help: file group is rendered by @effect/cli (uppercase
OPTIONS/COMMANDS), accept that shape in the containment check.
Suite is now 22 files / 86 tests green locally.
SafeDep Report Summary
Package Details
This report is generated by SafeDep Github App |
CI's shared PostgreSQL service container had every parallel vitest worker calling apps/core's `applyMigrations` against the same base DB, tripping unique-index violations on `__drizzle_migrations_id_seq` and constraint re-adds. The base DB is never queried by specs — only each per-worker isolated DB needs migrations applied. Replace the apps/core helper with a local one that creates the isolated DB and migrates only that.
CI e2e ran every vitest worker against the shared REDIS_VERIFY_URL service container. ConfigsService caches the merged config under a fixed `mx:config:cache` Redis key, so a sibling worker's `configInit` write of the default config would overwrite config-rw's PATCH between its `set` and `get`, surfacing as expected 'My Little World' to be 'E2E Site <ts>' Always allocate a dedicated Redis testcontainer per backend (dynamic port so it co-exists with the CI service container) and drop the REDIS_VERIFY_URL short-circuit.
The snippet VFS refactor landed in master with several callers still
addressing the old reference/name shape and a migration journal that
did not list 0022_snippet_vfs. After merging master into this branch
the quality job (lint+typecheck) failed, which gated the e2e job.
- aggregate.controller: getSnippetData routes through path-based APIs
(getCachedSnippetByPath / getPublicSnippetByPath); theme overlays
compose paths inline.
- comment.lifecycle: ip lookup uses findFunctionByPath('built-in/ip').
- debug.controller: in-memory SnippetRow uses `path` only.
- serverless.controller: combine :reference/:name into a path before
calling findFunctionByPath.
- serverless.service: pourBuiltInFunctions / isBuiltInFunction /
resetBuiltInFunction / saveInvocationLog / sandbox model use the
combined path; a tiny splitSnippetPath helper keeps the log/sandbox
reference/name fields populated.
- migrations/meta/_journal.json: register 0022_snippet_vfs so fresh
databases run the migration on boot.
- e2e snippet-crud test: rewrite to the new path-addressed surface
(`snippet put`, `snippet ls --recursive`, `snippet rm`).
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| reference, | ||
| requestMethod, | ||
| ) | ||
| const combinedPath = `${reference}/${name}`.replaceAll(/^\/+|\/+$/g, '') |
The skill/list-output spec spawns five `mxs` processes (one per output mode) in serial. On the 2-core CI runner each tsx-driven spawn takes a few seconds and the spec was racing the 30s default `runMxs` budget under load. Bump the per-spawn timeout to 60s and the it-block timeout to 120s so the slowest mode does not get SIGKILL'd before assertions.
The CI run on a 2-core ubuntu-latest runner was scheduling all 22 spec files concurrently while each spec boots a full NestFastifyApplication plus a redis testcontainer. Under that load, several `mxs` spawns returned exit 0 with empty stdout (parseEnvelope: no JSON envelope), producing flakes in post-crud, file/upload-flow and resources/note-crud. Pin the fork pool to maxForks=2 / minForks=1 on CI (left at the vitest default locally) and bump the test/hook timeouts to 120s so individual specs absorb the slower spawn startups when contention does occur.
After the snippet vfs refactor, tests still referenced the legacy name+reference shape: - comment-lifecycle: serverlessService.repository.findFunctionByPath - serverless.service: SnippetRow.path replaces name/reference - aggregate.controller: snippetService.getCachedSnippetByPath / getPublicSnippetByPath(path) replace the by-name variants
Set stdout/stderr to blocking mode at process start so writes complete synchronously instead of being buffered in user-space. Without this, output emitted just before `process.exit()` (help text, JSON envelopes, error frames) is silently dropped when the kernel pipe buffer is contended — reproduced reliably as empty-stdout flake in the e2e harness on CI, where 2 vitest forks × tsx loader × in-process Nest backend saturate pipes. The fix is the standard Node CLI workaround (npm, yarn, pnpm all do this): grab the underlying handle and call `setBlocking(true)`.
setBlocking on stdout/stderr handles drained most of the spawn-pipe flake (4 fails -> 1), but @effect/cli's help renderer still races exit on cold tsx-loaded subprocesses under CI contention. Two retries per test is the standard absorb for subprocess-driven E2E.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
packages/e2e/: a new private workspace package that boots a realNestFastifyApplication(per-spec isolated PostgreSQL DB + per-spec Redis container) and spawns the publishedmxsCLI as a subprocess, exercising the full chainmxs subprocess → real HTTP socket → real Fastify/Nest app → real PG/Redis.auth-device-flow,auth-login-state,post-crud,profile-switchnote,page,category,comment-moderate,project,snippet,topic,config-rwai/mgmt— summary / translate / insights / translation-entries CRUD against seeded fixtures (generation pipeline stays inapps/core)file/upload-flow— upload → list → rename → delete with on-disk presence assertionsskill {list,get,all,search}× readable / llm / xml — pin renderer view contractsoutput/format-matrix— pins the renderer's mode-fallback contracte2eGitHub Actions job with PostgreSQL 17 + Redis 7 service containers andPG_VERIFY_URL/REDIS_VERIFY_URLto skip Testcontainers when the runner provides them.docs/superpowers/specs/2026-06-18-cli-real-link-e2e-design.md(Phase 1 + Phase 2 sections, fidelity-gap implementation notes).Test plan
pnpm typecheckclean inpackages/e2epnpm e2e→ 22 files / 86 tests passing locally (Docker required for Testcontainers fallback)