diff --git a/vale_styles/config/vocabularies/Base/accept.txt b/vale_styles/config/vocabularies/Base/accept.txt index c5296f0e9..898c9ad0a 100644 --- a/vale_styles/config/vocabularies/Base/accept.txt +++ b/vale_styles/config/vocabularies/Base/accept.txt @@ -226,3 +226,4 @@ libpq cbshim stdin sudo +[Rr]eplayable diff --git a/versioned_docs/version-4.0.0/keploy-cloud/smart-set-agent.md b/versioned_docs/version-4.0.0/keploy-cloud/smart-set-agent.md index d1872dc14..e8d3ae1be 100644 --- a/versioned_docs/version-4.0.0/keploy-cloud/smart-set-agent.md +++ b/versioned_docs/version-4.0.0/keploy-cloud/smart-set-agent.md @@ -1,7 +1,7 @@ --- id: smart-set-agent -title: AI Agent for Smart Test Sets -sidebar_label: AI Agent (Smart Tests) +title: LLM Workflow (Smart Tests) +sidebar_label: LLM Workflow (Smart Tests) description: Let AI coding agents like Claude Code and Cursor diagnose failing smart-set replays and add new smart tests on a branch, using the Keploy MCP tools tags: - AI Agent @@ -26,7 +26,7 @@ import ProductTier from '@site/src/components/ProductTier'; ## Overview -Keploy's [smart test set](/docs/keploy-cloud/deduplication/) is a content-addressed test substrate: cases are keyed by a `schema_ref` (a hash of the contract shape — method, path, status, content-types, and the request/response body & query **shapes**), deduplicated per application, and edited **branch-natively** (the `main` view is read-only; edits live on a branch until a human or CI merges them). +Keploy's smart test set is a content-addressed test substrate: cases are keyed by a `schema_ref` (a hash of the contract shape — method, path, status, content-types, and the request/response body & query **shapes**), deduplicated per application, and edited **branch-natively** (the `main` view is read-only; edits live on a branch until a human or CI merges them). This page describes a ready-made **agent skill** that lets an AI coding assistant (Claude Code, Cursor, and similar) operate that substrate end-to-end. Given one of two plain-English prompts, the agent: @@ -35,64 +35,129 @@ This page describes a ready-made **agent skill** that lets an AI coding assistan The agent always stops at a **verified branch** and reports back. Merging to `main` stays a human/CI decision — it is intentionally not something the agent does. -## Prerequisites +## Before you start -- Keploy Enterprise with [smart test sets enabled](/docs/keploy-cloud/deduplication/) on the app (`EnableSmartTestSet=true`). -- The Keploy **MCP server** configured in your agent (see [MCP Server setup](/docs/running-keploy/agent-test-generation/#mcp-server-recommended-for-ai-agents)). The smart-set workflow uses the same `/client/v1/mcp` endpoint and the same authentication. -- A Personal Access Token (PAT) or API key with access to the app. -- The application's recording cluster reachable for `keploy cloud replay`. +- Keploy Enterprise with smart test sets enabled on the app (`EnableSmartTestSet=true`), and the app already [recording in your cluster](/docs/quickstart/k8s-proxy). +- A **Keploy PAT** — Dashboard → Settings → API Keys. Copy the `kep_...` value (shown only once). +- A **Skills-aware editor**: Cursor (cursor-agent CLI / Cursor IDE), Claude Code, Windsurf, Antigravity, or any agent that loads `.../skills//SKILL.md` on demand. Older `.cursorrules` / `.windsurfrules` always-on rules files work too but bill the full skill on every editor interaction — Skills are the modern, on-demand path and what this page recommends. -## What the agent needs from you +--- -The developer only ever says one of two things — the skill handles everything else (discovering the app, branch, failing run, and code changes) autonomously: +## Step 1 — Wire up the Keploy MCP server -| Prompt | Routine | -| --------------------------------------------------------------------- | ------------------------------------------------ | -| _"my keploy smart-set replay is failing, please analyze and fix it."_ | Routine A — diagnose & fix on a branch | -| _"Add new keploy smart tests for my changes."_ | Routine B — record, upload, validate on a branch | +All MCP-aware editors accept the exact same JSON config; only the config file path differs. -## Installing the skill +### MCP config path per editor -The skill is a single Markdown file that teaches your agent the smart-set workflow and guardrails. Drop it into your project so the agent picks it up automatically. +| Editor | Config file path | +| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| **Claude Code** | `~/.claude.json` (global) | +| **Cursor IDE** | `.cursor/mcp.json` in the repo (per-repo, committed) **or** `~/.cursor/mcp.json` (global) | +| **cursor-agent CLI** | `~/.cursor/mcp.json` (CLI reads the global path) | +| **Windsurf / Antigravity** | `~/.codeium/windsurf/mcp_config.json` or the editor's documented equivalent | +| **Other MCP-aware editors** | See [MCP Client Configuration](/docs/running-keploy/agent-test-generation#mcp-client-configuration) — same JSON shape works everywhere. | -### Cursor +### The JSON config (same for every editor) -Save the skill as `.cursor/skills/smart-set/SKILL.md` in your project root. Cursor auto-discovers [Agent Skills](https://cursor.com/docs/context/skills) from `.cursor/skills/` and invokes this one on demand when your prompt matches a failing smart-set replay or a request to add smart tests. This is the on-demand **skill** mechanism — distinct from always-on `.cursor/rules/*.mdc` project rules, which would bill the full skill on every turn. +```json +{ + "mcpServers": { + "keploy": { + "type": "http", + "url": "https://api.keploy.io/client/v1/mcp", + "headers": {"Authorization": "Bearer kep_..."} + } + } +} +``` -### Claude Code +Two things to swap for your environment: -Save the skill under your project's skills directory (e.g. `.claude/skills/smart-set/SKILL.md`) or reference its content from `CLAUDE.md`. Claude Code reads project-level skill and context files automatically. +- **`url`** — `https://api.keploy.io/client/v1/mcp` is the hosted Keploy endpoint. For **self-hosted** deployments use your api-server's URL (commonly `http://localhost:8083/client/v1/mcp` for a local dev install). +- **`kep_...`** — paste your PAT from Dashboard → Settings → API Keys (shown only once on creation; regenerate if you've lost it). -The full skill content is included at the end of this page under [Skill reference](#skill-reference). +Fully quit and reopen your editor after editing the config — MCP clients only re-read config on startup. -## How it works +### Verify the MCP wiring (do this BEFORE Step 2) -### Key concepts the agent relies on +After restarting your editor, ask the agent: -- **Branch-first, enforced by the substrate.** Every edit, delete, obsolete, or mock write is branch-scoped; a write without a `branch_id` is rejected. The Keploy branch name mirrors your git branch name (`git rev-parse --abbrev-ref HEAD`), and `create_branch` is idempotent (find-or-create). -- **`schema_ref` identity.** **Value edits** (response body, noise, assertions, mock re-links) keep the same `schema_ref` and are safe in place. **Shape edits** (changing method/path/status/content-type or the body/query structure) recompute the `schema_ref`; if the new ref collides with another case you get a typed `SchemaRefConflict` to resolve, not retry. -- **Non-destructive re-record.** Re-recording a same-shape contract replaces the case data in place and carries your noise/assertions/obsolete flags forward. A re-record that changes the shape lands a **new** `schema_ref` — the stale case is then deleted so the suite doesn't keep a red duplicate. -- **The boundary is the branch.** The agent never runs a merge or rebase to `main` — it reports a verified branch and the dashboard URLs, and you (or CI) merge. +> _"List your available tools — do you see any prefixed `keploy-` or `mcp__keploy`?"_ + +You should see Keploy MCP tools. Depending on the server version you'll see **either** the full catalog (`listApps`, `getApp`, `listTestReports`, `getTestReportFull`, `getMock`, `listSmartTestCases`, `updateSmartTestCase`, `upsertSmartMock`, `setSmartTestCaseObsolete`, etc.) **or** — when **tool-search mode** is on (the default on newer servers) — just the handful of meta-tools (`search_tools`, `get_tool_schema`, `invoke_tool`, `get_auth_status`, `get_setup_instructions`). **Either is fine**: the skill's Hard rule 0 handles both and reaches the full catalog by name. Only if you see **zero** keploy tools did the config fail to load — check: + +- The file path matches your editor's expected location (see the table above). +- The PAT in `Bearer kep_...` has no quotes / trailing-whitespace issues. +- The `url` is reachable from your machine: `curl -i ` should return an **HTTP response** (e.g. 200 or 401), NOT connection-refused. Any HTTP status means the server is up; connection-refused means the URL is wrong. + +> ⚠ **Do NOT proceed to Step 2 if Keploy MCP tools aren't visible.** Without native MCP the agent silently falls back to shell HTTP calls, which inflates per-turn token cost several-fold because every JSON-RPC envelope + auth token + response becomes new context instead of structured tool envelopes the cache can reuse. + +--- + +## Step 2 — Install the skill + +The skill teaches your agent to run the whole smart-set workflow autonomously from the two routine prompts. Without it, the agent has to rediscover the workflow on every call by reading each tool's description — slower and prone to skipping the branch-resolution step. + +The exact same file works on every Skills-aware editor; only the path changes. Skills load on demand when the user's prompt matches the skill's `description`, so it only enters your context when actually needed — not on every editor interaction. + +### Where the skill goes + +| Editor | Install path | +| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Cursor** | `.cursor/skills/smart-set/SKILL.md` in the repo (committed). Avoid always-on rule formats (`.cursorrules`, or `.cursor/rules/*.mdc` with `alwaysApply: true`) — they bill the full skill on every turn even for unrelated conversations. | +| **Claude Code** | `.claude/skills/smart-set/SKILL.md` in the repo (committed) **or** `~/.claude/skills/smart-set/SKILL.md` (global). Auto-loaded when the dev's prompt matches the skill's `description`. | +| **Windsurf / Antigravity / other Skills agents** | The agent's equivalent `skills//SKILL.md` path. Avoid global / always-on placements for the same token-cost reason. | + +Copy the skill from [Skill reference](#skill-reference) below into the file at the path you picked, then **fully restart the editor** — editors read skills only at startup. Commit the file when you want every teammate's agent to follow the same skill. + +--- -### Routine A — fix a failing replay +## Step 3 — Use the two prompts -1. **Resolve the failing run.** For a local failure, the agent fetches the newest `FAILED` report on the branch; for a CI failure, it extracts the `test_run_id` from the pasted CI/dashboard URL. -2. **Fetch the report**, projected to just the failing cases (a focused field set instead of the full ~34k-token report). -3. **Classify each failing case** after an unconditional working-tree check (`git status`/`git diff`): - - **Regression** — code changed and broke a correct contract → the agent **fixes the source and rebuilds**, never edits the test to match a bug. - - **Value drift** — a field/header/body value legitimately changed → `updateSmartTestCase` (golden body or `noise` for non-deterministic fields). - - **Shape drift** — the contract structure changed → `updateSmartTestCase` with the new request/response shape, resolving any `SchemaRefConflict`. - - **Mock drift** — a downstream response changed → `upsertSmartMock` (or re-record when the request itself changed). -4. **Verify on the branch** via `keploy cloud replay --replay-source smart-set` and iterate (capped retries). -5. **Report and stop** with a diagnosis table, the fixes applied, and dashboard URLs for the branch diff and run report. +That's it. From now on you only ever type one of: -### Routine B — add new smart tests +> **"my keploy smart-set replay is failing, please analyze and fix it."** -1. **Identify changed endpoints** from the git diff. -2. **Capture traffic** with `keploy record -c "" --sync --disable-mapping=false`, driving one realistic request per new/changed endpoint. -3. **Upload onto the branch** as a smart set (new contracts ingest as `imported-*`, deduplicated by `schema_ref`; existing ones are skipped). -4. **Validate on the branch** with `keploy cloud replay`. -5. **Report and stop** — you review the branch diff and merge; merge reconciles `imported-*` to stable `test-N`. +_or, when the failure was in CI:_ + +> **"the keploy smart-set pipeline is failing, please analyze and fix it."** + +or + +> **"Add new keploy smart tests for my changes."** + +What happens behind the scenes for each: + +### Prompt A — analyze and fix a failing smart-set replay (local or CI) + +| Phase | What the agent does | +| ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| A0 | Discovery — resolve `app_id` from `basename $(pwd)` + `listApps`, `branch_id` from `git rev-parse --abbrev-ref HEAD` + `create_branch` (find-or-create), and cluster identity from `getApp` (cached for the session). | +| A1 | Get a `test_run_id`. Local → `listTestReports({appId, branch_id, status: "FAILED"})`, newest id. CI → extract `test_run_id` from the pasted CI log / dashboard URL. | +| A2 | Fetch the report via `getTestReportFull` with a `fields=[...]` projection and `failed_only: true` (brings the ~34k-token report down to ~1-2k). For mock failures, a second call with `mock_mismatches_only: true` to surface the `mock-N` ids. | +| A3 | Classify each failing case after an unconditional `git status`/`git diff`: **Regression** (code broke a correct contract → fix the source, never the test), **Case A — value** (`respBody`, or `noiseJson` for non-deterministic fields — `schema_ref` preserved), **Case B — shape** (`requestJson`/`responseJson` — `schema_ref` recomputed; resolve a `SchemaRefConflict` by obsoleting/deleting the twin), **Case C — mock** (`upsertSmartMock`, or re-record when the outbound request moved). | +| A4 | Apply the fix on the branch via `updateSmartTestCase` / `upsertSmartMock` / `setSmartTestCaseObsolete` / `deleteSmartTestCase` (regression → edit source + rebuild image), then verify with `keploy cloud replay --replay-source smart-set --branch-name --cluster --freezeTime` (capped retries). | +| A5 | Report: diagnosis table (case per failing test) + fixes applied + next step + branch-diff URL + run-report URL. Merging to `main` stays with you or CI. | + +### Prompt B — author new smart tests + +| Phase | What the agent does | +| ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| B0 | Discovery (same as A0). | +| B1 | `git diff origin/main...HEAD --name-only` → new / changed endpoints (method + path). | +| B2 | Pre-flight: find the run command (Makefile / compose / Procfile / package.json / README), start the app, curl to confirm it serves, stop it. Then `keploy record -c "" --sync --disable-mapping=false`, drive one realistic request per new endpoint, SIGINT to flush. | +| B3 | `keploy upload test-set --smart-test-set --app --branch --test-set keploy/test-set-N` — new `schema_ref`s ingest as `imported-*` (deduplicated); existing ones are skipped. | +| B4 | `keploy cloud replay --replay-source smart-set --branch-name --cluster ` to validate. On failure, drop into Prompt A from Phase A2. | +| B5 | Report: captured endpoints + replay result + next step (open PR) + URLs. Merge reconciles `imported-*` → stable `test-N`. | + +--- + +## Key concepts the agent relies on + +- **Branch-first, enforced by the substrate.** Every edit, delete, obsolete, or mock write is branch-scoped; a write without a `branch_id` is rejected. The Keploy branch name mirrors your git branch name (`git rev-parse --abbrev-ref HEAD`), and `create_branch` is idempotent (find-or-create). +- **`schema_ref` identity.** **Value edits** (response body, noise, assertions, mock re-links) keep the same `schema_ref` and are safe in place. **Shape edits** (changing method/path/status/content-type or the body/query structure) recompute the `schema_ref`; if the new ref collides with another case you get a typed `SchemaRefConflict` to resolve, not retry. +- **Non-destructive re-record.** Re-recording a same-shape contract replaces the case data in place and carries your noise/assertions/obsolete flags forward. A shape-changing re-record lands a **new** `schema_ref` — the stale case is then deleted so the suite doesn't keep a red duplicate. +- **The boundary is the branch.** The agent never runs a merge or rebase to `main` — it reports a verified branch and the dashboard URLs, and you (or CI) merge. ## Replay flags @@ -117,7 +182,7 @@ When the agent runs `keploy cloud replay` for a smart-set app, these flags are r The complete skill file to install (`SKILL.md`): -```markdown +````markdown --- name: keploy-smart-set description: Keploy SMART-SET MCP workflow — when a smart-set cloud replay is failing (analyze and fix on a branch), or to add new smart tests for code changes. Drives schema_ref-keyed, branch-native record/replay and smart-case/mock edits via the Keploy MCP tools. The agent fixes on a branch and reports; merging to main is the dev's (or CI's) call. @@ -125,69 +190,292 @@ description: Keploy SMART-SET MCP workflow — when a smart-set cloud replay is # Keploy SMART-SET playbook — autonomous developer workflow -Smart test sets are Keploy's content-addressed test substrate: cases are keyed by `schema_ref` (a hash of the contract shape — method, path, status, content-types, request/response body & query SHAPES), deduped per app, and edited **branch-natively** (main is read-only; edits live on a branch until a human/CI merges). Re-recording is **non-destructive for same-shape refreshes** — it replaces a case's data in place by `schema_ref` and preserves history, so user edits (noise, assertions, obsolete) carry forward; but a re-record that changes the shape lands a new `schema_ref` and you must delete the stale old case (Hard rule 5). +Smart test sets are Keploy's content-addressed test substrate: cases are keyed by `schema_ref` (a hash of the contract shape — method, path, status, content-types, request/response body & query SHAPES), deduped per app, and edited **branch-natively** (main is read-only; edits live on a branch until a human/CI merges). Re-recording is **non-destructive for same-shape refreshes** — it replaces a case's data in place by `schema_ref` and preserves history, so user edits (noise, assertions, obsolete) carry forward; but a re-record that changes the shape lands a new `schema_ref` and you must delete the stale old case (Hard rule 5). This is the parallel to the legacy `keploy` skill; use THIS skill when the app/flow is on the smart-set substrate. ## Entry points The developer will only ever say one of two things to you: -- **Prompt A:** "my keploy smart-set replay is failing, please analyze and fix it." (local: find the latest failing test_run on the branch) OR "the keploy smart-set pipeline is failing, please analyze and fix it." (CI: extract `test_run_id` from the pasted CI log/dashboard URL). -- **Prompt B:** "Add new keploy smart tests for my changes." +**Prompt A:** "my keploy smart-set replay is failing, please analyze and fix it." OR "the keploy smart-set pipeline is failing, please analyze and fix it." — the first means the dev's last local replay on the branch failed (find the latest test_run via api-server), the second means a CI run failed (the dev pastes the CI log/dashboard URL; extract `test_run_id` from it). +**Prompt B:** "Add new keploy smart tests for my changes." -You handle EVERYTHING else autonomously — discover the app, the branch, the failing run, the code changes. Execute fixes **on a branch**, report what you did, and tell the dev to review & merge. +You handle EVERYTHING else autonomously — discover the app, the branch, the failing run, the code changes. Make decisions, execute fixes **on a branch**, report what you did, and tell the dev to review & merge. Do NOT ask follow-up questions unless truly blocked (see "When you may ask"). ## Hard rules -0. **Native MCP transport only.** Verify the Keploy MCP tools are loaded. If your tool list shows only the meta-tools (`get_auth_status`, `get_setup_instructions`, `search_tools`, `get_tool_schema`, `invoke_tool`) and none of the Smart-set names below, the real tools are hidden server-side to save context — fetch their schemas in ONE batched `get_tool_schema({names:[…]})` call, then run each via `invoke_tool({name, arguments})`. Smart-set names: `listApps`, `getApp`, `create_branch`, `listTestReports`, `getTestReportFull`, `listSmartTestCases`, `updateSmartTestCase`, `setSmartTestCaseObsolete`, `deleteSmartTestCase`, `upsertSmartMock`, `deleteSmartMock`, `getMock`, `uploadRecordingBundle`. -1. **Branch-first — the substrate ENFORCES it.** Every edit/delete/obsolete/mock-write is branch-scoped; a write without a `branch_id` is rejected. Resolve `branch_id` before any write. -2. **Keploy branch name = git branch name** (`git rev-parse --abbrev-ref HEAD`). Pass it to `create_branch` (find-or-create, idempotent); reuse the returned `branch_id`. Never target the reserved `main` branch. -3. **App resolution from cwd.** `basename $(pwd)` → `listApps({q: })`. One match → use it; zero/ambiguous → narrow by compose-service name, else ask once. -4. **schema_ref awareness.** VALUE edits keep `schema_ref` (`noiseJson`, `assertionsJson`, `description`, `mockReferencesJson`, `respBody`). SHAPE edits change it (`requestJson`/`responseJson`); a colliding new ref yields a `SchemaRefConflict` — don't retry blindly. All `*Json` args are STRINGIFIED JSON, not objects. -5. **Re-record replaces in place only if `schema_ref` is unchanged.** If the re-record changes the shape it lands a NEW `schema_ref` as a separate case — then `deleteSmartTestCase` the stale old one. -6. **Your boundary is the branch. NEVER merge or rebase.** After your fix is green on the branch, STOP and report — the dev/CI merges. -7. **Don't ask what you can find out** (`git log`, `git diff`, file reads, api-server calls). -8. **Always end with two dashboard URLs** — the branch diff page and the test-run report page. - -## Discovery (run once at the start) - -1. **App.** `basename $(pwd)` → `listApps({q})` → cache `app_id`. -2. **Branch.** `git rev-parse --abbrev-ref HEAD` → `create_branch({app_id, name})` → cache `branch_id`. -3. **App context (once).** `getApp({appId, fields:["name","namespace","deployment","origin.clusterName","origin.namespace","origin.deployment"]})` — you need `origin.clusterName` for `--cluster`. -4. **Canonical replay command — these flags on every replay** (drop `--freezeTime` for non-faketime apps): `keploy cloud replay --app --branch-name --cluster --replay-source smart-set --disableReportUpload=false --strict-failure [--freezeTime]`. Why each: `--replay-source smart-set` (the CLI defaults to latest-release), `--cluster` (from origin.clusterName; omit it and the CLI errors "no active clusters found"), `--disableReportUpload=false` (writes the /tr report row so the run shows on the dashboard), `--strict-failure` (don't silently demote response-divergent cases), and `--freezeTime` ONLY when the app is built with the Go faketime agent (omit it otherwise). These match the "Replay flags" table above. - -## Routine A — failing smart-set replay (ON A BRANCH) - -- **A1 — Resolve `test_run_id`.** Local → `listTestReports({appId, branch_id, status:"FAILED", limit:5})` exactly once, take `data[0].id` (`status` is case-sensitive). CI → extract from the pasted URL. -- **A2 — Fetch the report**, projected with `failed_only:true` + a `fields=` list (drops ~34k → ~1–2k tokens). For mock failures, a second call with `mock_mismatches_only:true` to get the `mock-N` ids. -- **A3 — Diagnose.** Unconditional working-tree check first (`git status -s`, `git diff`). **Code-change gate:** if a code change touches the same field the report says drifted, that's a regression by default — fix the source, don't bake it into the golden body. Classify each case: - - **Case 1 — App regression.** Edit/revert the application source, rebuild the image, replay. Don't touch the test. - - **Case A — Value drift.** `updateSmartTestCase` — `noiseJson` for non-deterministic fields, `respBody` for a real value change. - - **Case B — Shape drift.** `updateSmartTestCase` with `requestJson`/`responseJson`; resolve a `SchemaRefConflict` by obsoleting/deleting the twin, never by blind retry. - - **Case C — Mock drift.** `upsertSmartMock` for an in-place value drift; re-record when the outbound request changed or the match key can't be hand-authored. -- **A4 — Verify on the branch.** Rebuild first after a Case 1 edit. Replay with the **canonical command from Discovery (all flags)**, piping output through `tail`/`grep`. All cases failing "connection reset"/status 0 = a stale leftover replay container on the app port (`docker rm -f` it), not a code bug. Cap retries at 3. -- **A5 — Report and STOP.** Diagnosis table + fixes applied + the two dashboard URLs. Tell the dev to review & merge. - -## Routine B — add new smart tests - -- **B1 — Identify changes.** `git diff origin/main...HEAD --name-only`, filter to HTTP handlers, list each endpoint's method+path. -- **B2 — Capture traffic.** Pre-flight the run command, then `keploy record -c "" --sync --disable-mapping=false` (both flags mandatory), drive one realistic request per endpoint, stop the recorder by PID. -- **B3 — Upload onto the branch.** `keploy upload test-set --app --branch --test-set keploy/test-set-N --smart-test-set --name ` (ingests new contracts as `imported-*`, dedup by `schema_ref`). -- **B4 — Validate** with the **canonical replay command from Discovery (all flags)**. On failure, enter Routine A from A2. -- **B5 — Report and STOP.** Captured/skipped table + replay result + dashboard URLs; the dev merges (merge reconciles `imported-*` → `test-N`). - -## When you MAY ask the dev - -- PAT missing/invalid → ask for a fresh PAT. -- Detached `HEAD`/non-zero from `git rev-parse` → ask for a branch name once. -- `listApps` ambiguous and unnarrowable → list candidates, ask once. -- Pre-flight can't start the app → name the command + error, ask once. -- A `SchemaRefConflict` where both cases are legitimately distinct → surface it; "merge into existing" is the dev's call. +0. **Native MCP transport only — NEVER Python+urllib shell fallback.** Verify Keploy MCP tools are loaded. **If your tool list shows ONLY the meta-tools** (`get_auth_status`, `search_tools`, `get_tool_schema`, `invoke_tool`), the real tools are hidden server-side to save context. You already KNOW the names you need — fetch their schemas in ONE batched `get_tool_schema({names:[…]})` call, then run each via `invoke_tool({name, arguments})`. The smart-set names: `listApps`, `getApp`, `listBranches`, `create_branch`, `listTestReports`, `getTestReportFull`, `listSmartTestCases`, `updateSmartTestCase`, `setSmartTestCaseObsolete`, `deleteSmartTestCase`, `upsertSmartMock`, `deleteSmartMock`, `getMock`, `uploadRecordingBundle`. Do NOT use `search_tools` for a name you already have (fuzzy search returns 5–10 unrelated schemas, ~10× the tokens). Reserve `search_tools` ONLY for a name you genuinely don't know. -## Anti-patterns (refuse these) +1. **Branch-first — and the substrate ENFORCES it.** Main is read-only for smart-set: every edit/delete/obsolete/mock-write is branch-scoped and a write without a `branch_id` is REJECTED (`ErrSmartMutationRequiresBranch` / "requires a branch"). Resolve `branch_id` before any write. If a tool rejects the write for a missing branch, you skipped this — resolve the branch and retry, don't ask the dev. +2. **Keploy branch name = git branch name.** Detect via `git rev-parse --abbrev-ref HEAD`. Pass that string to `create_branch` (find-or-create, idempotent). Reuse the returned `branch_id` for every write this session. The branch literally named `main` is reserved — never target it. +3. **App resolution from cwd.** `basename $(pwd)` → `listApps({q: })`. One match → use it. Zero/ambiguous → narrow by compose-service name, else ask once. +4. **schema_ref awareness — know which edits move identity.** + - **VALUE edits keep `schema_ref`:** `noiseJson`, `assertionsJson`, `description`, mock relink (`mockReferencesJson`), and the golden response body (`respBody`). These are safe in-place edits. (All `*Json` args are STRINGIFIED JSON, not objects.) + - **SHAPE edits change `schema_ref`:** replacing `requestJson` or `responseJson` (method/path/status/content-type/body-or-query STRUCTURE). A shape edit recomputes the ref; if the new ref collides with another case in scope you get a typed `SchemaRefConflict` — DON'T retry blindly (see Case B). +5. **Re-record replaces in place only if the `schema_ref` is unchanged** (same shape → refreshes the case, your noise/assertions carry forward). If the re-record changes the shape it lands a NEW `schema_ref` as a separate case — then `deleteSmartTestCase` the stale old one or the suite keeps a red duplicate. Needed when a mock can't be patched in place (Case C) or the shape moved. +6. **Your boundary is the branch. NEVER merge or rebase.** Merging/rebasing to main is a human/CI-gated action (and is intentionally NOT an MCP tool). After your fix is green ON THE BRANCH, STOP and report — tell the dev to review & merge. Do not attempt `mergeBranch`/`rebaseBranch`, do not look for a merge CLI, do not edit on main. +7. **Don't ask what you can find out.** Use `git log`, `git diff`, file reads, and api-server calls. +8. **Always end with two dashboard URLs** — the branch diff page and the test-run report page: + - `Branch diff: https://app.keploy.io/api-testing/branch-diff?appId=&branchId=` + - `Run report: https://app.keploy.io/tr/?appId=` + +## Discovery (run once at the start, before either routine) + +1. **App.** `basename $(pwd)` → `listApps({q: })` → cache `app_id`. +2. **Branch.** `git rev-parse --abbrev-ref HEAD` → `create_branch({app_id, name: })` → cache `branch_id`. Detached `HEAD`/non-zero → ask the dev once. +3. **App context — MANDATORY before any `keploy cloud replay`, AT MOST ONCE per session.** `getApp({appId: app_id, fields: ["name","namespace","deployment","origin.clusterName","origin.namespace","origin.deployment"]})`. The full schema is ~16k tokens; this projection is ~300. You CANNOT issue `keploy cloud replay` without `origin.clusterName` (the CLI needs `--cluster `; its `no active clusters found` error actually means "you forgot `--cluster`"). The identity fields are sticky — never re-call `getApp`. +4. **`--replay-source smart-set` — MANDATORY on EVERY `keploy cloud replay`.** The CLI defaults to `latest-release`, which replays the raw per-release recordings, NOT the smart set. You MUST pass `--replay-source smart-set` so replay pulls the deduped smart-set cases (requires the app to have `EnableSmartTestSet=true`). Without it you're testing the legacy release sets, not the smart set, and the report won't reflect smart-set edits. +5. **`--freezeTime` — MANDATORY on EVERY `keploy cloud replay`** if the app is built with the Go faketime agent. Without it `time.Now()` drifts from the recording, timestamp-bearing mocks miss, and the call falls through to the real backend (looks like an app bug; isn't). Always: `keploy cloud replay --app --branch-name --cluster --replay-source smart-set --freezeTime --disableReportUpload=false --strict-failure`. + +All values are sticky. Don't re-discover unless the dev switches git branches. + +--- + +## Routine A — failing smart-set replay, analyze and fix (ON A BRANCH) + +### Phase A1 — Resolve the `test_run_id` + +- **Local form** → `listTestReports({appId: app_id, branch_id, status: "FAILED", limit: 5})` **EXACTLY ONCE**. Take `data[0].id` (newest first). `status` is CASE-SENSITIVE (`"FAILED"`). Do NOT retry permutations on empty; do NOT re-query after your own replay — parse the `View test report at: …/tr/` line from its stdout. +- **CI form** → extract `test_run_id` from the pasted CI/dashboard URL; fall back to the local lookup with `source: "ci"`. + +### Phase A2 — Fetch the full report + +Smart-set replays post the same `/tr` report shape as legacy, so the projection is identical — fetch everything you need in ONE call: + +``` +getTestReportFull({ + appId: app_id, reportId: test_run_id, + include_oss_report: true, failed_only: true, max_test_cases_per_set: 50, + fields: [ + "report.status","report.ci_metadata","report.failed_count","report.passed_count", + "test_sets[].test_set_name","test_sets[].test_set_id","test_sets[].status", + "test_sets[].test_cases[].test_case_name","test_sets[].test_cases[].test_case_id","test_sets[].test_cases[].status", + "test_sets[].test_cases[].oss_report.req","test_sets[].test_cases[].oss_report.resp", + "test_sets[].test_cases[].oss_report.result","test_sets[].test_cases[].oss_report.failure_info", + "test_sets[].test_cases[].oss_report.noise" + ] +}) +``` + +`failed_only: true` + the projection drops the call from ~34k → ~1–2k tokens. **Never** drop `fields=`. Path params (`appId`,`reportId`) are camelCase; query params (`include_oss_report`,`failed_only`,`mock_mismatches_only`,`max_test_cases_per_set`,`fields`) stay snake_case. For mock-driven failures, make a SECOND call with `mock_mismatches_only: true` projected to `…oss_report.mock_mismatches.actual_mocks[].name`/`.kind` to get the `mock-N` ids. The smart-set case name is `test-N` (stable across edits) — read `test_case_name`. + +### Phase A3 — Diagnose each failing case + +**Step 0 — Working-tree check (UNCONDITIONAL, FIRST).** `git status -s`, `git diff`, `git diff --cached` on the failing handler path. The empty result IS the evidence to advance. Only Phase A1 discovery MCP calls are allowed before Step 0; every classifier/write call comes after. + +**Verify intent before classifying.** Commit messages confirm "deliberate" (→ contract change) but never "bug." Tie the change to the failure (does the diff touch the field/header/status the report says drifted?). Expected change → fix test data on the branch; unexpected → fix the code. + +**⚠️ CODE-CHANGE GATE — decide this BEFORE reaching for any smart-set edit tool.** If Step 0 (or `git log`) shows a code change on the failing handler that touches the **same field the report says drifted**, that is **Case 1 (a regression) by default — REVERT/fix the source (`git checkout -- ` or an Edit) and rebuild. Do NOT `updateSmartTestCase`/`respBody` to make the app's new output the golden value.** Updating the test to match changed code **bakes the regression in as the new contract** and hides the bug. This skill is branch-edit-centric, but a code regression is NOT a test-data problem — resist the reflex to "just update the golden body." Only treat the code change as intentional (→ Case A/B test edit) when the **commit message explicitly claims that contract change** OR the dev asked for it in this session. If a code change is present and intent is genuinely ambiguous, **ASK the dev once** — do not silently update the test. + +Classify each failing case: + +#### Case 1 — App regression. You fix the code. + +The handler used to behave correctly; an uncommitted edit or recent commit broke it. Same as the legacy flow: (C1.1) edit/revert the **application source** (never Dockerfile/compose/keploy.yml/CI); (C1.2) verify the edit landed (`git status`/`git diff` clean for a revert); (C1.3) rebuild the image (`docker build -t `); (C1.4) replay on the branch. The recorded baseline is correct — do NOT touch the test. + +#### Case A — Value drift (test data is stale; identity unchanged). `updateSmartTestCase` on the branch. + +The contract changed in a way that keeps the same `schema_ref` (a field value, a header constant, a response body value). Tool: `updateSmartTestCase({appId, name: "test-N", branch_id, })`. +**`updateSmartTestCase` arg names — all the JSON-bearing fields are STRINGIFIED JSON** (not objects): `noiseJson`, `assertionsJson`, `mockReferencesJson`, `requestJson`, `responseJson` (each a JSON string), plus `respBody` (string), `description`, and the required `appId`, `name`, `branch_id`. Passing an object instead of a JSON string → `invalid request body`. + +- **Non-deterministic field** (timestamp, generated UUID, request id): add its JSONPath to **`noiseJson`** (a JSON map serialized as a string, e.g. `noiseJson: "{\"body.ts\":[]}"`) — the runner then ignores diffs on that path. +- **Real value change**: set **`respBody`** (the golden response body string) — a VALUE edit that never moves `schema_ref`. Read the case first via `listSmartTestCases` so you only change what the new contract dictates. +- Assertions / description: pass **`assertionsJson`** (JSON string) / `description`. + +#### Case B — Shape drift (the contract STRUCTURE changed; identity moves). `updateSmartTestCase` with `requestJson`/`responseJson`. + +Replacing **`requestJson`** (full HTTPReq as a JSON string) or **`responseJson`** (full HTTPResp JSON string) recomputes `schema_ref`. Two outcomes: + +- **Resolves to a new, free ref** → the case is updated and re-keyed; done. +- **`SchemaRefConflict`** (the new ref already belongs to a different case — including an obsolete one) → DON'T retry blindly. The conflict payload names `existingCaseId`/`existingName`. Decide: if the other case is the stale twin of this contract, `setSmartTestCaseObsolete` or `deleteSmartTestCase` it on the branch, then re-apply the shape edit; if both are legitimately distinct, leave them and report the conflict to the dev (an explicit "merge into existing" is a human action). The failed shape edit persists NOTHING — the case is untouched, so there is no partial state to clean up. + +#### Case C — Mock drift. `upsertSmartMock` on the branch. + +A downstream call's recorded response drifted (`mock_mismatches`). For a simple VALUE drift on an existing mock (same request shape), read the mock with `getMock({…, branch_id})` and write the corrected mock via `upsertSmartMock({appId, name: "mock-N", branch_id, })`; relink with `updateSmartTestCase`'s `mockReferences` (a VALUE edit). + +**Can't patch in place → re-record (Routine B on the branch), then `deleteSmartTestCase` the stale case if the shape moved (rule 5):** when the outbound REQUEST changed (a new/changed downstream call — no in-place mock to patch), or when the mock's match key is content-derived and can't be hand-authored — re-record so keploy re-derives it. + +Handle each failing case independently. Obsolete a contract that no longer exists with `setSmartTestCaseObsolete` (it carries forward across re-records). + +### Phase A4 — Verify (ON THE BRANCH) + +After a Case 1 source edit, rebuild the image first (`docker build -t `; Case A/B/C skip this). Then replay on the branch, **piping output through `tail`/`grep`** so the ~10–40k-token log doesn't enter context: + +```bash +keploy cloud replay --app --branch-name --cluster --replay-source smart-set --freezeTime --disableReportUpload=false --strict-failure 2>&1 \ + | tail -n 60 | grep -E "Total test|Failed Testcases|test passed|test failed|FAIL|ERROR|debug bundle|View test report" +``` + +`--disableReportUpload=false` (writes the `/tr` row), `--strict-failure` (don't let response-divergent-but-mock-drifted tests demote to OBSOLETE/PASSED), `--cluster` (from cached `origin.clusterName`), `--freezeTime` (if faketime) are all mandatory. If still failing, re-enter Phase A2 with the new `test_run_id`; cap retries at 3. **Sanity gate:** post-fix `total_tests` must be ≥ pre-fix (a "0 passed/0 failed" drop = coverage regression). + +**ALL cases failing "connection reset"/status 0 = a stale leftover replay container on the app port, not a code bug:** `docker ps -aq --filter publish= | xargs -r docker rm -f` and retry — don't rebuild or revert. + +### Phase A5 — Report and STOP (the dev merges) + +``` +### Diagnosis +| Case (test-N) | Class | Cause | +| --- | --- | --- | +| | 1 / A / B / C | | + +### Fixes applied (on branch ``) +- (Case 1) Edited `` — ``; rebuilt image; replay green. +- (Case A) `updateSmartTestCase` on `` — set noise on `` OR updated `respBody`. +- (Case B) `updateSmartTestCase` on `` — shape edit re-keyed schema_ref (or resolved a SchemaRefConflict by obsoleting ``). +- (Case C) `upsertSmartMock` on `` (or relinked `mockReferences`). +- `keploy cloud replay` on branch: `

/` tests passed. + +### Next step for you +- Review the branch diff. **Merge the branch to main yourself (or let CI merge it)** — I make changes on the branch only. +- (Case 1) Review the code edit at `` and push. +- (SchemaRefConflict left for you) Two cases share a contract — decide which to keep, or merge-into-existing in the UI. + +Branch diff: https://app.keploy.io/api-testing/branch-diff?appId=&branchId= +Run report: https://app.keploy.io/tr/?appId= +``` + +--- + +## Routine B — "add new keploy smart tests for my changes" + +### Phase B1 — Identify the changes + +`git diff origin/main...HEAD --name-only` (fall back to `main...HEAD`). Filter to HTTP-handler files; list each added/modified endpoint's `method`+`path`. Nothing handler-relevant → tell the dev, stop. + +### Phase B2 — Capture traffic + +Pre-flight: discover the run command (keploy.yml `command`, a `run.sh`, Makefile/compose/Procfile/package.json/README), start the app, curl an endpoint to confirm it serves, stop it cleanly. Make sure it's fully stopped before recording. Then `keploy record -c "" --sync --disable-mapping=false` (both flags MANDATORY — without `--disable-mapping=false` the test→mock mapping is silently dropped at upload), drive ONE realistic curl per new/changed endpoint, stop recording. Recording lands at `keploy/test-set-N/`. + +Background the recorder and stop it by PID (`echo $! > /tmp/rec.pid` … `kill -INT $(cat /tmp/rec.pid)`) — not `pkill -f 'keploy record'` (matches your own shell). If a re-replay doesn't reflect a change, clear the local cache: `rm -f keploy/smart-set/.mock-cache.json keploy/smart-set/mocks.yaml`. + +### Phase B3 — Upload as a smart set, onto the branch + +```bash +keploy upload test-set --app --branch --test-set keploy/test-set-N --smart-test-set --name +``` + +`--smart-test-set` (NOTE the dashes) ingests new contracts as `imported-*` on the branch, deduped by `schema_ref` (existing contracts are SKIPPED, reported per-case — re-uploading the same recording is a no-op, not a duplicate). Requires `--branch` and an app with smart test set enabled (else the server rejects with "requires the app to have smart test set enabled"). + +### Phase B4 — Validate (on the branch) + +```bash +keploy cloud replay --app --branch-name --cluster --replay-source smart-set --freezeTime --disableReportUpload=false 2>&1 \ + | tail -n 60 | grep -E "Total test|test passed|test failed|FAIL|ERROR|View test report" +``` + +If anything fails, enter Routine A from Phase A2. + +### Phase B5 — Report and STOP (the dev merges) -- Merging or rebasing the branch to main. -- Editing on `main` (every mutation needs `branch_id`). -- Treating a `SchemaRefConflict` as retryable. -- Re-recording a shape-changed contract but forgetting to delete the stale case. -- Editing handler code on a Case A/B/C (contract-change) failure. ``` +### Captured (imported on branch ``) +| Endpoint | Imported (test-N) | Skipped (already in set) | +| --- | --- | --- | +| | | | + +### Replay +

/ tests passed on branch ``. + +### Next step +Review the branch diff and **merge to main yourself (or via CI)** — merge reconciles the `imported-*` cases to stable `test-N`. I import & validate on the branch only. + +Branch diff: https://app.keploy.io/api-testing/branch-diff?appId=&branchId= +Run report: https://app.keploy.io/tr/?appId= +``` + +--- + +## When you MAY ask the dev (and only then) + +- PAT missing/invalid (auth error from the MCP tool) → ask for a fresh PAT. +- `git rev-parse --abbrev-ref HEAD` returns `HEAD`/non-zero → ask for a branch name ONCE. +- `listApps` ambiguous and unnarrowable by compose-service → list candidates, ask ONCE. +- Pre-flight cannot start the app → name the command + error, ask ONCE. +- A `SchemaRefConflict` where BOTH cases are legitimately distinct contracts → surface it; "merge into existing" is the dev's call. + +## Anti-patterns (refuse these) + +- **Merging or rebasing the branch to main** — not your job, not an MCP tool. Fix on the branch, report, let the dev/CI merge. +- **Editing on `main`** — every smart-set mutation needs `branch_id`; main is read-only. +- **Treating a `SchemaRefConflict` as retryable** — re-applying the same shape edit conflicts again. Resolve the twin (obsolete/delete) or report. +- **Re-recording a shape-changed contract but forgetting to delete the stale old case** — re-record lands a NEW `schema_ref`; the old case keeps failing as a duplicate. Delete/obsolete it after re-record (rule 5). +- **Editing handler code on a Case A/B/C failure** — contract changed → fix test data on the branch, not source. +- **Uploading fixtures from another branch / inventing a PAT or branch name.** +- **`search_tools` for a tool you already know** (use `get_tool_schema`), **`keploy --help`/`--version` dumps**, **reading `keploy/cloud-debug.log` or the local `keploy/` cache** (throwaway; use `getTestReportFull` for structured data). +```` + +## Scenarios + +Every scenario starts with the same two-prompt UX and ends at a **verified branch** (you or CI merge afterward). The variable part is the decision the agent makes in the middle — one per case it classifies. + +### Scenario 1 — App regression (the agent refuses to edit the test) + +A refactor accidentally emptied a response value that the app should still return (the downstream mock still carries the correct data). + +> _"my keploy smart-set replay is failing, please analyze and fix it."_ + +The agent reads the report, sees the field dropped with the **schema unchanged**, and confirms the downstream mock still provides the value → classifies a **regression**. It makes **no** smart-set edit — it flags the app bug (and fixes the source when the change was clearly unintended), rebuilds, and re-runs. Baking the bad value into the golden test is explicitly refused. + +### Scenario 2 — Value drift on the response body (Case A) + +You changed a response value on purpose (e.g. a version string now carries a `v` prefix). + +> _"my keploy smart-set replay is failing, please analyze and fix it."_ + +The recorded value differs from the app's new output with no regression signal → **Case A (value)**. The agent calls `updateSmartTestCase` with the new `respBody`; `schema_ref` is **preserved** (identity unchanged). Replay re-runs green. + +### Scenario 3 — Non-deterministic field (Case A, noise) + +A response field now varies every request (a per-request id, a deploy tag, a timestamp). + +> _"my keploy smart-set replay is failing, please analyze and fix it."_ + +The agent recognizes the value is non-deterministic (not a real contract change) → **Case A (noise)**. It calls `updateSmartTestCase` adding the field's JSONPath to `noiseJson` — rather than pinning a single value that would just re-fail next run — and **preserves** any existing noise entries. + +### Scenario 4 — Shape drift (Case B) + +You renamed (or removed / retyped) a response field. + +> _"the keploy smart-set pipeline is failing, please analyze and fix it."_ + +The contract **structure** moved → **Case B (shape)**. The agent calls `updateSmartTestCase` with the new `responseJson`; `schema_ref` is **recomputed** for the new shape. Replay re-runs green. + +> Note: purely **additive** fields do not fail replay — the response comparison is _expected ⊆ actual_ — so a shape scenario is triggered by a rename, removal, or type change, not by adding a new field. + +### Scenario 5 — Shape drift that collides with an existing case (`SchemaRefConflict`) + +A shape edit produces a `schema_ref` that already belongs to another case (often an obsolete twin). + +The agent does **not** blind-retry. It reads the conflict's `existingCaseId`, resolves the stale twin on the branch (`setSmartTestCaseObsolete` or `deleteSmartTestCase`), then re-applies the shape edit. If both cases are legitimately distinct, it reports the conflict for you to resolve rather than guessing. + +### Scenario 6 — Mock drift (Case C) + +The app's outbound call to a dependency changed, so the recorded mock no longer matches (`mock_mismatches` in the report). + +> _"my keploy smart-set replay is failing, please analyze and fix it."_ + +The agent confirms the app logic is intact and only the downstream contract moved → **Case C (mock)**. It calls `upsertSmartMock` to update the mock on the branch (or re-records when the request shape itself changed). Replay re-runs green. + +### Scenario 7 — Endpoint removed (obsolete) + +You deprecated an endpoint; its recorded contract can no longer replay. + +> _"my keploy smart-set replay is failing, please analyze and fix it."_ + +The agent recognizes an intentional removal (not a routing regression) → `setSmartTestCaseObsolete` on the branch. The case leaves the replayable set (kept in history), it is **not** deleted, and replay re-runs green. + +### Scenario 8 — Adding tests for a new endpoint (Routine B) + +You added a new route. + +> _"Add new keploy smart tests for my changes."_ + +The agent records one realistic request for the new endpoint, uploads it onto the branch (`keploy upload test-set --smart-test-set --branch ` — new `schema_ref`s ingest as `imported-*`, deduplicated), and validates with a branch replay. On merge, `imported-*` reconciles to a stable `test-N`. + +--- + +Across every scenario you only ever said one of two sentences. You push your code change (and, for Scenario 1, the agent's app-side fix); CI replays the branch on the PR and `keploy cloud branch-merge` lands the test data on `main` at merge. + +## Related workflows + +- **[Developer + LLM Workflow with Keploy Proxy](/docs/quickstart/k8s-proxy-llm-workflow)** — the same agent-driven loop for the broader Keploy Proxy flow: per-editor MCP config, the full playbook with phase-by-phase Routines A/B (report projection, exact report format), and worked scenarios. Start there for the general setup; use **this** page for the smart-set substrate specifics (`schema_ref` identity, branch-native edits, `SchemaRefConflict`). +- **[Developer Workflow with Keploy Proxy](/docs/quickstart/k8s-proxy-developer-workflow)** — the same flow done **manually** (dashboard / CLI, no agent): create a branch, edit test cases and mocks, replay, open a PR, merge. diff --git a/versioned_sidebars/version-4.0.0-sidebars.json b/versioned_sidebars/version-4.0.0-sidebars.json index 5390c359b..c1df70d9a 100644 --- a/versioned_sidebars/version-4.0.0-sidebars.json +++ b/versioned_sidebars/version-4.0.0-sidebars.json @@ -56,7 +56,6 @@ "keploy-cloud/auto-test-generation", "keploy-cloud/deduplication", "keploy-cloud/static-deduplication", - "keploy-cloud/smart-set-agent", "keploy-cloud/cloud-replay", { "type": "category", @@ -152,6 +151,7 @@ "quickstart/k8s-proxy", "quickstart/k8s-proxy-developer-workflow", "quickstart/k8s-proxy-llm-workflow", + "keploy-cloud/smart-set-agent", "running-keploy/k8s-proxy-daemonset-architecture" ] },