diff --git a/CLAUDE.md b/CLAUDE.md index 61267dcc..6030345e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -237,4 +237,18 @@ For significant releases, write an `ANNOUNCEMENT.md` for the GitHub Discussions > [!IMPORTANT] > Critical information users must know. ``` -4. After copying content to GitHub Discussions, delete `ANNOUNCEMENT.md` from the repo \ No newline at end of file +4. After copying content to GitHub Discussions, delete `ANNOUNCEMENT.md` from the repo + +## Agent skills + +### Issue tracker + +Issues are tracked in **Linear** (team: Product Engineering, key `CIP-`), not GitHub Issues. GitHub holds code and PRs only. See `docs/agents/issue-tracker.md`. + +### Triage labels + +**Defaults** — the five canonical roles, each label string equal to its name. Only `wontfix` exists in Linear today; the rest are created on demand. See `docs/agents/triage-labels.md`. + +### Domain docs + +**Multi-context** — the root `CONTEXT-MAP.md` points at per-package `CONTEXT.md` files under `packages/*/`. See `docs/agents/domain.md`. diff --git a/CONTEXT-MAP.md b/CONTEXT-MAP.md new file mode 100644 index 00000000..31c69ffa --- /dev/null +++ b/CONTEXT-MAP.md @@ -0,0 +1,87 @@ +# Context Map + +CipherStash Proxy is a Cargo workspace. Each context below owns a distinct domain +vocabulary. Read the `CONTEXT.md` for the context you're working in before exploring +its code. + +Per-context `CONTEXT.md` files are created lazily by `/domain-modeling` as terms get +resolved — a missing one is expected, not a gap to fill upfront. + +| Context | Path | Domain | +|---|---|---| +| Proxy | [`packages/cipherstash-proxy/`](./packages/cipherstash-proxy/CONTEXT.md) | PostgreSQL wire protocol, connection and message handling, client authentication, TLS, ZeroKMS key management, encrypt/decrypt of column values | +| EQL Mapper | [`packages/eql-mapper/`](./packages/eql-mapper/CONTEXT.md) | SQL parsing, type inference over statements, schema analysis, transformation rules that rewrite plaintext SQL into EQL v2 operations | +| Integration | `packages/cipherstash-proxy-integration/` | End-to-end test harness — container fixtures, encrypted-scenario coverage across the proxy and mapper together | +| Showcase | `packages/showcase/` | Healthcare example data model demonstrating EQL v2 encryption with realistic relationships | + +`packages/eql-mapper-macros/` is proc-macro support for EQL Mapper, not a context of its +own — treat it as part of the EQL Mapper context. + +## Relationships + +- **Proxy → EQL Mapper**: Proxy loads the database schema and hands it over with each + column marked native or encrypted; EQL Mapper returns a type-checked, rewritten + statement. Proxy then reads the per-node EQL term shapes back out to decide how to + encrypt each value. +- **Identity across the seam**: EQL Mapper's `TableColumn` and Proxy's `Identifier` are + the same `table.column` pair under two names. That pair is the only key joining a typed + AST node to its encryption config. +- **Capability across the seam — currently broken.** Proxy marks every encrypted column + with *all* `EqlTrait`s (`packages/cipherstash-proxy/src/proxy/schema/manager.rs:146`) + because it derives them from the PostgreSQL column type alone. The column encrypt + config, which knows the SEM terms actually configured, is loaded by a separate manager + that never meets the schema loader. EQL Mapper's bound checking is therefore + unreachable in production: a query needing `Ord` on a column with no ordering SEM term + type-checks cleanly and fails later. Read `EqlTraits` on a column as *intended* + capability, not observed. + +## Shared vocabulary + +Terms defined once for the whole system live here rather than in any one context. + +- **EQL v2** — Encrypt Query Language; the SQL-level encoding that makes encrypted + values searchable. +- **`eql_v2_encrypted`** — the PostgreSQL column type holding an encrypted value. +- **ZeroKMS** — CipherStash's key management service, which the proxy calls to encrypt + and decrypt. +- **Keyset** — the ZeroKMS key collection a workspace encrypts against. +- **SEM (searchable encrypted metadata)** — the encrypted material stored alongside a + value that makes some operation on it possible without decryption. A **SEM term** is + one such piece of metadata; ORE, unique, match and SteVec are SEM **types**. +- **EqlTrait** — a *capability* an encrypted column has: equality, ordering, token + match, JSON traversal, containment. Several SEM types can satisfy the same trait, so + the relationship is many-to-one: SEM terms are the storage, traits are what the storage + buys you. + + Say **capability/trait** when you mean what a query may do, and **SEM term** when you + mean what is written to the column. Do not call either an **index** — that word is + reserved for a PostgreSQL index. + + > The EQL config JSON and `cipherstash_client`'s `ColumnConfig` still spell SEM terms + > `indexes` / `IndexType`. That spelling is a wire and storage format shared with EQL + > and customer-authored config, so it is not ours to rename unilaterally — prefer SEM + > in prose and new code, and read `indexes` in those payloads as "SEM terms". + +## Cross-context term collisions + +Terms that mean different things depending on where you are. Resolve them against your +own context. + +- **Type** — a PostgreSQL wire-protocol type in Proxy; an inferred SQL expression type + in EQL Mapper. +- **Index** — means a PostgreSQL index in *both* contexts (EQL Mapper's only use of the + word is GIN indexes). Where it appears meaning searchable encryption — the `indexes` + key in EQL config, `IndexType`, "Unknown Index Term" — read it as **SEM term** and see + the shared vocabulary above. +- **Column** — an encrypted column's runtime config in Proxy; a schema column or a + projection column in EQL Mapper. Also `DataColumn` (a wire value) and + `RowDescriptionField` (a result descriptor) in Proxy. +- **Statement** — a type-analysed statement in Proxy; the parsed AST or a + `TypeCheckedStatement` in EQL Mapper. Four senses are live in Proxy alone. +- **Term** — the *shape* of an encrypted payload (`EqlTerm`) in EQL Mapper; a piece of + searchable encrypted metadata in Proxy and EQL config. Qualify as *EQL term* or + *SEM term*. +- **Projection** — a type in EQL Mapper's lattice; in Proxy, a positional list where a + missing entry means "native, do not encrypt". +- **Session** — banned in Proxy (see its `CONTEXT.md`); PostgreSQL owns the word, and the + code has used it for both a connection and a single statement. diff --git a/docs/agents/domain.md b/docs/agents/domain.md new file mode 100644 index 00000000..abe7c818 --- /dev/null +++ b/docs/agents/domain.md @@ -0,0 +1,61 @@ +# Domain Docs + +How the engineering skills should consume this repo's domain documentation when +exploring the codebase. + +This is a **multi-context** repo — a Cargo workspace whose packages carry distinct +domain vocabularies. + +## Before exploring, read these + +- **`CONTEXT-MAP.md`** at the repo root — it points at one `CONTEXT.md` per context. + Read each one relevant to the topic. +- **`docs/adr/`** — system-wide architectural decisions. +- **`packages//docs/adr/`** — context-scoped decisions for that package. + +If any of these files don't exist, **proceed silently**. Don't flag their absence; +don't suggest creating them upfront. The `/domain-modeling` skill (reached via +`/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms +or decisions actually get resolved. + +Note that `ARCHITECTURE.md` at the repo root already describes system structure. It is +not a glossary — read it for orientation, but domain *terms* belong in `CONTEXT.md`. + +## File structure + +This repo is a Cargo workspace, so contexts live under `packages/`, not `src/`: + +```text +/ +├── CONTEXT-MAP.md +├── docs/adr/ ← system-wide decisions +└── packages/ + ├── cipherstash-proxy/ + │ ├── CONTEXT.md + │ └── docs/adr/ ← context-specific decisions + └── eql-mapper/ + ├── CONTEXT.md + └── docs/adr/ +``` + +## Use the glossary's vocabulary + +When your output names a domain concept (in an issue title, a refactor proposal, a +hypothesis, a test name), use the term as defined in the relevant `CONTEXT.md`. Don't +drift to synonyms the glossary explicitly avoids. + +If the concept you need isn't in the glossary yet, that's a signal — either you're +inventing language the project doesn't use (reconsider) or there's a real gap (note it +for `/domain-modeling`). + +Watch for terms that mean different things in different contexts. "Type", for example, +means a PostgreSQL wire type in `cipherstash-proxy` and an inferred SQL type in +`eql-mapper`. That divergence is exactly why this repo is multi-context — resolve the +term against the context you're working in, not the other one. + +## Flag ADR conflicts + +If your output contradicts an existing ADR, surface it explicitly rather than silently +overriding: + +> _Contradicts ADR-000N (its title) — but worth reopening because…_ diff --git a/docs/agents/issue-tracker.md b/docs/agents/issue-tracker.md new file mode 100644 index 00000000..ed19719a --- /dev/null +++ b/docs/agents/issue-tracker.md @@ -0,0 +1,73 @@ +# Issue Tracker + +Issues for this repo live in **Linear**, not GitHub Issues. + +- **Workspace:** CipherStash +- **Team:** Product Engineering +- **Issue key prefix:** `CIP-` (e.g. `CIP-3233`) + +GitHub (`cipherstash/proxy`) holds code, branches, and pull requests. It is not the +issue tracker. Do not run `gh issue create` for this repo. + +## How to read and write issues + +Two access paths exist. **Prefer the CLI for writes.** + +### `linear-cli` (preferred) + +The `linear` command (`@schpet/linear-cli`, or `npx @schpet/linear-cli`) — see the +`linear-cli` skill for full usage. + +```bash +linear issue list +linear issue view CIP-3233 +linear issue create --title "..." --description-file /path/to/desc.md +linear issue update CIP-3233 --status "In Progress" +``` + +Always pass markdown via `--description-file` / `--body-file` rather than inline +arguments — inline content produces literal `\n` sequences in the Linear UI. + +### Linear MCP tools (fallback, and for querying) + +`mcp__claude_ai_Linear__*` — good for search and read-heavy work +(`list_issues`, `get_issue`, `save_issue`). Note these are +interactively authenticated and **may be unavailable in headless or cron runs**, so +anything scripted should prefer the CLI. + +## Statuses + +The Product Engineering workflow: + +| Status | Type | +|---|---| +| `Backlog` | backlog | +| `Todo` | unstarted | +| `In Progress` | started | +| `In Review` | started | +| `Done` | completed | +| `Canceled` | canceled | +| `Duplicate` | duplicate | + +## How the skills should use this + +- **`to-tickets`** — create one Linear issue per tracer-bullet ticket in team + Product Engineering, starting in `Todo`. Express blocking edges as **native Linear + blocked-by relations**, not as prose in the description and not as files under + `.scratch/`. Any issue whose blockers are `Done` is grabbable. +- **`implement`** — take an issue by its `CIP-####` key. Move it to `In Progress` when + work starts, and `In Review` when the PR opens. Leave the move to `Done` to the + human merging. +- **`to-spec`** — link the resulting spec back to the originating issue as a comment or + attachment on the Linear issue. +- **`code-review`** — the "Spec" axis reads the originating `CIP-####` issue for intent. + +## Linking PRs to issues + +Reference the issue key in the PR title or body (e.g. `CIP-3233`) so Linear's GitHub +integration attaches the PR to the issue automatically. + +## PRs as a request surface + +**Off.** External pull requests are not treated as incoming triage requests for this +repo. Flip this flag if that changes. diff --git a/docs/agents/triage-labels.md b/docs/agents/triage-labels.md new file mode 100644 index 00000000..dccc2505 --- /dev/null +++ b/docs/agents/triage-labels.md @@ -0,0 +1,23 @@ +# Triage Labels + +The skills speak in terms of five canonical triage roles. This file maps those roles to +the actual label strings used in this repo's issue tracker (Linear, team Product +Engineering — see [issue-tracker.md](./issue-tracker.md)). + +| Label in mattpocock/skills | Label in our tracker | Meaning | +| -------------------------- | -------------------- | ---------------------------------------- | +| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue | +| `needs-info` | `needs-info` | Waiting on reporter for more information | +| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent | +| `ready-for-human` | `ready-for-human` | Requires human implementation | +| `wontfix` | `wontfix` | Will not be actioned | + +When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the +corresponding label string from this table. + +## Creating the labels + +Only `wontfix` currently exists on the Product Engineering team. The other four are +created lazily — the first time `triage` needs one, create it on the team rather than +failing. These are workspace-level Linear labels, unparented; do not nest them under the +existing `Product` / `R&D` / `Issue type` groups. diff --git a/packages/cipherstash-proxy/CONTEXT.md b/packages/cipherstash-proxy/CONTEXT.md new file mode 100644 index 00000000..9289b0a0 --- /dev/null +++ b/packages/cipherstash-proxy/CONTEXT.md @@ -0,0 +1,115 @@ +# Proxy + +Speaks the PostgreSQL wire protocol to client applications, hands their SQL to EQL Mapper +for type checking and rewriting, encrypts and decrypts column values through ZeroKMS, and +forwards everything to the real database. It owns everything about *connections* and +*ciphertext*; it owns nothing about type inference. + +## Language + +### Connections and statements + +**Connection**: +One client application's TCP connection to Proxy, from startup to termination. All +per-client state hangs off it. +_Avoid_: session — PostgreSQL already owns that word for something else, and this +codebase has used it for both a connection and a single statement. + +**Context**: +The mutable state belonging to one connection — its prepared statements, portals, +pending describes and executes, and keyset. + +**Frontend** / **Backend**: +The two halves of the proxy pipeline by direction of travel: Frontend carries client to +database, Backend carries database to client. These are *not* the PostgreSQL protocol's +frontend/backend roles, and Backend is not the database. + +**Statement**: +A type-analysed SQL statement — its param columns, projection columns, literal columns +and PostgreSQL param type OIDs. The parsed AST and EQL Mapper's `TypeCheckedStatement` +are different things; do not call either of them a Statement here. + +**Portal**: +A prepared statement bound to parameter values, ready to execute. Either `Encrypted`, +carrying the analysed statement, or `Passthrough` when nothing in it is encrypted. + +**Statement metrics scope**: +The measurement window around a single statement — opened at Parse or Query, closed when +the statement completes. Many of these occur per connection. +_Avoid_: session (`start_session`, `SessionId` and the +`..._statements_session_duration_seconds` metric all use this sense and are misnamed). + +**Passthrough**: +Traffic forwarded without encryption or rewriting. Three independent conditions produce +it and they are not interchangeable: no encrypt config is loaded, mapping is disabled by +configuration, or `SET CIPHERSTASH.UNSAFE_DISABLE_MAPPING` was issued on this connection. +Name the condition rather than saying "passthrough" alone. + +### Encryption + +**Column**: +An encrypted column as this crate sees it — its `Identifier`, its column config, its +resolved PostgreSQL type, and the EQL term shape it takes in the current statement. +_Avoid_: confusing with EQL Mapper's schema/projection columns, or with `DataColumn` +(a value on the wire) and `RowDescriptionField` (a result descriptor). + +**Identifier**: +The `table.column` pair that keys the encrypt config. EQL Mapper calls the same thing a +`TableColumn`. + +**Encrypt credentials**: +The CipherStash client credentials Proxy authenticates to ZeroKMS with — client id, +client key, default keyset. Read from `[encrypt]` / `CS_ENCRYPT__*`. +_Avoid_: `EncryptConfig`, which currently names this *and* an unrelated struct. + +**Column encrypt config**: +The per-column encryption configuration — which columns are encrypted and with which SEM +terms — read from the EQL config table in the *database* at runtime, and reloaded when +DDL is observed. +_Avoid_: `EncryptConfig`; also "encrypt schema", despite `ReloadCommand::EncryptSchema` +naming the reload of this thing. + +**Client id**: +Ambiguous — always qualify. The **connection number** is the incrementing counter stamped +on log lines to correlate one connection's traffic. The **CipherStash client id** is the +credential UUID from `CS_ENCRYPT__CLIENT_ID`. They share a name and a field spelling and +are otherwise unrelated. + +**Keyset**: +The ZeroKMS collection of keys a workspace encrypts against. Addressed by UUID or by +name, and selectable per connection via `SET CIPHERSTASH.KEYSET_ID` / `KEYSET_NAME`. + +**Scoped cipher**: +A cipher bound to exactly one keyset. Cached per keyset, so switching keyset switches +cipher. + +**Plaintext**: +A value in its typed, pre-encryption form. The boundary type between PostgreSQL wire +values and ZeroKMS. + +**EQL ciphertext**: +An encrypted value in the shape stored in the database. What encryption produces and +decryption consumes. + +**Store** / **Query** operation: +The two directions of encryption. Storing writes a value with all its SEM terms; querying +produces only the term needed to match. The statement's EQL term shape decides which. + +### Control plane + +**`SET CIPHERSTASH.*`**: +Proxy's in-band control API, intercepted rather than forwarded — `KEYSET_ID`, +`KEYSET_NAME`, `UNSAFE_DISABLE_MAPPING`. These are the supported spellings; log messages +that print `CIPHERSTASH.DISABLE_MAPPING` are wrong. + +**Reload**: +Re-reading state from the database after observed DDL. Two independent things reload: the +database schema, and the column encrypt config. + +## Note on `session` + +This glossary bans `session` because the code uses it for two different things and +PostgreSQL uses it for a third. The identifiers `SessionId`, `start_session`, +`finish_session` and the Prometheus metric name all still carry it. Read them as +*statement metrics scope*; the doc comments on `Frontend` and `Backend` that say "session +context" mean *connection*. diff --git a/packages/eql-mapper/CONTEXT.md b/packages/eql-mapper/CONTEXT.md new file mode 100644 index 00000000..31e532b2 --- /dev/null +++ b/packages/eql-mapper/CONTEXT.md @@ -0,0 +1,102 @@ +# EQL Mapper + +Parses SQL, infers a type for every node, and rewrites statements that touch encrypted +columns into their EQL v2 equivalents. It knows nothing about the PostgreSQL wire +protocol, ZeroKMS, or ciphertext — it reasons about types and rewrites syntax. + +## Language + +### The type system + +**Type**: +A node's inferred type in this crate's own lattice — either a resolved `Value`, a +unification `Var`, or an `Associated` type. Never a PostgreSQL wire type. +_Avoid_: SQL type, Postgres type (those belong to Proxy). + +**Value**: +A *resolved* type — one of `Eql`, `Native`, `Array`, `Projection`, `SetOf`. It means +"value type", not a datum. The SQL AST's literal node is also called `Value`; when both +are in scope, alias the AST one. +_Avoid_: using bare `Value` for a SQL literal. + +**Native**: +A column or expression that is not encrypted. Native types satisfy *every* `EqlTrait` +bound — that is a deliberate escape hatch for the type checker, not a claim about the +database. + +**EqlValue**: +The identity of an encrypted column paired with the capabilities configured for it — +a `TableColumn` plus `EqlTraits`. + +**EqlTrait**: +A class of operation an encrypted value supports: `Eq`, `Ord`, `TokenMatch`, `JsonLike`, +`Contain`. A **capability**, not a storage structure — several SEM terms can satisfy one +trait. See the shared vocabulary in `CONTEXT-MAP.md`. +_Avoid_: index, index type (those name the storage, not the capability). + +**EqlTraits**: +A set of `EqlTrait`s. Read in two opposite directions depending on position: as +*required bounds* on a `Var`, and as *implemented capabilities* on an `EqlValue`. +_Avoid_: features, bounds, trait_impls as distinct concepts — they are all this one type. + +**EqlTerm**: +The shape of encrypted payload a node needs — `Full`, `Partial`, `JsonAccessor`, +`JsonPath`, `Tokenized`. `Partial` is a subset of `Full`, not an unresolved type. +_Avoid_: "term" meaning an encrypted search term; that sense belongs to Proxy. + +**Projection**: +The resultset shape of a statement or subquery — an ordered list of optionally-named +column types. + +**alias**: +The *effective name* of a projection column, which is the user's alias when one was +written and the underlying schema column name otherwise. Identifier resolution matches +against it. +_Avoid_: reading `alias` as "user-supplied alias only". + +### Naming and resolution + +**Relation**: +Anything you can select from — a table, view, CTE, or derived subquery — reduced to a +name plus a projection type. + +**Scope**: +The lexical frame holding the relations visible at a point in the statement. + +**TableColumn**: +The `table.column` identity of an encrypted column. It is the join key to Proxy's +encrypt config. +_Avoid_: identifier (Proxy uses that for its own type). + +**Schema**: +The database's tables and columns as this crate sees them, each column marked `Native` +or `Eql`. Loaded from the live database, not from a migration file. + +**Overlay**: +A per-transaction mask over the loaded `Schema` recording DDL that has run but not +committed. A table is either shadowed by a new definition or hidden as dropped. + +### Output + +**TypeCheckedStatement**: +The result of type checking — the projection, params, literals and node types, plus the +entry point that applies transformations. + +**TransformationRule**: +A composable mutation of the typed AST that rewrites plaintext SQL into EQL v2 +operations. Tuples of rules are themselves rules. +_Avoid_: using "rule" for a typing rule; those are operator/function signature +declarations. + +**Param**: +A `$N` placeholder position in the statement. Its PostgreSQL type OID is Proxy's +concern, not this crate's. + +## Known model gap + +The trait-bounds machinery here is complete — `satisfy_bounds` and `UnsatisfiedBounds` +exist and work — but Proxy currently feeds every encrypted column `EqlTraits::all()` +(`packages/cipherstash-proxy/src/proxy/schema/manager.rs:146`) rather than the traits its +configured SEM terms actually provide. So bound violations cannot be caught at type-check +time in production today. Treat `EqlTraits` on an `EqlValue` as *intended* capability, +not observed capability, until that join exists.