-
Notifications
You must be signed in to change notification settings - Fork 2
[1/3] docs: add domain glossaries for Proxy and EQL Mapper #420
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Context Map | ||
|
|
||
| CipherStash Proxy is a Cargo workspace. Each context below owns a distinct domain | ||
| vocabulary. Read the `CONTEXT.md` for the context you're working in before exploring | ||
| its code. | ||
|
|
||
| Per-context `CONTEXT.md` files are created lazily by `/domain-modeling` as terms get | ||
| resolved β a missing one is expected, not a gap to fill upfront. | ||
|
|
||
| | Context | Path | Domain | | ||
| |---|---|---| | ||
| | Proxy | [`packages/cipherstash-proxy/`](./packages/cipherstash-proxy/CONTEXT.md) | PostgreSQL wire protocol, connection and message handling, client authentication, TLS, ZeroKMS key management, encrypt/decrypt of column values | | ||
| | EQL Mapper | [`packages/eql-mapper/`](./packages/eql-mapper/CONTEXT.md) | SQL parsing, type inference over statements, schema analysis, transformation rules that rewrite plaintext SQL into EQL v2 operations | | ||
| | Integration | `packages/cipherstash-proxy-integration/` | End-to-end test harness β container fixtures, encrypted-scenario coverage across the proxy and mapper together | | ||
| | Showcase | `packages/showcase/` | Healthcare example data model demonstrating EQL v2 encryption with realistic relationships | | ||
|
|
||
| `packages/eql-mapper-macros/` is proc-macro support for EQL Mapper, not a context of its | ||
| own β treat it as part of the EQL Mapper context. | ||
|
|
||
| ## Relationships | ||
|
|
||
| - **Proxy β EQL Mapper**: Proxy loads the database schema and hands it over with each | ||
| column marked native or encrypted; EQL Mapper returns a type-checked, rewritten | ||
| statement. Proxy then reads the per-node EQL term shapes back out to decide how to | ||
| encrypt each value. | ||
| - **Identity across the seam**: EQL Mapper's `TableColumn` and Proxy's `Identifier` are | ||
| the same `table.column` pair under two names. That pair is the only key joining a typed | ||
| AST node to its encryption config. | ||
| - **Capability across the seam β currently broken.** Proxy marks every encrypted column | ||
| with *all* `EqlTrait`s (`packages/cipherstash-proxy/src/proxy/schema/manager.rs:146`) | ||
| because it derives them from the PostgreSQL column type alone. The column encrypt | ||
| config, which knows the SEM terms actually configured, is loaded by a separate manager | ||
| that never meets the schema loader. EQL Mapper's bound checking is therefore | ||
| unreachable in production: a query needing `Ord` on a column with no ordering SEM term | ||
| type-checks cleanly and fails later. Read `EqlTraits` on a column as *intended* | ||
| capability, not observed. | ||
|
|
||
| ## Shared vocabulary | ||
|
|
||
| Terms defined once for the whole system live here rather than in any one context. | ||
|
|
||
| - **EQL v2** β Encrypt Query Language; the SQL-level encoding that makes encrypted | ||
| values searchable. | ||
| - **`eql_v2_encrypted`** β the PostgreSQL column type holding an encrypted value. | ||
| - **ZeroKMS** β CipherStash's key management service, which the proxy calls to encrypt | ||
| and decrypt. | ||
| - **Keyset** β the ZeroKMS key collection a workspace encrypts against. | ||
| - **SEM (searchable encrypted metadata)** β the encrypted material stored alongside a | ||
| value that makes some operation on it possible without decryption. A **SEM term** is | ||
| one such piece of metadata; ORE, unique, match and SteVec are SEM **types**. | ||
| - **EqlTrait** β a *capability* an encrypted column has: equality, ordering, token | ||
| match, JSON traversal, containment. Several SEM types can satisfy the same trait, so | ||
| the relationship is many-to-one: SEM terms are the storage, traits are what the storage | ||
| buys you. | ||
|
|
||
| Say **capability/trait** when you mean what a query may do, and **SEM term** when you | ||
| mean what is written to the column. Do not call either an **index** β that word is | ||
| reserved for a PostgreSQL index. | ||
|
|
||
| > The EQL config JSON and `cipherstash_client`'s `ColumnConfig` still spell SEM terms | ||
| > `indexes` / `IndexType`. That spelling is a wire and storage format shared with EQL | ||
| > and customer-authored config, so it is not ours to rename unilaterally β prefer SEM | ||
| > in prose and new code, and read `indexes` in those payloads as "SEM terms". | ||
|
|
||
| ## Cross-context term collisions | ||
|
|
||
| Terms that mean different things depending on where you are. Resolve them against your | ||
| own context. | ||
|
|
||
| - **Type** β a PostgreSQL wire-protocol type in Proxy; an inferred SQL expression type | ||
| in EQL Mapper. | ||
| - **Index** β means a PostgreSQL index in *both* contexts (EQL Mapper's only use of the | ||
| word is GIN indexes). Where it appears meaning searchable encryption β the `indexes` | ||
| key in EQL config, `IndexType`, "Unknown Index Term" β read it as **SEM term** and see | ||
| the shared vocabulary above. | ||
| - **Column** β an encrypted column's runtime config in Proxy; a schema column or a | ||
| projection column in EQL Mapper. Also `DataColumn` (a wire value) and | ||
| `RowDescriptionField` (a result descriptor) in Proxy. | ||
| - **Statement** β a type-analysed statement in Proxy; the parsed AST or a | ||
| `TypeCheckedStatement` in EQL Mapper. Four senses are live in Proxy alone. | ||
| - **Term** β the *shape* of an encrypted payload (`EqlTerm`) in EQL Mapper; a piece of | ||
| searchable encrypted metadata in Proxy and EQL config. Qualify as *EQL term* or | ||
| *SEM term*. | ||
| - **Projection** β a type in EQL Mapper's lattice; in Proxy, a positional list where a | ||
| missing entry means "native, do not encrypt". | ||
| - **Session** β banned in Proxy (see its `CONTEXT.md`); PostgreSQL owns the word, and the | ||
| code has used it for both a connection and a single statement. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # Domain Docs | ||
|
|
||
| How the engineering skills should consume this repo's domain documentation when | ||
| exploring the codebase. | ||
|
|
||
| This is a **multi-context** repo β a Cargo workspace whose packages carry distinct | ||
| domain vocabularies. | ||
|
|
||
| ## Before exploring, read these | ||
|
|
||
| - **`CONTEXT-MAP.md`** at the repo root β it points at one `CONTEXT.md` per context. | ||
| Read each one relevant to the topic. | ||
| - **`docs/adr/`** β system-wide architectural decisions. | ||
| - **`packages/<name>/docs/adr/`** β context-scoped decisions for that package. | ||
|
|
||
| If any of these files don't exist, **proceed silently**. Don't flag their absence; | ||
| don't suggest creating them upfront. The `/domain-modeling` skill (reached via | ||
| `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms | ||
| or decisions actually get resolved. | ||
|
|
||
| Note that `ARCHITECTURE.md` at the repo root already describes system structure. It is | ||
| not a glossary β read it for orientation, but domain *terms* belong in `CONTEXT.md`. | ||
|
|
||
| ## File structure | ||
|
|
||
| This repo is a Cargo workspace, so contexts live under `packages/`, not `src/`: | ||
|
|
||
| ``` | ||
| / | ||
| βββ CONTEXT-MAP.md | ||
| βββ docs/adr/ β system-wide decisions | ||
| βββ packages/ | ||
| βββ cipherstash-proxy/ | ||
| β βββ CONTEXT.md | ||
| β βββ docs/adr/ β context-specific decisions | ||
| βββ eql-mapper/ | ||
| βββ CONTEXT.md | ||
| βββ docs/adr/ | ||
| ``` | ||
|
|
||
| ## Use the glossary's vocabulary | ||
|
|
||
| When your output names a domain concept (in an issue title, a refactor proposal, a | ||
| hypothesis, a test name), use the term as defined in the relevant `CONTEXT.md`. Don't | ||
| drift to synonyms the glossary explicitly avoids. | ||
|
|
||
| If the concept you need isn't in the glossary yet, that's a signal β either you're | ||
| inventing language the project doesn't use (reconsider) or there's a real gap (note it | ||
| for `/domain-modeling`). | ||
|
|
||
| Watch for terms that mean different things in different contexts. "Type", for example, | ||
| means a PostgreSQL wire type in `cipherstash-proxy` and an inferred SQL type in | ||
| `eql-mapper`. That divergence is exactly why this repo is multi-context β resolve the | ||
| term against the context you're working in, not the other one. | ||
|
|
||
| ## Flag ADR conflicts | ||
|
|
||
| If your output contradicts an existing ADR, surface it explicitly rather than silently | ||
| overriding: | ||
|
|
||
| > _Contradicts ADR-0007 (event-sourced orders) β but worth reopening becauseβ¦_ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| # Issue Tracker | ||
|
|
||
| Issues for this repo live in **Linear**, not GitHub Issues. | ||
|
|
||
| - **Workspace:** CipherStash | ||
| - **Team:** Product Engineering | ||
| - **Issue key prefix:** `CIP-` (e.g. `CIP-3233`) | ||
|
|
||
| GitHub (`cipherstash/proxy`) holds code, branches, and pull requests. It is not the | ||
| issue tracker. Do not run `gh issue create` for this repo. | ||
|
|
||
| ## How to read and write issues | ||
|
|
||
| Two access paths exist. **Prefer the CLI for writes.** | ||
|
|
||
| ### `linear-cli` (preferred) | ||
|
|
||
| The `linear` command (`@schpet/linear-cli`, or `npx @schpet/linear-cli`) β see the | ||
| `linear-cli` skill for full usage. | ||
|
|
||
| ```bash | ||
| linear issue list | ||
| linear issue view CIP-3233 | ||
| linear issue create --title "..." --description-file /path/to/desc.md | ||
| linear issue update CIP-3233 --status "In Progress" | ||
| ``` | ||
|
|
||
| Always pass markdown via `--description-file` / `--body-file` rather than inline | ||
| arguments β inline content produces literal `\n` sequences in the Linear UI. | ||
|
|
||
| ### Linear MCP tools (fallback, and for querying) | ||
|
|
||
| `mcp__claude_ai_Linear__*` β good for search and read-heavy work | ||
| (`list_issues`, `get_issue`, `save_issue`). Note these are | ||
| interactively authenticated and **may be unavailable in headless or cron runs**, so | ||
| anything scripted should prefer the CLI. | ||
|
|
||
| ## Statuses | ||
|
|
||
| The Product Engineering workflow: | ||
|
|
||
| | Status | Type | | ||
| |---|---| | ||
| | `Backlog` | backlog | | ||
| | `Todo` | unstarted | | ||
| | `In Progress` | started | | ||
| | `In Review` | started | | ||
| | `Done` | completed | | ||
| | `Canceled` | canceled | | ||
| | `Duplicate` | duplicate | | ||
|
|
||
| ## How the skills should use this | ||
|
|
||
| - **`to-tickets`** β create one Linear issue per tracer-bullet ticket in team | ||
| Product Engineering, starting in `Todo`. Express blocking edges as **native Linear | ||
| blocked-by relations**, not as prose in the description and not as files under | ||
| `.scratch/`. Any issue whose blockers are `Done` is grabbable. | ||
| - **`implement`** β take an issue by its `CIP-####` key. Move it to `In Progress` when | ||
| work starts, and `In Review` when the PR opens. Leave the move to `Done` to the | ||
| human merging. | ||
| - **`to-spec`** β link the resulting spec back to the originating issue as a comment or | ||
| attachment on the Linear issue. | ||
| - **`code-review`** β the "Spec" axis reads the originating `CIP-####` issue for intent. | ||
|
|
||
| ## Linking PRs to issues | ||
|
|
||
| Reference the issue key in the PR title or body (e.g. `CIP-3233`) so Linear's GitHub | ||
| integration attaches the PR to the issue automatically. | ||
|
|
||
| ## PRs as a request surface | ||
|
|
||
| **Off.** External pull requests are not treated as incoming triage requests for this | ||
| repo. Flip this flag if that changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Triage Labels | ||
|
|
||
| The skills speak in terms of five canonical triage roles. This file maps those roles to | ||
| the actual label strings used in this repo's issue tracker (Linear, team Product | ||
| Engineering β see [issue-tracker.md](./issue-tracker.md)). | ||
|
|
||
| | Label in mattpocock/skills | Label in our tracker | Meaning | | ||
| | -------------------------- | -------------------- | ---------------------------------------- | | ||
| | `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue | | ||
| | `needs-info` | `needs-info` | Waiting on reporter for more information | | ||
| | `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent | | ||
| | `ready-for-human` | `ready-for-human` | Requires human implementation | | ||
| | `wontfix` | `wontfix` | Will not be actioned | | ||
|
|
||
| When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the | ||
| corresponding label string from this table. | ||
|
|
||
| Edit the right-hand column to match whatever vocabulary you actually use. | ||
|
|
||
| ## Creating the labels | ||
|
|
||
| Only `wontfix` currently exists on the Product Engineering team. The other four are | ||
| created lazily β the first time `triage` needs one, create it on the team rather than | ||
| failing. These are workspace-level Linear labels, unparented; do not nest them under the | ||
| existing `Product` / `R&D` / `Issue type` groups. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| # Proxy | ||
|
|
||
| Speaks the PostgreSQL wire protocol to client applications, hands their SQL to EQL Mapper | ||
| for type checking and rewriting, encrypts and decrypts column values through ZeroKMS, and | ||
| forwards everything to the real database. It owns everything about *connections* and | ||
| *ciphertext*; it owns nothing about type inference. | ||
|
|
||
| ## Language | ||
|
|
||
| ### Connections and statements | ||
|
|
||
| **Connection**: | ||
| One client application's TCP connection to Proxy, from startup to termination. All | ||
| per-client state hangs off it. | ||
| _Avoid_: session β PostgreSQL already owns that word for something else, and this | ||
| codebase has used it for both a connection and a single statement. | ||
|
|
||
| **Context**: | ||
| The mutable state belonging to one connection β its prepared statements, portals, | ||
| pending describes and executes, and keyset. | ||
|
|
||
| **Frontend** / **Backend**: | ||
| The two halves of the proxy pipeline by direction of travel: Frontend carries client to | ||
| database, Backend carries database to client. These are *not* the PostgreSQL protocol's | ||
| frontend/backend roles, and Backend is not the database. | ||
|
|
||
| **Statement**: | ||
| A type-analysed SQL statement β its param columns, projection columns, literal columns | ||
| and PostgreSQL param type OIDs. The parsed AST and EQL Mapper's `TypeCheckedStatement` | ||
| are different things; do not call either of them a Statement here. | ||
|
|
||
| **Portal**: | ||
| A prepared statement bound to parameter values, ready to execute. Either `Encrypted`, | ||
| carrying the analysed statement, or `Passthrough` when nothing in it is encrypted. | ||
|
|
||
| **Statement metrics scope**: | ||
| The measurement window around a single statement β opened at Parse or Query, closed when | ||
| the statement completes. Many of these occur per connection. | ||
| _Avoid_: session (`start_session`, `SessionId` and the | ||
| `..._statements_session_duration_seconds` metric all use this sense and are misnamed). | ||
|
|
||
| **Passthrough**: | ||
| Traffic forwarded without encryption or rewriting. Three independent conditions produce | ||
| it and they are not interchangeable: no encrypt config is loaded, mapping is disabled by | ||
| configuration, or `SET CIPHERSTASH.UNSAFE_DISABLE_MAPPING` was issued on this connection. | ||
| Name the condition rather than saying "passthrough" alone. | ||
|
|
||
| ### Encryption | ||
|
|
||
| **Column**: | ||
| An encrypted column as this crate sees it β its `Identifier`, its column config, its | ||
| resolved PostgreSQL type, and the EQL term shape it takes in the current statement. | ||
| _Avoid_: confusing with EQL Mapper's schema/projection columns, or with `DataColumn` | ||
| (a value on the wire) and `RowDescriptionField` (a result descriptor). | ||
|
|
||
| **Identifier**: | ||
| The `table.column` pair that keys the encrypt config. EQL Mapper calls the same thing a | ||
| `TableColumn`. | ||
|
|
||
| **Encrypt credentials**: | ||
| The CipherStash client credentials Proxy authenticates to ZeroKMS with β client id, | ||
| client key, default keyset. Read from `[encrypt]` / `CS_ENCRYPT__*`. | ||
| _Avoid_: `EncryptConfig`, which currently names this *and* an unrelated struct. | ||
|
|
||
| **Column encrypt config**: | ||
| The per-column encryption configuration β which columns are encrypted and with which SEM | ||
| terms β read from the EQL config table in the *database* at runtime, and reloaded when | ||
| DDL is observed. | ||
| _Avoid_: `EncryptConfig`; also "encrypt schema", despite `ReloadCommand::EncryptSchema` | ||
| naming the reload of this thing. | ||
|
|
||
| **Client id**: | ||
| Ambiguous β always qualify. The **connection number** is the incrementing counter stamped | ||
| on log lines to correlate one connection's traffic. The **CipherStash client id** is the | ||
| credential UUID from `CS_ENCRYPT__CLIENT_ID`. They share a name and a field spelling and | ||
| are otherwise unrelated. | ||
|
|
||
| **Keyset**: | ||
| The ZeroKMS collection of keys a workspace encrypts against. Addressed by UUID or by | ||
| name, and selectable per connection via `SET CIPHERSTASH.KEYSET_ID` / `KEYSET_NAME`. | ||
|
|
||
| **Scoped cipher**: | ||
| A cipher bound to exactly one keyset. Cached per keyset, so switching keyset switches | ||
| cipher. | ||
|
|
||
| **Plaintext**: | ||
| A value in its typed, pre-encryption form. The boundary type between PostgreSQL wire | ||
| values and ZeroKMS. | ||
|
|
||
| **EQL ciphertext**: | ||
| An encrypted value in the shape stored in the database. What encryption produces and | ||
| decryption consumes. | ||
|
|
||
| **Store** / **Query** operation: | ||
| The two directions of encryption. Storing writes a value with all its SEM terms; querying | ||
| produces only the term needed to match. The statement's EQL term shape decides which. | ||
|
|
||
| ### Control plane | ||
|
|
||
| **`SET CIPHERSTASH.*`**: | ||
| Proxy's in-band control API, intercepted rather than forwarded β `KEYSET_ID`, | ||
| `KEYSET_NAME`, `UNSAFE_DISABLE_MAPPING`. These are the supported spellings; log messages | ||
| that print `CIPHERSTASH.DISABLE_MAPPING` are wrong. | ||
|
|
||
| **Reload**: | ||
| Re-reading state from the database after observed DDL. Two independent things reload: the | ||
| database schema, and the column encrypt config. | ||
|
|
||
| ## Note on `session` | ||
|
|
||
| This glossary bans `session` because the code uses it for two different things and | ||
| PostgreSQL uses it for a third. The identifiers `SessionId`, `start_session`, | ||
| `finish_session` and the Prometheus metric name all still carry it. Read them as | ||
| *statement metrics scope*; the doc comments on `Frontend` and `Backend` that say "session | ||
| context" mean *connection*. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
π Maintainability & Code Quality | π‘ Minor | β‘ Quick win
Specify a language for the directory-tree fence.
Line 28 triggers markdownlint MD040. Use
```textfor the tree diagram.Proposed fix
π§° Tools
πͺ markdownlint-cli2 (0.23.0)
[warning] 28-28: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
π€ Prompt for AI Agents
Source: Linters/SAST tools