Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This document describes the internal architecture of CipherStash Proxy. It's int

## Overview

CipherStash Proxy sits between an application and PostgreSQL. It intercepts SQL statements over the PostgreSQL wire protocol, determines which columns are encrypted, rewrites queries to use [EQL v2](https://github.com/cipherstash/encrypt-query-language) operations, encrypts literals and parameters, forwards the transformed query to PostgreSQL, and decrypts results before returning them to the application.
CipherStash Proxy sits between an application and PostgreSQL. It intercepts SQL statements over the PostgreSQL wire protocol, determines which columns are encrypted, rewrites queries to use [EQL v3](https://github.com/cipherstash/encrypt-query-language) operations, encrypts literals and parameters, forwards the transformed query to PostgreSQL, and decrypts results before returning them to the application.

The two most interesting pieces of the system are:

Expand Down Expand Up @@ -116,10 +116,12 @@ The current rules:

| Rule | What it does |
|---|---|
| `CastLiteralsAsEncrypted` | Replaces plaintext literals with `eql_v2.cast_as_encrypted(ciphertext)` |
| `CastParamsAsEncrypted` | Wraps parameter placeholders (`$1`, `$2`, ...) with encrypted casts |
| `RewriteContainmentOps` | Transforms `col @> val` to `eql_v2.jsonb_contains(col, val)` |
| `RewriteStandardSqlFnsOnEqlTypes` | Rewrites `min()`, `max()`, `jsonb_path_query()` etc. to `eql_v2.*` equivalents |
| `RewriteEqlComparisonOps` | Rewrites scalar comparisons: `col <op> x` β†’ `eql_v3.<term>(col) <op> eql_v3.<term>(x)` (term chosen from the column's domain: `eq_term`/`ord_term`/`ord_term_ore`) |
| `RewriteEqlMatchOps` | Rewrites `LIKE`/`ILIKE`/`@@` to `eql_v3.match_term(a) @> eql_v3.match_term(b)` |
| `RewriteContainmentOps` | Rewrites JSON `@>`/`<@` to `eql_v3.jsonb_contains`/`jsonb_contained_by`, and `->`/`->>` to `eql_v3."->"`/`"->>"` |
| `RewriteStandardSqlFnsOnEqlTypes` | Rewrites `min()`, `max()`, `jsonb_path_query()` etc. to their `eql_v3.*` counterparts (`count()` stays native) |
| `CastLiteralsAsEncrypted` | Replaces plaintext literals with the ciphertext cast to the column's v3 domain (`::public.eql_v3_*`) or, for a query operand, its query twin (`::eql_v3.query_*`) |
| `CastParamsAsEncrypted` | Wraps parameter placeholders (`$1`, `$2`, ...) with the same v3 domain casts |
| `PreserveEffectiveAliases` | Maintains column aliases through transformations |
| `FailOnPlaceholderChange` | Postcondition check that prepared statement placeholders weren't corrupted |

Expand Down Expand Up @@ -191,7 +193,7 @@ When encrypting values for a statement, many columns may be `NULL` or non-encryp

## Schema Management

The proxy discovers the database schema at startup and reloads it periodically. Schema loading queries PostgreSQL's `information_schema` to discover tables and columns, then checks `eql_v2_configuration` to determine which columns are encrypted and what index types they support.
The proxy discovers the database schema at startup and reloads it periodically. Schema loading queries PostgreSQL's `information_schema` to discover tables and columns, including each column's domain type. EQL v3 columns are self-configuring domain types (e.g. `eql_v3_text_search`), so both the type-checker's capability view and the encrypt config are inferred from that single schema load β€” the column's domain name determines which columns are encrypted, their token type, and their searchable capabilities. There is no `eql_v2_configuration` table.

Schema state is stored behind an `ArcSwap`, which provides lock-free reads with atomic updates. This means query processing never blocks on a schema reload β€” readers always get a consistent snapshot.

Expand Down
19 changes: 12 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ CipherStash Proxy is a PostgreSQL proxy that provides **transparent, searchable

Key capabilities:
- Zero-change SQL queries - applications connect to Proxy instead of directly to PostgreSQL
- EQL v2 (Encrypt Query Language) for searchable encryption using CipherStash ZeroKMS
- EQL v3 (Encrypt Query Language) for searchable encryption using CipherStash ZeroKMS
- Support for encrypted equality, comparison, ordering, and grouping operations
- Written in Rust for performance with strongly-typed SQL statement mapping

Expand All @@ -34,14 +34,14 @@ Key capabilities:
- Language-specific integration tests (Python, Go)

**Showcase (`packages/showcase/`):**
- Healthcare data model demonstrating EQL v2 encryption
- Healthcare data model demonstrating EQL v3 encryption
- Example of realistic encrypted application with foreign keys and relationships

### Request Flow

1. Application connects to Proxy (port 6432) using standard PostgreSQL protocol
2. Proxy intercepts SQL statements and uses EQL Mapper to analyze query structure
3. For encrypted columns, Proxy transforms SQL using EQL v2 operations
3. For encrypted columns, Proxy transforms SQL using EQL v3 operations
4. Encrypted queries are sent to actual PostgreSQL database
5. Results are decrypted before returning to application

Expand Down Expand Up @@ -166,12 +166,17 @@ Available targets: `DEVELOPMENT`, `AUTHENTICATION`, `CONFIG`, `CONTEXT`, `ENCODI

## EQL Integration

CipherStash Proxy uses EQL v2 for searchable encryption. Key concepts:
CipherStash Proxy uses EQL v3 for searchable encryption. Key concepts:

- **Plaintext columns** - standard PostgreSQL data types
- **Encrypted columns** - use `eql_v2_encrypted` type in schema
- **Searchable operations** - equality, comparison, ordering work on encrypted data
- **Index support** - ORE (Order Revealing Encryption) and Match indexes for performance
- **Encrypted columns** - use a self-configuring EQL v3 domain type in the schema
(e.g. `eql_v3_text_search`, `eql_v3_integer_ord`, `eql_v3_json_search`); the
domain encodes the token type and searchable capabilities, so there is no
separate `add_search_config` call
- **Searchable operations** - equality, comparison, ordering, text match, and JSON
traversal work on encrypted data, gated by the column's domain capability
- **Index support** - functional indexes over the term-extraction functions
(e.g. `CREATE INDEX ON t (eql_v3.ord_term(col))`)

EQL is automatically downloaded and installed during setup. Use `CS_EQL_PATH` to point to local EQL development version.

Expand Down
29 changes: 17 additions & 12 deletions CONTEXT-MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ resolved β€” a missing one is expected, not a gap to fill upfront.
| Context | Path | Domain |
|---|---|---|
| Proxy | [`packages/cipherstash-proxy/`](./packages/cipherstash-proxy/CONTEXT.md) | PostgreSQL wire protocol, connection and message handling, client authentication, TLS, ZeroKMS key management, encrypt/decrypt of column values |
| EQL Mapper | [`packages/eql-mapper/`](./packages/eql-mapper/CONTEXT.md) | SQL parsing, type inference over statements, schema analysis, transformation rules that rewrite plaintext SQL into EQL v2 operations |
| EQL Mapper | [`packages/eql-mapper/`](./packages/eql-mapper/CONTEXT.md) | SQL parsing, type inference over statements, schema analysis, transformation rules that rewrite plaintext SQL into EQL v3 operations |
| Integration | `packages/cipherstash-proxy-integration/` | End-to-end test harness β€” container fixtures, encrypted-scenario coverage across the proxy and mapper together |
| Showcase | `packages/showcase/` | Healthcare example data model demonstrating EQL v2 encryption with realistic relationships |
| Showcase | `packages/showcase/` | Healthcare example data model demonstrating EQL v3 encryption with realistic relationships |

`packages/eql-mapper-macros/` is proc-macro support for EQL Mapper, not a context of its
own β€” treat it as part of the EQL Mapper context.
Expand All @@ -26,22 +26,27 @@ own β€” treat it as part of the EQL Mapper context.
- **Identity across the seam**: EQL Mapper's `TableColumn` and Proxy's `Identifier` are
the same `table.column` pair under two names. That pair is the only key joining a typed
AST node to its encryption config.
- **Capability across the seam β€” currently broken.** Proxy marks every encrypted column
with *all* `EqlTrait`s (`packages/cipherstash-proxy/src/proxy/schema/manager.rs:146`)
because it derives them from the PostgreSQL column type alone. The column encrypt
config, which knows the SEM terms actually configured, is loaded by a separate manager
that never meets the schema loader. EQL Mapper's bound checking is therefore
unreachable in production: a query needing `Ord` on a column with no ordering SEM term
type-checks cleanly and fails later. Read `EqlTraits` on a column as *intended*
capability, not observed.
- **Capability across the seam.** Under EQL v3 each encrypted column is a self-configuring
domain type (e.g. `eql_v3_text_search`) whose typname encodes both the token type and the
searchable capabilities. The schema loader resolves that domain to a `DomainIdentity` and
the exact `EqlTraits` it supports (`packages/cipherstash-proxy/src/proxy/schema/manager.rs`,
via `proxy/schema/eql_domains.rs`), so the traits handed to EQL Mapper are *observed*, not
a blanket grant. EQL Mapper's bound checking is therefore effective in production: a query
needing `Ord` on a column whose domain has no ordering capability is rejected at type-check
time. The encrypt config is derived from the same domain type
(`proxy/encrypt_config/from_domain.rs`), so schema view and encrypt config no longer
disagree.

## Shared vocabulary

Terms defined once for the whole system live here rather than in any one context.

- **EQL v2** β€” Encrypt Query Language; the SQL-level encoding that makes encrypted
- **EQL v3** β€” Encrypt Query Language; the SQL-level encoding that makes encrypted
values searchable.
- **`eql_v2_encrypted`** β€” the PostgreSQL column type holding an encrypted value.
- **EQL v3 domain types** β€” encrypted columns are self-configuring PostgreSQL DOMAINs over
`jsonb` (e.g. `eql_v3_text_search`, `eql_v3_int8_ord`, `eql_v3_json_search`). The domain's
typname encodes the token type and the searchable capabilities, replacing EQL v2's opaque
`eql_v2_encrypted` type plus a separate `eql_v2_configuration` table.
- **ZeroKMS** β€” CipherStash's key management service, which the proxy calls to encrypt
and decrypt.
- **Keyset** β€” the ZeroKMS key collection a workspace encrypts against.
Expand Down
4 changes: 2 additions & 2 deletions docs/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -479,7 +479,7 @@ For example:

## Unknown Column <a id='encrypt-unknown-column'></a>

The column has an encrypted type (PostgreSQL `eql_v2_encrypted` type ) with no encryption configuration.
The column has an encrypted type (an EQL v3 encrypted domain type, e.g. `eql_v3_text_search`) with no encryption configuration.

Without the configuration, Cipherstash Proxy does not know how to encrypt the column.
Any data is unprotected and unencrypted.
Expand All @@ -506,7 +506,7 @@ Column 'column_name' in table 'table_name' has no Encrypt configuration

## Unknown Table <a id='encrypt-unknown-table'></a>

The table has one or more encrypted columns (PostgreSQL `eql_v2_encrypted` type ) with no encryption configuration.
The table has one or more encrypted columns (an EQL v3 encrypted domain type, e.g. `eql_v3_text_search`) with no encryption configuration.

Without the configuration, Cipherstash Proxy does not know how to encrypt the column.
Any data is unprotected and unencrypted.
Expand Down
73 changes: 19 additions & 54 deletions docs/how-to/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ You can also install EQL by running [the installation script](https://github.com
Once you have installed EQL, you can see what version is installed by querying the database:

```sql
SELECT eql_v2.version();
SELECT eql_v3.version();
```

This will output the version of EQL installed.
Expand All @@ -182,79 +182,44 @@ This will output the version of EQL installed.

In your existing PostgreSQL database, you store your data in tables and columns.
Those columns have types like `integer`, `text`, `timestamp`, and `boolean`.
When storing encrypted data in PostgreSQL with Proxy, you use a special column type called `eql_v2_encrypted`, which is [provided by EQL](#setting-up-the-database-schema).
`eql_v2_encrypted` is a container column type that can be used for any type of encrypted data you want to store or search, whether they are numbers (`int`, `small_int`, `big_int`), text (`text`), dates and times (`date`. `timestamp`), or booleans (`boolean`).
When storing encrypted data in PostgreSQL with Proxy, you use one of EQL's **encrypted domain types**, which are [provided by EQL](#setting-up-the-database-schema).

Create a table with an encrypted column for `email`:
In EQL v3 these domain types are **self-configuring**: the type you choose for a column both marks it as encrypted *and* declares which searches it supports. This replaces EQL v2's model of a single opaque `eql_v2_encrypted` container type plus a separate `eql_v2.add_search_config` call per index β€” there is no separate index-configuration step, and no `eql_v2_configuration` table.

Domain types follow the naming pattern `eql_v3_<token>_<capability>`:

- **Storage only** β€” `eql_v3_text`, `eql_v3_integer`, `eql_v3_bigint`, `eql_v3_date`, `eql_v3_boolean`, and so on store an encrypted value that can be read back but not searched. (`boolean` is always storage-only: a two-value column would leak its distribution under any index.)
- **Ordering and range** β€” the `_ord` suffix (e.g. `eql_v3_integer_ord`, `eql_v3_date_ord`) adds ordering (`ORDER BY`) and range comparisons (`<`, `<=`, `>`, `>=`), and also supports equality (`=`). This is the recommended default and uses CLLW-OPE ordering.
- **Ordering and range via ORE** β€” the `_ord_ore` suffix (e.g. `eql_v3_integer_ord_ore`) is an alternative ordering scheme backed by block-ORE. Choose `_ord` or `_ord_ore` for a column, not both.
- **Full text search** β€” for `text`, `eql_v3_text_search` bundles equality, ordering, and fuzzy `LIKE`/`ILIKE` match in one type; `eql_v3_text_search_ore` is the ORE-backed variant, and `eql_v3_text_ord_ope` provides OPE ordering.
- **Encrypted JSON** β€” `eql_v3_json_search` stores encrypted JSON with SteVec containment (`@>`, `<@`) and path (`->`, `->>`) search. See [Searchable JSON](../reference/searchable-json.md).

Create a `users` table with an encrypted, fully-searchable `email` column:

```sql
CREATE TABLE users (
id SERIAL PRIMARY KEY,
email eql_v2_encrypted
email eql_v3_text_search
)
```

This creates a `users` table with two columns:

- `id`, an autoincrementing integer column that is the primary key for the record
- `email`, a `eql_v2_encrypted` column
- `email`, an encrypted `text` column that supports equality (`=`), ordering, and fuzzy `LIKE`/`ILIKE` matching β€” because it uses the `eql_v3_text_search` domain type

There are important differences between the plaintext columns you've traditionally used in PostgreSQL and encrypted columns with CipherStash Proxy:

- **Plaintext columns can be searched if they don't have an index**, albeit with the performance cost of a full table scan.
- **Encrypted columns cannot be searched without an encrypted index**, and the encrypted indexes you define determine what kind of searches you can do on encrypted data.

In the previous step we created a table with an encrypted column, but without any encrypted indexes.

Now you can add an encrypted index for that encrypted column:

```sql
SELECT eql_v2.add_search_config(
'users',
'email',
'unique',
'text'
);
```

This statement adds a `unique` index for the `email` column in the `users` table, which has an underlying data type of `text`.

`unique` indexes are used to find records with columns with unique values, like with the `=` operator.

There are other types of encrypted indexes you can use on `text` data:

```sql
SELECT eql_v2.add_search_config(
'users',
'email',
'match',
'text'
);

SELECT eql_v2.add_search_config(
'users',
'email',
'ore',
'text'
);

SELECT eql_v2.add_search_config(
'users',
'email',
'ope',
'text'
);
```
- **An encrypted column can only be searched in the ways its domain type allows.** Choose the domain type up front to match the queries you need: `eql_v3_text` if you only store and retrieve the value, `eql_v3_text_search` if you also need to compare and match it.

The first SQL statement adds a `match` index, which is used for partial matches with `LIKE`.
The second SQL statement adds an `ore` index, which is used for ordering with `ORDER BY` and range comparisons (`<`, `<=`, `>`, `>=`).
The third SQL statement adds an `ope` index, which supports the same range and ordering operators as `ore`.
If you only needed equality on `email` β€” for example a lookup by exact address β€” you could store it as a scalar ordering type such as `eql_v3_text_ord_ope`, or use `eql_v3_text_search` when you also want partial matches with `LIKE`.

`ore` and `ope` are alternatives for range and ordering queries β€” add one or the other to a column, not both. `ore` is the recommended default. `ope` produces ciphertexts that sort under PostgreSQL's native byte ordering, which makes ordering and range scans cheaper, but as an order-preserving scheme it reveals more about the relative order of stored values than `ore` does. Choose based on your performance and threat-model requirements; see the [EQL `INDEX` documentation](https://github.com/cipherstash/encrypt-query-language/blob/main/docs/reference/INDEX.md) for the full tradeoffs.
`_ord` (CLLW-OPE) produces ciphertexts that sort under PostgreSQL's native byte ordering, which makes ordering and range scans cheaper, but as an order-preserving scheme it reveals more about the relative order of stored values than the block-ORE `_ord_ore` variant does. Choose based on your performance and threat-model requirements; see the [EQL `INDEX` documentation](https://github.com/cipherstash/encrypt-query-language/blob/main/docs/reference/INDEX.md) for the full tradeoffs.


> [!IMPORTANT]
> Adding, updating, or deleting encrypted indexes on columns that already contain encrypted data will not re-index that data. To use the new indexes, you must `SELECT` the data out of the column, and `UPDATE` it again.
> The searches an encrypted column supports are fixed by its domain type. To change them you change the column's type (e.g. `ALTER TABLE users ALTER COLUMN email TYPE eql_v3_text_search`), and any data already stored must be re-encrypted under the new type β€” `SELECT` it out of the column and `UPDATE` it back β€” before the new capabilities apply to it.

To learn how to use encrypted indexes for other encrypted data types like `text`, `int`, `boolean`, `date`, and `jsonb`, see the [EQL documentation](https://github.com/cipherstash/encrypt-query-language/blob/main/docs/reference/INDEX.md).

Expand Down
4 changes: 2 additions & 2 deletions docs/reference/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -470,10 +470,10 @@ The parameter is always scoped to the connection `SESSION` - mapping is only eve

CipherStash Proxy and EQL do provide some protection against writing plaintext into and reading plaintext from encrypted columns.

Always use `eql_v2.add_encrypted_constraint(table, column)` when defining encrypted columns to ensure plaintext data cannot be written.
In EQL v3 this protection is built into the column's domain type: an `eql_v3_*` domain is a checked `jsonb` domain, so PostgreSQL rejects a plaintext value that is not a valid encrypted payload. There is no separate `eql_v2.add_encrypted_constraint` call to apply β€” defining the column with an `eql_v3_*` type is sufficient.

Unmapped `SELECT` statements should always return the encrypted payload.
If the constraint has been applied, unmapped `INSERT`/`UPDATE` statements should return a PostgreSQL type error.
Unmapped `INSERT`/`UPDATE` statements that try to write plaintext should return a PostgreSQL type error.


### Disable mapping
Expand Down
Loading
Loading