From d2374779656af46fa798254565f1017ced888c54 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 24 Jun 2026 13:10:50 +0300 Subject: [PATCH 1/5] docs(design-proposals): add external database exposure model Record the external-exposure design for managed databases: Gateway API TLS-passthrough routed by SNI on native ports, collapsing N per-database LoadBalancer IPs onto one tenant Gateway IP, with end-to-end TLS by reusing the operator-issued certificate (passthrough does not terminate). Assisted-By: Claude Signed-off-by: Aleksei Sviridkin --- .../external-database-exposure/README.md | 227 ++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 design-proposals/external-database-exposure/README.md diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md new file mode 100644 index 0000000..ee64e5d --- /dev/null +++ b/design-proposals/external-database-exposure/README.md @@ -0,0 +1,227 @@ + +# External database exposure via Gateway API TLS-passthrough (SNI) and end-to-end TLS + +- **Title:** `External database exposure via Gateway API TLS-passthrough (SNI) and end-to-end TLS` +- **Author(s):** `@lexfrei` +- **Date:** `2026-06-24` +- **Status:** Draft + +## Overview + +Today every managed database a tenant exposes externally gets its own `LoadBalancer` Service, and therefore its own public IP. A tenant with ten external databases burns ten IPs, and there is no managed end-to-end TLS story that does not terminate somewhere in the middle. This proposal exposes managed databases through the Gateway API TLS-passthrough listeners that Cozystack already operates, routed by SNI on each engine's native port. Multiple databases per tenant collapse onto a single LoadBalancer IP, and because a passthrough listener never terminates TLS, the certificate the external client validates is byte-identical to the operator-issued server certificate the database already presents inside the cluster — end-to-end TLS with no second certificate, no re-encryption hop, and no private key held at the edge. + +This is the design-proposal artifact required by `cozystack/cozystack#2816`, and it records the decision that issue asks for. The trade-off that issue frames is CNI mesh encryption (datapath lock-in) versus application-level TLS: this proposal chooses **application-level, operator-owned TLS carried through a non-terminating gateway** for the external leg — no edge termination, no second certificate, no private key at the edge, and no dependency on a particular CNI. The other half of that framing — in-cluster (east-west) pod-to-pod encryption, which never leaves the cluster and is necessarily datapath-specific — is recorded and executed separately under `cozystack/cozystack#2977` (PR `cozystack/cozystack#2984`); it complements this proposal rather than competing with it. It is the external-exposure half of epic `cozystack/cozystack#2811`; the certificate/PKI half is covered by the sibling proposal `design-proposals/unified-tls-pki`, on which this one depends for the trust-anchor object. + +## Scope and related proposals + +- **Depends on:** `design-proposals/unified-tls-pki` — provides the `-ca-cert` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. +- **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). +- **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). + +All repository paths below refer to the `cozystack/cozystack` repository. + +## Context + +### TLS-passthrough already exists + +Cozystack already runs Gateway API TLS-passthrough for layer-7 system services. The `TenantGateway` controller (`internal/controller/tenantgateway/reconciler.go`) renders, for each entry in `TenantGateway.Spec.TLSPassthroughServices`, a listener named `tls-` on port 443 with `Protocol: TLS`, `Mode: Passthrough`, hostname `.`, and `AllowedRoutes` restricted to `TLSRoute`. The default services are `api`, `vm-exportproxy`, and `cdi-uploadproxy`, each with a `TLSRoute` that attaches by `sectionName: tls-` (for example `packages/system/cozystack-api/templates/api-tlsroute.yaml`). The platform runs Cilium 1.19.3 with Gateway API v1.5.1 CRDs and GatewayClass `cilium`; `TLSRoute` and passthrough are supported. + +The mechanism this proposal needs is therefore already in production for layer-7 services. The work is to extend it to databases, which speak on native (non-443) ports. + +### Each external database burns its own IP today + +When a database chart sets `external: true`, it provisions a dedicated `Service` of `type: LoadBalancer`: postgres `-external-write` on 5432 (`packages/apps/postgres/templates/external-svc.yaml`), redis `-external-lb` on 6379 (`packages/apps/redis/templates/service.yaml`), mongodb `-external` on 27017 (`packages/apps/mongodb/templates/external-svc.yaml`), kafka on 9094 plus a LoadBalancer per broker (`packages/apps/kafka/templates/kafka.yaml`), and mariadb via a `primaryService` of `type: LoadBalancer` (`packages/apps/mariadb/templates/mariadb.yaml`). Each is a separate IP from the shared MetalLB (or Cilium LB-IPAM) pool; there is no per-tenant IP isolation and no SNI multiplexing. + +### The certificate hooks already exist + +The external hostname is `.<_namespace.host>`, where `_namespace.host` is the tenant apex. The SAN-injection hook already exists on `main` for postgres: it adds the hostname to the CNPG `Cluster` via `spec.certificates.serverAltDNSNames` in `packages/apps/postgres/templates/db.yaml`, gated by the `tls.enabled` tri-state in `_tls.tpl` (which defaults on when `external` is true). The other engines acquire the same SAN hook through the per-app TLS series tracked by the `unified-tls-pki` proposal — on `main` today redis and mariadb carry no TLS templates at all, and their certificate/SAN support lives in the open PRs `cozystack/cozystack#2729` and `cozystack/cozystack#2680`. The trust anchor (`ca.crt`) is delivered to the tenant through that same `unified-tls-pki` contract. So this proposal builds on hooks that are present for postgres and arriving for the rest — it does not invent new ones. + +### The one-IP-per-tenant ceiling + +Multiple per-tenant Gateways cannot share a single LoadBalancer IP on current Cilium: every Gateway claims `443/TCP`, so `lbipam.cilium.io/sharing-key` is inactive on the port collision (`packages/extra/gateway/README.md`; upstream cilium#21270, cilium#42756). Within a single Gateway, a parent and its inheriting children share one IP. `Gateway.spec.listeners` is hard-capped at 64. The practical consequence: the unit of IP sharing is one tenant Gateway, and a tenant's database fan-out draws from the 64-listener budget shared with the existing http/https/https-apex and per-child-apex wildcard listeners. + +### The problem + +Two concrete problems, both solved by SNI-passthrough plus certificate reuse: + +1. **IPv4 scarcity.** N externally-exposed databases cost N public IPs per tenant. This does not scale, and IPs are the scarce resource. +2. **No managed end-to-end TLS.** Exposing a database externally today either lacks a managed-TLS-to-the-client story or would require terminating at the edge — which means a second certificate, a re-encryption hop, and the edge holding a private key for a database it does not own. + +## Goals + +- Multiple external databases per tenant share a single LoadBalancer IP, distinguished by SNI on their native ports. +- End-to-end TLS using the **same** operator-issued certificate client-side and pod-side: no edge termination, no edge-held private key, no second certificate. +- Trust established through the existing `ca.crt`-only object from `unified-tls-pki` — no new PKI, issuer, or rotation machinery. +- A minimal, auditable API surface that reuses the existing passthrough listener and SAN-injection hooks. +- Per-engine honesty: ship what fits the model, defer what does not, and say why. + +### Non-goals + +- **East-west / pod-to-pod CNI encryption** — separate workstream (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`); complementary, not designed here. +- **Kafka external SNI exposure** — deferred; its per-broker advertised-address topology does not fit a single SNI entrypoint (see the matrix). Kafka keeps today's per-broker LoadBalancer behavior. +- **MongoDB non-sharded replica-set SNI exposure** — deferred; per-member LoadBalancers plus `rs.conf` rewrite are the same class of problem as Kafka. Only the sharded `mongos` topology fits. +- **Edge TLS termination / re-encryption / `BackendTLSPolicy`** — explicitly rejected; the recorded decision lands on non-terminating passthrough. +- **New PKI, issuer, or certificate-rotation machinery** — out of scope; this reuses `unified-tls-pki` entirely. +- **Multi-Gateway single-IP sharing** — out of scope until Cilium implements ListenerSet; the design targets one IP per tenant Gateway. +- **Replacing `TLSPassthroughServices`** — the existing layer-7 field stays; this adds a parallel field. +- **SNI confidentiality (ECH)** — database clients do not use ECH; hiding the external hostname is not a goal. +- **Non-Cilium GatewayClasses** — the design assumes GatewayClass `cilium`. + +## Design + +### 1. Listener and port model + +```mermaid +flowchart TB + subgraph ext["External clients"] + C1["psql
SNI=db.alice.example.org:5432"] + C2["redis-cli
SNI=cache.alice.example.org:6379"] + end + subgraph gw["Tenant Gateway — ONE LoadBalancer IP"] + L1["listener tls-db
5432 / Passthrough"] + L2["listener tls-cache
6379 / Passthrough"] + end + subgraph pods["Operator-owned database pods"] + P1["postgres
presents CNPG server cert"] + P2["redis
presents operator server cert"] + end + CA["<release>-ca-cert (ca.crt only)
projected via tenantsecrets"] + C1 -->|"raw TLS, SNI-routed"| L1 -->|"no termination"| P1 + C2 -->|"raw TLS, SNI-routed"| L2 -->|"no termination"| P2 + CA -.->|"verifies"| C1 + CA -.->|"verifies"| C2 +``` + +A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI). SNI-based routing therefore works on any TCP port, not only 443 — a `TLSRoute` selects its backend by hostname regardless of the listener's port. "Share one IP via SNI" and "use the native port" are orthogonal, so the design takes both: one passthrough listener per engine on its native port — `tls-` on 5432 for postgres, 6379 for redis, 27017 for sharded mongo, 3306 for mariadb — each `mode: Passthrough`, hostname `.`, SNI-routed via a per-release `TLSRoute`. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. + +Native ports are chosen over forcing everything onto 443 because the latter buys nothing: it does not improve IP consolidation (SNI already does that on any port) and it breaks client ergonomics and tooling defaults (`psql -h host` assumes 5432) while still demanding direct-TLS. The all-on-443 variant is retained as a documented opt-in for operators who want a single-port firewall surface (see Alternatives). + +**The Postgres caveat is load-bearing.** libpq has historically performed a StartTLS-style negotiation: it sends a plaintext `SSLRequest` and waits for the server's single-byte reply *before* the TLS `ClientHello`. There is no SNI in the first packet, so a passthrough listener cannot route it. This is resolved by `sslnegotiation=direct` (libpq, PostgreSQL 17+), which sends the `ClientHello` immediately, carrying SNI. Postgres-over-passthrough is therefore conditional on direct-TLS-capable clients and is opt-in, not default-on. + +### 2. Certificate reuse is a property of passthrough (the WS5 core) + +Passthrough *is* the certificate reuse. Because the Gateway in `mode: Passthrough` does not terminate TLS, it forwards the raw handshake bytes to the backend pod. The certificate the external client validates is byte-identical to the operator-issued server certificate the database already presents internally — CNPG-managed for postgres, Strimzi-managed for kafka, PSMDB-managed for mongo. There is no second certificate, no re-encryption, and no new issuance. End-to-end TLS is a consequence of not terminating, not a feature to build. + +WS5 adds nothing beyond two hooks that already exist: + +1. **Trust-anchor delivery** — the `-ca-cert` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. +2. **SAN coverage** — the chart already injects `.` into the operator-issued certificate (postgres `serverAltDNSNames` in `db.yaml`, gated by `tls.enabled`). The SAN the client's SNI will carry is already in the certificate. + +Stated plainly, where this does **not** work: + +- **Kafka** — the external listener advertises per-broker addresses; after bootstrap the client connects directly to each broker. A single SNI endpoint cannot represent N per-broker endpoints. Deferred. +- **MongoDB non-sharded** — one LoadBalancer per member plus an `rs.conf` rewrite; the driver reaches each member by its advertised address. A single SNI front cannot represent the replica-set topology. Only sharded `mongos` fits. +- **Clients that omit or mis-emit SNI** — a passthrough listener has no certificate to fall back to, so a missing SNI is a hard connection failure, not a downgrade. This includes pre-direct-TLS Postgres clients. + +### 3. IP consolidation and its limits + +The headline payoff is IPv4 scarcity relief. Today ten external databases cost ten public IPs (more for Kafka and Mongo replica sets, which add per-broker / per-member IPs). Under this proposal, N SNI-routed databases on one tenant Gateway share one IP, differentiated by SNI on their native ports. Ten databases become one IP. + +The ceiling, stated honestly: + +- `Gateway.spec.listeners` is capped at 64. The tenant Gateway already spends slots on the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. Database listeners draw from the same budget; an operator near the cap should split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true`, as the controller already advises. +- Multi-Gateway IP sharing is not possible today (the one-IP-per-tenant blocker above). The win is one IP per tenant Gateway. Lifting it to share an IP across Gateways depends on Cilium implementing ListenerSet (experimental in Gateway API v1.5.1, not implemented by Cilium 1.19.3). + +### 4. Per-engine treatment + +| Engine | Native port | Fits SNI-passthrough? | Client requirement | Recommendation | +| --- | --- | --- | --- | --- | +| Redis | 6379 | Yes — immediate TLS, no pre-TLS negotiation | `--tls` + `ca.crt`; modern client emits SNI from a hostname | default-on candidate (cleanest fit) | +| PostgreSQL (CNPG) | 5432 | Yes, only with direct-TLS | `sslnegotiation=direct` + `sslmode=verify-full` + `ca.crt` (libpq PG17+) | opt-in (pre-PG17 clients fail closed) | +| MongoDB — sharded (mongos) | 27017 | Yes — single stateless endpoint | `tls=true` + `tlsCAFile`; seed pointing at the SNI hostname | opt-in (sharded only) | +| MariaDB | 3306 | Likely — modern connectors do TLS-first with SNI | TLS connector emitting SNI + `ca.crt`; verify per connector | opt-in pending connector conformance | +| MongoDB — replica set | 27017 | No — per-member LB + `rs.conf` rewrite | n/a | deferred | +| Kafka (Strimzi) | 9094 | No — per-broker advertised addresses | n/a | deferred | + +Kafka is deferred because its wire protocol redirects: the client connects to a bootstrap endpoint, receives metadata listing per-broker advertised host:port pairs, then opens direct connections to each broker. Strimzi's `type: loadbalancer` external listener provisions a LoadBalancer per broker plus a bootstrap LB precisely for this. A single SNI front cannot satisfy the per-broker fan-out. Making Kafka fit would require either per-broker SNI listeners and TLSRoutes (N listeners against the 64-listener cap, partially defeating the consolidation goal) plus rewriting Strimzi's advertised listeners, or a Kafka-aware re-advertising proxy — both out of scope. Revisit if per-broker SNI proves worth the listener budget. + +### 5. API and controller extension + +The trigger and listener synthesis stay in the controller; database charts do not render Gateway plumbing. Database charts already receive the `_cluster` values channel and read parts of it (for example `packages/apps/postgres/templates/db.yaml` reads `_cluster.scheduling`, and mongodb reads `_cluster["cluster-domain"]`), but they do not read the gateway-discovery keys that `cozystack-api` consumes (the gateway-enabled flag and the gateway name) and have no logic to locate the tenant Gateway. Teaching every database chart that discovery dance would duplicate it across five charts and couple application charts to networking topology. The controller already owns listener synthesis; keep it there. + +The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak for databases — a bare service name hardcodes the layer-7 convention of port 443 and hostname `.`. Add one structured field alongside it, leaving the existing field untouched for backward compatibility: + +```go +// TLSPassthroughBackend declares a per-engine passthrough listener on a +// native port, SNI-routed to a backend Service. +type TLSPassthroughBackend struct { + Name string `json:"name"` // listener suffix -> "tls-" + Port int32 `json:"port"` // native port (5432/6379/27017/3306) + Hostname string `json:"hostname,omitempty"` // default "." + BackendRef ... `json:"backendRef"` // Service + port (cross-namespace via ReferenceGrant) +} + +// TLSPassthroughBackends extends TLSPassthroughServices for engines that need +// a native (non-443) port. Each entry renders one Passthrough listener +// "tls-" on .Port; a TLSRoute attaches by sectionName. +// +optional +TLSPassthroughBackends []TLSPassthroughBackend `json:"tlsPassthroughBackends,omitempty"` +``` + +The controller change is to extend the existing passthrough-listener loop to also iterate `TLSPassthroughBackends`, rendering each listener with the supplied `Port` instead of the hardcoded 443. The `TLSRoute` attaches by `sectionName: tls-`, identical to the existing `api-tlsroute.yaml` pattern. The field is populated by the Tenant / HelmRelease orchestration layer that already knows both the tenant Gateway and the database release — not the database chart and not the human — so a database's surface stays a single `external`-adjacent toggle. + +### 6. Trust-anchor and SAN flow + +End to end: the chart injects the external hostname `.` into the operator-issued certificate's SAN list (gated by the `tls.enabled` tri-state, which defaults on with `external`). The operator issues the server certificate with that SAN. The `ca.crt`-only object is projected to the tenant. The external client connects with SNI `.` and verifies the presented certificate against that `ca.crt`. The passthrough listener forwards the raw handshake; the pod presents the SAN-matching certificate. No hop in the path terminates TLS. + +## User-facing changes + +A database gains an `external`-adjacent toggle to select passthrough/SNI mode. Per-engine connection recipes are documented: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=."`, `redis-cli --tls --cacert ca.crt -h .`, `mongosh --tls --tlsCAFile ca.crt --host .`, `mysql --ssl-mode=VERIFY_IDENTITY --ssl-ca=ca.crt -h .`. Kafka and non-sharded MongoDB keep today's per-LoadBalancer behavior. + +## Upgrade and rollback compatibility + +The default stays today's per-database LoadBalancer, so no existing external database changes its IP on upgrade. Passthrough/SNI mode is opt-in. Migrating an existing external database to SNI mode is a breaking, opt-in change — the IP changes and the client must be reconfigured (new host, direct-TLS for Postgres); it is not automatic. The existing `TLSPassthroughServices` field continues to work unchanged, and reverting the feature removes the new listener and field without touching the database's own PKI. + +## Security + +The edge never holds the database's private key — the central strength of passthrough over termination. SNI is sent in cleartext on TLS 1.3 except under ECH, which database clients do not use, so the external hostname `.` is observable on the wire; this is no worse than DNS or SNI exposure for any TLS service, and the payload stays encrypted end-to-end. A missing or mis-emitted SNI is a hard connection failure (a passthrough listener has no certificate to fall back to) — fail-closed, which is the secure default. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material. + +## Failure and edge cases + +- A pre-PG17 or non-direct-TLS Postgres client sends no SNI → no route → connection reset/timeout (document the symptom). +- The 64-listener budget is exceeded → the listener is rejected; mitigation is to split the high-fanout subtree onto its own Gateway. +- `tls.enabled=false` while exposing externally → certificate SAN mismatch; the tri-state should auto-enable TLS with `external`. +- An operator expects multi-Gateway IP sharing → each Gateway still gets its own IP (expected under the Cilium constraint). +- A database is deleted → its `TLSPassthroughBackends` entry and listener must be cleaned up so they do not orphan against the 64-listener cap. + +## Testing + +- Helm-template assertions that the certificate SAN includes `.` per engine, mirroring the existing TLS test fixtures. +- A controller unit test that a `TLSPassthroughBackends` entry renders a listener on the native port with `mode: Passthrough` and `AllowedRoutes` restricted to `TLSRoute`. +- An end-to-end test per fitting engine: connect from outside the cluster with SNI and `ca.crt`, and assert that the serial of the presented certificate equals the operator-issued internal certificate — proving reuse, not re-issuance. +- A negative test: a client without SNI fails closed. + +## Rollout + +1. API field plus controller listener rendering, no engine wired. **Gate:** confirm Cilium 1.19.3 actually renders a working `mode: Passthrough` listener on a non-443 port (for example 5432) and routes it by SNI — the existing passthrough services all run on 443, so this is unproven on a native database port and must be validated before any engine is wired. +2. Redis (the cleanest fit, default-on candidate) behind an opt-in. +3. PostgreSQL (direct-TLS) and sharded MongoDB, opt-in. +4. MariaDB after connector-conformance validation. + +Kafka and non-sharded MongoDB are explicitly out of this rollout. Each phase ships documentation, a connection recipe, and an end-to-end gate. + +## Open questions + +- **MariaDB / MySQL connector conformance** — do the dominant connectors (libmysqlclient, MariaDB Connector/C, JDBC, Go drivers) emit SNI and do TLS-first such that passthrough routes correctly, or does any do a server-greeting-then-STARTTLS dance like pre-direct-TLS Postgres? Default-on versus opt-in for MariaDB hinges on this. +- **Direct-TLS client floor for Postgres** — what fraction of the tenant base predates `sslnegotiation=direct`? Is a per-release attestation gate enough, given there is no graceful downgrade on a passthrough listener? +- **Cilium passthrough on a non-443 port** — the existing passthrough listeners all run on 443; whether Cilium 1.19.3's Gateway implementation honors a `mode: Passthrough` listener on a native database port (5432/6379/27017/3306) and SNI-routes it is unverified and is the single biggest implementation risk. It is the Phase 1 validation gate in Rollout. +- **TLSRoute ownership** — does the controller synthesize the `TLSRoute` alongside the listener (single source of truth, but the controller must know the backend Service), or does the database chart render it (the chart owns the backend selector, but must learn a gateway name/namespace it does not have today)? This decides whether database charts gain gateway-discovery logic. +- **Who writes `TLSPassthroughBackends`** — the orchestration layer auto-appending on `external: true` plus passthrough mode, an operator-managed explicit list, or both; and which reconciliation owns add/remove and orphaned-listener cleanup on database deletion. +- **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, default passthrough services, and N database listeners, what is the realistic per-tenant ceiling, and should the controller surface a status condition as the budget nears 64? +- **ListenerSet timeline** — multi-Gateway IP sharing is blocked on Cilium implementing ListenerSet. Do we design the field and status to be ListenerSet-ready now, or revisit when Cilium ships it? This determines whether one-IP-per-tenant is a permanent or temporary ceiling. + +## Alternatives considered + +- **Edge TLS termination plus re-encryption (`BackendTLSPolicy`).** Rejected: the edge holds keys, there are two certificates, and it defeats reuse. This is the recorded decision WS5 asks for — passthrough chooses application-level, CNI-agnostic TLS over edge termination. +- **All-on-443 SNI.** Mirrors the existing three services exactly with no new port plumbing, retained as a documented opt-in, but rejected as the default for the client-ergonomics and direct-TLS-everywhere tax. +- **Route-driven implicit listeners** — the controller watches `TLSRoute` objects (it already does) and auto-synthesizes a passthrough listener for any route targeting a non-443 port. Less API surface, but it inverts the explicit-intent model and makes the listener set implicit and hard to audit. Offered as a future ergonomic; the explicit field is recommended for v1. +- **Per-broker SNI for Kafka.** Rejected for now (see the matrix); revisit if the listener-budget cost is justified. +- **Status quo: a LoadBalancer per database.** The IPv4-scarcity problem this proposal exists to solve. + +--- + + From 55aeda2200b7fa654f253f41d3adc199fcd4cc59 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 24 Jun 2026 18:43:25 +0300 Subject: [PATCH 2/5] docs(external-database-exposure): share listeners, reject insecure TLS Replace the one-listener-per-database model with one shared passthrough listener per engine type, with per-release routing on standard TLSRoute objects keyed by SNI hostname. Listener consumption drops from O(database instances) to O(engine types), so the 64-listener Gateway cap stops being a per-database ceiling. Rename the API field to TLSPassthroughListeners, declare the route backend as a standard Gateway API BackendObjectReference instead of a placeholder, and rework the diagram, budget, failure, testing, and open-questions sections to match. Reject an explicit tls.enabled=false combined with external=true at admission via a ValidatingAdmissionPolicy rather than silently overriding it, so a tenant cannot stand up a plaintext external endpoint; an unset tri-state still auto-enables TLS together with external. Assisted-By: Claude Signed-off-by: Aleksei Sviridkin --- .../external-database-exposure/README.md | 86 +++++++++++-------- 1 file changed, 52 insertions(+), 34 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index ee64e5d..651a66c 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -74,25 +74,38 @@ Two concrete problems, both solved by SNI-passthrough plus certificate reuse: ```mermaid flowchart TB subgraph ext["External clients"] - C1["psql
SNI=db.alice.example.org:5432"] - C2["redis-cli
SNI=cache.alice.example.org:6379"] + C1["psql
SNI=db1.alice.example.org:5432"] + C2["psql
SNI=db2.alice.example.org:5432"] + C3["redis-cli
SNI=cache.alice.example.org:6379"] end - subgraph gw["Tenant Gateway — ONE LoadBalancer IP"] - L1["listener tls-db
5432 / Passthrough"] - L2["listener tls-cache
6379 / Passthrough"] + subgraph gw["Tenant Gateway — ONE LoadBalancer IP, one listener per engine type"] + L1["listener tls-postgres
5432 / Passthrough
hostname *.alice.example.org"] + L2["listener tls-redis
6379 / Passthrough
hostname *.alice.example.org"] + end + subgraph rt["Per-release TLSRoutes (SNI hostname)"] + R1["TLSRoute db1"] + R2["TLSRoute db2"] + R3["TLSRoute cache"] end subgraph pods["Operator-owned database pods"] - P1["postgres
presents CNPG server cert"] - P2["redis
presents operator server cert"] + P1["postgres db1
presents CNPG server cert"] + P2["postgres db2
presents CNPG server cert"] + P3["redis cache
presents operator server cert"] end CA["<release>-ca-cert (ca.crt only)
projected via tenantsecrets"] - C1 -->|"raw TLS, SNI-routed"| L1 -->|"no termination"| P1 - C2 -->|"raw TLS, SNI-routed"| L2 -->|"no termination"| P2 + C1 -->|"raw TLS, SNI"| L1 + C2 -->|"raw TLS, SNI"| L1 + C3 -->|"raw TLS, SNI"| L2 + L1 -->|"SNI db1"| R1 -->|"no termination"| P1 + L1 -->|"SNI db2"| R2 -->|"no termination"| P2 + L2 -->|"SNI cache"| R3 -->|"no termination"| P3 CA -.->|"verifies"| C1 - CA -.->|"verifies"| C2 + CA -.->|"verifies"| C3 ``` -A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI). SNI-based routing therefore works on any TCP port, not only 443 — a `TLSRoute` selects its backend by hostname regardless of the listener's port. "Share one IP via SNI" and "use the native port" are orthogonal, so the design takes both: one passthrough listener per engine on its native port — `tls-` on 5432 for postgres, 6379 for redis, 27017 for sharded mongo, 3306 for mariadb — each `mode: Passthrough`, hostname `.`, SNI-routed via a per-release `TLSRoute`. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. +A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **multiple `TLSRoute` objects can attach to one listener**, each selecting its backend by its own `spec.hostnames`. SNI-based routing therefore works on any TCP port, not only 443, and — crucially — it does not need a listener per database. The design uses **one passthrough listener per engine type**, on that engine's native port, with a wildcard hostname: `tls-postgres` on 5432, `tls-redis` on 6379, `tls-mongos` on 27017, `tls-mariadb` on 3306 — each `mode: Passthrough`, hostname `*.` (or unrestricted), `AllowedRoutes` limited to `TLSRoute`. Every database release of that engine then attaches a per-release `TLSRoute` carrying `spec.hostnames: ["."]`, and the Gateway SNI-routes each connection to the right backend. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. + +This keeps listener consumption at **O(engine types)** — at most four or five — rather than O(database instances). A tenant running thirty Postgres releases spends one `tls-postgres` slot, not thirty, so the 64-listener budget (§3) stops being a per-database ceiling and the consolidation goal scales with instance count, not against it. Native ports are chosen over forcing everything onto 443 because the latter buys nothing: it does not improve IP consolidation (SNI already does that on any port) and it breaks client ergonomics and tooling defaults (`psql -h host` assumes 5432) while still demanding direct-TLS. The all-on-443 variant is retained as a documented opt-in for operators who want a single-port firewall surface (see Alternatives). @@ -119,7 +132,7 @@ The headline payoff is IPv4 scarcity relief. Today ten external databases cost t The ceiling, stated honestly: -- `Gateway.spec.listeners` is capped at 64. The tenant Gateway already spends slots on the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. Database listeners draw from the same budget; an operator near the cap should split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true`, as the controller already advises. +- `Gateway.spec.listeners` is capped at 64. Because database listeners are **one per engine type** (§1), not one per release, their contribution to that budget is bounded by the four or five supported engines no matter how many instances a tenant runs; the per-release fan-out lands on `TLSRoute` objects, which are not capped. The remaining slots go to the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. A tenant therefore approaches the cap through child-apex fan-out, not database fan-out; the controller's existing advice to split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true` still applies there. - Multi-Gateway IP sharing is not possible today (the one-IP-per-tenant blocker above). The win is one IP per tenant Gateway. Lifting it to share an IP across Gateways depends on Cilium implementing ListenerSet (experimental in Gateway API v1.5.1, not implemented by Cilium 1.19.3). ### 4. Per-engine treatment @@ -139,26 +152,30 @@ Kafka is deferred because its wire protocol redirects: the client connects to a The trigger and listener synthesis stay in the controller; database charts do not render Gateway plumbing. Database charts already receive the `_cluster` values channel and read parts of it (for example `packages/apps/postgres/templates/db.yaml` reads `_cluster.scheduling`, and mongodb reads `_cluster["cluster-domain"]`), but they do not read the gateway-discovery keys that `cozystack-api` consumes (the gateway-enabled flag and the gateway name) and have no logic to locate the tenant Gateway. Teaching every database chart that discovery dance would duplicate it across five charts and couple application charts to networking topology. The controller already owns listener synthesis; keep it there. -The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak for databases — a bare service name hardcodes the layer-7 convention of port 443 and hostname `.`. Add one structured field alongside it, leaving the existing field untouched for backward compatibility: +Two objects implement the model, at different cardinalities: one shared listener per engine type, and one `TLSRoute` per database release. + +The **listener** is the new API surface. The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak — a bare service name hardcodes the layer-7 convention of port 443 and hostname `.`. Add one structured field alongside it, leaving the existing field untouched for backward compatibility: ```go -// TLSPassthroughBackend declares a per-engine passthrough listener on a -// native port, SNI-routed to a backend Service. -type TLSPassthroughBackend struct { - Name string `json:"name"` // listener suffix -> "tls-" - Port int32 `json:"port"` // native port (5432/6379/27017/3306) - Hostname string `json:"hostname,omitempty"` // default "." - BackendRef ... `json:"backendRef"` // Service + port (cross-namespace via ReferenceGrant) +// TLSPassthroughListener declares one shared passthrough listener for a +// database engine type, on its native port, with a wildcard SNI hostname. +// Every release of that engine attaches its own TLSRoute by SNI hostname. +type TLSPassthroughListener struct { + Name string `json:"name"` // listener suffix -> "tls-" (e.g. "postgres") + Port int32 `json:"port"` // native port (5432/6379/27017/3306) + Hostname string `json:"hostname,omitempty"` // wildcard SNI match, default "*." } -// TLSPassthroughBackends extends TLSPassthroughServices for engines that need -// a native (non-443) port. Each entry renders one Passthrough listener -// "tls-" on .Port; a TLSRoute attaches by sectionName. +// TLSPassthroughListeners renders one Passthrough listener "tls-" on +// .Port per entry, AllowedRoutes restricted to TLSRoute. Independent of the +// layer-7 TLSPassthroughServices field. // +optional -TLSPassthroughBackends []TLSPassthroughBackend `json:"tlsPassthroughBackends,omitempty"` +TLSPassthroughListeners []TLSPassthroughListener `json:"tlsPassthroughListeners,omitempty"` ``` -The controller change is to extend the existing passthrough-listener loop to also iterate `TLSPassthroughBackends`, rendering each listener with the supplied `Port` instead of the hardcoded 443. The `TLSRoute` attaches by `sectionName: tls-`, identical to the existing `api-tlsroute.yaml` pattern. The field is populated by the Tenant / HelmRelease orchestration layer that already knows both the tenant Gateway and the database release — not the database chart and not the human — so a database's surface stays a single `external`-adjacent toggle. +The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` attaches to the shared `tls-` listener by `sectionName`, `spec.hostnames: ["."]` carries the SNI match, and `spec.rules[].backendRefs` is a standard `gwapiv1alpha2.BackendRef` (whose embedded `BackendObjectReference` points at the database Service on its native port, cross-namespace via `ReferenceGrant`). + +The controller change is to render, per `TLSPassthroughListeners` entry, one Passthrough listener on the supplied `Port` instead of the hardcoded 443, and to attach each per-release `TLSRoute` by `sectionName: tls-` — the same attachment pattern as the existing `api-tlsroute.yaml`, except that many routes share one listener and the Gateway disambiguates them by SNI hostname. Both are populated by the Tenant / HelmRelease orchestration layer that already knows the tenant Gateway and the database release — not the database chart and not the human — so a database's surface stays a single `external`-adjacent toggle. The engine-type listener is created on first exposure of that engine and removed once its last release is gone; per-release add/remove only touches the route, never the shared listener. ### 6. Trust-anchor and SAN flow @@ -170,24 +187,25 @@ A database gains an `external`-adjacent toggle to select passthrough/SNI mode. P ## Upgrade and rollback compatibility -The default stays today's per-database LoadBalancer, so no existing external database changes its IP on upgrade. Passthrough/SNI mode is opt-in. Migrating an existing external database to SNI mode is a breaking, opt-in change — the IP changes and the client must be reconfigured (new host, direct-TLS for Postgres); it is not automatic. The existing `TLSPassthroughServices` field continues to work unchanged, and reverting the feature removes the new listener and field without touching the database's own PKI. +The default stays today's per-database LoadBalancer, so no existing external database changes its IP on upgrade. Passthrough/SNI mode is opt-in. Migrating an existing external database to SNI mode is a breaking, opt-in change — the IP changes and the client must be reconfigured (new host, direct-TLS for Postgres); it is not automatic. The existing `TLSPassthroughServices` field continues to work unchanged, and reverting the feature removes the per-release routes and the engine-type listeners without touching the database's own PKI. ## Security -The edge never holds the database's private key — the central strength of passthrough over termination. SNI is sent in cleartext on TLS 1.3 except under ECH, which database clients do not use, so the external hostname `.` is observable on the wire; this is no worse than DNS or SNI exposure for any TLS service, and the payload stays encrypted end-to-end. A missing or mis-emitted SNI is a hard connection failure (a passthrough listener has no certificate to fall back to) — fail-closed, which is the secure default. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material. +The edge never holds the database's private key — the central strength of passthrough over termination. SNI is sent in cleartext on TLS 1.3 except under ECH, which database clients do not use, so the external hostname `.` is observable on the wire; this is no worse than DNS or SNI exposure for any TLS service, and the payload stays encrypted end-to-end. A missing or mis-emitted SNI is a hard connection failure (a passthrough listener has no certificate to fall back to) — fail-closed, which is the secure default. Exposing a database externally with TLS explicitly off is not silently corrected: when `tls.enabled` is left unset the tri-state turns TLS on together with `external`, but an explicit `tls.enabled: false` combined with `external: true` is **rejected by a ValidatingAdmissionPolicy** — the same admission mechanism that already enforces tenant hostname/IP isolation — so a tenant cannot stand up a plaintext external endpoint by accident or by overriding the default, and the conflict surfaces as a clear admission error rather than a silent override the operator never sees. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material. ## Failure and edge cases - A pre-PG17 or non-direct-TLS Postgres client sends no SNI → no route → connection reset/timeout (document the symptom). -- The 64-listener budget is exceeded → the listener is rejected; mitigation is to split the high-fanout subtree onto its own Gateway. -- `tls.enabled=false` while exposing externally → certificate SAN mismatch; the tri-state should auto-enable TLS with `external`. +- The 64-listener budget is exceeded → the listener is rejected; because listeners are one per engine type this is reached only through child-apex fan-out, and the mitigation is to split that subtree onto its own Gateway. +- An explicit `tls.enabled: false` together with `external: true` → rejected at admission by a ValidatingAdmissionPolicy, not silently overridden; an unset `tls` tri-state auto-enables TLS with `external`. - An operator expects multi-Gateway IP sharing → each Gateway still gets its own IP (expected under the Cilium constraint). -- A database is deleted → its `TLSPassthroughBackends` entry and listener must be cleaned up so they do not orphan against the 64-listener cap. +- A database is deleted → its per-release `TLSRoute` is removed; the shared engine-type listener is removed only when its last release is gone, so a single deletion never orphans a listener. ## Testing - Helm-template assertions that the certificate SAN includes `.` per engine, mirroring the existing TLS test fixtures. -- A controller unit test that a `TLSPassthroughBackends` entry renders a listener on the native port with `mode: Passthrough` and `AllowedRoutes` restricted to `TLSRoute`. +- A controller unit test that a `TLSPassthroughListeners` entry renders a shared listener on the native port with `mode: Passthrough` and `AllowedRoutes` restricted to `TLSRoute`, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. +- An admission test that `tls.enabled: false` with `external: true` is rejected by the ValidatingAdmissionPolicy, while an unset `tls` is admitted and auto-enables. - An end-to-end test per fitting engine: connect from outside the cluster with SNI and `ca.crt`, and assert that the serial of the presented certificate equals the operator-issued internal certificate — proving reuse, not re-issuance. - A negative test: a client without SNI fails closed. @@ -205,9 +223,9 @@ Kafka and non-sharded MongoDB are explicitly out of this rollout. Each phase shi - **MariaDB / MySQL connector conformance** — do the dominant connectors (libmysqlclient, MariaDB Connector/C, JDBC, Go drivers) emit SNI and do TLS-first such that passthrough routes correctly, or does any do a server-greeting-then-STARTTLS dance like pre-direct-TLS Postgres? Default-on versus opt-in for MariaDB hinges on this. - **Direct-TLS client floor for Postgres** — what fraction of the tenant base predates `sslnegotiation=direct`? Is a per-release attestation gate enough, given there is no graceful downgrade on a passthrough listener? - **Cilium passthrough on a non-443 port** — the existing passthrough listeners all run on 443; whether Cilium 1.19.3's Gateway implementation honors a `mode: Passthrough` listener on a native database port (5432/6379/27017/3306) and SNI-routes it is unverified and is the single biggest implementation risk. It is the Phase 1 validation gate in Rollout. -- **TLSRoute ownership** — does the controller synthesize the `TLSRoute` alongside the listener (single source of truth, but the controller must know the backend Service), or does the database chart render it (the chart owns the backend selector, but must learn a gateway name/namespace it does not have today)? This decides whether database charts gain gateway-discovery logic. -- **Who writes `TLSPassthroughBackends`** — the orchestration layer auto-appending on `external: true` plus passthrough mode, an operator-managed explicit list, or both; and which reconciliation owns add/remove and orphaned-listener cleanup on database deletion. -- **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, default passthrough services, and N database listeners, what is the realistic per-tenant ceiling, and should the controller surface a status condition as the budget nears 64? +- **TLSRoute ownership** — does the controller synthesize the per-release `TLSRoute` alongside the shared listener (single source of truth, but the controller must know the backend Service), or does the database chart render it (the chart owns the backend selector, but must learn a gateway name/namespace it does not have today)? This decides whether database charts gain gateway-discovery logic. +- **Who writes the listener/route set** — the orchestration layer auto-creating the engine-type listener on first exposure and a per-release route on each `external: true` + passthrough, an operator-managed explicit list, or both; and which reconciliation owns route add/remove plus reference-counting the shared listener down to zero on the last database deletion. +- **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, and default passthrough services now joined by one listener per database engine type (not per instance), what is the realistic per-tenant ceiling, and should the controller still surface a status condition as the budget nears 64? - **ListenerSet timeline** — multi-Gateway IP sharing is blocked on Cilium implementing ListenerSet. Do we design the field and status to be ListenerSet-ready now, or revisit when Cilium ships it? This determines whether one-IP-per-tenant is a permanent or temporary ceiling. ## Alternatives considered From 0f3d8c9794dcae51f437db4f46e18b8bd1d2c950 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 26 Jun 2026 18:37:34 +0300 Subject: [PATCH 3/5] docs(external-database-exposure): correct Cilium passthrough constraints Reframe the implementation risk: a mode: Passthrough listener on a native database port is supported (Cilium since 1.14), so the open question is not the port but SNI-hostname isolation across listeners. The flat *. scheme overlaps the engine-type passthrough listeners with the HTTPS-terminate listeners, which Cilium isolates correctly only from 1.20, so record that dependency and keep distinct per-engine subdomains as the documented fallback. Move the Phase 1 gate to validate that case on the target Cilium version. Fix the LB-IPAM citations (the sharing-key port-collision rule is in the Cilium docs, not the cited issues), re-date ListenerSet (graduated toward Standard in Gateway API v1.5; Cilium does not implement it on stable) and TLSRoute (GA as v1 since v1.5.0, though Cilium consumes the experimental channel), and add the Postgres direct-TLS prerequisites (server 17+, ALPN, sslmode floor, hostname SNI). Add a validation contract to the listener type, gate the per-engine ca.crt dependency on unified-tls-pki convergence, and reconcile the redis opt-in wording. Assisted-By: Claude Signed-off-by: Aleksei Sviridkin --- .../external-database-exposure/README.md | 57 ++++++++++++------- 1 file changed, 38 insertions(+), 19 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index 651a66c..54aac9e 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -14,7 +14,7 @@ This is the design-proposal artifact required by `cozystack/cozystack#2816`, and ## Scope and related proposals -- **Depends on:** `design-proposals/unified-tls-pki` — provides the `-ca-cert` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. +- **Depends on:** `design-proposals/unified-tls-pki` — provides the `-ca-cert` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract, and the path differs by engine — redis self-publishes a key-free `-ca-cert` through its forked operator, while postgres and mongodb obtain theirs through the extraction controller in `unified-tls-pki`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. - **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). - **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). @@ -24,10 +24,16 @@ All repository paths below refer to the `cozystack/cozystack` repository. ### TLS-passthrough already exists -Cozystack already runs Gateway API TLS-passthrough for layer-7 system services. The `TenantGateway` controller (`internal/controller/tenantgateway/reconciler.go`) renders, for each entry in `TenantGateway.Spec.TLSPassthroughServices`, a listener named `tls-` on port 443 with `Protocol: TLS`, `Mode: Passthrough`, hostname `.`, and `AllowedRoutes` restricted to `TLSRoute`. The default services are `api`, `vm-exportproxy`, and `cdi-uploadproxy`, each with a `TLSRoute` that attaches by `sectionName: tls-` (for example `packages/system/cozystack-api/templates/api-tlsroute.yaml`). The platform runs Cilium 1.19.3 with Gateway API v1.5.1 CRDs and GatewayClass `cilium`; `TLSRoute` and passthrough are supported. +Cozystack already runs Gateway API TLS-passthrough for layer-7 system services. The `TenantGateway` controller (`internal/controller/tenantgateway/reconciler.go`) renders, for each entry in `TenantGateway.Spec.TLSPassthroughServices`, a listener named `tls-` on port 443 with `Protocol: TLS`, `Mode: Passthrough`, hostname `.`, and `AllowedRoutes` restricted to `TLSRoute`. The default services are `api`, `vm-exportproxy`, and `cdi-uploadproxy`, each with a `TLSRoute` that attaches by `sectionName: tls-` (for example `packages/system/cozystack-api/templates/api-tlsroute.yaml`). The platform runs Cilium 1.19.3 with GatewayClass `cilium`. `TLSRoute` is GA in the Gateway API Standard channel as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0; Cilium 1.19.x still consumes it from the **experimental-channel** CRD, so that CRD must be installed and the operative API version for this design is Cilium's channel, not the upstream graduation. `mode: Passthrough` TLSRoute is supported in Cilium since 1.14. The mechanism this proposal needs is therefore already in production for layer-7 services. The work is to extend it to databases, which speak on native (non-443) ports. +### Passthrough on a native port works; the real constraint is SNI-hostname overlap + +Routing a `mode: Passthrough` listener on a non-443 port is supported, not exotic: Cilium has shipped TLSRoute passthrough since 1.14, its own conformance fixtures exercise passthrough listeners on a non-443 port, and field reports route passthrough on database ports (for example a mongo `27017` case). The native port (5432/6379/27017/3306) is not the risk. + +The real constraint is **SNI-hostname uniqueness across listeners on one Gateway**. On every released Cilium up to and including 1.19.x, two listeners that share a hostname but differ in port — or a passthrough listener that shares a hostname with an HTTPS-terminate listener — collide and do not isolate correctly (the cross-port hostname-isolation defect; fixed by an upstream change that targets Cilium 1.20 and is not in any release tag as of this writing). This matters here because the listener model below puts every engine-type passthrough listener on the wildcard hostname `*.`, which is the same hostname the tenant Gateway's HTTPS-terminate listeners already use. This proposal keeps the flat `*.` scheme — for clean `.` connection hostnames — and therefore **depends on Cilium 1.20** for correct multi-listener isolation. If that dependency is unacceptable, the documented fallback is distinct per-engine subdomains (`*.pg.`, `*.redis.`, …), which sidestep the overlap at the cost of longer hostnames and per-engine SAN/recipe changes. Either way this is the Phase 1 validation gate (Rollout): the gate validates the shared-hostname case on the target Cilium version, not merely that passthrough binds on a non-443 port. + ### Each external database burns its own IP today When a database chart sets `external: true`, it provisions a dedicated `Service` of `type: LoadBalancer`: postgres `-external-write` on 5432 (`packages/apps/postgres/templates/external-svc.yaml`), redis `-external-lb` on 6379 (`packages/apps/redis/templates/service.yaml`), mongodb `-external` on 27017 (`packages/apps/mongodb/templates/external-svc.yaml`), kafka on 9094 plus a LoadBalancer per broker (`packages/apps/kafka/templates/kafka.yaml`), and mariadb via a `primaryService` of `type: LoadBalancer` (`packages/apps/mariadb/templates/mariadb.yaml`). Each is a separate IP from the shared MetalLB (or Cilium LB-IPAM) pool; there is no per-tenant IP isolation and no SNI multiplexing. @@ -38,7 +44,7 @@ The external hostname is `.<_namespace.host>`, where `_namespace.host` ### The one-IP-per-tenant ceiling -Multiple per-tenant Gateways cannot share a single LoadBalancer IP on current Cilium: every Gateway claims `443/TCP`, so `lbipam.cilium.io/sharing-key` is inactive on the port collision (`packages/extra/gateway/README.md`; upstream cilium#21270, cilium#42756). Within a single Gateway, a parent and its inheriting children share one IP. `Gateway.spec.listeners` is hard-capped at 64. The practical consequence: the unit of IP sharing is one tenant Gateway, and a tenant's database fan-out draws from the 64-listener budget shared with the existing http/https/https-apex and per-child-apex wildcard listeners. +Multiple per-tenant Gateways cannot share a single LoadBalancer IP on current Cilium. Cilium LB-IPAM shares one IP across Services via `lbipam.cilium.io/sharing-key` only when their ports do not conflict; every Gateway claims `443/TCP`, so the shared-IP path is inactive on that port collision (`packages/extra/gateway/README.md`; the port-conflict rule is documented in the Cilium LB-IPAM docs). Within a single Gateway, a parent and its inheriting children share one IP. `Gateway.spec.listeners` is hard-capped at 64 (Gateway API CRD `MaxItems`). The practical consequence: the unit of IP sharing is one tenant Gateway, and a tenant's database fan-out draws from the 64-listener budget shared with the existing http/https/https-apex and per-child-apex wildcard listeners. ### The problem @@ -107,14 +113,18 @@ A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **m This keeps listener consumption at **O(engine types)** — at most four or five — rather than O(database instances). A tenant running thirty Postgres releases spends one `tls-postgres` slot, not thirty, so the 64-listener budget (§3) stops being a per-database ceiling and the consolidation goal scales with instance count, not against it. +**The shared-hostname dependency is load-bearing.** Putting `tls-postgres` (5432) and `tls-redis` (6379) both on `*.` means several listeners share one hostname across ports, and that hostname also overlaps the Gateway's HTTPS-terminate listeners. As noted in Context, Cilium isolates that case correctly only from 1.20 onward, so this scheme depends on Cilium 1.20. The per-release `TLSRoute` hostnames (`.`) are always distinct and are not the issue; the overlap is at the listener level. The flat scheme is chosen for clean connection hostnames; distinct per-engine subdomains are the documented fallback if the Cilium 1.20 dependency is unacceptable. + Native ports are chosen over forcing everything onto 443 because the latter buys nothing: it does not improve IP consolidation (SNI already does that on any port) and it breaks client ergonomics and tooling defaults (`psql -h host` assumes 5432) while still demanding direct-TLS. The all-on-443 variant is retained as a documented opt-in for operators who want a single-port firewall surface (see Alternatives). -**The Postgres caveat is load-bearing.** libpq has historically performed a StartTLS-style negotiation: it sends a plaintext `SSLRequest` and waits for the server's single-byte reply *before* the TLS `ClientHello`. There is no SNI in the first packet, so a passthrough listener cannot route it. This is resolved by `sslnegotiation=direct` (libpq, PostgreSQL 17+), which sends the `ClientHello` immediately, carrying SNI. Postgres-over-passthrough is therefore conditional on direct-TLS-capable clients and is opt-in, not default-on. +**The Postgres caveat is load-bearing.** libpq has historically performed a StartTLS-style negotiation: it sends a plaintext `SSLRequest` and waits for the server's single-byte reply *before* the TLS `ClientHello`. There is no SNI in the first packet, so a passthrough listener cannot route it. This is resolved by `sslnegotiation=direct` (libpq, PostgreSQL 17+), which sends the `ClientHello` immediately, carrying SNI. It comes with hard prerequisites, not preferences: the **server must also be PostgreSQL 17+**; `sslnegotiation=direct` requires `sslmode=require` or stronger; direct SSL mandates ALPN (`postgresql`); and SNI is emitted only when the client dials by **hostname** with `sslsni=1` (the default) — an IP literal carries no SNI. Driver coverage is uneven (recent pgjdbc 42.7.x and Npgsql 9.0+ support it; node-postgres historically did not). Any client older than these, against a pre-17 server, or with `sslmode` below `require`, falls back to the legacy negotiation and cannot be SNI-routed. Postgres-over-passthrough is therefore conditional on direct-TLS-capable clients and is opt-in, not default-on. ### 2. Certificate reuse is a property of passthrough (the WS5 core) Passthrough *is* the certificate reuse. Because the Gateway in `mode: Passthrough` does not terminate TLS, it forwards the raw handshake bytes to the backend pod. The certificate the external client validates is byte-identical to the operator-issued server certificate the database already presents internally — CNPG-managed for postgres, Strimzi-managed for kafka, PSMDB-managed for mongo. There is no second certificate, no re-encryption, and no new issuance. End-to-end TLS is a consequence of not terminating, not a feature to build. +One subtlety the "single certificate" framing hides: an external client doing `sslmode=verify-full` (or the equivalent) validates the presented certificate's SANs against the hostname it dialed. CNPG presents one server certificate across its `-rw`/`-r`/`-ro` services with no per-listener split, so the external passthrough hostname `.` **must be in that certificate's `serverAltDNSNames`** for verify-full to pass — which is exactly what the SAN hook below injects. Engines with a per-listener certificate model (Strimzi `brokerCertChainAndKey`) could serve a different cert externally, but the fitting engines here present one server cert, so "reuse" is literal. + WS5 adds nothing beyond two hooks that already exist: 1. **Trust-anchor delivery** — the `-ca-cert` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. @@ -132,21 +142,21 @@ The headline payoff is IPv4 scarcity relief. Today ten external databases cost t The ceiling, stated honestly: -- `Gateway.spec.listeners` is capped at 64. Because database listeners are **one per engine type** (§1), not one per release, their contribution to that budget is bounded by the four or five supported engines no matter how many instances a tenant runs; the per-release fan-out lands on `TLSRoute` objects, which are not capped. The remaining slots go to the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. A tenant therefore approaches the cap through child-apex fan-out, not database fan-out; the controller's existing advice to split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true` still applies there. -- Multi-Gateway IP sharing is not possible today (the one-IP-per-tenant blocker above). The win is one IP per tenant Gateway. Lifting it to share an IP across Gateways depends on Cilium implementing ListenerSet (experimental in Gateway API v1.5.1, not implemented by Cilium 1.19.3). +- `Gateway.spec.listeners` is capped at 64. Because database listeners are **one per engine type** (§1), not one per release, their contribution to that budget is bounded by the four or five supported engines no matter how many instances a tenant runs; the per-release fan-out lands on `TLSRoute` objects, which are not capped. The remaining slots go to the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. A tenant therefore approaches the cap through child-apex fan-out, not database fan-out; the controller's existing advice to split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true` still applies there. (The 64 cap is on one Gateway's inline `listeners` array; a future ListenerSet could raise the effective count, see Open questions.) +- Multi-Gateway IP sharing is not possible today (the one-IP-per-tenant blocker above). The win is one IP per tenant Gateway. Lifting it depends on Cilium implementing ListenerSet, which would let many tenants' listeners be consolidated onto **one shared Gateway** (and thus one IP) — it does not make N separate Gateways share an IP. ListenerSet graduated toward the Gateway API Standard channel in v1.5, but Cilium does not implement it on any stable release (tracked by the Cilium CFP `cilium#42756`, implementation PR `cilium#46303` in progress). ### 4. Per-engine treatment | Engine | Native port | Fits SNI-passthrough? | Client requirement | Recommendation | | --- | --- | --- | --- | --- | -| Redis | 6379 | Yes — immediate TLS, no pre-TLS negotiation | `--tls` + `ca.crt`; modern client emits SNI from a hostname | default-on candidate (cleanest fit) | -| PostgreSQL (CNPG) | 5432 | Yes, only with direct-TLS | `sslnegotiation=direct` + `sslmode=verify-full` + `ca.crt` (libpq PG17+) | opt-in (pre-PG17 clients fail closed) | +| Redis | 6379 | Yes — immediate TLS, no pre-TLS negotiation | `--tls` + `ca.crt`; modern client emits SNI from a hostname | cleanest fit; ships opt-in first, default-on candidate once validated | +| PostgreSQL (CNPG) | 5432 | Yes, only with direct-TLS | `sslnegotiation=direct` + `sslmode=verify-full` + `ca.crt`, hostname (not IP); libpq PG17+ **and** server PG17+ | opt-in (pre-PG17 client or server fails closed) | | MongoDB — sharded (mongos) | 27017 | Yes — single stateless endpoint | `tls=true` + `tlsCAFile`; seed pointing at the SNI hostname | opt-in (sharded only) | | MariaDB | 3306 | Likely — modern connectors do TLS-first with SNI | TLS connector emitting SNI + `ca.crt`; verify per connector | opt-in pending connector conformance | | MongoDB — replica set | 27017 | No — per-member LB + `rs.conf` rewrite | n/a | deferred | | Kafka (Strimzi) | 9094 | No — per-broker advertised addresses | n/a | deferred | -Kafka is deferred because its wire protocol redirects: the client connects to a bootstrap endpoint, receives metadata listing per-broker advertised host:port pairs, then opens direct connections to each broker. Strimzi's `type: loadbalancer` external listener provisions a LoadBalancer per broker plus a bootstrap LB precisely for this. A single SNI front cannot satisfy the per-broker fan-out. Making Kafka fit would require either per-broker SNI listeners and TLSRoutes (N listeners against the 64-listener cap, partially defeating the consolidation goal) plus rewriting Strimzi's advertised listeners, or a Kafka-aware re-advertising proxy — both out of scope. Revisit if per-broker SNI proves worth the listener budget. +Kafka is deferred because its wire protocol redirects: the client connects to a bootstrap endpoint, receives metadata listing per-broker advertised host:port pairs, then opens direct connections to each broker. Strimzi's `type: loadbalancer` external listener provisions a LoadBalancer per broker plus a bootstrap LB precisely for this. A single SNI front cannot satisfy the per-broker fan-out — this is a topology problem, not a certificate one (Strimzi could serve any cert per listener). Making Kafka fit would require either per-broker SNI listeners and TLSRoutes (N listeners against the 64-listener cap, partially defeating the consolidation goal) plus rewriting Strimzi's advertised listeners, or a Kafka-aware re-advertising proxy — both out of scope. Revisit if per-broker SNI proves worth the listener budget. ### 5. API and controller extension @@ -160,6 +170,13 @@ The **listener** is the new API surface. The existing `TLSPassthroughServices [] // TLSPassthroughListener declares one shared passthrough listener for a // database engine type, on its native port, with a wildcard SNI hostname. // Every release of that engine attaches its own TLSRoute by SNI hostname. +// +// Validation contract (enforced by CEL / admission): +// - Name: DNS-1123 label; unique within the list; renders as "tls-". +// - Port: 1..65535; unique within the list; must not collide with a +// synthesized layer-7 listener port (443) or another entry. +// - Hostname: optional; a valid DNS wildcard or exact hostname; defaults +// to "*." when empty. type TLSPassthroughListener struct { Name string `json:"name"` // listener suffix -> "tls-" (e.g. "postgres") Port int32 `json:"port"` // native port (5432/6379/27017/3306) @@ -173,7 +190,7 @@ type TLSPassthroughListener struct { TLSPassthroughListeners []TLSPassthroughListener `json:"tlsPassthroughListeners,omitempty"` ``` -The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` attaches to the shared `tls-` listener by `sectionName`, `spec.hostnames: ["."]` carries the SNI match, and `spec.rules[].backendRefs` is a standard `gwapiv1alpha2.BackendRef` (whose embedded `BackendObjectReference` points at the database Service on its native port, cross-namespace via `ReferenceGrant`). +The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` attaches to the shared `tls-` listener by `sectionName`, `spec.hostnames: ["."]` carries the SNI match, and `spec.rules[].backendRefs` is a standard `BackendRef` (whose embedded `BackendObjectReference` points at the database Service on its native port, cross-namespace via `ReferenceGrant`). `TLSRoute` is GA as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0, but Cilium 1.19.x consumes the experimental-channel CRD — pin to whatever API version the targeted Cilium ships, not the upstream `v1` graduation. The controller change is to render, per `TLSPassthroughListeners` entry, one Passthrough listener on the supplied `Port` instead of the hardcoded 443, and to attach each per-release `TLSRoute` by `sectionName: tls-` — the same attachment pattern as the existing `api-tlsroute.yaml`, except that many routes share one listener and the Gateway disambiguates them by SNI hostname. Both are populated by the Tenant / HelmRelease orchestration layer that already knows the tenant Gateway and the database release — not the database chart and not the human — so a database's surface stays a single `external`-adjacent toggle. The engine-type listener is created on first exposure of that engine and removed once its last release is gone; per-release add/remove only touches the route, never the shared listener. @@ -183,7 +200,7 @@ End to end: the chart injects the external hostname `.` into the ## User-facing changes -A database gains an `external`-adjacent toggle to select passthrough/SNI mode. Per-engine connection recipes are documented: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=."`, `redis-cli --tls --cacert ca.crt -h .`, `mongosh --tls --tlsCAFile ca.crt --host .`, `mysql --ssl-mode=VERIFY_IDENTITY --ssl-ca=ca.crt -h .`. Kafka and non-sharded MongoDB keep today's per-LoadBalancer behavior. +A database gains an `external`-adjacent toggle to select passthrough/SNI mode. Per-engine connection recipes are documented: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=."` (libpq and server both PG17+), `redis-cli --tls --cacert ca.crt -h .`, `mongosh --tls --tlsCAFile ca.crt --host .`, `mysql --ssl-mode=VERIFY_IDENTITY --ssl-ca=ca.crt -h .`. Kafka and non-sharded MongoDB keep today's per-LoadBalancer behavior. ## Upgrade and rollback compatibility @@ -195,8 +212,9 @@ The edge never holds the database's private key — the central strength of pass ## Failure and edge cases -- A pre-PG17 or non-direct-TLS Postgres client sends no SNI → no route → connection reset/timeout (document the symptom). +- A pre-PG17 (client or server) or non-direct-TLS Postgres client sends no SNI → no route → connection reset/timeout (document the symptom). - The 64-listener budget is exceeded → the listener is rejected; because listeners are one per engine type this is reached only through child-apex fan-out, and the mitigation is to split that subtree onto its own Gateway. +- A passthrough listener shares a hostname with a terminate listener on Cilium <1.20 → routing does not isolate correctly; mitigated by the Cilium 1.20 dependency or the distinct-subdomain fallback, and caught by the Phase 1 gate. - An explicit `tls.enabled: false` together with `external: true` → rejected at admission by a ValidatingAdmissionPolicy, not silently overridden; an unset `tls` tri-state auto-enables TLS with `external`. - An operator expects multi-Gateway IP sharing → each Gateway still gets its own IP (expected under the Cilium constraint). - A database is deleted → its per-release `TLSRoute` is removed; the shared engine-type listener is removed only when its last release is gone, so a single deletion never orphans a listener. @@ -205,14 +223,14 @@ The edge never holds the database's private key — the central strength of pass - Helm-template assertions that the certificate SAN includes `.` per engine, mirroring the existing TLS test fixtures. - A controller unit test that a `TLSPassthroughListeners` entry renders a shared listener on the native port with `mode: Passthrough` and `AllowedRoutes` restricted to `TLSRoute`, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. -- An admission test that `tls.enabled: false` with `external: true` is rejected by the ValidatingAdmissionPolicy, while an unset `tls` is admitted and auto-enables. +- An admission test that `tls.enabled: false` with `external: true` is rejected by the ValidatingAdmissionPolicy, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name/port uniqueness, port range, 443 collision) is enforced. - An end-to-end test per fitting engine: connect from outside the cluster with SNI and `ca.crt`, and assert that the serial of the presented certificate equals the operator-issued internal certificate — proving reuse, not re-issuance. - A negative test: a client without SNI fails closed. ## Rollout -1. API field plus controller listener rendering, no engine wired. **Gate:** confirm Cilium 1.19.3 actually renders a working `mode: Passthrough` listener on a non-443 port (for example 5432) and routes it by SNI — the existing passthrough services all run on 443, so this is unproven on a native database port and must be validated before any engine is wired. -2. Redis (the cleanest fit, default-on candidate) behind an opt-in. +1. API field plus controller listener rendering, no engine wired. **Gate:** on the target Cilium version, stand up a `mode: Passthrough` listener on a native database port (for example 5432) whose hostname `*.` overlaps the Gateway's terminate listeners, and confirm it SNI-routes to the right backend with exactly one Envoy filter chain. The risk is not the non-443 port (supported since Cilium 1.14) but the shared-hostname isolation that lands in Cilium 1.20; this gate must pass on the version actually deployed before any engine is wired, or the distinct-subdomain fallback adopted. +2. Redis (the cleanest fit) behind an opt-in; default-on candidate once the gate and client-emits-SNI behavior are confirmed in practice. 3. PostgreSQL (direct-TLS) and sharded MongoDB, opt-in. 4. MariaDB after connector-conformance validation. @@ -220,17 +238,18 @@ Kafka and non-sharded MongoDB are explicitly out of this rollout. Each phase shi ## Open questions +- **Cilium shared-hostname isolation on the target version** — the listener model overlaps `*.` across the engine-type passthrough listeners and the HTTPS-terminate listeners, which Cilium isolates correctly only from 1.20. Do we pin Cilium 1.20 as a hard prerequisite for this feature, or ship the distinct-per-engine-subdomain scheme so it runs on 1.19.x? This is the single biggest implementation risk and the Phase 1 gate. - **MariaDB / MySQL connector conformance** — do the dominant connectors (libmysqlclient, MariaDB Connector/C, JDBC, Go drivers) emit SNI and do TLS-first such that passthrough routes correctly, or does any do a server-greeting-then-STARTTLS dance like pre-direct-TLS Postgres? Default-on versus opt-in for MariaDB hinges on this. -- **Direct-TLS client floor for Postgres** — what fraction of the tenant base predates `sslnegotiation=direct`? Is a per-release attestation gate enough, given there is no graceful downgrade on a passthrough listener? -- **Cilium passthrough on a non-443 port** — the existing passthrough listeners all run on 443; whether Cilium 1.19.3's Gateway implementation honors a `mode: Passthrough` listener on a native database port (5432/6379/27017/3306) and SNI-routes it is unverified and is the single biggest implementation risk. It is the Phase 1 validation gate in Rollout. +- **Direct-TLS client/server floor for Postgres** — what fraction of the tenant base predates `sslnegotiation=direct` on either client (libpq PG17+, driver support) or server (PG17+)? Is a per-release attestation gate enough, given there is no graceful downgrade on a passthrough listener? - **TLSRoute ownership** — does the controller synthesize the per-release `TLSRoute` alongside the shared listener (single source of truth, but the controller must know the backend Service), or does the database chart render it (the chart owns the backend selector, but must learn a gateway name/namespace it does not have today)? This decides whether database charts gain gateway-discovery logic. - **Who writes the listener/route set** — the orchestration layer auto-creating the engine-type listener on first exposure and a per-release route on each `external: true` + passthrough, an operator-managed explicit list, or both; and which reconciliation owns route add/remove plus reference-counting the shared listener down to zero on the last database deletion. -- **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, and default passthrough services now joined by one listener per database engine type (not per instance), what is the realistic per-tenant ceiling, and should the controller still surface a status condition as the budget nears 64? -- **ListenerSet timeline** — multi-Gateway IP sharing is blocked on Cilium implementing ListenerSet. Do we design the field and status to be ListenerSet-ready now, or revisit when Cilium ships it? This determines whether one-IP-per-tenant is a permanent or temporary ceiling. +- **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, and default passthrough services now joined by one listener per database engine type (not per instance), what is the realistic per-tenant ceiling, and should the controller surface a status condition as the budget nears 64? +- **ListenerSet timeline** — consolidating onto one shared Gateway/IP is blocked on Cilium implementing ListenerSet (CFP `cilium#42756`, PR `cilium#46303`). Do we design the field and status to be ListenerSet-ready now, or revisit when Cilium ships it? This determines whether one-IP-per-tenant is a permanent or temporary ceiling. ## Alternatives considered - **Edge TLS termination plus re-encryption (`BackendTLSPolicy`).** Rejected: the edge holds keys, there are two certificates, and it defeats reuse. This is the recorded decision WS5 asks for — passthrough chooses application-level, CNI-agnostic TLS over edge termination. +- **Distinct per-engine subdomains for the passthrough listeners** (`*.pg.`, `*.redis.`, …). Sidesteps the Cilium shared-hostname isolation defect on 1.19.x without waiting for 1.20, but at the cost of longer connection hostnames and per-engine SAN/recipe changes. Retained as the documented fallback; the flat `*.` scheme is preferred for clean hostnames where Cilium 1.20 is available. - **All-on-443 SNI.** Mirrors the existing three services exactly with no new port plumbing, retained as a documented opt-in, but rejected as the default for the client-ergonomics and direct-TLS-everywhere tax. - **Route-driven implicit listeners** — the controller watches `TLSRoute` objects (it already does) and auto-synthesizes a passthrough listener for any route targeting a non-443 port. Less API surface, but it inverts the explicit-intent model and makes the listener set implicit and hard to audit. Offered as a future ergonomic; the explicit field is recommended for v1. - **Per-broker SNI for Kafka.** Rejected for now (see the matrix); revisit if the listener-budget cost is justified. From b287ef3f91a84b23f94db43126b2ca45131f36be Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 2 Jul 2026 12:54:45 +0300 Subject: [PATCH 4/5] docs(external-database-exposure): default to subdomain hostnames, exclude MariaDB Make distinct per-engine subdomain hostnames (*.postgres.) the v1 listener scheme: they avoid the cross-port hostname-isolation defect on the deployed Cilium 1.19.x, while the flat *. scheme becomes an additive later phase gated on Cilium 1.20. Issue both name shapes as certificate SANs from day one so the later flat adoption changes no client. Exclude MariaDB from SNI exposure entirely: the classic MySQL/MariaDB protocol is server-speaks-first (plaintext greeting, then SSLRequest, then TLS), so no ClientHello ever reaches a raw passthrough listener and no client version changes that. This closes the connector-conformance open question negatively. Re-anchor the value proposition on same-engine multiplexing and hostname-stable access: LB-IPAM sharing-key already consolidates across engine types (added to Alternatives) but cannot put two instances of one engine behind one IP. Weigh edge termination honestly: client trust-bootstrap and CA-rotation costs are booked, and a terminate listener dissolves neither the Postgres nor the MariaDB wire constraint. Correct facts against main and upstream: Cilium is 1.19.5; the existing 443 passthrough listeners allow both HTTPRoute and TLSRoute (the cilium/cilium#45559 workaround) while new per-engine listeners are safely TLSRoute-only; mongodb's external Service is already mongos-only; redis-cli emits SNI only with an explicit --sni flag; Cilium's ListenerSet implementation merged upstream and ships in 1.20. Name the real admission enforcement point: apps.cozystack.io kinds are served by the aggregated cozystack-api, whose writes bypass kube-apiserver admission, so the plaintext-external rejection lands in cozystack-api's write path. Reference the structured expose model as the natural trigger surface and route owner. Assisted-By: Claude Signed-off-by: Aleksei Sviridkin --- .../external-database-exposure/README.md | 144 ++++++++++-------- 1 file changed, 80 insertions(+), 64 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index 54aac9e..d87a071 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -4,17 +4,20 @@ - **Title:** `External database exposure via Gateway API TLS-passthrough (SNI) and end-to-end TLS` - **Author(s):** `@lexfrei` - **Date:** `2026-06-24` -- **Status:** Draft +- **Status:** Review ## Overview -Today every managed database a tenant exposes externally gets its own `LoadBalancer` Service, and therefore its own public IP. A tenant with ten external databases burns ten IPs, and there is no managed end-to-end TLS story that does not terminate somewhere in the middle. This proposal exposes managed databases through the Gateway API TLS-passthrough listeners that Cozystack already operates, routed by SNI on each engine's native port. Multiple databases per tenant collapse onto a single LoadBalancer IP, and because a passthrough listener never terminates TLS, the certificate the external client validates is byte-identical to the operator-issued server certificate the database already presents inside the cluster — end-to-end TLS with no second certificate, no re-encryption hop, and no private key held at the edge. +Today every managed database a tenant exposes externally gets its own `LoadBalancer` Service, and therefore its own public IP. Cross-engine consolidation of those IPs is partly achievable already — Cilium LB-IPAM can share one IP across Services whose ports do not conflict (see Context) — but nothing today can put two Postgres instances behind one IP (same port, guaranteed conflict), and nothing gives the tenant a stable, per-database hostname with a managed end-to-end TLS story that does not terminate somewhere in the middle. This proposal exposes managed databases through the Gateway API TLS-passthrough listeners that Cozystack already operates, routed by SNI on each engine's native port. Any number of databases per tenant — including multiple instances of the same engine, the case port-based IP sharing structurally cannot serve — collapse onto a single LoadBalancer IP, and because a passthrough listener never terminates TLS, the certificate the external client validates is byte-identical to the operator-issued server certificate the database already presents inside the cluster — end-to-end TLS with no second certificate, no re-encryption hop, and no private key held at the edge. + +The design ships on the Cilium the platform runs today: passthrough listeners use distinct per-engine subdomain hostnames (`*..`), which avoid the listener-hostname overlap that Cilium isolates correctly only from 1.20 onward. The flat `*.` scheme — shorter `.` connection hostnames — is recorded as the refinement to adopt once the platform is on Cilium 1.20, and the SAN plan below makes that adoption additive rather than client-breaking. This is the design-proposal artifact required by `cozystack/cozystack#2816`, and it records the decision that issue asks for. The trade-off that issue frames is CNI mesh encryption (datapath lock-in) versus application-level TLS: this proposal chooses **application-level, operator-owned TLS carried through a non-terminating gateway** for the external leg — no edge termination, no second certificate, no private key at the edge, and no dependency on a particular CNI. The other half of that framing — in-cluster (east-west) pod-to-pod encryption, which never leaves the cluster and is necessarily datapath-specific — is recorded and executed separately under `cozystack/cozystack#2977` (PR `cozystack/cozystack#2984`); it complements this proposal rather than competing with it. It is the external-exposure half of epic `cozystack/cozystack#2811`; the certificate/PKI half is covered by the sibling proposal `design-proposals/unified-tls-pki`, on which this one depends for the trust-anchor object. ## Scope and related proposals - **Depends on:** `design-proposals/unified-tls-pki` — provides the `-ca-cert` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract, and the path differs by engine — redis self-publishes a key-free `-ca-cert` through its forked operator, while postgres and mongodb obtain theirs through the extraction controller in `unified-tls-pki`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. +- **Related:** `design-proposals/structured-external-exposure` (community pull request #29) — replaces the chart-level `external` boolean with a structured, additive `expose` list riding `ExposureClass` / `ServiceExposure` (`cozystack/cozystack#3081`), and lists Gateway/SNI consolidation as the forward-compatible future path. The two proposals meet at the trigger surface: once `expose` lands, "expose this database via SNI-passthrough" is naturally one `expose` entry (an exposure class or scope backed by the tenant Gateway) rather than a new chart toggle, and the per-release route rendering naturally belongs to that orchestration layer. This proposal defines the Gateway-side mechanics either trigger drives; it does not depend on `expose` landing first. - **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). - **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). @@ -24,51 +27,57 @@ All repository paths below refer to the `cozystack/cozystack` repository. ### TLS-passthrough already exists -Cozystack already runs Gateway API TLS-passthrough for layer-7 system services. The `TenantGateway` controller (`internal/controller/tenantgateway/reconciler.go`) renders, for each entry in `TenantGateway.Spec.TLSPassthroughServices`, a listener named `tls-` on port 443 with `Protocol: TLS`, `Mode: Passthrough`, hostname `.`, and `AllowedRoutes` restricted to `TLSRoute`. The default services are `api`, `vm-exportproxy`, and `cdi-uploadproxy`, each with a `TLSRoute` that attaches by `sectionName: tls-` (for example `packages/system/cozystack-api/templates/api-tlsroute.yaml`). The platform runs Cilium 1.19.3 with GatewayClass `cilium`. `TLSRoute` is GA in the Gateway API Standard channel as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0; Cilium 1.19.x still consumes it from the **experimental-channel** CRD, so that CRD must be installed and the operative API version for this design is Cilium's channel, not the upstream graduation. `mode: Passthrough` TLSRoute is supported in Cilium since 1.14. +Cozystack already runs Gateway API TLS-passthrough for layer-7 system services. The `TenantGateway` controller (`internal/controller/tenantgateway/reconciler.go`) renders, for each entry in `TenantGateway.Spec.TLSPassthroughServices`, a listener named `tls-` on port 443 with `Protocol: TLS`, `Mode: Passthrough`, and hostname `.`. One deliberate quirk: those listeners' `AllowedRoutes` spans **both** `HTTPRoute` and `TLSRoute` — every listener on port 443 carries the same kind set, because Cilium collapses same-port listeners whose route kinds diverge and drops the HTTP routes (`cilium/cilium#45559`; see the comment in the reconciler). The default services are `api`, `vm-exportproxy`, and `cdi-uploadproxy`, each with a `TLSRoute` that attaches by `sectionName: tls-` (for example `packages/system/cozystack-api/templates/api-tlsroute.yaml`). The platform runs Cilium 1.19.5 with GatewayClass `cilium`. `TLSRoute` is GA in the Gateway API Standard channel as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0; Cilium 1.19.x still consumes it from the **experimental-channel** CRD, so that CRD must be installed and the operative API version for this design is Cilium's channel, not the upstream graduation. `mode: Passthrough` TLSRoute is supported in Cilium since 1.14. The mechanism this proposal needs is therefore already in production for layer-7 services. The work is to extend it to databases, which speak on native (non-443) ports. ### Passthrough on a native port works; the real constraint is SNI-hostname overlap -Routing a `mode: Passthrough` listener on a non-443 port is supported, not exotic: Cilium has shipped TLSRoute passthrough since 1.14, its own conformance fixtures exercise passthrough listeners on a non-443 port, and field reports route passthrough on database ports (for example a mongo `27017` case). The native port (5432/6379/27017/3306) is not the risk. +Routing a `mode: Passthrough` listener on a non-443 port is supported, not exotic: Cilium has shipped TLSRoute passthrough since 1.14, its own conformance fixtures exercise passthrough listeners on a non-443 port, and field reports route passthrough on database ports (for example a mongo `27017` case). The native port (5432/6379/27017) is not the risk. -The real constraint is **SNI-hostname uniqueness across listeners on one Gateway**. On every released Cilium up to and including 1.19.x, two listeners that share a hostname but differ in port — or a passthrough listener that shares a hostname with an HTTPS-terminate listener — collide and do not isolate correctly (the cross-port hostname-isolation defect; fixed by an upstream change that targets Cilium 1.20 and is not in any release tag as of this writing). This matters here because the listener model below puts every engine-type passthrough listener on the wildcard hostname `*.`, which is the same hostname the tenant Gateway's HTTPS-terminate listeners already use. This proposal keeps the flat `*.` scheme — for clean `.` connection hostnames — and therefore **depends on Cilium 1.20** for correct multi-listener isolation. If that dependency is unacceptable, the documented fallback is distinct per-engine subdomains (`*.pg.`, `*.redis.`, …), which sidestep the overlap at the cost of longer hostnames and per-engine SAN/recipe changes. Either way this is the Phase 1 validation gate (Rollout): the gate validates the shared-hostname case on the target Cilium version, not merely that passthrough binds on a non-443 port. +The real constraint is **hostname overlap across listeners on one Gateway**. On every released Cilium up to and including 1.19.x, two listeners that share a hostname but differ in port — or a passthrough listener that shares a hostname with an HTTPS-terminate listener — collide and do not isolate correctly (the cross-port hostname-isolation defect; fixed on `main` by `cilium/cilium#44889`, which targets Cilium 1.20 — v1.20.0 is at `pre.3` with the final release proposed for late July 2026 — and has an open v1.19 backport, `cilium/cilium#46826`, so no release tag carries it as of this writing). A flat scheme that puts every engine-type passthrough listener on `*.` — the same hostname string the tenant Gateway's HTTPS-terminate listeners already use — walks straight into that defect. This proposal therefore defaults to **distinct per-engine subdomain hostnames** (`*.postgres.`, `*.redis.`, …), which are different hostname strings from `*.` and sidestep the overlap on the Cilium deployed today. The flat scheme is retained as the refinement to adopt once the platform runs Cilium 1.20 (see Rollout and Alternatives); the SAN plan in §6 deliberately covers both name shapes from day one so that adoption is additive, not a client migration. Note the existing wildcard DNS record also needs no change: a DNS wildcard matches across multiple labels (RFC 4592), so `db1.postgres.` already resolves through the `*.` record operators provision today. ### Each external database burns its own IP today -When a database chart sets `external: true`, it provisions a dedicated `Service` of `type: LoadBalancer`: postgres `-external-write` on 5432 (`packages/apps/postgres/templates/external-svc.yaml`), redis `-external-lb` on 6379 (`packages/apps/redis/templates/service.yaml`), mongodb `-external` on 27017 (`packages/apps/mongodb/templates/external-svc.yaml`), kafka on 9094 plus a LoadBalancer per broker (`packages/apps/kafka/templates/kafka.yaml`), and mariadb via a `primaryService` of `type: LoadBalancer` (`packages/apps/mariadb/templates/mariadb.yaml`). Each is a separate IP from the shared MetalLB (or Cilium LB-IPAM) pool; there is no per-tenant IP isolation and no SNI multiplexing. +When a database chart sets `external: true`, it provisions a dedicated `Service` of `type: LoadBalancer`: postgres `-external-write` on 5432 (`packages/apps/postgres/templates/external-svc.yaml`), redis `-external-lb` on 6379 (`packages/apps/redis/templates/service.yaml`), mongodb `-external` on 27017 (`packages/apps/mongodb/templates/external-svc.yaml`, rendered only in sharding mode — it fronts `mongos`; non-sharded replica sets are exposed by the Percona operator as one LoadBalancer per member), kafka on 9094 plus a LoadBalancer per broker (`packages/apps/kafka/templates/kafka.yaml`), and mariadb via a `primaryService` of `type: LoadBalancer` (`packages/apps/mariadb/templates/mariadb.yaml`). Each is a separate IP from the shared MetalLB (or Cilium LB-IPAM) pool; there is no per-tenant IP isolation and no SNI multiplexing. + +One consolidation lever exists today without any Gateway involvement: Cilium LB-IPAM shares one IP across Services that carry the same `lbipam.cilium.io/sharing-key` annotation, as long as their ports do not conflict (conflicting Services are allocated an additional IP into the same sharing-key set; sharing across namespaces requires the `lbipam.cilium.io/sharing-cross-namespace` annotation on both Services). Stamped onto the per-engine LoadBalancer Services above, that would already collapse one Postgres + one Redis + one MongoDB onto one IP — with end-to-end TLS for free, since a plain LoadBalancer to the pod is already non-terminating. What sharing-key structurally cannot do is put two instances of the **same** engine behind one IP (identical port, guaranteed conflict), and it offers no hostname-based routing at all — clients dial bare IPs, and every scale-out of the same engine buys a new one. That same-engine, hostname-routed case is precisely what this proposal adds; see Alternatives. ### The certificate hooks already exist -The external hostname is `.<_namespace.host>`, where `_namespace.host` is the tenant apex. The SAN-injection hook already exists on `main` for postgres: it adds the hostname to the CNPG `Cluster` via `spec.certificates.serverAltDNSNames` in `packages/apps/postgres/templates/db.yaml`, gated by the `tls.enabled` tri-state in `_tls.tpl` (which defaults on when `external` is true). The other engines acquire the same SAN hook through the per-app TLS series tracked by the `unified-tls-pki` proposal — on `main` today redis and mariadb carry no TLS templates at all, and their certificate/SAN support lives in the open PRs `cozystack/cozystack#2729` and `cozystack/cozystack#2680`. The trust anchor (`ca.crt`) is delivered to the tenant through that same `unified-tls-pki` contract. So this proposal builds on hooks that are present for postgres and arriving for the rest — it does not invent new ones. +The external hostname today is `.<_namespace.host>`, where `_namespace.host` is the tenant apex. The SAN-injection hook already exists on `main` for postgres: it adds the hostname to the CNPG `Cluster` via `spec.certificates.serverAltDNSNames` in `packages/apps/postgres/templates/db.yaml`, gated on `tls.enabled` **and** `external` (the tri-state in the chart's own `_tls.tpl` resolves `tls.enabled` to the value of `external` when unset). The other engines acquire the same SAN hook through the per-app TLS series tracked by the `unified-tls-pki` proposal — on `main` today redis and mariadb carry no TLS templates at all, and their certificate/SAN support lives in the open PRs `cozystack/cozystack#2729` and `cozystack/cozystack#2680`. The trust anchor (`ca.crt`) is delivered to the tenant through that same `unified-tls-pki` contract. So this proposal builds on hooks that are present for postgres and arriving for the rest — it does not invent new ones; what changes under the subdomain scheme is the SAN value set those hooks inject (§6). ### The one-IP-per-tenant ceiling -Multiple per-tenant Gateways cannot share a single LoadBalancer IP on current Cilium. Cilium LB-IPAM shares one IP across Services via `lbipam.cilium.io/sharing-key` only when their ports do not conflict; every Gateway claims `443/TCP`, so the shared-IP path is inactive on that port collision (`packages/extra/gateway/README.md`; the port-conflict rule is documented in the Cilium LB-IPAM docs). Within a single Gateway, a parent and its inheriting children share one IP. `Gateway.spec.listeners` is hard-capped at 64 (Gateway API CRD `MaxItems`). The practical consequence: the unit of IP sharing is one tenant Gateway, and a tenant's database fan-out draws from the 64-listener budget shared with the existing http/https/https-apex and per-child-apex wildcard listeners. +Multiple per-tenant Gateways cannot share a single LoadBalancer IP on current Cilium: every Gateway claims `443/TCP`, so the sharing-key path is inactive on that port collision (`packages/extra/gateway/README.md`). Within a single Gateway, a parent and its inheriting children share one IP. `Gateway.spec.listeners` is hard-capped at 64 (Gateway API CRD `MaxItems`). The practical consequence: the unit of IP sharing is one tenant Gateway, and a tenant's database fan-out draws from the 64-listener budget shared with the listeners the controller already renders (in the wildcard cert modes: http, https, https-apex, and one wildcard listener per inheriting child apex; in HTTP-01 mode: one per-app listener per attached hostname, which is the fan-out that approaches the cap first). + +Both halves of that ceiling now have a concrete upstream answer converging on the same release: Gateway API v1.5.0 graduated `ListenerSet` to the Standard channel (listeners defined as separate, per-namespace objects merged onto a shared Gateway, explicitly lifting the 64-listener limit), and Cilium merged its ListenerSet implementation on 2026-06-26 (`cilium/cilium#46303`, closing CFP `cilium/cilium#42756`), expected to ship in Cilium 1.20. ListenerSet consolidates many tenants' listeners onto **one shared Gateway** (and thus one IP); it does not make N separate Gateways share an IP. ### The problem Two concrete problems, both solved by SNI-passthrough plus certificate reuse: -1. **IPv4 scarcity.** N externally-exposed databases cost N public IPs per tenant. This does not scale, and IPs are the scarce resource. -2. **No managed end-to-end TLS.** Exposing a database externally today either lacks a managed-TLS-to-the-client story or would require terminating at the edge — which means a second certificate, a re-encryption hop, and the edge holding a private key for a database it does not own. +1. **IPv4 scarcity, specifically for same-engine fan-out and hostname-stable access.** N externally-exposed databases cost N public IPs per tenant. Port-based IP sharing (sharing-key) can consolidate across engine types but is structurally blind to the same-engine case — two Postgres instances always cost two IPs — and it routes by port on a bare IP, so there is no per-database hostname and no stable connection identity. This does not scale, and IPs are the scarce resource. +2. **No managed end-to-end TLS with a verifiable identity.** Exposing a database externally today either lacks a managed-TLS-to-the-client story or would require terminating at the edge — which means a second certificate, a re-encryption hop, and the edge holding a private key for a database it does not own. ## Goals -- Multiple external databases per tenant share a single LoadBalancer IP, distinguished by SNI on their native ports. +- Multiple external databases per tenant — including multiple instances of the same engine — share a single LoadBalancer IP, distinguished by SNI on their native ports. - End-to-end TLS using the **same** operator-issued certificate client-side and pod-side: no edge termination, no edge-held private key, no second certificate. - Trust established through the existing `ca.crt`-only object from `unified-tls-pki` — no new PKI, issuer, or rotation machinery. +- Ships on the Cilium the platform runs today (1.19.x); the flat-hostname refinement is additive when Cilium 1.20 arrives, not a client migration. - A minimal, auditable API surface that reuses the existing passthrough listener and SAN-injection hooks. - Per-engine honesty: ship what fits the model, defer what does not, and say why. ### Non-goals - **East-west / pod-to-pod CNI encryption** — separate workstream (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`); complementary, not designed here. -- **Kafka external SNI exposure** — deferred; its per-broker advertised-address topology does not fit a single SNI entrypoint (see the matrix). Kafka keeps today's per-broker LoadBalancer behavior. -- **MongoDB non-sharded replica-set SNI exposure** — deferred; per-member LoadBalancers plus `rs.conf` rewrite are the same class of problem as Kafka. Only the sharded `mongos` topology fits. -- **Edge TLS termination / re-encryption / `BackendTLSPolicy`** — explicitly rejected; the recorded decision lands on non-terminating passthrough. +- **Kafka external SNI exposure** — deferred; its per-broker advertised-address topology needs per-broker routes plus advertised-listener coordination (see the matrix). Kafka keeps today's per-broker LoadBalancer behavior. +- **MongoDB non-sharded replica-set SNI exposure** — deferred; per-member LoadBalancers plus `rs.conf` rewrite are the same advertised-address class of problem as Kafka. Only the sharded `mongos` topology fits. +- **MariaDB SNI exposure** — not achievable at the wire-protocol level (the server speaks first; see the matrix); excluded until upstream grows a TLS-first mode. +- **Edge TLS termination / re-encryption / `BackendTLSPolicy`** — rejected after weighing (see Alternatives); the recorded decision lands on non-terminating passthrough. - **New PKI, issuer, or certificate-rotation machinery** — out of scope; this reuses `unified-tls-pki` entirely. -- **Multi-Gateway single-IP sharing** — out of scope until Cilium implements ListenerSet; the design targets one IP per tenant Gateway. +- **Multi-Gateway single-IP sharing** — out of scope; the design targets one IP per tenant Gateway, with ListenerSet-based consolidation onto a shared Gateway as the future lever (see Open questions). - **Replacing `TLSPassthroughServices`** — the existing layer-7 field stays; this adds a parallel field. - **SNI confidentiality (ECH)** — database clients do not use ECH; hiding the external hostname is not a goal. - **Non-Cilium GatewayClasses** — the design assumes GatewayClass `cilium`. @@ -80,13 +89,13 @@ Two concrete problems, both solved by SNI-passthrough plus certificate reuse: ```mermaid flowchart TB subgraph ext["External clients"] - C1["psql
SNI=db1.alice.example.org:5432"] - C2["psql
SNI=db2.alice.example.org:5432"] - C3["redis-cli
SNI=cache.alice.example.org:6379"] + C1["psql
SNI=db1.postgres.alice.example.org:5432"] + C2["psql
SNI=db2.postgres.alice.example.org:5432"] + C3["redis client
SNI=cache.redis.alice.example.org:6379"] end subgraph gw["Tenant Gateway — ONE LoadBalancer IP, one listener per engine type"] - L1["listener tls-postgres
5432 / Passthrough
hostname *.alice.example.org"] - L2["listener tls-redis
6379 / Passthrough
hostname *.alice.example.org"] + L1["listener tls-postgres
5432 / Passthrough
hostname *.postgres.alice.example.org"] + L2["listener tls-redis
6379 / Passthrough
hostname *.redis.alice.example.org"] end subgraph rt["Per-release TLSRoutes (SNI hostname)"] R1["TLSRoute db1"] @@ -109,58 +118,61 @@ flowchart TB CA -.->|"verifies"| C3 ``` -A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **multiple `TLSRoute` objects can attach to one listener**, each selecting its backend by its own `spec.hostnames`. SNI-based routing therefore works on any TCP port, not only 443, and — crucially — it does not need a listener per database. The design uses **one passthrough listener per engine type**, on that engine's native port, with a wildcard hostname: `tls-postgres` on 5432, `tls-redis` on 6379, `tls-mongos` on 27017, `tls-mariadb` on 3306 — each `mode: Passthrough`, hostname `*.` (or unrestricted), `AllowedRoutes` limited to `TLSRoute`. Every database release of that engine then attaches a per-release `TLSRoute` carrying `spec.hostnames: ["."]`, and the Gateway SNI-routes each connection to the right backend. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. +A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **multiple `TLSRoute` objects can attach to one listener**, each selecting its backend by its own `spec.hostnames`. SNI-based routing therefore works on any TCP port, not only 443, and — crucially — it does not need a listener per database. The design uses **one passthrough listener per engine type**, on that engine's native port, with a per-engine wildcard hostname: `tls-postgres` on 5432 with hostname `*.postgres.`, `tls-redis` on 6379 with `*.redis.`, `tls-mongos` on 27017 with `*.mongo.` — each `mode: Passthrough`, `AllowedRoutes` limited to `TLSRoute`. Every database release of that engine then attaches a per-release `TLSRoute` carrying `spec.hostnames: [".."]`, and the Gateway SNI-routes each connection to the right backend. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP. (Restricting the new listeners to `TLSRoute` is safe where the 443 listeners could not be: the `cilium/cilium#45559` same-port kind-divergence collapse applies per port, and each engine listener is alone on its port.) This keeps listener consumption at **O(engine types)** — at most four or five — rather than O(database instances). A tenant running thirty Postgres releases spends one `tls-postgres` slot, not thirty, so the 64-listener budget (§3) stops being a per-database ceiling and the consolidation goal scales with instance count, not against it. -**The shared-hostname dependency is load-bearing.** Putting `tls-postgres` (5432) and `tls-redis` (6379) both on `*.` means several listeners share one hostname across ports, and that hostname also overlaps the Gateway's HTTPS-terminate listeners. As noted in Context, Cilium isolates that case correctly only from 1.20 onward, so this scheme depends on Cilium 1.20. The per-release `TLSRoute` hostnames (`.`) are always distinct and are not the issue; the overlap is at the listener level. The flat scheme is chosen for clean connection hostnames; distinct per-engine subdomains are the documented fallback if the Cilium 1.20 dependency is unacceptable. +**Why per-engine subdomains, and why the flat scheme waits.** Distinct per-engine hostnames (`*.postgres.` ≠ `*.`) keep every passthrough listener's hostname string disjoint from the HTTPS-terminate listeners', which is what makes the model work on the deployed Cilium 1.19.x — the cross-port hostname-isolation defect (Context) triggers on listeners sharing a hostname, and here none do. The flat `*.` scheme buys shorter connection hostnames (`db1.`) at the price of exactly that overlap, so it is gated on a Cilium carrying the isolation fix (1.20, or a 1.19.x patch if the open backport lands — see Open questions) and recorded as a later, additive phase: new flat-hostname listeners join the Gateway, per-release routes gain the flat hostname, and existing subdomain hostnames keep resolving and matching — nothing already connected has to move (§6 keeps both name shapes in the certificate SANs from the start). The per-release `TLSRoute` hostnames are always distinct from each other and are not the issue; the overlap risk is at the listener level. (Whether to skip the subdomain generation entirely and wait for the fixed Cilium to ship flat-only from day one is recorded as an open question — it is a scheduling call, not a design fork.) One spec note for precision: a Gateway API wildcard hostname is a *suffix* match covering one **or more** labels (`*.example.com` matches `test.example.com` and `foo.test.example.com` alike), so once a flat `*.` listener exists it also covers the subdomain-shaped names; SNI match precedence (most-specific first) keeps the routing deterministic. Native ports are chosen over forcing everything onto 443 because the latter buys nothing: it does not improve IP consolidation (SNI already does that on any port) and it breaks client ergonomics and tooling defaults (`psql -h host` assumes 5432) while still demanding direct-TLS. The all-on-443 variant is retained as a documented opt-in for operators who want a single-port firewall surface (see Alternatives). -**The Postgres caveat is load-bearing.** libpq has historically performed a StartTLS-style negotiation: it sends a plaintext `SSLRequest` and waits for the server's single-byte reply *before* the TLS `ClientHello`. There is no SNI in the first packet, so a passthrough listener cannot route it. This is resolved by `sslnegotiation=direct` (libpq, PostgreSQL 17+), which sends the `ClientHello` immediately, carrying SNI. It comes with hard prerequisites, not preferences: the **server must also be PostgreSQL 17+**; `sslnegotiation=direct` requires `sslmode=require` or stronger; direct SSL mandates ALPN (`postgresql`); and SNI is emitted only when the client dials by **hostname** with `sslsni=1` (the default) — an IP literal carries no SNI. Driver coverage is uneven (recent pgjdbc 42.7.x and Npgsql 9.0+ support it; node-postgres historically did not). Any client older than these, against a pre-17 server, or with `sslmode` below `require`, falls back to the legacy negotiation and cannot be SNI-routed. Postgres-over-passthrough is therefore conditional on direct-TLS-capable clients and is opt-in, not default-on. +**The Postgres caveat is load-bearing.** libpq has historically performed a StartTLS-style negotiation: it sends a plaintext `SSLRequest` and waits for the server's single-byte reply *before* the TLS `ClientHello`. There is no SNI in the first packet, so a passthrough listener cannot route it. This is resolved by `sslnegotiation=direct` (libpq, PostgreSQL 17+), which sends the `ClientHello` immediately, carrying SNI. It comes with hard prerequisites, not preferences: the **server must also be PostgreSQL 17+**; `sslnegotiation=direct` requires `sslmode=require` or stronger; direct SSL mandates ALPN (`postgresql`); and SNI is emitted only when the client dials by **hostname** with `sslsni=1` (the default) — an IP literal carries no SNI. Driver coverage is broad by now — pgjdbc 42.7.4+, Npgsql 9.0+, node-postgres, pgx v5.7.5+, lib/pq, and psycopg linked against a libpq 17 all support direct negotiation — with stragglers (sqlx has an open request; asyncpg offers only a lower-level flag without the standard ALPN). Any client older than these, against a pre-17 server, or with `sslmode` below `require`, falls back to the legacy negotiation and cannot be SNI-routed. Postgres-over-passthrough is therefore conditional on direct-TLS-capable clients and is opt-in, not default-on. ### 2. Certificate reuse is a property of passthrough (the WS5 core) Passthrough *is* the certificate reuse. Because the Gateway in `mode: Passthrough` does not terminate TLS, it forwards the raw handshake bytes to the backend pod. The certificate the external client validates is byte-identical to the operator-issued server certificate the database already presents internally — CNPG-managed for postgres, Strimzi-managed for kafka, PSMDB-managed for mongo. There is no second certificate, no re-encryption, and no new issuance. End-to-end TLS is a consequence of not terminating, not a feature to build. -One subtlety the "single certificate" framing hides: an external client doing `sslmode=verify-full` (or the equivalent) validates the presented certificate's SANs against the hostname it dialed. CNPG presents one server certificate across its `-rw`/`-r`/`-ro` services with no per-listener split, so the external passthrough hostname `.` **must be in that certificate's `serverAltDNSNames`** for verify-full to pass — which is exactly what the SAN hook below injects. Engines with a per-listener certificate model (Strimzi `brokerCertChainAndKey`) could serve a different cert externally, but the fitting engines here present one server cert, so "reuse" is literal. +One subtlety the "single certificate" framing hides: an external client doing `sslmode=verify-full` (or the equivalent) validates the presented certificate's SANs against the hostname it dialed. CNPG presents one server certificate across its `-rw`/`-r`/`-ro` services with no per-listener split, so the external passthrough hostname **must be in that certificate's `serverAltDNSNames`** for verify-full to pass — which is exactly what the SAN hook below injects. Engines with a per-listener certificate model (Strimzi `brokerCertChainAndKey`) could serve a different cert externally, but the fitting engines here present one server cert, so "reuse" is literal. WS5 adds nothing beyond two hooks that already exist: 1. **Trust-anchor delivery** — the `-ca-cert` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. -2. **SAN coverage** — the chart already injects `.` into the operator-issued certificate (postgres `serverAltDNSNames` in `db.yaml`, gated by `tls.enabled`). The SAN the client's SNI will carry is already in the certificate. +2. **SAN coverage** — the chart already injects the external hostname into the operator-issued certificate (postgres `serverAltDNSNames` in `db.yaml`, gated on TLS and external). Under this proposal the hook injects **both** name shapes — `..` and `.` — so the certificate is valid for the subdomain scheme shipping now and the flat scheme arriving with Cilium 1.20 (§6). Stated plainly, where this does **not** work: -- **Kafka** — the external listener advertises per-broker addresses; after bootstrap the client connects directly to each broker. A single SNI endpoint cannot represent N per-broker endpoints. Deferred. -- **MongoDB non-sharded** — one LoadBalancer per member plus an `rs.conf` rewrite; the driver reaches each member by its advertised address. A single SNI front cannot represent the replica-set topology. Only sharded `mongos` fits. +- **Kafka** — the external listener advertises per-broker addresses; after bootstrap the client connects directly to each broker (see the matrix for what making it fit would take). Deferred. +- **MongoDB non-sharded** — one LoadBalancer per member plus an `rs.conf` rewrite; the driver reaches each member by its advertised address. Only sharded `mongos` fits. +- **MariaDB** — the wire protocol precludes it entirely (see the matrix). - **Clients that omit or mis-emit SNI** — a passthrough listener has no certificate to fall back to, so a missing SNI is a hard connection failure, not a downgrade. This includes pre-direct-TLS Postgres clients. ### 3. IP consolidation and its limits -The headline payoff is IPv4 scarcity relief. Today ten external databases cost ten public IPs (more for Kafka and Mongo replica sets, which add per-broker / per-member IPs). Under this proposal, N SNI-routed databases on one tenant Gateway share one IP, differentiated by SNI on their native ports. Ten databases become one IP. +The headline payoff is IPv4 scarcity relief where no existing mechanism reaches: same-engine fan-out behind one IP, with a stable hostname per database. Cross-engine consolidation alone does not need this design — sharing-key delivers that today (Context, Alternatives). What sharing-key cannot deliver is ten Postgres instances on one IP (same port, guaranteed conflict) or any hostname-based identity; under this proposal ten SNI-routed databases of any mix share one tenant-Gateway IP, each reachable at its own name. The ceiling, stated honestly: -- `Gateway.spec.listeners` is capped at 64. Because database listeners are **one per engine type** (§1), not one per release, their contribution to that budget is bounded by the four or five supported engines no matter how many instances a tenant runs; the per-release fan-out lands on `TLSRoute` objects, which are not capped. The remaining slots go to the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. A tenant therefore approaches the cap through child-apex fan-out, not database fan-out; the controller's existing advice to split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true` still applies there. (The 64 cap is on one Gateway's inline `listeners` array; a future ListenerSet could raise the effective count, see Open questions.) -- Multi-Gateway IP sharing is not possible today (the one-IP-per-tenant blocker above). The win is one IP per tenant Gateway. Lifting it depends on Cilium implementing ListenerSet, which would let many tenants' listeners be consolidated onto **one shared Gateway** (and thus one IP) — it does not make N separate Gateways share an IP. ListenerSet graduated toward the Gateway API Standard channel in v1.5, but Cilium does not implement it on any stable release (tracked by the Cilium CFP `cilium#42756`, implementation PR `cilium#46303` in progress). +- `Gateway.spec.listeners` is capped at 64. Because database listeners are **one per engine type** (§1), not one per release, their contribution to that budget is bounded by the four or five supported engines no matter how many instances a tenant runs; the per-release fan-out lands on `TLSRoute` objects, which are not capped. The remaining slots go to the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. A tenant therefore approaches the cap through child-apex fan-out, not database fan-out; the controller's existing advice to split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true` still applies there. The cap itself is on one Gateway's inline `listeners` array — `ListenerSet` (Standard channel since Gateway API v1.5.0) exists precisely to lift it, and Cilium's implementation merged on 2026-06-26 (`cilium/cilium#46303`), expected in 1.20. +- Multi-Gateway IP sharing is not possible today (the one-IP-per-tenant blocker above). The win is one IP per tenant Gateway. The future lever is ListenerSet-based consolidation of many tenants' listeners onto **one shared Gateway** (and thus one IP) — it does not make N separate Gateways share an IP. With the Cilium implementation merged and shipping in 1.20, this moves from "speculative" to "a platform-upgrade decision" (see Open questions). ### 4. Per-engine treatment | Engine | Native port | Fits SNI-passthrough? | Client requirement | Recommendation | | --- | --- | --- | --- | --- | -| Redis | 6379 | Yes — immediate TLS, no pre-TLS negotiation | `--tls` + `ca.crt`; modern client emits SNI from a hostname | cleanest fit; ships opt-in first, default-on candidate once validated | +| Redis | 6379 | Yes — immediate TLS, no pre-TLS negotiation | `--tls` + `ca.crt`; library clients (go-redis, redis-py, Lettuce, node-redis) emit SNI from the hostname; `redis-cli` sends SNI **only** with an explicit `--sni` flag | cleanest fit; ships opt-in first, default-on candidate once validated | | PostgreSQL (CNPG) | 5432 | Yes, only with direct-TLS | `sslnegotiation=direct` + `sslmode=verify-full` + `ca.crt`, hostname (not IP); libpq PG17+ **and** server PG17+ | opt-in (pre-PG17 client or server fails closed) | -| MongoDB — sharded (mongos) | 27017 | Yes — single stateless endpoint | `tls=true` + `tlsCAFile`; seed pointing at the SNI hostname | opt-in (sharded only) | -| MariaDB | 3306 | Likely — modern connectors do TLS-first with SNI | TLS connector emitting SNI + `ca.crt`; verify per connector | opt-in pending connector conformance | +| MongoDB — sharded (mongos) | 27017 | Yes — single stateless endpoint (the chart's external Service is already mongos-only) | `tls=true` + `tlsCAFile`; seed pointing at the SNI hostname | opt-in (sharded only) | | MongoDB — replica set | 27017 | No — per-member LB + `rs.conf` rewrite | n/a | deferred | -| Kafka (Strimzi) | 9094 | No — per-broker advertised addresses | n/a | deferred | +| MariaDB | 3306 | **No — wire protocol precludes it** | n/a | excluded until upstream grows TLS-first | +| Kafka (Strimzi) | 9094 | Not in v1 — per-broker advertised addresses need per-broker routes + advertised-listener coordination | n/a | deferred | -Kafka is deferred because its wire protocol redirects: the client connects to a bootstrap endpoint, receives metadata listing per-broker advertised host:port pairs, then opens direct connections to each broker. Strimzi's `type: loadbalancer` external listener provisions a LoadBalancer per broker plus a bootstrap LB precisely for this. A single SNI front cannot satisfy the per-broker fan-out — this is a topology problem, not a certificate one (Strimzi could serve any cert per listener). Making Kafka fit would require either per-broker SNI listeners and TLSRoutes (N listeners against the 64-listener cap, partially defeating the consolidation goal) plus rewriting Strimzi's advertised listeners, or a Kafka-aware re-advertising proxy — both out of scope. Revisit if per-broker SNI proves worth the listener budget. +**MariaDB (and MySQL) cannot be SNI-routed, at any client version.** The classic protocol on 3306 is server-speaks-first: the server sends a plaintext initial handshake packet, the client answers with an `SSLRequest`, and only then does the TLS handshake begin. A raw passthrough listener waits for a `ClientHello` that will never come while the client waits for a greeting the listener will never send — a mutual deadlock, not a degraded mode. No TLS-first wire mode exists in any MySQL or MariaDB release; MySQL 8.1 added client-side SNI (`MYSQL_OPT_TLS_SNI_SERVERNAME` / `--tls-sni-servername`), but that only names the server inside the eventual `ClientHello` — the plaintext preamble still precedes it, so the listener never gets to route. (The upstream feature request to allow TLS-first handshakes, MySQL bug #84849, remains open.) MySQL's X Protocol on 33060 is also STARTTLS-style (a plaintext capabilities exchange precedes TLS), so it offers no escape hatch either. Routing MariaDB by SNI would take a protocol-aware proxy that fabricates the server greeting — out of scope. MariaDB keeps its dedicated LoadBalancer; sharing-key remains the only consolidation available to it. This also closes the "MariaDB connector conformance" question the first draft left open: no connector can conform, because the server, not the client, owns the first packet. + +**Kafka** is deferred as a scope decision, not an impossibility. The wire protocol redirects: the client connects to a bootstrap endpoint, receives metadata listing per-broker advertised host:port pairs, then opens direct connections to each broker. Strimzi's `type: loadbalancer` external listener provisions a LoadBalancer per broker plus a bootstrap LB precisely for this. Under the shared-listener model the Gateway-side cost of fitting Kafka is modest — one bootstrap route plus N per-broker `TLSRoute`s on the one engine listener (routes are uncapped; and since a Gateway API wildcard hostname is a multi-label suffix match, `broker-0.mykafka.kafka.` still matches the `*.kafka.` listener). The real coordination burden is on the Kafka side: every broker must advertise its own SNI hostname (Strimzi supports per-broker `advertisedHost` overrides, and its `type: ingress` listener already implements exactly this SNI-passthrough pattern against ingress-nginx), the certificate SANs must cover the per-broker names, and route lifecycle must track broker scaling. That is real integration work with an operator-owned moving part, and none of the payoff differs from the simpler engines — so it is deferred, to be revisited once the model is proven on Redis/Postgres/mongos. The same advertised-address shape applies to non-sharded MongoDB replica sets. ### 5. API and controller extension -The trigger and listener synthesis stay in the controller; database charts do not render Gateway plumbing. Database charts already receive the `_cluster` values channel and read parts of it (for example `packages/apps/postgres/templates/db.yaml` reads `_cluster.scheduling`, and mongodb reads `_cluster["cluster-domain"]`), but they do not read the gateway-discovery keys that `cozystack-api` consumes (the gateway-enabled flag and the gateway name) and have no logic to locate the tenant Gateway. Teaching every database chart that discovery dance would duplicate it across five charts and couple application charts to networking topology. The controller already owns listener synthesis; keep it there. +The trigger and listener synthesis stay in the controller; database charts do not render Gateway plumbing. Database charts already receive the `_cluster` values channel and read parts of it (for example `packages/apps/postgres/templates/db.yaml` reads `_cluster.scheduling`, and mongodb reads `_cluster["cluster-domain"]`), but they do not read the gateway-discovery keys (the gateway-enabled flag and the gateway name) and have no logic to locate the tenant Gateway. Teaching every database chart that discovery dance would duplicate it across five charts and couple application charts to networking topology. The controller already owns listener synthesis; keep it there. Two objects implement the model, at different cardinalities: one shared listener per engine type, and one `TLSRoute` per database release. @@ -176,11 +188,13 @@ The **listener** is the new API surface. The existing `TLSPassthroughServices [] // - Port: 1..65535; unique within the list; must not collide with a // synthesized layer-7 listener port (443) or another entry. // - Hostname: optional; a valid DNS wildcard or exact hostname; defaults -// to "*." when empty. +// to "*.." when empty (the per-engine subdomain +// scheme; see §1 for why the flat "*." form is gated +// on Cilium 1.20). type TLSPassthroughListener struct { Name string `json:"name"` // listener suffix -> "tls-" (e.g. "postgres") - Port int32 `json:"port"` // native port (5432/6379/27017/3306) - Hostname string `json:"hostname,omitempty"` // wildcard SNI match, default "*." + Port int32 `json:"port"` // native port (5432/6379/27017) + Hostname string `json:"hostname,omitempty"` // wildcard SNI match, default "*.." } // TLSPassthroughListeners renders one Passthrough listener "tls-" on @@ -190,69 +204,71 @@ type TLSPassthroughListener struct { TLSPassthroughListeners []TLSPassthroughListener `json:"tlsPassthroughListeners,omitempty"` ``` -The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` attaches to the shared `tls-` listener by `sectionName`, `spec.hostnames: ["."]` carries the SNI match, and `spec.rules[].backendRefs` is a standard `BackendRef` (whose embedded `BackendObjectReference` points at the database Service on its native port, cross-namespace via `ReferenceGrant`). `TLSRoute` is GA as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0, but Cilium 1.19.x consumes the experimental-channel CRD — pin to whatever API version the targeted Cilium ships, not the upstream `v1` graduation. +The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` attaches to the shared `tls-` listener by `sectionName`, `spec.hostnames: [".."]` carries the SNI match, and `spec.rules[].backendRefs` is a standard `BackendRef` (whose embedded `BackendObjectReference` points at the database Service on its native port, cross-namespace via `ReferenceGrant`). `TLSRoute` is GA as `gateway.networking.k8s.io/v1` since Gateway API v1.5.0, but Cilium 1.19.x consumes the experimental-channel CRD — pin to whatever API version the targeted Cilium ships, not the upstream `v1` graduation. -The controller change is to render, per `TLSPassthroughListeners` entry, one Passthrough listener on the supplied `Port` instead of the hardcoded 443, and to attach each per-release `TLSRoute` by `sectionName: tls-` — the same attachment pattern as the existing `api-tlsroute.yaml`, except that many routes share one listener and the Gateway disambiguates them by SNI hostname. Both are populated by the Tenant / HelmRelease orchestration layer that already knows the tenant Gateway and the database release — not the database chart and not the human — so a database's surface stays a single `external`-adjacent toggle. The engine-type listener is created on first exposure of that engine and removed once its last release is gone; per-release add/remove only touches the route, never the shared listener. +The controller change is to render, per `TLSPassthroughListeners` entry, one Passthrough listener on the supplied `Port` instead of the hardcoded 443, and to attach each per-release `TLSRoute` by `sectionName: tls-` — the same attachment pattern as the existing `api-tlsroute.yaml`, except that many routes share one listener and the Gateway disambiguates them by SNI hostname. Both are populated by the orchestration layer that already knows the tenant Gateway and the database release — not the database chart and not the human. The natural shape of that orchestration is the structured `expose` model of `design-proposals/structured-external-exposure` (community pull request #29): an `expose` entry whose class or scope selects SNI-passthrough is the tenant-facing trigger, and the layer that reconciles `expose` entries into `ServiceExposure` objects is the same layer that renders the per-release `TLSRoute` and reference-counts the shared engine listener. Until that model lands, the interim trigger is the engine's `external`-adjacent toggle, with the same rendering responsibility held by the Tenant / HelmRelease orchestration. Either way, the engine-type listener is created on first exposure of that engine and removed once its last release is gone; per-release add/remove only touches the route, never the shared listener. ### 6. Trust-anchor and SAN flow -End to end: the chart injects the external hostname `.` into the operator-issued certificate's SAN list (gated by the `tls.enabled` tri-state, which defaults on with `external`). The operator issues the server certificate with that SAN. The `ca.crt`-only object is projected to the tenant. The external client connects with SNI `.` and verifies the presented certificate against that `ca.crt`. The passthrough listener forwards the raw handshake; the pod presents the SAN-matching certificate. No hop in the path terminates TLS. +End to end: the chart injects the external hostnames into the operator-issued certificate's SAN list (gated on TLS and external) — under this proposal that is **both** `..` (the subdomain scheme shipping now) and `.` (the flat scheme arriving with Cilium 1.20), so a later listener-scheme change never invalidates issued certificates or forces client reconfiguration. The operator issues the server certificate with those SANs. The `ca.crt`-only object is projected to the tenant. The external client connects with SNI `..` and verifies the presented certificate against that `ca.crt`. The passthrough listener forwards the raw handshake; the pod presents the SAN-matching certificate. No hop in the path terminates TLS. ## User-facing changes -A database gains an `external`-adjacent toggle to select passthrough/SNI mode. Per-engine connection recipes are documented: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=."` (libpq and server both PG17+), `redis-cli --tls --cacert ca.crt -h .`, `mongosh --tls --tlsCAFile ca.crt --host .`, `mysql --ssl-mode=VERIFY_IDENTITY --ssl-ca=ca.crt -h .`. Kafka and non-sharded MongoDB keep today's per-LoadBalancer behavior. +A database gains an `external`-adjacent toggle to select passthrough/SNI mode (expected to become an `expose` entry once the structured-exposure model lands; see §5). Per-engine connection recipes are documented against the subdomain hostnames: `psql "sslnegotiation=direct sslmode=verify-full sslrootcert=ca.crt host=.postgres."` (libpq and server both PG17+), `redis-cli --tls --cacert ca.crt --sni .redis. -h .redis.` (the explicit `--sni` matters: redis-cli does not derive SNI from `-h`; library clients do), `mongosh --tls --tlsCAFile ca.crt --host .mongo.`. Kafka, MariaDB, and non-sharded MongoDB keep today's per-LoadBalancer behavior. ## Upgrade and rollback compatibility -The default stays today's per-database LoadBalancer, so no existing external database changes its IP on upgrade. Passthrough/SNI mode is opt-in. Migrating an existing external database to SNI mode is a breaking, opt-in change — the IP changes and the client must be reconfigured (new host, direct-TLS for Postgres); it is not automatic. The existing `TLSPassthroughServices` field continues to work unchanged, and reverting the feature removes the per-release routes and the engine-type listeners without touching the database's own PKI. +The default stays today's per-database LoadBalancer, so no existing external database changes its IP on upgrade. Passthrough/SNI mode is opt-in. Migrating an existing external database to SNI mode is a breaking, opt-in change — the IP changes and the client must be reconfigured (new host, direct-TLS for Postgres); it is not automatic. The later flat-hostname phase is additive by construction (both SANs issued from day one; subdomain hostnames keep working). The existing `TLSPassthroughServices` field continues to work unchanged, and reverting the feature removes the per-release routes and the engine-type listeners without touching the database's own PKI. ## Security -The edge never holds the database's private key — the central strength of passthrough over termination. SNI is sent in cleartext on TLS 1.3 except under ECH, which database clients do not use, so the external hostname `.` is observable on the wire; this is no worse than DNS or SNI exposure for any TLS service, and the payload stays encrypted end-to-end. A missing or mis-emitted SNI is a hard connection failure (a passthrough listener has no certificate to fall back to) — fail-closed, which is the secure default. Exposing a database externally with TLS explicitly off is not silently corrected: when `tls.enabled` is left unset the tri-state turns TLS on together with `external`, but an explicit `tls.enabled: false` combined with `external: true` is **rejected by a ValidatingAdmissionPolicy** — the same admission mechanism that already enforces tenant hostname/IP isolation — so a tenant cannot stand up a plaintext external endpoint by accident or by overriding the default, and the conflict surfaces as a clear admission error rather than a silent override the operator never sees. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material. +The edge never holds the database's private key — the central strength of passthrough over termination. SNI is sent in cleartext on TLS 1.3 except under ECH, which database clients do not use, so the external hostname is observable on the wire; this is no worse than DNS or SNI exposure for any TLS service, and the payload stays encrypted end-to-end. A missing or mis-emitted SNI is a hard connection failure (a passthrough listener has no certificate to fall back to) — fail-closed, which is the secure default. + +Exposing a database externally with TLS explicitly off is not silently corrected: when `tls.enabled` is left unset the tri-state turns TLS on together with `external`, but an explicit `tls.enabled: false` combined with `external: true` is **rejected at admission**, so a tenant cannot stand up a plaintext external endpoint by accident, and the conflict surfaces as a clear error rather than a silent override. Where that rejection binds deserves precision, because the obvious answer does not work: the database kinds (`apps.cozystack.io/v1alpha1` `Postgres`, `Redis`, …) are served by the **aggregated** `cozystack-api` apiserver, whose writes are proxied past the kube-apiserver admission chain — a `ValidatingAdmissionPolicy` on those typed kinds would never see the request. (The existing tenant-isolation VAPs in `packages/system/cozystack-basics` work because they bind to the *real* CRDs — `Gateway`, `HTTPRoute`, `TLSRoute` — and they enforce hostname isolation, not IP restrictions.) The enforcement point for this rule is therefore `cozystack-api` itself, which owns the typed kinds' write path and their conversion to HelmRelease values — the natural place for typed validation; a CEL policy on the underlying `HelmRelease` objects is the fallback if a policy-shaped guard is preferred. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material. ## Failure and edge cases - A pre-PG17 (client or server) or non-direct-TLS Postgres client sends no SNI → no route → connection reset/timeout (document the symptom). -- The 64-listener budget is exceeded → the listener is rejected; because listeners are one per engine type this is reached only through child-apex fan-out, and the mitigation is to split that subtree onto its own Gateway. -- A passthrough listener shares a hostname with a terminate listener on Cilium <1.20 → routing does not isolate correctly; mitigated by the Cilium 1.20 dependency or the distinct-subdomain fallback, and caught by the Phase 1 gate. -- An explicit `tls.enabled: false` together with `external: true` → rejected at admission by a ValidatingAdmissionPolicy, not silently overridden; an unset `tls` tri-state auto-enables TLS with `external`. +- Any MariaDB/MySQL client dials a passthrough listener → mutual deadlock (client waits for the server greeting, listener waits for a ClientHello) → timeout; prevented by never rendering a MariaDB listener (matrix exclusion). +- The 64-listener budget is exceeded → the listener is rejected; because listeners are one per engine type this is reached only through child-apex fan-out, and the mitigation is to split that subtree onto its own Gateway (until ListenerSet lands with Cilium 1.20). +- A flat `*.` passthrough listener is created on Cilium <1.20 → hostname overlap with the terminate listeners; routing does not isolate correctly. Prevented by the subdomain default; the flat scheme is gated on the platform reaching Cilium 1.20 (Rollout). +- An explicit `tls.enabled: false` together with `external: true` → rejected at admission in `cozystack-api` (see Security), not silently overridden; an unset `tls` tri-state auto-enables TLS with `external`. - An operator expects multi-Gateway IP sharing → each Gateway still gets its own IP (expected under the Cilium constraint). - A database is deleted → its per-release `TLSRoute` is removed; the shared engine-type listener is removed only when its last release is gone, so a single deletion never orphans a listener. ## Testing -- Helm-template assertions that the certificate SAN includes `.` per engine, mirroring the existing TLS test fixtures. -- A controller unit test that a `TLSPassthroughListeners` entry renders a shared listener on the native port with `mode: Passthrough` and `AllowedRoutes` restricted to `TLSRoute`, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. -- An admission test that `tls.enabled: false` with `external: true` is rejected by the ValidatingAdmissionPolicy, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name/port uniqueness, port range, 443 collision) is enforced. +- Helm-template assertions that the certificate SAN includes both `..` and `.` per engine, mirroring the existing TLS test fixtures. +- A controller unit test that a `TLSPassthroughListeners` entry renders a shared listener on the native port with `mode: Passthrough`, the `*..` default hostname, and `AllowedRoutes` restricted to `TLSRoute`, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. +- An admission test that `tls.enabled: false` with `external: true` is rejected in the `cozystack-api` write path, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name/port uniqueness, port range, 443 collision) is enforced. - An end-to-end test per fitting engine: connect from outside the cluster with SNI and `ca.crt`, and assert that the serial of the presented certificate equals the operator-issued internal certificate — proving reuse, not re-issuance. - A negative test: a client without SNI fails closed. ## Rollout -1. API field plus controller listener rendering, no engine wired. **Gate:** on the target Cilium version, stand up a `mode: Passthrough` listener on a native database port (for example 5432) whose hostname `*.` overlaps the Gateway's terminate listeners, and confirm it SNI-routes to the right backend with exactly one Envoy filter chain. The risk is not the non-443 port (supported since Cilium 1.14) but the shared-hostname isolation that lands in Cilium 1.20; this gate must pass on the version actually deployed before any engine is wired, or the distinct-subdomain fallback adopted. -2. Redis (the cleanest fit) behind an opt-in; default-on candidate once the gate and client-emits-SNI behavior are confirmed in practice. +1. API field plus controller listener rendering, no engine wired. **Gate:** on the deployed Cilium (1.19.x), stand up a `mode: Passthrough` listener on a native database port (for example 5432) with a per-engine subdomain hostname (`*.postgres.`) alongside the Gateway's terminate listeners, and confirm it SNI-routes to the right backend with exactly one Envoy filter chain. The gate validates the subdomain scheme on the version actually deployed; no part of Phase 1 depends on an unreleased Cilium. +2. Redis (the cleanest fit) behind an opt-in; default-on candidate once the gate and library-client SNI behavior are confirmed in practice. 3. PostgreSQL (direct-TLS) and sharded MongoDB, opt-in. -4. MariaDB after connector-conformance validation. +4. Flat-hostname refinement, gated on the platform's Cilium carrying the isolation fix (see Open questions): add `*.` passthrough listeners, extend per-release routes with the flat hostname, re-run the Phase 1 gate against the overlap case (a passthrough listener sharing `*.` with the terminate listeners, exactly one filter chain). Certificates already carry both SANs, so this phase changes no client. -Kafka and non-sharded MongoDB are explicitly out of this rollout. Each phase ships documentation, a connection recipe, and an end-to-end gate. +Kafka, MariaDB, and non-sharded MongoDB are explicitly out of this rollout. Each phase ships documentation, a connection recipe, and an end-to-end gate. ## Open questions -- **Cilium shared-hostname isolation on the target version** — the listener model overlaps `*.` across the engine-type passthrough listeners and the HTTPS-terminate listeners, which Cilium isolates correctly only from 1.20. Do we pin Cilium 1.20 as a hard prerequisite for this feature, or ship the distinct-per-engine-subdomain scheme so it runs on 1.19.x? This is the single biggest implementation risk and the Phase 1 gate. -- **MariaDB / MySQL connector conformance** — do the dominant connectors (libmysqlclient, MariaDB Connector/C, JDBC, Go drivers) emit SNI and do TLS-first such that passthrough routes correctly, or does any do a server-greeting-then-STARTTLS dance like pre-direct-TLS Postgres? Default-on versus opt-in for MariaDB hinges on this. +- **Ship now on subdomains, or wait and ship flat-only** — the flat `*.` hostnames need a Cilium release carrying the isolation fix `cilium/cilium#44889`: 1.20 (final proposed for late July 2026), or a 1.19.x patch if the open v1.19 backport (`cilium/cilium#46826`) merges first. That leaves a genuine fork: implement the subdomain scheme now on 1.19.x and adopt flat later as the additive phase, or wait for the fixed Cilium and ship flat-only from day one — no dual scheme, no per-engine subdomain legacy to carry. Waiting may well be free in calendar terms, because engine wiring is gated on `unified-tls-pki` convergence anyway (Scope), which lands on a similar horizon. This document specifies the subdomain path as the no-wait default; the fork is a scheduling decision to settle at the next sync, not a design blocker. - **Direct-TLS client/server floor for Postgres** — what fraction of the tenant base predates `sslnegotiation=direct` on either client (libpq PG17+, driver support) or server (PG17+)? Is a per-release attestation gate enough, given there is no graceful downgrade on a passthrough listener? -- **TLSRoute ownership** — does the controller synthesize the per-release `TLSRoute` alongside the shared listener (single source of truth, but the controller must know the backend Service), or does the database chart render it (the chart owns the backend selector, but must learn a gateway name/namespace it does not have today)? This decides whether database charts gain gateway-discovery logic. -- **Who writes the listener/route set** — the orchestration layer auto-creating the engine-type listener on first exposure and a per-release route on each `external: true` + passthrough, an operator-managed explicit list, or both; and which reconciliation owns route add/remove plus reference-counting the shared listener down to zero on the last database deletion. -- **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, and default passthrough services now joined by one listener per database engine type (not per instance), what is the realistic per-tenant ceiling, and should the controller surface a status condition as the budget nears 64? -- **ListenerSet timeline** — consolidating onto one shared Gateway/IP is blocked on Cilium implementing ListenerSet (CFP `cilium#42756`, PR `cilium#46303`). Do we design the field and status to be ListenerSet-ready now, or revisit when Cilium ships it? This determines whether one-IP-per-tenant is a permanent or temporary ceiling. +- **TLSRoute ownership and the `expose` model** — the natural owner of per-release route rendering is the orchestration layer that reconciles `expose` entries (community pull request #29), which also answers who reference-counts the shared listener down to zero. If this proposal is implemented before `expose` lands, does the interim Tenant / HelmRelease orchestration carry that logic temporarily, or do we sequence this proposal's engine wiring after `expose`? +- **64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, and default passthrough services now joined by one listener per database engine type (not per instance), what is the realistic per-tenant ceiling on 1.19.x, and should the controller surface a status condition as the budget nears 64? (ListenerSet dissolves the cap once the platform is on Cilium 1.20.) +- **ListenerSet adoption** — Cilium's implementation merged upstream (2026-06-26) and ships in 1.20; Gateway API v1.5.0 carries it in Standard. Do we design this field and its status to be ListenerSet-ready now (listeners as per-namespace objects merged onto a shared Gateway, lifting both the 64 cap and the one-IP-per-tenant ceiling), or revisit after the platform's Cilium bump? This determines whether one-IP-per-tenant is a permanent or temporary ceiling. ## Alternatives considered -- **Edge TLS termination plus re-encryption (`BackendTLSPolicy`).** Rejected: the edge holds keys, there are two certificates, and it defeats reuse. This is the recorded decision WS5 asks for — passthrough chooses application-level, CNI-agnostic TLS over edge termination. -- **Distinct per-engine subdomains for the passthrough listeners** (`*.pg.`, `*.redis.`, …). Sidesteps the Cilium shared-hostname isolation defect on 1.19.x without waiting for 1.20, but at the cost of longer connection hostnames and per-engine SAN/recipe changes. Retained as the documented fallback; the flat `*.` scheme is preferred for clean hostnames where Cilium 1.20 is available. +- **`lbipam.cilium.io/sharing-key` on the existing per-engine LoadBalancer Services.** Genuinely delivers part of the goal today: one IP across Services whose ports do not conflict, with end-to-end TLS for free (a plain LB to the pod is already non-terminating), no new API, and no controller change. Rejected as the *complete* answer, not as a mechanism: it is structurally blind to the same-engine case (two Postgres instances collide on 5432 and are allocated separate IPs into the sharing-key set), it routes by port on a bare IP so there is no per-database hostname or stable connection identity, and cross-namespace sharing needs mutual annotations. It is complementary — an operator can use sharing-key for cross-engine consolidation today, and this design for same-engine fan-out and hostname-routed access; the two do not conflict. +- **Edge TLS termination plus re-encryption (`BackendTLSPolicy`).** Weighed, not dismissed. What termination with a publicly-trusted edge certificate genuinely buys: zero client-side trust bootstrap (no `ca.crt` distribution, no `sslrootcert=`), CA-rotation blast radius contained in-cluster (an interior CA rotation never breaks an external client), and familiarity. What it costs: the edge holds a private key for a database it does not own, a second certificate whose lifecycle diverges from the operator-issued one, a re-encryption hop, and — decisively — it does **not** dissolve the hard client constraints, because a raw terminate listener still expects a TLS-first client: a legacy Postgres client still opens with a plaintext `SSLRequest` and a MariaDB client still waits for a server greeting, so the engines that are hard for passthrough are equally hard for generic edge termination (only a protocol-aware proxy changes that, which is out of scope either way). Passthrough's real cost is booked openly: every external client must obtain and pin the interior `ca.crt`, and an interior CA rotation requires external clients to re-fetch it — that is the trade `cozystack/cozystack#2816` frames ("end-to-end without terminating anywhere in between"), and this proposal records the decision in passthrough's favor with that cost stated rather than hidden. +- **Flat `*.` hostnames as the v1 default.** Shorter connection hostnames (`db1.`), but the listener hostname then overlaps the HTTPS-terminate listeners, which Cilium isolates correctly only from 1.20 (unreleased at the time of writing) — making the feature's default depend on an unreleased CNI. Adopted instead as the Phase 4 refinement once the platform is on 1.20; the dual-SAN plan (§6) makes that adoption additive. The per-engine subdomain scheme is the v1 default because it works on the Cilium actually deployed. - **All-on-443 SNI.** Mirrors the existing three services exactly with no new port plumbing, retained as a documented opt-in, but rejected as the default for the client-ergonomics and direct-TLS-everywhere tax. - **Route-driven implicit listeners** — the controller watches `TLSRoute` objects (it already does) and auto-synthesizes a passthrough listener for any route targeting a non-443 port. Less API surface, but it inverts the explicit-intent model and makes the listener set implicit and hard to audit. Offered as a future ergonomic; the explicit field is recommended for v1. -- **Per-broker SNI for Kafka.** Rejected for now (see the matrix); revisit if the listener-budget cost is justified. +- **Per-broker SNI for Kafka.** Deferred, not rejected on cost grounds: routes are uncapped on the shared listener, and Strimzi natively supports the advertised-address overrides it would need — but it is real integration work against an operator-owned moving part, with no payoff the simpler engines do not already prove (see the matrix). Revisit after Redis/Postgres/mongos validate the model. - **Status quo: a LoadBalancer per database.** The IPv4-scarcity problem this proposal exists to solve. --- From a143c055b03dca8e7d20ea941a491c9d377c0304 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 2 Jul 2026 14:12:37 +0300 Subject: [PATCH 5/5] docs(external-database-exposure): correct the admission enforcement point MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A ValidatingAdmissionPolicy bound to the typed apps.cozystack.io kinds does fire, contrary to the previous revision's claim. The kube-apiserver proxies aggregated writes before its own admission chain, but the aggregated cozystack-api is built on k8s.io/apiserver RecommendedOptions, whose delegated admission chain includes the ValidatingAdmissionPolicy plugin — the extension apiserver evaluates cluster VAPs itself. Verified at runtime with a scoped deny-all canary policy matching apps.cozystack.io tenants: it denies the typed-kind write exactly as it denies a core-resource control, and the existing cozystack-tenant-host-policy already relies on this path in production. The plaintext-external rejection therefore ships as a VAP on the typed kinds, consistent with the existing tenant-isolation policies, with registry-level validation in cozystack-api as a complement rather than the only option. Assisted-By: Claude Signed-off-by: Aleksei Sviridkin --- design-proposals/external-database-exposure/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index d87a071..f8bb52d 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -224,7 +224,7 @@ The default stays today's per-database LoadBalancer, so no existing external dat The edge never holds the database's private key — the central strength of passthrough over termination. SNI is sent in cleartext on TLS 1.3 except under ECH, which database clients do not use, so the external hostname is observable on the wire; this is no worse than DNS or SNI exposure for any TLS service, and the payload stays encrypted end-to-end. A missing or mis-emitted SNI is a hard connection failure (a passthrough listener has no certificate to fall back to) — fail-closed, which is the secure default. -Exposing a database externally with TLS explicitly off is not silently corrected: when `tls.enabled` is left unset the tri-state turns TLS on together with `external`, but an explicit `tls.enabled: false` combined with `external: true` is **rejected at admission**, so a tenant cannot stand up a plaintext external endpoint by accident, and the conflict surfaces as a clear error rather than a silent override. Where that rejection binds deserves precision, because the obvious answer does not work: the database kinds (`apps.cozystack.io/v1alpha1` `Postgres`, `Redis`, …) are served by the **aggregated** `cozystack-api` apiserver, whose writes are proxied past the kube-apiserver admission chain — a `ValidatingAdmissionPolicy` on those typed kinds would never see the request. (The existing tenant-isolation VAPs in `packages/system/cozystack-basics` work because they bind to the *real* CRDs — `Gateway`, `HTTPRoute`, `TLSRoute` — and they enforce hostname isolation, not IP restrictions.) The enforcement point for this rule is therefore `cozystack-api` itself, which owns the typed kinds' write path and their conversion to HelmRelease values — the natural place for typed validation; a CEL policy on the underlying `HelmRelease` objects is the fallback if a policy-shaped guard is preferred. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material. +Exposing a database externally with TLS explicitly off is not silently corrected: when `tls.enabled` is left unset the tri-state turns TLS on together with `external`, but an explicit `tls.enabled: false` combined with `external: true` is **rejected at admission**, so a tenant cannot stand up a plaintext external endpoint by accident, and the conflict surfaces as a clear error rather than a silent override. Where that rejection binds deserves precision: the database kinds (`apps.cozystack.io/v1alpha1` `Postgres`, `Redis`, …) are served by the **aggregated** `cozystack-api` apiserver, and the kube-apiserver proxies those writes *before* its own admission chain runs — so it is not kube-apiserver that enforces a policy there. A `ValidatingAdmissionPolicy` bound to the typed kinds still works, because `cozystack-api` is built on the generic apiserver library (`k8s.io/apiserver` `RecommendedOptions`), whose delegated admission chain includes the ValidatingAdmissionPolicy plugin — the extension apiserver evaluates cluster VAPs itself. Verified at runtime, not assumed: a scoped deny-all canary VAP matching `apps.cozystack.io` `tenants` fires on a live cluster exactly as it does on a core resource, and the existing `cozystack-tenant-host-policy` (`packages/system/cozystack-basics`) already relies on this path in production. So the rule ships as a VAP on the typed kinds, consistent with the existing tenant-isolation policies (which enforce hostname isolation — not IP restrictions — via VAPs on the Gateway API kinds and on `tenants`); registry-level validation in `cozystack-api`'s write path remains available as a belt-and-suspenders complement. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material. ## Failure and edge cases @@ -232,7 +232,7 @@ Exposing a database externally with TLS explicitly off is not silently corrected - Any MariaDB/MySQL client dials a passthrough listener → mutual deadlock (client waits for the server greeting, listener waits for a ClientHello) → timeout; prevented by never rendering a MariaDB listener (matrix exclusion). - The 64-listener budget is exceeded → the listener is rejected; because listeners are one per engine type this is reached only through child-apex fan-out, and the mitigation is to split that subtree onto its own Gateway (until ListenerSet lands with Cilium 1.20). - A flat `*.` passthrough listener is created on Cilium <1.20 → hostname overlap with the terminate listeners; routing does not isolate correctly. Prevented by the subdomain default; the flat scheme is gated on the platform reaching Cilium 1.20 (Rollout). -- An explicit `tls.enabled: false` together with `external: true` → rejected at admission in `cozystack-api` (see Security), not silently overridden; an unset `tls` tri-state auto-enables TLS with `external`. +- An explicit `tls.enabled: false` together with `external: true` → rejected at admission by a ValidatingAdmissionPolicy on the typed kind, evaluated in `cozystack-api`'s admission chain (see Security), not silently overridden; an unset `tls` tri-state auto-enables TLS with `external`. - An operator expects multi-Gateway IP sharing → each Gateway still gets its own IP (expected under the Cilium constraint). - A database is deleted → its per-release `TLSRoute` is removed; the shared engine-type listener is removed only when its last release is gone, so a single deletion never orphans a listener. @@ -240,7 +240,7 @@ Exposing a database externally with TLS explicitly off is not silently corrected - Helm-template assertions that the certificate SAN includes both `..` and `.` per engine, mirroring the existing TLS test fixtures. - A controller unit test that a `TLSPassthroughListeners` entry renders a shared listener on the native port with `mode: Passthrough`, the `*..` default hostname, and `AllowedRoutes` restricted to `TLSRoute`, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends. -- An admission test that `tls.enabled: false` with `external: true` is rejected in the `cozystack-api` write path, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name/port uniqueness, port range, 443 collision) is enforced. +- An admission test that `tls.enabled: false` with `external: true` is rejected by the typed-kind ValidatingAdmissionPolicy on the aggregated path, while an unset `tls` is admitted and auto-enables; and that the listener validation contract (name/port uniqueness, port range, 443 collision) is enforced. - An end-to-end test per fitting engine: connect from outside the cluster with SNI and `ca.crt`, and assert that the serial of the presented certificate equals the operator-issued internal certificate — proving reuse, not re-issuance. - A negative test: a client without SNI fails closed.