feat(kubernetes): create SandboxClaims for matching warm pools

architecture/compute-runtimes.md

@@ -118,7 +118,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
 |---|---|---|---|
 | Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
 | Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
-| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
+| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, GPU resources, and optionally Agent Sandbox v1beta1 warm-pool claims. |
 | VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
 | Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
 
@@ -203,6 +203,77 @@ identity is absent, malformed, unknown, ambiguous, or resolves to UID/GID 0.
 The supervisor itself remains root so it can establish isolation before
 starting unprivileged children.
 
+Kubernetes supervisors authenticate to the gateway in two stages. They first
+call `RegisterSupervisorPod` with the projected ServiceAccount token; the
+gateway validates the pod-bound token and live Agent Sandbox owner state, then
+activates already-bound cold pods by streaming back a gateway-minted sandbox
+JWT. The supervisor installs that JWT in memory and starts the normal
+`ConnectSupervisor` session as the activated sandbox. The gateway does not dial
+pod IPs or require an inbound activation port; activation is supervisor
+initiated over the existing outbound gRPC connection.
+
+When compatible OpenShell-enabled Agent Sandbox warm pools exist, the
+Kubernetes driver can create a `SandboxClaim` instead of a direct `Sandbox`.
+The driver watches warm pools and templates into a local cache so create
+requests do not list or fetch extension objects on the hot path. This remains
+behind the compute-driver boundary: other drivers and the driver-agnostic
+gateway lifecycle continue to operate on OpenShell sandbox identity and status.
+Warm-pool activation uses the same trust boundary as direct Kubernetes
+activation. The direct path validates `pod token -> live pod -> owning Sandbox
+CR -> Sandbox sandbox-id metadata`; the warm path validates `pod token -> live
+pod -> owning Sandbox CR -> associated SandboxClaim -> SandboxClaim sandbox-id
+metadata`. In both paths, sandbox-id metadata is not cryptographic proof by
+itself. It is trusted because the gateway and Agent Sandbox controllers own the
+relevant Kubernetes objects, and RBAC must prevent sandbox workloads and
+untrusted users from creating or mutating trusted `Sandbox`, `SandboxClaim`, pod
+metadata, or sandbox service-account identity in the gateway-managed namespace.
+Under that model, warm pooling does not require a separate anti-spoofing token
+or gateway-side claim mapping beyond the same live Kubernetes ownership and
+metadata consistency checks used by the direct path. If an operator grants
+untrusted principals write access to those objects, both warm and direct
+activation require a broader common proof model.
+Claim activation is level-triggered rather than dependent on a single watch
+event. The controller periodically relists OpenShell-managed claims and retries
+failed watch streams, because a claim can become visible before its selected
+`Sandbox`, pod, or outbound supervisor registration. A newly authenticated
+warm-pod registration clears completed-claim hints and triggers an immediate
+relist, so both a restarted process with the same pod UID and a replacement pod
+with a new UID can reactivate against an existing claim. It reconciles different
+claims concurrently, deduplicates work by claim UID, and caps concurrency so
+one late registration or slow Kubernetes lookup cannot block activation for
+the namespace or create unbounded work.
+The gateway retains activated pod-UID tombstones for one hour to reject
+duplicate registration and activation races, then prunes them opportunistically
+on registry access so long-running gateways do not accumulate one entry per
+historical pod. A new authenticated registration supersedes its pod-UID
+tombstone and receives a new local session ID. Activation and failure delivery
+are conditional on that session ID, preventing work started for an old process
+from consuming or terminating its replacement stream.
+Warm claim creation is idempotent by the deterministic claim name. After a
+create timeout, transport failure, server error, or conflict, the driver reads
+that name back and retries the same claim create when it is not yet visible. It
+never switches to direct Sandbox creation after an ambiguous claim write. An
+existing claim is accepted only when its OpenShell identity, workspace,
+allocation marker, and warm-pool reference match the request. If Kubernetes
+still cannot determine whether the claim exists, the driver returns
+`Unavailable` and the gateway retains the durable `Provisioning` record. This
+prevents another create from assigning a new sandbox ID to the same name while
+the original Kubernetes write may still be live; the normal driver watcher can
+subsequently reconcile an accepted claim.
+The Kubernetes driver can optionally run a separate ConfigMap profile
+reconciler that turns admin-authored TOML profiles into generated
+`SandboxTemplate` and `SandboxWarmPool` resources. That reconciler is
+independent of the create-path matcher: admins can use it, Helm, kubectl, or
+another controller to create warm pools, and the matcher only considers the
+resulting enabled warm-pool resources and their fingerprints. Sandbox creation
+and profile reconciliation share one workspace-to-namespace mapping boundary;
+the current shared-namespace implementation maps every workspace to the
+configured Kubernetes namespace.
+The gateway starts the profile reconciler only on the replica holding the
+shared reconciler lease and stops it before returning that replica to standby.
+The read-side warm-pool cache still runs on every replica so local create
+requests do not depend on the lease holder.
+
 Kubernetes can run the supervisor in the default combined topology or in a
 sidecar topology. Combined mode keeps network and process supervision in the
 agent container. Sidecar mode runs network enforcement, the proxy, and gateway

architecture/gateway.md

@@ -155,24 +155,41 @@ token is reported as connected but unauthenticated.
 Sandbox supervisor RPCs authenticate with explicit sandbox credentials; mTLS
 does not grant sandbox identity. Kubernetes deployments use the
 gateway-minted JWT bootstrap path: the supervisor starts with a projected
-ServiceAccount token, exchanges it for a gateway-minted sandbox JWT, and uses
-that JWT on subsequent gateway RPCs.
+ServiceAccount token, registers the pod with the gateway, receives a
+gateway-minted sandbox JWT after activation, and uses that JWT on subsequent
+gateway RPCs.
 User-facing mutations are authorized by role policy when OIDC or edge identity
 is enabled.
 
 Sandbox secrets are gateway-signed JWTs bound to a single sandbox ID. Docker,
 Podman, and VM drivers deliver the initial token through supervisor-only
 runtime material; Kubernetes supervisors exchange a projected ServiceAccount
-token through `IssueSandboxToken`. The gateway validates that projected token
-with Kubernetes `TokenReview`, requires the configured sandbox service account,
-checks the returned pod binding against the live pod UID, and verifies the pod's
-controlling `Sandbox` ownerReference against the live Sandbox CR UID and
-sandbox-id label before minting the gateway JWT. The bootstrap path accepts
-both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
-controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
-deployments. Supervisors renew gateway JWTs in memory before expiry only while
-the sandbox record still exists. Older tokens are not server-revoked; shared
-deployments bound replay exposure with short `gateway_jwt.ttl_secs` lifetimes.
+token through `RegisterSupervisorPod`. The gateway validates that projected
+token with Kubernetes `TokenReview`, requires the configured sandbox service
+account, checks the returned pod binding against the live pod UID, and verifies
+the pod's controlling `Sandbox` ownerReference against the live Sandbox CR UID
+and sandbox-id label before sending an activation message with the gateway JWT.
+For warm-pooled Kubernetes sandboxes, the same check is re-anchored through the
+claim that adopted the warm pod: the live pod must still be controlled by a
+live `Sandbox`, that `Sandbox` must be associated with the live
+`SandboxClaim`, and the sandbox-id metadata on that claim must identify the
+same OpenShell sandbox record before activation can complete. This is not a
+separate warm-only spoofing defense; it is the warm-path equivalent of the
+direct path's owner-reference and sandbox-id consistency check. Both paths
+depend on the same Kubernetes RBAC boundary: sandbox workloads and untrusted
+users must not be able to create or mutate trusted Agent Sandbox objects, pod
+metadata, or the configured sandbox ServiceAccount in the gateway-managed
+namespace. If that boundary is weakened, a fix must cover both direct
+`Sandbox` and warm `SandboxClaim` activation rather than adding a warm-only
+proof.
+The bootstrap path accepts both `agents.x-k8s.io/v1beta1` ownerReferences from
+newer Agent Sandbox controllers and `agents.x-k8s.io/v1alpha1` ownerReferences
+from existing deployments. `IssueSandboxToken` remains as a compatibility shim
+for older supervisor images and mints through the same already-bound pod
+activation logic. Supervisors renew gateway JWTs in memory before expiry only
+while the sandbox record still exists. Older tokens are not server-revoked;
+shared deployments bound replay exposure with short `gateway_jwt.ttl_secs`
+lifetimes.
 The config default is
 `gateway_jwt.ttl_secs = 0` for local single-player Docker, Podman, and VM
 gateways; those tokens carry `exp = 0` and do not expire. Kubernetes and other

crates/openshell-cli/tests/ensure_providers_integration.rs

@@ -564,6 +564,17 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
+    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::PodActivationMessage, Status>,
+    >;
+
+    async fn register_supervisor_pod(
+        &self,
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        Err(Status::unimplemented("not implemented in test"))
+    }
+
     async fn refresh_sandbox_token(
         &self,
         _request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,

crates/openshell-cli/tests/mtls_integration.rs

@@ -449,6 +449,17 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
+    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::PodActivationMessage, Status>,
+    >;
+
+    async fn register_supervisor_pod(
+        &self,
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        Err(Status::unimplemented("not implemented in test"))
+    }
+
     async fn refresh_sandbox_token(
         &self,
         _request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,

crates/openshell-cli/tests/provider_commands_integration.rs

@@ -967,6 +967,17 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
+    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::PodActivationMessage, Status>,
+    >;
+
+    async fn register_supervisor_pod(
+        &self,
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        Err(Status::unimplemented("not implemented in test"))
+    }
+
     async fn refresh_sandbox_token(
         &self,
         _request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,

crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs

@@ -637,6 +637,17 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
+    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::PodActivationMessage, Status>,
+    >;
+
+    async fn register_supervisor_pod(
+        &self,
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        Err(Status::unimplemented("not implemented in test"))
+    }
+
     async fn refresh_sandbox_token(
         &self,
         _request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,

crates/openshell-cli/tests/sandbox_name_fallback_integration.rs

@@ -537,6 +537,17 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
+    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::PodActivationMessage, Status>,
+    >;
+
+    async fn register_supervisor_pod(
+        &self,
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        Err(Status::unimplemented("not implemented in test"))
+    }
+
     async fn refresh_sandbox_token(
         &self,
         _request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,

crates/openshell-core/src/error.rs

@@ -119,6 +119,10 @@ pub enum ComputeDriverError {
     /// A precondition for the operation was not met.
     #[error("{0}")]
     Precondition(String),
+    /// The backend may have accepted the operation, but its result could not be
+    /// determined. Callers must preserve durable intent for reconciliation.
+    #[error("{0}")]
+    Unavailable(String),
     /// Generic error message.
     #[error("{0}")]
     Message(String),
@@ -130,6 +134,7 @@ impl From<ComputeDriverError> for tonic::Status {
             ComputeDriverError::AlreadyExists => Self::already_exists("sandbox already exists"),
             ComputeDriverError::InvalidArgument(m) => Self::invalid_argument(m),
             ComputeDriverError::Precondition(m) => Self::failed_precondition(m),
+            ComputeDriverError::Unavailable(m) => Self::unavailable(m),
             ComputeDriverError::Message(m) => Self::internal(m),
         }
     }