feat: support in-place supervisor upgrade via re-exec

[mrunalp]

Problem Statement

Upgrading the openshell-sandbox supervisor today requires destroying and recreating the sandbox, which kills the agent workload running inside it. We want the supervisor to be able to replace its own binary in place — bring in a new executable, execve() it, and continue supervising the same child processes — analogous to systemctl daemon-reexec. Because execve destroys the entire address space, this requires serializing supervisor state so the incoming binary can resume rather than bootstrap.

This spike assesses feasibility, maps the exact state that must cross the exec boundary, and identifies the design decisions that need human input before implementation.

Technical Context

openshell-sandbox runs inside the workload container in one of three topologies: combined (--mode=network,process, usually PID 1, root with SYS_ADMIN/NET_ADMIN/SYS_PTRACE/SYSLOG), K8s sidecar (two supervisors — non-root --mode=network plus --mode=process — coupled by a one-shot Unix control socket), and --mode=network-init (one-shot, exits immediately).

The structural fact that dominates this design: privileged bootstrap is strictly ordered before a one-way seccomp door. Everything requiring mount/bpf/pivot_root/kexec_* happens during startup, and then apply_supervisor_startup_hardening() (crates/openshell-supervisor-process/src/run.rs:111) installs a filter that is inherited across both fork and execve. A re-exec'd supervisor therefore comes up already hardened and cannot redo those bootstrap steps.

Feasibility verdict: yes, with a hard boundary. execve preserves what the kernel owns and destroys what userspace holds. That split determines the entire design.

Affected Components

Component	Key Files	Role
Supervisor entry	crates/openshell-sandbox/src/main.rs	argv/env parsing, tracing setup, tokio runtime, copy-self, debug-rpc
Orchestrator	crates/openshell-sandbox/src/lib.rs	policy load, netns, provider creds, aggregators, poll loop, sidecar control, exit code
Process leaf	crates/openshell-supervisor-process/src/{run,process,ssh,supervisor_session,managed_children}.rs	hardening prelude, entrypoint spawn/wait, SSH daemon, gateway relay, reaper
Network leaf	crates/openshell-supervisor-network/src/{run,proxy,identity,opa,inference_routes}.rs	ephemeral CA, CONNECT proxy, TOFU cache, OPA generations
Namespace/kernel state	crates/openshell-supervisor-process/src/netns/mod.rs, sandbox/linux/{seccomp,landlock,mod}.rs	netns + veth + nftables, seccomp prelude, child Landlock
Sidecar coupling	crates/openshell-sandbox/src/sidecar_control.rs	one-shot authenticated channel between the two supervisors
Gateway session	crates/openshell-server/src/supervisor_session.rs	session registry, supersede, pending-relay replay
Binary delivery	crates/openshell-driver-{kubernetes,docker,podman,vm}	how the supervisor binary reaches the container

Technical Investigation

Architecture Overview

Control flow: main() (main.rs:457) → tracing setup → multi-thread tokio runtime (main.rs:513) → run_sandbox() (lib.rs:89) → run_networking() (supervisor-network/src/run.rs:76) and run_process() (supervisor-process/src/run.rs:53). run_sandbox blocks on the entrypoint child's exit and returns its code, which becomes the container exit code (main.rs:643).

What survives execve for free: PID identity — so the entrypoint and SSH-session processes remain direct children (single fork+exec, confirmed at process.rs:730 and ssh.rs:870); netns membership; cgroup; nftables rules; mount-namespace membership; no_new_privs; and any fd with FD_CLOEXEC cleared.

Also load-bearing: execve runs no Rust destructors, so kill_on_drop(true) (process.rs:603), ProxyHandle::drop (proxy.rs:402), and NetworkNamespace::drop (netns/mod.rs:359) never fire. That is exactly what we want here — and also why a failed resume leaks the netns.

What dies: all tokio state and all TLS/SSH crypto state. Established SSH sessions (russh state is userspace) and in-flight proxy connections (CONNECT tunnels, TLS-terminated L7 relays) cannot be handed to a new process image under any design short of moving them into separate processes.

Capability caveat. execve recomputes the permitted capability set. For a euid-0 process the kernel's legacy-root rule restores it, so the combined topology is fine. But the K8s network sidecar runs as a non-root uid with SYS_PTRACE + DAC_READ_SEARCH (driver-kubernetes/src/driver.rs:1743-1766), and there are no ambient capabilities configured anywhere in this repo. On re-exec those capabilities would be lost, and they are needed at runtime by procfs::read_process_binary (supervisor-network/src/procfs.rs:93-160), which fails hard when the exe symlink is unreadable — the sidecar would silently lose binary-aware policy enforcement and start failing closed. This claim is kernel- and runtime-config dependent and should be verified empirically before designing around it.

Code References

Location	Description
crates/openshell-sandbox/src/main.rs:457	main(); PID 1; std::process::exit(exit_code) at :643
crates/openshell-sandbox/src/main.rs:25	COPY_SELF_SUBCOMMAND — existing pre-clap dispatch; natural home for resume/verify-state
crates/openshell-sandbox/src/main.rs:240	copy_self() — writes a 0755 copy of current_exe() to an arbitrary path
crates/openshell-sandbox/src/main.rs:498-508	RollingFileAppender + non_blocking writer thread; buffered lines lost on execve (guards at :546 never drop)
crates/openshell-sandbox/src/main.rs:513	multi-thread tokio runtime — execve from a worker thread destroys all other threads
crates/openshell-sandbox/src/lib.rs:89	run_sandbox() — the sequence a resume path must partition into SKIP/REDO/ADOPT
crates/openshell-sandbox/src/lib.rs:180	load_policy() — gateway fetch, 5-attempt retry (:1790), fatal on exhaustion (:1906)
crates/openshell-sandbox/src/lib.rs:207-280	provider credential fetch; degrades gracefully — would silently strip live creds on resume (:244-262)
crates/openshell-sandbox/src/lib.rs:309-314	create_netns_for_proxy — must be SKIPPED on resume
crates/openshell-sandbox/src/lib.rs:713-731	process supervisor exits fatally when the sidecar control channel closes
crates/openshell-sandbox/src/lib.rs:741-753	network sidecar exits 1 when the control task ends, forcing a K8s restart
crates/openshell-sandbox/src/lib.rs:2011-2043	"no in-memory last-known-good generation during startup, so both configured modes necessarily fail closed"
crates/openshell-supervisor-process/src/run.rs:111	apply_supervisor_startup_hardening() — the one-way door
crates/openshell-supervisor-process/src/run.rs:146-204	zombie reaper: waitid(WNOWAIT) + managed_children::is_managed
crates/openshell-supervisor-process/src/run.rs:226-297	SSH spawn + 10 s readiness gate; startup fails if the socket cannot bind
crates/openshell-supervisor-process/src/run.rs:363-405	wait_for_process_exit_or_shutdown → handle.wait() on a tokio::process::Child that cannot be reconstructed post-execve
crates/openshell-supervisor-process/src/process.rs:600-603	Stdio::inherit() ×3 + kill_on_drop(true) — entrypoint output does not flow through the supervisor
crates/openshell-supervisor-process/src/process.rs:687-727	child pre_exec: setpgid, setns, privilege drop, Landlock restrict_self + child seccomp
crates/openshell-supervisor-process/src/process.rs:288,316,331-341,410	SUPERVISOR_IDENTITY_MOUNT_NS; prepare_..._from_env() has zero callers repo-wide (see Risks)
crates/openshell-supervisor-process/src/sandbox/linux/seccomp.rs:95-121	prelude blocklist: mount, fsopen, fsconfig, fsmount, fspick, move_mount, open_tree, pivot_root, umount2, bpf, perf_event_open, userfaultfd, init_module, finit_module, delete_module, kexec_load, kexec_file_load
crates/openshell-supervisor-process/src/sandbox/linux/seccomp.rs:538-552	test asserting setns, clone, unshare, ptrace remain available; execve, memfd_create, pidfd_open are also unlisted → available
crates/openshell-supervisor-process/src/sandbox/linux/mod.rs:25-58	Landlock applied only inside child pre_exec — the supervisor itself is never Landlock-restricted
crates/openshell-supervisor-process/src/ssh.rs:51	PrivateKey::random(&mut rng, Algorithm::Ed25519) — fresh host key every start, never persisted
crates/openshell-supervisor-process/src/ssh.rs:1399-1411	test proving an abstract socket cannot be rebound while the old fd is open
crates/openshell-supervisor-process/src/managed_children.rs:16	LazyLock<Mutex<HashSet<i32>>> — process-global, must be reconstructed
crates/openshell-supervisor-process/src/supervisor_session.rs:302-336	reconnect loop, 1 s → 30 s exponential backoff
crates/openshell-supervisor-process/src/supervisor_session.rs:351-399	full re-registration: new channel, new instance_id, SupervisorHello, SessionAccepted
crates/openshell-supervisor-process/src/supervisor_session.rs:699-708	SSH relay peer-PID check — survives re-exec because the PID is retained
crates/openshell-server/src/supervisor_session.rs:705-717,735-740	gateway supersedes the old session; only not-yet-accepted relays are replayed
crates/openshell-supervisor-process/src/netns/mod.rs:63,87	NetworkNamespace::create() runs ip netns add, which bind-mounts → requires mount(2)
crates/openshell-supervisor-process/src/netns/mod.rs:156-167	open("/var/run/netns/<name>", O_RDONLY) — the ns is nameable and re-openable by name
crates/openshell-supervisor-process/src/netns/mod.rs:330-356	bind_tcp_in_netns — dedicated thread + setns; still works post-prelude
crates/openshell-supervisor-network/src/run.rs:207-272	SandboxCa::generate() + write_ca_files — a new ephemeral CA on every start
crates/openshell-supervisor-network/src/run.rs:203	BinaryIdentityCache::new() — TOFU map, in-memory
crates/openshell-supervisor-network/src/proxy.rs:218	TcpListener::bind(http_addr) on the netns host-side veth IP
crates/openshell-supervisor-network/src/opa.rs:125,240-254	generation: Arc<AtomicU64> + per-tunnel PolicyGenerationGuard
crates/openshell-supervisor-network/src/inference_routes.rs:33,341	routes refresh every 5 s — no need to serialize
crates/openshell-sandbox/src/sidecar_control.rs:361-388	one-shot: after the first accept() the listener is dropped and the socket path unlinked
crates/openshell-driver-kubernetes/src/driver.rs:1451-1465	supervisor emptyDir mounted readOnly: true in the agent container
crates/openshell-driver-kubernetes/src/driver.rs:1743-1767	network sidecar: non-root, drop: ALL, add: [SYS_PTRACE, DAC_READ_SEARCH]
crates/openshell-driver-kubernetes/src/driver.rs:2538-2554	combined supervisor: runAsUser: 0, add: [SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYSLOG]
crates/openshell-driver-docker/src/lib.rs:2043 / crates/openshell-driver-podman/src/container.rs:960	supervisor binary mounted read-only (:ro,z / rw: false)

Current Behavior

1. main() dispatches copy-self / debug-rpc pre-clap, parses Args, handles --mode=network-init, builds the rolling appender and tokio runtime, calls run_sandbox, then std::process::exit.
2. run_sandbox(): set_ctx OCSF context → (process-only sidecar) connect_process_client, fatal on failure → load_policy (5 retries then fatal), build OpaEngine, install middleware registries → resolve_process_identity → provider env fetch → create_netns_for_proxy (ip netns add, veth, routes, nft bypass rules via nsenter) → run_networking (symlink task, BinaryIdentityCache, SandboxCa::generate(), ProxyHandle::start_with_bind_addr binding 10.200.0.1:3128) → (network sidecar) sidecar_control::spawn_server → aggregators + policy poll loop → optional GCE metadata listener → run_process.
3. run_process(): passwd/group updates, identity validation, prepare_filesystem_with_identity, install_initial_agent_skill, apply_supervisor_startup_hardening(), bypass monitor, PID-limit check, zombie reaper, SSH server + readiness gate, supervisor_session::spawn, ProcessHandle::spawn, wait_for_process_exit_or_shutdown.
4. Exit: entrypoint status → drop(networking) → main exits. NetworkNamespace::drop tears down the netns.

What Would Need to Change

Resume disposition matrix — the core deliverable:

Step	Disposition	Why
tracing/appender, tokio runtime	REDO	New address space; accept loss of buffered log lines
OCSF ctx	REDO from serialized fields	Trivially reconstructible
load_policy gateway fetch	SKIP → ADOPT	Re-fetch is fatal-on-unreachable and resets has_last_valid_policy, which can silently swap a running policy for restrictive_default_policy()
Middleware registry connect	REDO, degraded-tolerant	Already has a NeedsReconciliation path
Provider credentials	ADOPT	Re-fetch failure silently empties the credential map
create_netns_for_proxy	SKIP; ADOPT by name	ip netns add needs mount(2), blocked by the inherited prelude
nft bypass rules	SKIP	Rules live in the kernel netns
SandboxCa::generate()	SKIP; ADOPT	Regenerating breaks the running workload's trust store — see Risks
ProxyHandle listener	ADOPT fd or REBIND	Rebinding loses queued backlog and in-flight tunnels
Sidecar control channel	ADOPT fd (mandatory)	One-shot listener is unlinked; reconnect is impossible
Aggregators / channels	REDO	Bounded loss, ≤ one flush interval
GCE metadata listener	ADOPT fd or REBIND	bind_tcp_in_netns still works post-prelude
passwd/group/filesystem prep	SKIP (idempotent if redone)
apply_supervisor_startup_hardening	SKIP	Already inherited; re-applying stacks a duplicate filter
Bypass monitor	REDO	Loses /dev/kmsg position; may double- or under-report
Zombie reaper	REDO + initial full sweep	Children that exited during the exec window are already zombies
managed_children	ADOPT	Otherwise the reaper steals the entrypoint's exit status
SSH server	CONDITIONAL	Rebinding works (old fd is gone), but the host key changes and existing sessions are dead
supervisor_session	REDO	Already reconnects and re-registers; gateway supersedes and replays pending relays
ProcessHandle::spawn	SKIP; ADOPT PID	Must pidfd_open/waitpid the existing child
entrypoint wait	REWRITE	tokio::process::Child is unreconstructible

Per-component work:

• openshell-sandbox/main.rs: a pre-clap resume --state-fd <n> and verify-state subcommand alongside copy-self. A state fd is preferable to a state path — no writable file, no TOCTOU, no path the workload can point at.
• openshell-sandbox/lib.rs: run_sandbox is a ~700-line straight-line function with interleaved conditionals. It must be split into bootstrap() and resume() sharing a runtime-assembly step, or every SKIP/REDO decision becomes an if resuming branch inside an already-dense function. This refactor is the single largest cost and should land first, independently.
• New crates/openshell-sandbox/src/handoff.rs: a versioned HandoffState + fd table, serialized over a memfd/pipe, with an explicit schema version and an unknown-field rejection policy.
• supervisor-process/run.rs: an AdoptedEntrypoint variant that skips both ProcessHandle::spawn and the hardening prelude.
• supervisor-process/process.rs: ProcessHandle needs a constructor backed by pidfd_open(2) (available post-prelude) or a waitpid loop; Drop (:899) must not unregister an adopted child during handoff.
• supervisor-process/managed_children.rs: add hydrate(&[i32]).
• supervisor-process/ssh.rs: ssh_server_init (:44) accepts a pre-bound UnixListener and a persisted host key.
• supervisor-process/netns/mod.rs: NetworkNamespace::adopt(name) that re-opens /var/run/netns/<name> without ip netns add, plus suppression of Drop teardown during handoff.
• supervisor-network/run.rs: SandboxCa must become adoptable — the biggest correctness change on the network side.
• supervisor-network/proxy.rs: accept a pre-bound TcpListener.
• supervisor-network/identity.rs: BinaryIdentityCache export/import.
• supervisor-network/opa.rs: carry the generation counter forward so numbering stays monotonic.
• Drivers: a writable, workload-inaccessible location for the replacement binary plus digest verification. No such location exists today in any driver.

Alternative Approaches Considered

(a) Full state handoff with fd passing. Serialize HandoffState + fd table; execve with FD_CLOEXEC cleared. Preserves the proxy listener backlog, SSH listener, sidecar control channel, netns fd, and (if the CA key is carried) TLS trust continuity. In-flight TLS/SSH crypto state still dies regardless. Cost: highest — ~25-40 files, requires the run_sandbox refactor and a versioned wire format, and moves the CA private key across a serialization boundary.

(b) Minimal re-exec: keep the entrypoint, drop everything session-shaped. Serialize only what is unrecoverable or dangerous to re-derive: entrypoint PID + managed children, policy proto + generation + last-known-good flag, provider credential snapshot, netns name, CA key + cert, TOFU map. Rebind the proxy and SSH listeners; let the gateway session reconnect on its own. Accept dropped SSH sessions, dropped proxy connections, a changed SSH host key, and seconds of degraded egress. The right first increment for the combined topology. Explicitly excludes the K8s sidecar topology.

(c) Split supervisor: thin supervising parent + replaceable worker child. A tiny never-upgraded PID-1 parent owning the netns, listeners, and entrypoint; a worker child holding the replaceable logic. Pros: eliminates the PID-1 fatality risk structurally (the parent can retry or roll back), the worker can be Landlock-restricted, and state handoff becomes an ordinary fd/socket protocol instead of an exec-time trick. Cons: the entrypoint becomes a grandchild of PID 1 unless the parent spawns it, changing reaping (run.rs:146-204), the expected_ssh_peer_pid check (supervisor_session.rs:699), and every /proc/<pid> identity assumption in procfs.rs. Larger change, simpler steady state. Worth deciding on before committing to (a).

Relationship to RFC #981 ("Split Supervisor and Agent into Separate Pods with gVisor Isolation"). That RFC proposes a pod-level split for isolation reasons; option (c) here is a process-level split within the supervisor for upgradability. They are not the same proposal, but they push in the same architectural direction and would interact: the split-pod model already sideloads a trusted OpenShell binary into the agent pod as its entrypoint, which is a second binary-delivery path this feature would need to account for. If #981 is accepted, option (c) becomes substantially more attractive and this spike should be re-scoped against it rather than built independently.

(d) Drain and recreate the sandbox. Zero new supervisor code and no writable-binary surface, but it kills the workload — precisely what this issue exists to avoid.

Patterns to Follow

• Pre-clap subcommand dispatch matching copy-self / debug-rpc (main.rs:461-481).
• OwnedFd where ownership transfers, RawFd only where existing code already does; clear FD_CLOEXEC with the existing fcntl idiom (bypass_monitor/mod.rs:562-568).
• Readiness signalling via oneshot, as the SSH server does (run.rs:239, ssh.rs:134), so resume failures surface as errors rather than hangs.
• OCSF per AGENTS.md: ConfigStateChangeBuilder for re-exec initiated/completed/state-adopted, AppLifecycleBuilder for the new supervisor coming up, dual-emitted DetectionFindingBuilder for binary-verification failure. Never log the CA key or credential material.
• Graceful-degradation logging (lib.rs:244-262) for non-fatal adoption failures — and its inverse: anything that would silently loosen policy must be fatal, not degraded.
• Fail-closed on missing prior state, mirroring lib.rs:2020-2024.
• #[allow(unsafe_code)] with an explicit SAFETY comment on every libc call.

Proposed Approach

Sequence this rather than building the full handoff up front. First, land the run_sandbox split into bootstrap/assembly phases as an independent refactor — it is separately valuable and separately testable, and every later step depends on it. Second, persist the SSH host key, which is correct regardless of whether re-exec ships. Third, introduce a versioned HandoffState plus a verify-state subcommand with no exec wired up, so the serialization format can be validated in isolation. Fourth, implement combined-topology re-exec along option (b), adopting the entrypoint PID, netns, CA, and TOFU map while letting sessions drop. Sidecar-topology coordination comes last and is gated on a maintainer decision about the one-shot control socket.

Scope Assessment

• Complexity: High — not because any single piece is hard, but because correctness depends on getting a ~30-item SKIP/REDO/ADOPT matrix right across three topologies, and the largest prerequisite is refactoring a 700-line function that currently encodes startup order implicitly.
• Confidence: High for the code-behavior claims (each is line-cited). Medium for the execve capability-transformation claim and for the blast radius of CA regeneration on real workloads.
• Estimated files to change: 12-18 for combined-topology-only (option b); 25-40 including sidecar, drivers, Helm, proto, gateway, and docs.
• Issue type: feat, preceded by a refactor PR splitting run_sandbox.

Risks & Open Questions

1. PID-1 fatality with no rollback. Every fatal path in the current startup sequence becomes a workload-killing path on resume — load_policy retry exhaustion (lib.rs:1906), SSH bind failure (run.rs:283-295), sidecar control checks (lib.rs:713-753). Needs an explicit design answer. Mitigations: a verify-state dry run in a forked child before the point of no return; the outgoing supervisor validates the new binary's version and schema acceptance before exec; option (c) removes the risk structurally.
2. Ephemeral CA regeneration breaks the running workload's TLS. SandboxCa::generate() runs on every start (supervisor-network/run.rs:207-272). Long-lived processes (Node, Python requests, Java) read the CA bundle at startup via SSL_CERT_FILE/NODE_EXTRA_CA_CERTS; after a resume with a fresh CA, every TLS request they make through the proxy fails validation. Either serialize the CA private key — a supervisor-private key crossing a serialization boundary — or accept the breakage.
3. Binary delivery is the whole security story. Today no writable executable path exists in any driver (K8s readOnly: true, Docker :ro,z, Podman rw: false), and nothing authenticates the binary beyond image-pull integrity. Introducing a writable path is CWE-494 territory. Requires: writable only by a non-workload identity, digest verification against a gateway-minted allow-list, and an OCSF DetectionFinding on mismatch. If the trigger is gateway-initiated and LLM-influenceable, assess prompt-injection reachability (OWASP LLM06). Note the marginal risk of the exec primitive itself is near zero — memfd_create and execveat are already unblocked in the prelude.
4. Capability loss on non-root re-exec. If confirmed, the K8s network sidecar cannot use this mechanism without ambient capabilities, which is itself a hardening regression. Verify empirically.
5. Sidecar one-shot control socket makes independent re-exec impossible. Changing it weakens a deliberate anti-impersonation property.
6. Entrypoint exit during the handoff window. The child can exit between the outgoing supervisor's last wait and the incoming supervisor's first waitpid. The exit status must not be lost or stolen by the new reaper — managed_children hydration must happen before the reaper starts.
7. TOFU cache loss re-trusts binaries. A binary swapped during the handoff window would be re-trusted at its new hash. Narrow but real integrity weakening.
8. State blob as an untrusted input. Even fd-passed, the deserializer runs as root pre-Landlock. Version skew is the point of the feature, so the format must reject unknown/mismatched schemas rather than best-effort parse.
9. Multi-threaded execve semantics. PID is retained even when a non-leader thread execs, but a thread stuck in an uninterruptible syscall can stall it. Drive the exec from a controlled point after quiescing.
10. Log continuity. The non_blocking appender's buffer is lost (main.rs:498-508). Acceptable, but operators debugging an upgrade will see a gap — document it.

Pre-existing defect encountered during this spike — already tracked

While mapping the mount-namespace machinery, this spike independently rediscovered that prepare_supervisor_identity_mount_namespace_from_env (process.rs:316) has zero callers repo-wide, so supervisor_identity_mount_from_env (:331-341) hard-fails entrypoint and SSH spawn whenever PROVIDER_SPIFFE_WORKLOAD_API_SOCKET is set in privileged mode. This is already filed as #2571, which has a live-cluster reproduction. This spike adds one datum to that report: the regression was introduced by 1ca23bc5 (#1650, the process/network subcrate split), which dropped the call site that previously sat at crates/openshell-sandbox/src/lib.rs:615 immediately before apply_supervisor_startup_hardening().

It is noted here only because it is adjacent code — it is not a blocker for this work, and fixing it is #2571's scope, not this issue's.

Disposition Readiness

• Assessment: The spike establishes that the feature is technically feasible, identifies the exact kernel/userspace boundary that constrains the design, produces a complete SKIP/REDO/ADOPT matrix with line-level evidence, and surfaces the decisions a human must make. That is sufficient for an accept/decline decision.
• Missing evidence: One empirical check would firm up the scope — whether a non-root execve actually loses SYS_PTRACE/DAC_READ_SEARCH in the K8s network sidecar (Risk 4). It affects whether the sidecar topology is in scope at all, but does not block disposition of the combined-topology work.
• Label note: the create-spike skill specifies state:validated / state:accepted / state:needs-info and agent:* labels, but none of those exist in this repository. Filed with state:review-ready, matching the convention used by other spike issues (feat: add warm-pool provisioning for Kubernetes sandboxes #2157, feat(policy): optional protocol-aware inbound authorization for gateway-less deployments #2144). The skill's label taxonomy appears to have drifted from the repo and is worth reconciling via sync-agent-infra.

Test Considerations

Existing infrastructure. Unit tests are colocated (#[cfg(test)] mod tests). Behavioral seccomp tests fork children and assert errnos (seccomp.rs:579-610). One integration test exercises the real binary (crates/openshell-sandbox/tests/stdout_logging.rs, via CARGO_BIN_EXE_openshell-sandbox). E2E lives in e2e/rust/tests/ (mise run e2e) with directly relevant precedents: gateway_resume.rs, podman_gateway_resume.rs, vm_gateway_resume.rs, local_driver_token_restart.rs, live_policy_update.rs, bypass_detection.rs.

Levels needed:

• Unit: HandoffState round-trip; version-skew rejection (older→newer, newer→older, unknown fields); managed_children::hydrate; NetworkNamespace::adopt; the SKIP/REDO/ADOPT decision table as a pure function so it is testable without a container.
• Integration (real binary): verify-state accepts a valid blob and rejects a malformed or foreign-version one; resume with no state fails closed.
• E2E (new, modelled on gateway_resume.rs): start a sandbox with a long-running workload process, trigger re-exec, then assert (i) the entrypoint PID is unchanged, (ii) egress still works and is still policy-enforced, (iii) a denied destination is still denied, (iv) the entrypoint exit code is correctly reported after re-exec, (v) the netns and veth were not recreated. Plus a negative test: re-exec into a deliberately broken binary must not kill the workload — this is the test that validates the rollback design.
• Seccomp: a behavioral test proving the prelude is still enforced after re-exec and that mount(2) still returns EPERM in the resumed process. This is the regression that would silently reopen the one-way door.

Infrastructure gaps. There is no harness for "run the supervisor, replace its binary, re-exec it" — the existing e2e resume tests restart the gateway, not the supervisor. Something new is needed, likely a test-only subcommand or env-gated trigger plus a second build artifact to exec into.

LSM fragility. Any new test that forks and execs a system binary and then reads /proc/<pid>/exe will fail on SELinux-enforcing hosts (domain transition makes the link ENOENT, not EACCES). Follow the existing guard at bypass_monitor/mod.rs:592-602. For re-exec tests, prefer exec'ing the same binary — same label, no transition — which is also the real-world case.

LSM impact on the feature itself. A replacement binary written into a new writable volume inherits that volume's label; on container-selinux hosts an emptyDir is container_file_t and executable by container_t, but a host bind-mount without :z/:Z may not be. The K8s driver sets per-container appArmorProfile (driver.rs:1792,1848,2551), which can deny x on the new path — document the required rule. /proc/<pid>/exe across domains is already load-bearing in procfs.rs:93-160; the resumed supervisor must treat ENOENT as "unreadable", not "gone".

Documentation impact. architecture/sandbox.md "Startup Flow" (lines 24-35) and "Runtime Model" describe a single-shot startup and would need a resume path. docs/reference/gateway-config.mdx needs updating only if the feature introduces gateway TOML keys (e.g. an allowed-supervisor-digest list); driver-level changes would land near gateway-config.mdx:324-329. Per AGENTS.md's Cluster Infrastructure rule, a writable supervisor volume in the pod spec makes updating debug-openshell-cluster mandatory; helm-dev-environment and openshell-cli need updating if the pod spec or a trigger command changes.

---

Created by spike investigation. state:validated means the issue is ready for human disposition. A human applies state:accepted if OpenShell should pursue the work and places it on the roadmap separately. To queue unattended agent planning, a human applies agent:plan-requested; a direct request to an agent does not require that label.