Skip to content

VM sandbox SSH session disconnects with broken pipe #2587

Open
Bug
Open
VM sandbox SSH session disconnects with broken pipe#2587
Bug
Labels
state:triage-neededOpened without agent diagnostics and needs triage

Description

@denniskniep
denniskniep
opened on Aug 2, 2026

Agent Diagnostic

  • Skills loaded: create-github-issue (repo skill), direct code investigation of crates/openshell-supervisor-process,
    openshell-supervisor-network, openshell-driver-vm, openshell-sandbox, openshell-server
  • Latest release checked: OpenShell: 0.0.96 on 6.18.33.2-microsoft-standard-WSL2
  • Findings: SSH session to a VM-driver sandbox completes a full handshake and shell setup (pty_request/env_request/shell_request all
    succeed), then dies ~14.5s later with write to target failed: Broken pipe (os error 32) on the relay bridge to
    unix:/run/openshell/ssh.sock. No timeout/keepalive/rekey constant in supervisor_session.rs, the gateway's relay-pending timeouts
    (grpc/sandbox.rs), or the embedded SSH server config (ssh.rs) matches this window — the only 10s/15s timeouts found gate
    pre-establishment (waiting for the relay to be claimed), not an already-bridging channel. The sandbox supervisor process itself is
    confirmed still alive well past the failure (background Fetching inference route bundle polling continues uninterrupted every ~5s for
    the following minute), which rules out a whole-process/VM-guest death and scopes the bug to a single relay bridge/channel.
  • Remaining reason for filing: an interactive SSH session to a keep-alive entrypoint (tail -f /dev/null) should stay open until the
    user disconnects; a ~15s spontaneous teardown with no matching timeout in the code and no client-side activity in that window indicates
    a real defect in per-channel session/lifecycle handling under the VM driver, not expected behavior.

Description

Actual behavior: After openshell sandbox create (VM compute driver), the auto-connected SSH session completes handshake and shell
setup successfully, then disconnects ~14-15 seconds later with:
Warning: Permanently added 'sandbox' (ED25519) to the list of known hosts.
Connection to sandbox closed by remote host.
Connection to sandbox closed.
client_loop: send disconnect: Broken pipe
The gateway logs ForwardTcp: inbound stream ended and the sandbox logs a relay bridge failure (write to target failed: Broken pipe (os error 32)) at the same moment, while the sandbox process itself keeps running normally afterward.

Expected behavior: The SSH session should remain open indefinitely (entrypoint is tail -f /dev/null, a keep-alive command) until
the user disconnects.

Reproduction Steps

  1. Use a gateway.toml with the VM compute driver: 
    [openshell.gateway]
compute_drivers = ["vm"]
log_level = "debug"

[openshell.drivers.vm]
mem_mib = 4096
vcpus = 2
default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:fffb6b2"
bootstrap_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:fffb6b2"
  2. Run openshell -v sandbox create.
  3. Observe the SSH session connect, then disconnect ~14-15 seconds later.

Environment

Environment

  • OS: 6.18.33.2-microsoft-standard-WSL2
  • OpenShell: 0.0.96

Logs

cli

openshell -v sandbox create

Created sandbox: dreamy-halibut

✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
Warning: Permanently added 'sandbox' (ED25519) to the list of known hosts.
Connection to sandbox closed by remote host.
Connection to sandbox closed.
client_loop: send disconnect: Broken pipe

gateway/sandbox

12:35:58 gateway INFO  minted sandbox JWT
12:35:58 gateway INFO  CreateSandbox request completed successfully
12:35:59 sandbox INFO  Starting sandbox command=["tail", "-f", "/dev/null"]
12:35:59 sandbox INFO  Fetching sandbox policy via gRPC endpoint=https://host.openshell.internal:17670/ sandbox_id=cbea7e3a-fda6-4e20-90e4-6816a3dc3984
12:35:59 gateway DEBUG GetSandboxConfig: no policy configured, returning empty response
12:35:59 gateway INFO  UpdateConfig: backfilled spec.policy from sandbox-discovered policy
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway INFO  GetSandboxProviderEnvironment request completed successfully
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway INFO  ReportPolicyStatus: sandbox reported policy load result
12:35:59 gateway INFO  supervisor session: accepted
12:36:00 gateway INFO  supervisor session: relay opened successfully
12:36:00 gateway INFO  relay stream: claimed pending relay, bridging
12:35:59 sandbox OCSF  CONFIG:DISCOVERY [INFO] Server returned no policy; attempting local discovery
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Loaded sandbox policy from container disk [path:/etc/openshell/policy.yaml]
12:35:59 sandbox INFO  Creating OPA engine from proto policy data
12:35:59 sandbox INFO  Configured OPA runtime binary identity mode require_binary_identity=true source=proto
12:35:59 sandbox OCSF  CONFIG:CONFIGURED [INFO] OPA runtime binary identity mode configured [source:proto require_binary_identity:true] [source:proto]
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: claude_code.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: github_rest_api.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: github_ssh_over_https.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Fetched provider environment [env_count:0]
12:35:59 sandbox OCSF  CONFIG:CREATING [INFO] Creating network namespace [ns:sandbox-61660b7c host_veth:veth-h-61660b7c sandbox_veth:veth-s-61660b7c]
12:35:59 sandbox OCSF  CONFIG:CREATED [INFO] Network namespace created [ns:sandbox-61660b7c host_ip:10.200.0.1 sandbox_ip:10.200.0.2]
12:35:59 sandbox WARN  non-required nft command failed in namespace (continuing) command=add rule inet openshell_bypass output tcp flags syn limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid error=Error: syntax error, unexpected colon, expecting end of file or newline or semicolon
add rule inet openshell_bypass output tcp flags syn limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid
                                                                                                             ^ netns=sandbox-61660b7c
12:35:59 sandbox WARN  non-required nft command failed in namespace (continuing) command=add rule inet openshell_bypass output meta l4proto udp limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid error=Error: syntax error, unexpected colon, expecting end of file or newline or semicolon
add rule inet openshell_bypass output meta l4proto udp limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid
                                                                                                                ^ netns=sandbox-61660b7c
12:35:59 sandbox OCSF  CONFIG:INSTALLED [INFO] Bypass detection rules installed [ns:sandbox-61660b7c]
12:35:59 sandbox OCSF  CONFIG:ENABLED [INFO] TLS termination enabled: ephemeral CA generated
12:35:59 sandbox INFO  Fetching inference route bundle from gateway endpoint=https://host.openshell.internal:17670/
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Loaded inference route bundle [route_count:0 revision:d1fba762150c532c]
12:35:59 sandbox OCSF  CONFIG:WAITING [INFO] Inference route bundle is empty; keeping routing enabled and waiting for refresh
12:35:59 sandbox OCSF  CONFIG:ENABLED [INFO] Inference routing enabled with local execution [route_count:0]
12:35:59 sandbox OCSF  NET:LISTEN [INFO] 10.200.0.1:3128
12:35:59 sandbox WARN  host.openshell.internal maps to a non-link-local IP; trusted-gateway SSRF exemption disabled ip=192.168.127.254
12:35:59 sandbox OCSF  CONFIG:VALIDATED [INFO] Validated 'sandbox' user exists in image
12:35:59 sandbox OCSF  CONFIG:VALIDATED [INFO] Validated 'sandbox' group exists in image
12:35:59 sandbox WARN  runtime cgroup pids.max is unavailable (No such file or directory (os error 2)); configure the compute driver or container runtime to enforce a PID limit
12:35:59 sandbox OCSF  SSH:LISTEN [INFO]
12:35:59 sandbox OCSF  LIFECYCLE:INSTALL [INFO] OpenShell Sandbox Supervisor success
12:35:59 sandbox INFO  supervisor session task spawned
12:35:59 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:35:59 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Acknowledged initial policy revision as loaded [version:1] [hash:27380a8cbc68351c4aa9a60cdfebe905ec647600bb2e5d527e13f1a298998792]
12:35:59 sandbox OCSF  NET:OPEN [INFO] host.openshell.internal:17670
12:35:59 sandbox OCSF  NET:OPEN [INFO] [msg:ssh relay open (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c, target=unix:/run/openshell/ssh.sock)]
12:35:59 sandbox OCSF  SSH:OPEN [INFO] ALLOWED
12:36:00 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:36:04 sandbox WARN  Entrypoint PID never published; binary symlink resolution skipped. Policy binary paths will be matched literally.
12:36:09 gateway DEBUG GetSandboxConfig served from policy history
12:36:14 gateway DEBUG ForwardTcp: inbound stream ended
12:36:14 sandbox OCSF  NET:FAIL [LOW] [msg:ssh relay bridge failed (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c, target=unix:/run/openshell/ssh.sock): write to...]
12:36:14 sandbox WARN  supervisor session: relay bridge failed channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c error=write to target failed: Broken pipe (os error 32) sandbox_id=cbea7e3a-fda6-4e20-90e4-6816a3dc3984
12:36:19 gateway DEBUG GetSandboxConfig served from policy history
12:36:26 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:26 gateway DEBUG GetSandboxConfig served from policy history
12:35:58 gateway INFO  minted sandbox JWT
12:35:58 gateway INFO  CreateSandbox request completed successfully
12:35:59 sandbox INFO  Starting sandbox command=["tail", "-f", "/dev/null"]
12:35:59 sandbox INFO  Fetching sandbox policy via gRPC endpoint=https://host.openshell.internal:17670/ sandbox_id=cbea7e3a-fda6-4e20-90e4-6816a3dc3984
12:35:59 gateway DEBUG GetSandboxConfig: no policy configured, returning empty response
12:35:59 gateway INFO  UpdateConfig: backfilled spec.policy from sandbox-discovered policy
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway INFO  GetSandboxProviderEnvironment request completed successfully
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway INFO  ReportPolicyStatus: sandbox reported policy load result
12:35:59 gateway INFO  supervisor session: accepted
12:36:00 gateway INFO  supervisor session: relay opened successfully
12:36:00 gateway INFO  relay stream: claimed pending relay, bridging
12:35:59 sandbox OCSF  CONFIG:DISCOVERY [INFO] Server returned no policy; attempting local discovery
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Loaded sandbox policy from container disk [path:/etc/openshell/policy.yaml]
12:35:59 sandbox INFO  Creating OPA engine from proto policy data
12:35:59 sandbox INFO  Configured OPA runtime binary identity mode require_binary_identity=true source=proto
12:35:59 sandbox OCSF  CONFIG:CONFIGURED [INFO] OPA runtime binary identity mode configured [source:proto require_binary_identity:true] [source:proto]
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: claude_code.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: github_rest_api.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: github_ssh_over_https.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Fetched provider environment [env_count:0]
12:35:59 sandbox OCSF  CONFIG:CREATING [INFO] Creating network namespace [ns:sandbox-61660b7c host_veth:veth-h-61660b7c sandbox_veth:veth-s-61660b7c]
12:35:59 sandbox OCSF  CONFIG:CREATED [INFO] Network namespace created [ns:sandbox-61660b7c host_ip:10.200.0.1 sandbox_ip:10.200.0.2]
12:35:59 sandbox WARN  non-required nft command failed in namespace (continuing) command=add rule inet openshell_bypass output tcp flags syn limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid error=Error: syntax error, unexpected colon, expecting end of file or newline or semicolon
add rule inet openshell_bypass output tcp flags syn limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid
                                                                                                             ^ netns=sandbox-61660b7c
12:35:59 sandbox WARN  non-required nft command failed in namespace (continuing) command=add rule inet openshell_bypass output meta l4proto udp limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid error=Error: syntax error, unexpected colon, expecting end of file or newline or semicolon
add rule inet openshell_bypass output meta l4proto udp limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid
                                                                                                                ^ netns=sandbox-61660b7c
12:35:59 sandbox OCSF  CONFIG:INSTALLED [INFO] Bypass detection rules installed [ns:sandbox-61660b7c]
12:35:59 sandbox OCSF  CONFIG:ENABLED [INFO] TLS termination enabled: ephemeral CA generated
12:35:59 sandbox INFO  Fetching inference route bundle from gateway endpoint=https://host.openshell.internal:17670/
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Loaded inference route bundle [route_count:0 revision:d1fba762150c532c]
12:35:59 sandbox OCSF  CONFIG:WAITING [INFO] Inference route bundle is empty; keeping routing enabled and waiting for refresh
12:35:59 sandbox OCSF  CONFIG:ENABLED [INFO] Inference routing enabled with local execution [route_count:0]
12:35:59 sandbox OCSF  NET:LISTEN [INFO] 10.200.0.1:3128
12:35:59 sandbox WARN  host.openshell.internal maps to a non-link-local IP; trusted-gateway SSRF exemption disabled ip=192.168.127.254
12:35:59 sandbox OCSF  CONFIG:VALIDATED [INFO] Validated 'sandbox' user exists in image
12:35:59 sandbox OCSF  CONFIG:VALIDATED [INFO] Validated 'sandbox' group exists in image
12:35:59 sandbox WARN  runtime cgroup pids.max is unavailable (No such file or directory (os error 2)); configure the compute driver or container runtime to enforce a PID limit
12:35:59 sandbox OCSF  SSH:LISTEN [INFO]
12:35:59 sandbox OCSF  LIFECYCLE:INSTALL [INFO] OpenShell Sandbox Supervisor success
12:35:59 sandbox INFO  supervisor session task spawned
12:35:59 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:35:59 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Acknowledged initial policy revision as loaded [version:1] [hash:27380a8cbc68351c4aa9a60cdfebe905ec647600bb2e5d527e13f1a298998792]
12:35:59 sandbox OCSF  NET:OPEN [INFO] host.openshell.internal:17670
12:35:59 sandbox OCSF  NET:OPEN [INFO] [msg:ssh relay open (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c, target=unix:/run/openshell/ssh.sock)]
12:35:59 sandbox OCSF  SSH:OPEN [INFO] ALLOWED
12:36:00 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:36:04 sandbox WARN  Entrypoint PID never published; binary symlink resolution skipped. Policy binary paths will be matched literally.
12:36:09 gateway DEBUG GetSandboxConfig served from policy history
12:36:14 gateway DEBUG ForwardTcp: inbound stream ended
12:36:14 sandbox OCSF  NET:FAIL [LOW] [msg:ssh relay bridge failed (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c, target=unix:/run/openshell/ssh.sock): write to...]
12:36:14 sandbox WARN  supervisor session: relay bridge failed channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c error=write to target failed: Broken pipe (os error 32) sandbox_id=cbea7e3a-fda6-4e20-90e4-6816a3dc3984
12:36:19 gateway DEBUG GetSandboxConfig served from policy history
12:36:26 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:26 gateway DEBUG GetSandboxConfig served from policy history
12:36:28 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:28 gateway DEBUG GetSandboxConfig served from policy history
12:36:28 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:30 gateway DEBUG GetSandboxConfig served from policy history
12:36:30 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:30 gateway DEBUG GetSandboxConfig served from policy history
12:36:30 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:32 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:32 gateway DEBUG GetSandboxConfig served from policy history
12:36:32 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:34 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:34 gateway DEBUG GetSandboxConfig served from policy history

rootfs-console.log

  2026-08-02T12:36:00.013Z DEBUG log: handler.shell_request ChannelId(2)
  2026-08-02T12:36:00.016Z DEBUG log: channel_success ... (shell established)
  2026-08-02T12:36:04.702Z WARN openshell_supervisor_network::run: Entrypoint PID never published; binary symlink resolution skipped.
  2026-08-02T12:36:14.514Z OCSF NET:FAIL [msg:ssh relay bridge failed (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c,
  target=unix:/run/openshell/ssh.sock): write to...]
  2026-08-02T12:36:14.516Z WARN openshell_supervisor_process::supervisor_session: supervisor session: relay bridge failed
  channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c error=write to target failed: Broken pipe (os error 32)
  2026-08-02T12:36:19.725Z DEBUG openshell_core::grpc_client: Fetching inference route bundle   <- background task confirmed still alive
  after the failure, repeating every ~5s through 12:37:10+

  Full raw debug logs (gateway + rootfs-console, including h2/TLS trace-level noise) available on request.

Agent-First Checklist

  • I pointed my agent at the repo and had it investigate this issue
  • I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
  • I checked the latest OpenShell release and either reproduced the issue there or explained why I cannot upgrade/test it
  • I searched existing issues for possible duplicates or explained why I could not
  • My agent could not resolve this — the diagnostic above explains why

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    Bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions