feat: dynamic sandbox log routing and ClickHouse-backed log reads by rguliyev · Pull Request #3236 · e2b-dev/infra

rguliyev · 2026-07-09T02:22:18Z

What

Makes sandbox log write routing dynamic and adds a ClickHouse-backed read path for local-cluster sandbox/build logs. Both are controlled by LaunchDarkly flags and remain no-ops when the flags are unset.

Write path — `logs-write-config` JSON flag

HTTPWriter and the hyperloop /logs forwarder resolve their destination per write through a cached LogWriteConfigResolver instead of a fixed collector address.
The config supports one synchronous primary plus optional fire-and-forget shadow destinations.
Null or invalid config falls back to the legacy LOGS_COLLECTOR_ADDRESS behavior.
Shadow writes are bounded, never block the primary result, and are covered by route/result metrics.

Read path — `logs-read-config` bool flag

Adds clickhouse/pkg/sandboxlogs, a reader for the sandbox_logs table.
The API uses ClickHouse for local-cluster sandbox/build log reads only when the flag is enabled and CLICKHOUSE_CONNECTION_STRING is configured.
If the flag is disabled or no ClickHouse DSN is configured, reads stay on Loki.

Why

Lets the log pipeline migrate collectors and storage backends with flag flips instead of redeploys, while keeping every step reversible and preserving the legacy Loki path as the default.

Testing

Unit tests for flag parsing, config resolution caching, URL validation, dynamic writer routing, shadow concurrency limits, and ClickHouse sandbox/build log queries.
Local validation: go test ./packages/clickhouse/pkg/sandboxlogs ./packages/api/internal/clusters ./packages/api/internal/handlers and targeted golangci-lint.
Previously verified end-to-end in a dev cluster: logs written through the routed pipeline into ClickHouse and read back through the API with the read flag enabled.

codecov · 2026-07-09T02:23:29Z

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
3394	5	3389	9

View the top 3 failed test(s) by shortest run time

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory

Stack Traces | 0.02s run time

=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir

Stack Traces | 6.52s run time

=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
--- FAIL: TestListDir (6.52s)

View the full list of 1 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Flake rate in main: 30.77% (Passed 9 times, Failed 4 times)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

gemini-code-assist

Code Review

This pull request introduces dynamic log write routing and ClickHouse-backed log reading, both controlled by LaunchDarkly feature flags (logs-write-config and logs-read-config). It adds a minimal ClickHouse reader for sandbox and build logs, integrates feature flags into the local cluster resource provider, and updates the hyperloop server and HTTP log writer to support dynamic routing to primary and shadow destinations with bounded concurrency. Feedback on the changes suggests using context.WithoutCancel as the base context in the HTTP log writer to ensure that critical logs emitted during the application shutdown phase are successfully delivered even if the root context is cancelled.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e540816764

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be583e9859

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

matthewlouisbrockman

preexisting bug it looks like; there's an issue where there's a difference in the clickhouse vs loki ingestion, where loki's rejecting based on age > 7d, clickhouse based on when ingested, and logs before the timestamp on sandbox start is updated can get dropped in loki, and then also logs from when sandbox was pausing/snapshotting and still not flushed can also get into clickhouse and correctly rejected from loki - not sure how to handle that; not ur fault but will cause issues.

linear-code · 2026-07-16T03:53:49Z

EN-1802

rguliyev requested review from ValentaTomas, dobrac and jakubno as code owners July 9, 2026 02:22

cla-bot Bot added the cla-signed label Jul 9, 2026

rguliyev marked this pull request as draft July 9, 2026 02:23

gemini-code-assist Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread packages/shared/pkg/logger/exporter.go

claude Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/hyperloopserver/handlers/logs.go

Comment thread packages/api/internal/clusters/resources_local.go

Comment thread packages/shared/pkg/logger/exporter_test.go Outdated

Comment thread packages/orchestrator/pkg/hyperloopserver/handlers/store.go

rguliyev changed the title ~~Logs clickhouse routing~~ feat: dynamic sandbox log routing and ClickHouse-backed log reads Jul 9, 2026

rguliyev force-pushed the logs-clickhouse-routing branch 2 times, most recently from bd0a2e4 to e540816 Compare July 14, 2026 17:50

rguliyev marked this pull request as ready for review July 14, 2026 17:52

chatgpt-codex-connector Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread packages/clickhouse/pkg/sandboxlogs/sandboxlogs.go

Comment thread packages/shared/pkg/featureflags/flags.go

rguliyev force-pushed the logs-clickhouse-routing branch 4 times, most recently from 51543d6 to cb652db Compare July 15, 2026 22:18

rguliyev requested review from matthewlouisbrockman and wj-e2b July 15, 2026 22:26

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread packages/clickhouse/pkg/sandboxlogs/sandboxlogs.go

feat(logs): route sandbox logs through ClickHouse

696abc1

rguliyev force-pushed the logs-clickhouse-routing branch from be583e9 to 696abc1 Compare July 15, 2026 22:34

rguliyev requested a review from a team as a code owner July 15, 2026 22:34

matthewlouisbrockman approved these changes Jul 16, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: dynamic sandbox log routing and ClickHouse-backed log reads#3236

feat: dynamic sandbox log routing and ClickHouse-backed log reads#3236
rguliyev wants to merge 1 commit into
mainfrom
logs-clickhouse-routing

rguliyev commented Jul 9, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

matthewlouisbrockman left a comment

Uh oh!

linear-code Bot commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rguliyev commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Write path — logs-write-config JSON flag

Read path — logs-read-config bool flag

Why

Testing

Uh oh!

codecov Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 5 Tests Failed:

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

matthewlouisbrockman left a comment

Choose a reason for hiding this comment

Uh oh!

linear-code Bot commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rguliyev commented Jul 9, 2026 •

edited

Loading

Write path — `logs-write-config` JSON flag

Read path — `logs-read-config` bool flag

codecov Bot commented Jul 9, 2026 •

edited

Loading