feat(observability): export gateway traces over OTLP - #2534
Open
krishicks wants to merge 1 commit into
Open
Conversation
Add an opt-in OTLP/gRPC trace exporter to the gateway. Export is enabled
by the presence of an `[openshell.gateway.otlp]` table with an endpoint;
there is no separate toggle.
Instrumented:
- Request spans, one root per entrypoint, named for the RPC
(`$service/$method`) or `{method} {path}` for plain HTTP. gRPC requests
also carry `rpc.system`, `rpc.service`, and `rpc.method`, which RPC-aware
backends use to build service maps.
- Compute driver calls (create, delete, list, get, validate, watch) as
client spans anchored on the `ComputeDriver` contract.
- Store reads and writes.
- Work with no inbound request: the sandbox reconcile sweep, the provider
credential refresh tick, and driver watch events. Each roots its own
trace, so the store calls they make are attributable instead of
arriving as anonymous single-span traces.
This is deliberately not exhaustive. Auth, policy evaluation, and
middleware remain uninstrumented, as do store lifecycle calls (`ping`,
`close`) that a readiness poll would turn into a span per tick. The aim
is a working design at a reviewable size; coverage can grow against real
traces.
Design notes:
- The TOML table owns whether and where to export. The SDK's `OTEL_*`
variables own how: sampling, batching, and limits are not mirrored into
gateway config.
- Telemetry never blocks the gateway. A malformed endpoint degrades to
export-off rather than failing startup.
- Failed spans carry error status but no separate `error.type` attribute.
The cause stays filterable through attributes those spans already carry
(`grpc.code`, `http.response.status_code`). Status is set three ways,
by what the span can observe: `#[instrument(err)]` on the reconcile and
refresh sweeps, where any `Err` is a real failure; `record_error` for
spans named at runtime or built from a status code rather than a
`Result`; and `store_dispatch_traced!` for store calls, which must
exempt `UniqueViolation` and `Conflict`. Those are how the store
reports contention — a lease already held, a version conflict driving a
retry — so marking them would export a lease every follower loses, and
every gateway restart, as a failure.
- Runtime export failures are silent: the SDK's `internal-logs` feature
emitted nothing when enabled for testing, and its retry sits behind an
experimental feature, so a dropped batch stays dropped.
- The compute driver is reachable only through `TracedDriver::call`, so a
call cannot skip its span. This is the client half of a client/server
pair; when drivers move out of process it is also the single place
trace context needs injecting.
- Tests share one process-wide subscriber and in-memory exporter.
`tracing` caches callsite interest globally, so a per-test subscriber
leaves callsites permanently dark once an untraced test touches them.
Trace context is not propagated yet: no W3C `traceparent` is injected into
outbound driver calls, so spans from an out-of-process driver would not join
the gateway's trace. The driver seam is the one place that injection needs
to happen when it lands.
The Helm chart is intentionally unchanged, so this cannot yet be enabled
on a chart-deployed gateway. Adds a `mise run observability:*` task group
that stands up Jaeger in k3d for local verification. Those tasks pin the
kube context rather than using whatever is current, since the teardown
deletes a namespace whose name is plausible in a real cluster.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
krishicks
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
July 28, 2026 23:38
|
🌿 Preview your docs: https://nvidia-preview-pr-2534.docs.buildwithfern.com/openshell |
| /// | ||
| /// When `otlp_provider` is supplied, spans are additionally exported over | ||
| /// OTLP. The layer sits behind the same `EnvFilter` as the other layers, | ||
| /// so the configured log level gates exported spans too, and the |
Contributor
There was a problem hiding this comment.
think this is fine to couple the log-level/trace-level, trying to think of when I'd want those to be necessarily separated. I guess it depends how many places we end up instrumenting with spans
| service_name = "openshell-gateway" | ||
| ``` | ||
|
|
||
| `endpoint` is required and must be a valid URI. The gateway rejects a malformed endpoint at startup rather than falling back to disabled export. It does not connect at startup: an unreachable collector produces export failures, never a failure to serve. |
Contributor
There was a problem hiding this comment.
I thought we were failing + continuing if it has issues connecting? or do we do something special for the first-connection-attempt?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an opt-in OTLP/gRPC trace exporter to the gateway. Export is enabled by the presence of an
[openshell.gateway.otlp]table with an endpoint; there is no separate toggle.Related Issue
#2507
Changes
Instrumented:
$service/$method) or{method} {path}for plain HTTP. gRPC requests also carryrpc.system,rpc.service, andrpc.method, which RPC-aware backends use to build service maps.ComputeDrivercontract.This is deliberately not exhaustive. Auth, policy evaluation, and middleware remain uninstrumented, as do store lifecycle calls (
ping,close) that a readiness poll would turn into a span per tick. The aim is a working design at a reviewable size; coverage can grow against real traces.Design notes:
OTEL_*variables own how: sampling, batching, and limits are not mirrored into gateway config.error.typeattribute. The cause stays filterable through attributes those spans already carry (grpc.code,http.response.status_code). Status is set three ways,by what the span can observe:
#[instrument(err)]on the reconcile and refresh sweeps, where anyErris a real failure;record_errorfor spans named at runtime or built from a status code rather than aResult; andstore_dispatch_traced!for store calls, which must exemptUniqueViolationandConflict. Those are how the store reports contention — a lease already held, a version conflict driving a retry — so marking them would export a lease every follower loses, and every gateway restart, as a failure.- The compute driver is reachable only throughTracedDriver::call, so a call cannot skip its span. This is the client half of a client/server pair; when drivers move out of process it is also the single place trace context needs injecting.internal-logsfeature emitted nothing when enabled for testing, and its retry sits behind an experimental feature, so a dropped batch stays dropped.TracedDriver::call, so a call cannot skip its span. This is the client half of a client/server pair; when drivers move out of process it is also the single place trace context needs injecting.tracingcaches callsite interest globally, so a per-test subscriber leaves callsites permanently dark once an untraced test touches them.Trace context is not propagated yet: no W3C
traceparentis injected into outbound driver calls, so spans from an out-of-process driver would not join the gateway's trace. The driver seam is the one place that injection needs to happen when it lands.The Helm chart is intentionally unchanged, so this cannot yet be enabled on a chart-deployed gateway. Adds a
mise run observability:*task group that stands up Jaeger in k3d for local verification.Testing
mise run pre-commitpassesChecklist