oss_14_processor_export_lock_timeout#15
Conversation
SimpleSpanProcessor.on_end previously called span_exporter.export with no synchronization, so two threads ending sampled spans could invoke export concurrently, which the specification forbids. Guard the export call with a per-processor lock so concurrent on_end calls are serialized. The batch processor read OTEL_BSP_EXPORT_TIMEOUT into configuration but never applied it, so a hung exporter could block the worker thread and force_flush indefinitely. Run each export on a dedicated single-worker executor and await it with the configured timeout as a deadline; on timeout the batch worker stops waiting, records the failure, and moves on instead of blocking forever. A non-positive timeout disables the deadline. This affects both BatchSpanProcessor and BatchLogRecordProcessor via the shared BatchProcessor. Add tests: SimpleSpanProcessor concurrent on_end never overlaps export, and a hung batch export is bounded by the export timeout.
The batch export now runs on a separate ThreadPoolExecutor thread, but ThreadPoolExecutor.submit does not copy the caller's contextvars into the worker thread. As a result the _SUPPRESS_INSTRUMENTATION_KEY attached by the batch worker was not visible to the exporter, so an auto-instrumented transport used by the exporter could emit telemetry that feeds back into the processor. Capture the current context on the worker thread and run the export inside it via contextvars.Context.run, restoring suppression during export. Add a regression test asserting the exporter observes _SUPPRESS_INSTRUMENTATION_KEY as True when driven through the BatchProcessor.
|
🔒 Internal (dash0) — not for upstream.
|
|
📣 Public-facing draft — to be opened upstream in Title: Serialize What
WhyThe spec requires that Export not be called concurrently for the same exporter, and that Export not block indefinitely. Notes / limitationsPython cannot cancel a running synchronous call, so a timed-out export keeps running on the executor thread (bounded, single-thread back-pressure); the worker and TestingTests cover: |
1 participant
Closes #12
Fixes span-processor export serialization + timeout (Linear OSS-14, findings T2/T3): SimpleSpanProcessor.export() is now serialized behind a lock (T2); BatchProcessor enforces OTEL_BSP_EXPORT_TIMEOUT via a bounded executor so a hung export can't block forever (T3), with instrumentation-suppression context preserved across the executor thread.
Linear issue: https://linear.app/dash0/issue/OSS-14/t2-t3-simplespanprocessor-concurrent-export-bsp-export-timeout-not