Skip to content

Synchronize cloud telemetry lifecycle to fix shared-state races#1019

Open
jnunemaker wants to merge 2 commits into
mainfrom
telemetry-shared-state-race
Open

Synchronize cloud telemetry lifecycle to fix shared-state races#1019
jnunemaker wants to merge 2 commits into
mainfrom
telemetry-shared-state-race

Conversation

@jnunemaker

Copy link
Copy Markdown
Collaborator

Flipper::Cloud::Telemetry reassigned and tore down shared state (@metric_storage/@pool/@timer) with no synchronization, so concurrent fork detection could double up restarts (leaking a pool and timer with live threads) and readers could observe an inconsistent storage/pool/timer trio. This guards the lifecycle (start/stop/restart) with a mutex via internal start!/stop!/restart! variants and uses double-checked locking in detect_forking so exactly one thread restarts on fork. The hot record path stays lock-free (it only touches the mutex when a fork is actually detected), and the worker paths snapshot the swapped ivars into locals. Adds specs proving exactly one restart under concurrent fork detection and that records aren't lost while a fork restart is in progress.

🤖 Generated with Claude Code

record, post_to_pool, and post_to_cloud read @metric_storage/@pool/@Timer
while restart (on fork) and stop (on a telemetry-shutdown header) reassign or
tear them down with no synchronization. Concurrent fork detection could double
up restarts, leaking a pool+timer with live threads, and readers could observe
an inconsistent storage/pool/timer trio or write into a swapped-out storage.

Guard the lifecycle (start/stop/restart) with a mutex via internal start!/stop!/
restart! variants, use double-checked locking in detect_forking so exactly one
thread restarts while keeping the hot record path lock-free, and snapshot the
swapped ivars into locals in the worker paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jnunemaker jnunemaker force-pushed the telemetry-shared-state-race branch from e8566fe to 283b3bd Compare July 8, 2026 06:26
record, post_to_pool, and post_to_cloud read @metric_storage/@pool/@Timer
while restart (on fork) and stop (on a telemetry-shutdown header) reassign or
tear them down with no synchronization. Concurrent fork detection could double
up restarts, leaking a pool+timer with live threads.

Guard the lifecycle (start/stop/restart) with a mutex via internal start!/stop!/
restart! variants, and use double-checked locking in detect_forking so exactly
one thread restarts while keeping the hot record path lock-free.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jnunemaker jnunemaker force-pushed the telemetry-shared-state-race branch from 283b3bd to 71b80c4 Compare July 8, 2026 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant