Bugfix: workflow transactions recovered before commit were not exactly-once

[rjhuijsman]

Before this fix, with a failure at the wrong time, a workflow's idempotent transaction call executed — and commits — twice. Discovered by a flaky test: tasks_tests_py::TasksTestCase.test_workflow_idempotent_writer_and_transaction_survive_recovery was flaky ever since it was written, and pointed us to this real bug.

The race was as follows:

1. A workflow calls a transaction with an idempotency key. The transaction's response returns to the workflow as soon as two-phase commit's prepare is durable everywhere; the commit becomes durable asynchronously (via the coordinator's commit control loop and/or the participant's watch task, both funneling into SidecarStateManager.transaction_participant_commit).
2. The application goes down inside that window (in the test, rbt.down() cancels the pending commit tasks), leaving the transaction durably prepared but uncommitted — its idempotent-mutation record (key → response) persisted only inside the prepared transaction.
3. On restart, recovery registers the prepared transaction and finishes its commit in the background — while the recovered workflow re-executes concurrently. The workflow's duplicate transaction call runs check_for_idempotent_mutation() before that commit lands: the committed-mutations lookup misses, and the in-flight-transaction guard (reboot-dev/mono#5361) misses too, so the call starts a new transaction, waits behind the recovered transaction's exclusive lock, and re-executes the user's transaction method after the recovered one commits.

In practice this bug is rare to see, because the prepared transaction being recovered typically completes its commit before the workflow re-executes - so the test usually passes. It's possible however for the workflow re-execution to win, in which case the transaction effects occur twice. 😬

There are two bugs that combined to produce this race:

1. C++: DatabaseService::RecoverTransactionIdempotentMutations scanned a recovered transaction's write batch only under the idempotent-mutation-v4 key prefix. Commit db327a7 moved workflow-scoped mutations under workflow-idempotent-mutation* key prefixes and updated Store, Export, and RecoverIdempotentMutations accordingly — but not RecoverTransactionIdempotentMutations. A recovered prepared transaction therefore reported empty uncommitted_idempotent_mutations for workflow calls. That same commit also introduced the flaky test, so the flake has existed since this code was written.
2. Python: the fix we made for reboot-dev/mono#5361 in check_for_idempotent_mutation() matches only transaction.idempotency_key, which is never persisted and is therefore None for a recovered transaction.

Fixes:

• reboot/server/database.cc: scan the write batch once per idempotent-mutation key prefix (plain, expiring, and the three workflow-scoped ones) so a recovered prepared transaction knows all of its uncommitted idempotent mutations.
• reboot/aio/state_managers.py: the in-flight guard also matches context.idempotency_key in transaction.idempotent_mutations; a duplicate call then awaits the recovered transaction and serves its cached response.
• reboot/aio/state_managers.py: transaction_participant_commit now tolerates the per-state idempotent-mutations cache entry being absent — possible once recovered transactions carry idempotent mutations and commit before any call has touched that state — instead of raising KeyError in the exception-intolerant commit section. Skipping the update is safe: any later lookup recovers from the database, which at that point includes the committed mutations.
• tests/reboot/tasks_tests.py: new deterministic regression test.

This ports reboot-dev/mono#5617 to the public repo.

[aviator-app]

Current Aviator status

Aviator will automatically update this comment as the status of the PR changes.
Comment /aviator refresh to force Aviator to re-examine your PR (or learn about other /aviator commands).

This PR is not ready to merge (currently in state pending): this PR has not been approved.

Pending Status Checks

• ⏳ Build public repo for MacOS arm64 / Build Reboot for MacOS (pending)
• ⏳ Test React Native mobile app (Maestro) / GitHub-hosted Runner (pending)
• ⏳ Build public repo for Linux x86_64 / GitHub-hosted Runner (pending)
• ⏳ Check Code Style / GitHub-hosted Runner (pending)
• ⏳ Build public repo for Linux arm64 / GitHub-hosted Runner (pending)
• ⏳ 1 other pending tests
• ✅ 2 tests passing!

---

See the real-time status of this PR on the Aviator webapp.

Use the Aviator Chrome Extension to see the status of your PR within GitHub.