Skip to content

Fix #291: accept EVENTSENT/EVENTRAISED confirmation for entity calls on classic (Azure Storage) backend#294

Merged
YunchuWang merged 5 commits into
mainfrom
yunchuwang-fix-291-entity-eventsent-replay
Jul 10, 2026
Merged

Fix #291: accept EVENTSENT/EVENTRAISED confirmation for entity calls on classic (Azure Storage) backend#294
YunchuWang merged 5 commits into
mainfrom
yunchuwang-fix-291-entity-eventsent-replay

Conversation

@YunchuWang

Copy link
Copy Markdown
Member

Fixes #291.

Problem

callEntity is not replay-deterministic on the classic (Azure Storage / DurableTask.Core) backend. The worker throws a NonDeterminismError whenever an entity call is confirmed via an EVENTSENT history event, so entities only work on the DTS backend.

Root cause

On the classic backend, entity operations do not use the DTS entity protocol. The Durable Functions gRPC shim (ProtobufUtils.ToHistoryEventProto) has no EntityOperationCalled case, so a callEntity round-trips as a classic send-event:

  • The entity-send confirmation replays as an EVENTSENT history event.
  • The entity result replays as an EVENTRAISED event whose name == requestId and whose input is a DTFx ResponseMessage JSON.

But handleEventSent in orchestration-executor.ts popped the pending action by id and required it to be a sendEvent action — when it was a sendEntityMessage it threw:

A previous execution called sendEvent with ID=1, but the current execution is instead trying to call a different method as part of rebuilding its history.

And handleEventRaised only correlated external events by name, so it never routed the entity result back to the pending callEntity task. Only the DTS ENTITYOPERATIONCALLED / ENTITYOPERATIONCOMPLETED path worked.

Fix (two halves — core SDK only, surgical)

Half 1 — request confirmation (handleEventSent): When the pending action popped for an EVENTSENT is a sendEntityMessage (call / signal / lock / unlock), treat it as a valid confirmation and remove the pending action instead of throwing. Genuinely-wrong action types (e.g. a ScheduleTask/createTimer confirmed via EVENTSENT) still throw NonDeterminismError — the existing guard is preserved.

Half 2 — response routing (handleEventRaised): When an EVENTRAISED's (lowercased) name matches a pending entity call (keyed by requestId), route it to handleEntityResponseFromEventRaised, which decodes the DTFx ResponseMessage wrapper and resolves/rejects the callEntity task. On DTS this branch never fires (responses arrive as ENTITYOPERATIONCOMPLETED, which removes the pending call first), so there is no double-handling and no behavior change for DTS or ordinary external events.

The correlation needs no new map: pendingEntityCalls is already keyed by requestId, and both JS newGuid() and .NET Guid.ToString() emit lowercase GUIDs, so the lowercased EVENTRAISED name matches directly.

ResponseMessage decoding (the flagged-uncertain part)

Mirrors the Java SDK's handleEntityResponseFromEventRaised (authoritative; handles both DTFx variants). Wrapper shape:

  • result — serialized operation result (double-encoded string; may be null/absent).
  • exceptionType — present ⇒ failure. Misleading key name: it carries the error message (the C# property is ErrorMessage but a [DataMember(Name = "exceptionType")] annotation overrides the JSON key). Omitted when null.
  • failureDetails — optional structured { ErrorType, ErrorMessage, StackTrace, ... } (PascalCase).

Decode logic:

  1. JSON.parse the raw input. If it isn't an object or has no result key ⇒ fall back to treating the parsed value directly as the result (matches Java's fallback for a possible future raw-result format).
  2. If exceptionType or failureDetails is present ⇒ fail with EntityOperationFailedException (errorType from failureDetails.ErrorType else "unknown"; errorMessage from failureDetails.ErrorMessage else the exceptionType text).
  3. Otherwise ⇒ success: extract the inner (double-encoded) result string and JSON.parse it once (nullundefined), with a defensive fallback to the raw string.

Cross-SDK reference

All three other gRPC SDKs already handle this; this ports their semantics:

  • Java (TaskOrchestrationExecutor.java): handleEventSent is a type-agnostic no-op that just removes the pending action; the result arrives via EVENTRAISED and is decoded by handleEntityResponseFromEventRaised.
  • Python (durabletask/worker.py): the eventSent handler tolerates a sendEntityMessage confirmation and records correlation state so the later EVENTRAISED result is routed.
  • .NET (durabletask-dotnet): converts both EVENTSENT and ENTITYOPERATIONCALLED into the same EventSentEvent before replay, so no type mismatch arises.

Tests

Added to test/orchestration_executor.spec.ts (new describe block next to the existing EventSent Handler tests):

  • Replay: EVENTSENT confirming a callEntity CALL action removes the pending action and does not throw.
  • Replay: same for a signalEntity SIGNAL action.
  • End-to-end: callEntity confirmed via EVENTSENT, result delivered via EVENTRAISED (ResponseMessage {result}) resolves the task with the decoded value.
  • End-to-end: a failureDetails ResponseMessage rejects with EntityOperationFailedException (correct errorType/errorMessage).
  • End-to-end: an exceptionType-only ResponseMessage rejects with errorType = "unknown" and the message from exceptionType.

The existing test asserting that an EVENTSENT confirming a non-entity, non-sendEvent action (a ScheduleTask) still throws NonDeterminismError is preserved and passing.

Validation

  • npm run build:core — ✅
  • npm run lint — ✅
  • Core unit suite (test:unit, 60 suites / 1075 tests) — ✅ no regressions
  • Workspace unit tests — ✅

Notes / follow-up

  • Out of scope (follow-up): routing lock-grant responses that arrive as EVENTRAISED so lockEntities fully works on the classic backend. signalEntity/lock/unlock confirmations no longer crash (Half 1), but wiring the lock-grant response on classic is a separate change (Java handles it via an AutoCloseable data-type branch). Filed as a follow-up rather than bundled here to keep this fix surgical.

…on classic (Azure Storage) backend

On the classic backend (Azure Storage via the Durable Functions gRPC shim), a
callEntity round-trips as a classic send-event rather than the DTS entity
protocol: the request confirmation replays as an EVENTSENT history event and the
result as an EVENTRAISED (name = requestId, input = a DTFx ResponseMessage JSON).

Previously handleEventSent required the pending action to be a sendEvent and threw
NonDeterminismError for a sendEntityMessage action, and handleEventRaised never
routed the entity result. Net effect: entities only replayed on DTS, never on
Azure Storage.

This mirrors the Java/Python/.NET SDKs (all of which handle this) with a
two-half, core-only fix:
- Request half: handleEventSent now treats an EVENTSENT confirming a
  sendEntityMessage action (call/signal/lock/unlock) as valid and removes the
  pending action instead of throwing. Genuinely-wrong action types (e.g. a
  ScheduleTask confirmed via EVENTSENT) still throw.
- Response half: handleEventRaised routes an EVENTRAISED whose name matches a
  pending entity call (by requestId) into the callEntity task, decoding the DTFx
  ResponseMessage wrapper (result / exceptionType / failureDetails) to
  resolve or reject with EntityOperationFailedException.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 9, 2026 17:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes entity-call replay determinism for the classic (Azure Storage / DurableTask.Core) backend by tolerating the legacy EVENTSENT confirmation shape for entity message actions and by correlating entity call responses that arrive as EVENTRAISED (name = requestId) containing a DTFx ResponseMessage JSON wrapper. This brings the JS SDK behavior in line with other language SDKs while keeping DTS behavior unchanged.

Changes:

  • Route classic-backend entity call responses delivered via EVENTRAISED (name = requestId) to pending callEntity tasks, decoding the DTFx ResponseMessage wrapper.
  • Accept EVENTSENT confirmations for sendEntityMessage actions (call/signal/lock/unlock) and clear the pending action instead of throwing NonDeterminismError.
  • Add unit tests covering the classic EVENTSENT/EVENTRAISED entity-call path, including success and failure variants.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File Description
packages/durabletask-js/src/worker/orchestration-executor.ts Adds classic-backend entity response routing + ResponseMessage decoding; relaxes EVENTSENT validation to accept entity-message confirmations.
packages/durabletask-js/test/orchestration_executor.spec.ts Adds regression tests for classic backend entity confirmation (EVENTSENT) and result delivery (EVENTRAISED) including failure decoding.

Comment thread packages/durabletask-js/test/orchestration_executor.spec.ts
Comment thread packages/durabletask-js/test/orchestration_executor.spec.ts
Comment thread packages/durabletask-js/src/worker/orchestration-executor.ts Outdated
Comment thread packages/durabletask-js/src/worker/orchestration-executor.ts
Comment thread packages/durabletask-js/src/worker/orchestration-executor.ts
YunchuWang and others added 4 commits July 9, 2026 15:25
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…ResponseMessage

Autofix commit fd8f39a added `stackTrace: decoded.stackTrace` at the
EntityOperationFailedException call site in handleEntityResponseFromEventRaised
but never updated decodeEntityResponseMessage to return it, breaking the build:
  error TS2339: Property 'stackTrace' does not exist on type
  '{ isFailure: true; errorType: string; errorMessage: string; }'.

Over gRPC the classic (Azure Storage) backend uses DurableTask.Core native
entities, whose ResponseMessage.FailureDetails carries a real StackTrace. The
wire JSON is {"result":null,"failureDetails":{"ErrorType":..,"ErrorMessage":..,"StackTrace":..}}.
Extract failureDetails.StackTrace in the decoder and return it in the failure
arm (return type widened to include an optional stackTrace) so the call site
type-checks and the trace propagates into
EntityOperationFailedException.failureDetails.stackTrace.

This intentionally goes one step beyond the Java SDK (which passes null for the
stack trace); documented in a JSDoc note so it isn't "fixed" back for parity.

Tests: the failureDetails-variant test now includes a StackTrace and asserts it
propagates; the exceptionType-only (WebJobs-variant) test asserts stackTrace
stays undefined.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@YunchuWang YunchuWang merged commit dd8f53e into main Jul 10, 2026
28 checks passed
@YunchuWang YunchuWang deleted the yunchuwang-fix-291-entity-eventsent-replay branch July 10, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

callEntity/signalEntity is not replay-deterministic — entity send action is never cleared from _pendingActions

3 participants