callEntity/signalEntity is not replay-deterministic — entity send action is never cleared from _pendingActions

[YunchuWang]

Summary

Calling an entity from an orchestrator (ctx.entities.callEntity(...), or the classic context.df.callEntity(...) in the durable-functions v4 compat layer) is not replay-deterministic. The orchestration runs correctly the first time, but the moment it replays (i.e. after the entity response event comes back), the backend rejects the replayed action stream and the orchestration ends in Failed:

Could not process the event EVENTSENT due to error:
  A previous execution called sendEvent with ID=1, but the current execution is instead
  trying to call a different method as part of rebuilding its history.
Reason: DurableTask.Core.Exceptions.OrchestrationFailureException

A .NET-hosted test harness surfaces this as Orchestration reached Failed state when test was expecting Completed.

Key discriminator — entity execution works; only the replay of the calling orchestration is broken

The entity function itself executes fine over gRPC (Executed 'Functions.TestEntity' (Succeeded) ... State: Completed). So this is not an entity-dispatch problem. The defect is specifically in how the calling orchestration's entity-message action is reconciled against history on replay.

Root cause (concrete)

Every other completion handler removes its emitted action from _pendingActions when the corresponding history event is replayed, so the action is not re-emitted on subsequent turns. The entity-completion handlers do not.

• Activity path — packages/durabletask-js/src/worker/orchestration-executor.ts, handleTaskScheduled:

  const taskId = event.getEventid();
  const action = ctx._pendingActions[taskId];
  delete ctx._pendingActions[taskId];   // "Remove ... so we don't schedule it again."

• Entity path — same file, handleEntityOperationCompleted / handleEntityOperationFailed:

  const pendingCall = ctx._entityFeature.pendingEntityCalls.get(requestId);
  ...
  ctx._entityFeature.pendingEntityCalls.delete(requestId);  // deletes ONLY the requestId->task map entry
  pendingCall.task.complete(result);                         // never deletes ctx._pendingActions[<actionId>]

callEntity emits the message action with a deterministic id (nextSequenceNumber()) and a replay-safe requestId (newGuid()), and stores the action in _pendingActions[actionId] (runtime-orchestration-context.ts, callEntity ~line 774–797). But pendingEntityCalls only stores { task, entityId, operationName } — not the actionId:

readonly pendingEntityCalls: Map<
  string,
  { task: CompletableTask<any>; entityId: EntityInstanceId; operationName: string }
>;

Because the completion handler has only the requestId and the map entry does not carry the actionId, it cannot remove the emitted SendEntityMessage action from _pendingActions. getActions() returns Object.values(this._pendingActions), so the already-satisfied entity send action is re-emitted on the next turn, desyncing the action stream from recorded history and tripping the backend's non-determinism guard (sendEvent ID=1 ... trying to call a different method).

This is symmetric with signalEntity and lockEntities, which emit the same class of SendEntityMessage action.

Affected scenarios

• Class-based / function-based entity orchestrations that callEntity then read state (SetState → GetState)
• Entity error-handling flows (caught / uncaught / retried entity exceptions) — all reach Failed via this path
• Any orchestration that calls an entity and then continues (i.e. requires at least one replay)

Minimal repro

Classic v4 (durable-functions) surface:

import * as df from "durable-functions";

df.app.orchestration("EntityReplayRepro", function* (context) {
    const id = new df.EntityId("TestEntity", "singleton");
    yield context.df.callEntity(id, "SetState", 42);
    const result = yield context.df.callEntity(id, "GetState"); // replay diverges here
    return result;
});

df.app.entity("TestEntity", (context) => {
    if (context.df.operationName === "SetState") {
        const s = context.df.getState(() => ({ state: "" }));
        s.state = String(context.df.getInput());
        context.df.setState(s);
    } else if (context.df.operationName === "GetState") {
        context.df.return(context.df.getState());
    }
});

Because the defect is in core entity-response handling, a pure-core orchestration that yields ctx.entities.callEntity(...) twice should reproduce it identically (no compat layer required). A regression test that drives an entity-calling orchestrator through ≥ 2 turns and asserts it reaches Completed would cover it.

Suggested fix

Make the entity-message action replay-consistent, mirroring the activity path:

1. Store the emitted actionId in the pendingEntityCalls entry ({ task, entityId, operationName, actionId }), and
2. In handleEntityOperationCompleted / handleEntityOperationFailed, delete ctx._pendingActions[actionId] when the response is processed (and do the analogous cleanup for lockEntities grants).

Add a replay unit test that runs an entity-calling orchestrator through ≥ 2 executions and asserts a stable action stream / Completed terminal state.

Attribution / environment

Pre-existing on main — this is in core (packages/durabletask-js/src/worker/), not in the packages/azure-functions-durable compat layer. git diff <base>..HEAD for the compat PR does not touch runtime-orchestration-context.ts; the file was last modified by unrelated commits (#273 / #267 / #180). Observed against a local gRPC worker + WebJobs.Extensions.DurableTask host running the classic v3 sample suite.