feat(agent): long-horizon readiness — tool heartbeat, shell timeout, telegram persist, LLM retry resilience, graceful budget exit, stall detection, compaction default-on - #123
Merged
Conversation
…bounded probes, telegram per-turn persist Four gaps found in a long-horizon task review (per-turn session persistence for CLI/REPL/serve landed separately in #122): 1. Tool-running heartbeat (internal/loop, internal/render, odek.go, cmd/odek/telegram.go): a single long tool call (e.g. shell running a test suite for minutes) showed a static spinner with zero feedback inside the shell tool's 30-minute buffered window. The loop now wraps each tool call with a watchdog that emits a 'tool_running' SignalEvent every 60s (toolHeartbeatInterval, test-overridable) until the call returns — closed on normal, error, and panic paths and on engine-context cancel, so no goroutine leaks. CLI renders '⏳ <tool> still running (<elapsed>)'; Telegram surfaces it even when skills.verbose is off (a silent multi-minute call looks like a hang); serve forwards it generically. 2. shell timeout_seconds (cmd/odek/shell.go): the LLM can now bound long commands explicitly (schema + Description advise it for builds/test suites). Clamped to [1, 1800]s via clampShellTimeoutSeconds, mirroring parallel_shell's cap; zero/ negative treated as absent. Output stays fully buffered — the heartbeat is the progress signal. 3. Bounded Docker availability probes: dockerAvailable() in cmd/odek/main_test.go and internal/sandbox/sandbox_test.go, plus the 'docker image inspect' probe in internal/sandbox, now use exec.CommandContext with a 5s timeout — a stopped/starting daemon can no longer hang the test suite or sandbox setup. 4. Telegram per-turn persistence (internal/telegram/session.go, cmd/odek/telegram.go): SessionManager.SaveNoIndex mirrors Save via Store.SaveNoIndex (no per-iteration embedding call) but does NOT advance the user-visible TurnCount — only a completed turn's final Save may. handleChatMessage wires SetMessagesPersistCallback with dropDanglingToolCalls (extracted from persistPartialMessages) and a skip-on-trim length check against the latest persisted state. /stop, restarts, and crashes now resume from the last completed step; SIGKILL may still lose the in-flight step (documented in docs/CLI.md). Docs: docs/CLI.md continue-row resume semantics; AGENTS.md testing guidance (scope test runs, -race on changed packages only, buffered shell + timeout_seconds) and shell.go layout line. Tests: TestToolHeartbeat_LongRunningToolEmitsSignals (fires during, stops after), shell timeout/clamp/non-positive tests, TestSessionManager_SaveNoIndex (no vector indexing, no TurnCount inflation, final Save indexes). Verified with clean env: go build ./... + go vet ./..., affected package tests, -race on internal/loop + internal/telegram, full ./cmd/odek suite (24.4s), golangci-lint 0 issues.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
odek | 70f5bd0 | Commit Preview URL Branch Preview URL |
Jul 28 2026, 08:46 PM |
…teration-budget exit, stall detection, compaction default-on A. LLM client resilience (internal/llm) — the retry budget was sized for second-scale blips and killed long unattended runs on the first sustained incident: - Retryable statuses now include 408, 500, and 529 (Anthropic overloaded) alongside 429/502/503/504. - maxRetries 3 -> 7 (8 attempts), exponential backoff 1s..30s with +/-25% jitter (crypto/rand), worst-case ~114s; Retry-After honored verbatim, cap raised 60s -> 120s. - Malformed JSON bodies and zero-choice 200 responses (transient gateway artifacts) now consume retry attempts instead of aborting immediately; validated inside the retry loop via validateCompletionBody with error strings unchanged. - retrySleep package var makes backoff injectable for fast tests. B. Iteration ceiling + stall guard (internal/loop): - max_iterations exhaustion no longer returns a bare error: the engine makes one final tool-less LLM call (30s bound, compaction-style side-call) for a partial-progress summary and returns it as the final answer marked '[Iteration budget reached - partial summary]'. Summarizer failure (or a non-compliant tool-call response) falls back to the original error unchanged. - Stall detection: 3 consecutive identical SUCCESSFUL tool calls (name+args fingerprint) inject a corrective hint and emit a tool_recovery signal, closing the polling-loop blind spot where only failing tools were detected. Hint, not enforcement - never aborts. C. Compaction ON by default + serve digest persistence: - ResolvedConfig seeds compaction=true; explicit false from ~/.odek/config.json, ./odek.json, ODEK_COMPACTION=false, or the new --no-compaction flag still disables (plumbing was already *bool end-to-end). odek.Config/loop.Engine stay explicit. - Config templates: global pins "compaction": true; local omits the key so projects inherit (an explicit false would actively disable). - odek serve's persist filter now preserves compaction digest system messages ([Compacted earlier context: prefix) instead of stripping them - resumed web sessions keep their compacted history. Other injected system messages (skills/memory/episodes) still dropped. - AGENTS.md: corrected stale '300 iterations' claim (actual: 90), documented the partial-summary exit, stall detection, compaction default, and serve digest preservation. docs/CLI.md + docs/CONFIG.md updated for --no-compaction and the new default. Tests: llm retry tests for 529/500/parse/zero-choice + jitter bounds; loop tests for partial-summary success/fallback/tool-call-rejection/ persistence and stall streak + reset; config tests for default-on and explicit-false-wins; serve filter tests for digest preservation. Verified with clean env: go build ./... + go vet ./..., go test -race ./internal/llm ./internal/loop ./internal/config, root package, full ./cmd/odek suite (multiple consecutive -count=1 passes). Note: go test -count=2 ./cmd/odek fails on path-confinement tests identically on clean main (pre-existing cross-test interference, not introduced here).
jkyberneees
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Long-horizon readiness review for 3h+ unattended sessions (triggered by an agent appearing "stuck" running
go test -racevia the shell tool). Per-turn session persistence for CLI/REPL/serve landed separately in #122; this PR closes the remaining gaps.Batch 1 — observability & persistence
shellcommand showed a static spinner inside a 30-min buffered window.docker infowith no timeout could hang tests/sandbox setup)./stoplost the in-progress turn).Batch 2 — 3h+ session survival
5. LLM retry budget sized for blips: 4 attempts over ~7s; 529 (Anthropic overloaded)/500/408 not retried; malformed/zero-choice completions aborted immediately.
6.
max_iterations(default 90) exhaustion returned a bare error, discarding all progress; no detection of repeated identical successful tool calls (polling loops).7. Compaction off by default → long sessions silently forgot earlier work;
odek servestripped compaction digests from persisted snapshots, so resumed web sessions lost compacted history.Changes
Tool-running heartbeat (
internal/loop,internal/render,odek.go,cmd/odek/telegram.go) — each tool call gets a watchdog emitting atool_runningsignal every 60s until return (leak-free on error/panic/cancel;-raceverified). CLI + Telegram surface it; serve forwards generically.shelltimeout_seconds— optional per-command timeout clamped to [1, 1800]s, mirroringparallel_shell; docs advise it for builds/test suites.Bounded Docker probes —
dockerAvailable()(both copies) and thedocker image inspectprobe use 5sCommandContext.Telegram per-turn persistence —
SessionManager.SaveNoIndex(no per-step embedding call, noTurnCountinflation) wired to the #122 persist callback withdropDanglingToolCalls+ skip-on-trim.LLM retry resilience (
internal/llm) — retryable statuses + 408/500/529; 8 attempts with 1s→30s exponential backoff, ±25% jitter (crypto/rand), worst case ~114s;Retry-Afterhonored verbatim (cap 120s); malformed JSON / zero-choice 200s retried through the same budget viavalidateCompletionBody.Graceful iteration-budget exit (
internal/loop) — onmax_iterationsexhaustion the engine makes one final tool-less LLM call (30s bound) and returns a partial-progress summary marked[Iteration budget reached — partial summary]; summarizer failure falls back to the original error.Stall detection (
internal/loop) — 3 consecutive identical successful tool calls (name+args) inject a corrective hint +tool_recoverysignal. Hint, not enforcement.Compaction default-on —
ResolvedConfigseedscompaction=true; explicitfalsefrom any config layer,ODEK_COMPACTION=false, or new--no-compactionstill disables. Global config template pins"compaction": true; local template omits it (inherit). Engine/library semantics stay explicit.serve digest preservation — the persist filter keeps
[Compacted earlier context:digest system messages while still dropping skills/memory/episode injections; resumed web sessions keep compacted history.Docs — AGENTS.md (stale "300 iterations" → 90, partial-summary exit, stall detection, compaction default, serve digests, testing guidance),
docs/CLI.md,docs/CONFIG.md.Tests
Heartbeat fires-during/stops-after; shell timeout/clamp/non-positive;
SaveNoIndexno-index/no-TurnCount-inflation; llm 529/500/parse/zero-choice retries + jitter bounds; loop partial-summary success/fallback/tool-call-rejection/persist + stall streak/reset; config default-on/explicit-false-wins; serve digest filter. Verified with clean env:go build ./...,go vet ./...,-raceoninternal/llm/internal/loop/internal/config/internal/telegram, root package, full./cmd/odeksuite (multiple consecutive passes), golangci-lint 0 issues.Known pre-existing issue (also on main, not introduced here):
go test -count=2 ./cmd/odekfails on path-confinement tests due to cross-test cwd interference.