fix(cli): actionable console nudge + positional reply text (inbox eval findings)#252
Merged
Conversation
… text (inbox eval findings) The inbox eval (PR #250) showed both Claude and Gemini ignore the count-only 📬 nudge when nothing in their task mentions the console: they read the line, finish the literal task, and the human's message is never applied (0/3 runs). It also showed both agents' first reply attempt is positional (`reply <scope> "text"`), which failed and cost a --help round-trip (3/3 runs). - inbox-nudge: when the unread header fires, fetch the unread events (?all=1 — peek, URL-gated) and print the message content inline plus an explicit act-before-finishing instruction. Feedback inside tool output the agent is already reading is the one channel that reliably lands (same pattern that fixed --type selection). Content fetch is best-effort; the imperative nudge prints regardless. Ack semantics unchanged (only a real `termchart inbox` read advances the receipt). - reply: bare words are now the message (joined); --message still wins; unknown flags still rejected; the missing-message error names both forms. - termchart front-door skill: document the 📬 contract (apply + mark read + reply before finishing); plugin 0.16.3. Verified end-to-end: eval TC-INBOX-UNPROMPTED flips FAIL(3/3)→PASS with the agent visibly acting on the inline nudge, and TC-INBOX-PROMPTED passes with zero reply --help detours (was 100%). Unit: 10/10 on the touched files.
Owner
Author
Gemini leg — fix confirmed agent-agnosticSame eval (
Both agents that exhibited the failure now pass both cases. 4/4 eval cases green across Claude + Gemini. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Product fixes for the two gaps the bidirectional-inbox eval (#250) surfaced. Both were agent-agnostic (Claude and Gemini):
📬nudge was ignored 3/3 runs when unprompted. Agents saw📬 1 console message …in push output and finished the literal task anyway — the human's console message was never applied. The count-only hint reads as informational.termchart replypositional-arg trap, 3/3 runs. Both agents' first attempt wasreply <scope> "text"→ error →--helpround-trip →--message.Fixes
inbox-nudge.ts— when thex-termchart-inbox-unreadheader fires, the CLI now fetches the unread events (GET /inbox?all=1, a peek: ack semantics unchanged) and prints the message content inline plus an explicit instruction:--typeselection in Close the push→display validation gap: reject anything the viewer can't render #219/Guide agents to the right --type: did-you-mean hint + rich types in --help #229). Content fetch is best-effort — the imperative nudge prints even if it fails. Still stderr-only (never pollutes piped stdout).reply.ts— bare words are now the message (termchart reply --project p --agent a did the thing);--messagestill wins; unknown flags still rejected; the missing-message error names both forms.plugin/skills/termchart/SKILL.md— the front-door skill now documents the 📬 contract (it never mentioned the console). Plugin 0.16.3.Verification (eval-gated)
Re-ran the #250 eval against this CLI (
SEED_INBOX=1 CASE_SET=termchart-inbox CONCURRENCY=1):termchart inbox, board carries the console-only content (deterministicgrepgate)--helpdetour every run)--helpdetoursGemini leg running; will post results as a comment.
Also live-verified against a local viewer (content-nudge output above is a real capture) + probes: no-message → exit 3 naming both forms; unknown flag still rejected; nudge disappears after a real
termchart inboxread (ack unchanged). Unit tests: 10/10 on touched files (newinbox-nudge.test.tscovers silent-when-zero, content-inline, action-ref rendering, 3-event cap, fetch-failure fallback).Note: full CLI suite is green in CI-conditions; local full-suite runs flaked on 5s network-stub timeouts under load-average ~180 (unrelated background jobs) — the failing set varies run to run and all touched files pass in isolation.