Skip to content

fix(cli): actionable console nudge + positional reply text (inbox eval findings)#252

Merged
ivanmkc merged 2 commits into
masterfrom
fix/inbox-nudge-reply
Jul 8, 2026
Merged

fix(cli): actionable console nudge + positional reply text (inbox eval findings)#252
ivanmkc merged 2 commits into
masterfrom
fix/inbox-nudge-reply

Conversation

@ivanmkc

@ivanmkc ivanmkc commented Jul 8, 2026

Copy link
Copy Markdown
Owner

What

Product fixes for the two gaps the bidirectional-inbox eval (#250) surfaced. Both were agent-agnostic (Claude and Gemini):

  1. The 📬 nudge was ignored 3/3 runs when unprompted. Agents saw 📬 1 console message … in push output and finished the literal task anyway — the human's console message was never applied. The count-only hint reads as informational.
  2. termchart reply positional-arg trap, 3/3 runs. Both agents' first attempt was reply <scope> "text" → error → --help round-trip → --message.

Fixes

  • inbox-nudge.ts — when the x-termchart-inbox-unread header fires, the CLI now fetches the unread events (GET /inbox?all=1, a peek: ack semantics unchanged) and prints the message content inline plus an explicit instruction:
    📬 1 unread console message from the human — ACT ON THIS BEFORE FINISHING:
       › Please add a maintenance banner for Sunday 2-4am.
       Apply what it asks in this scope, mark it read with `termchart inbox --project demo --agent a1`,
       then acknowledge: termchart reply --project demo --agent a1 --message "<what you changed>"
    
    In-tool-output feedback is the one channel that reliably lands (same pattern that fixed --type selection in Close the push→display validation gap: reject anything the viewer can't render #219/Guide agents to the right --type: did-you-mean hint + rich types in --help #229). Content fetch is best-effort — the imperative nudge prints even if it fails. Still stderr-only (never pollutes piped stdout).
  • reply.ts — bare words are now the message (termchart reply --project p --agent a did the thing); --message still wins; unknown flags still rejected; the missing-message error names both forms.
  • plugin/skills/termchart/SKILL.md — the front-door skill now documents the 📬 contract (it never mentioned the console). Plugin 0.16.3.

Verification (eval-gated)

Re-ran the #250 eval against this CLI (SEED_INBOX=1 CASE_SET=termchart-inbox CONCURRENCY=1):

Case (Claude) Before After
TC-INBOX-UNPROMPTED ❌ FAIL 3/3 (inbox never read) PASS — agent reacts to the inline nudge, runs termchart inbox, board carries the console-only content (deterministic grep gate)
TC-INBOX-PROMPTED ✅ (with --help detour every run) ✅ PASS — positional reply accepted first-try, zero --help detours

Gemini leg running; will post results as a comment.

Also live-verified against a local viewer (content-nudge output above is a real capture) + probes: no-message → exit 3 naming both forms; unknown flag still rejected; nudge disappears after a real termchart inbox read (ack unchanged). Unit tests: 10/10 on touched files (new inbox-nudge.test.ts covers silent-when-zero, content-inline, action-ref rendering, 3-event cap, fetch-failure fallback).

Note: full CLI suite is green in CI-conditions; local full-suite runs flaked on 5s network-stub timeouts under load-average ~180 (unrelated background jobs) — the failing set varies run to run and all touched files pass in isolation.

… text (inbox eval findings)

The inbox eval (PR #250) showed both Claude and Gemini ignore the count-only
📬 nudge when nothing in their task mentions the console: they read the line,
finish the literal task, and the human's message is never applied (0/3 runs).
It also showed both agents' first reply attempt is positional
(`reply <scope> "text"`), which failed and cost a --help round-trip (3/3 runs).

- inbox-nudge: when the unread header fires, fetch the unread events
  (?all=1 — peek, URL-gated) and print the message content inline plus an
  explicit act-before-finishing instruction. Feedback inside tool output the
  agent is already reading is the one channel that reliably lands (same
  pattern that fixed --type selection). Content fetch is best-effort; the
  imperative nudge prints regardless. Ack semantics unchanged (only a real
  `termchart inbox` read advances the receipt).
- reply: bare words are now the message (joined); --message still wins;
  unknown flags still rejected; the missing-message error names both forms.
- termchart front-door skill: document the 📬 contract (apply + mark read +
  reply before finishing); plugin 0.16.3.

Verified end-to-end: eval TC-INBOX-UNPROMPTED flips FAIL(3/3)→PASS with the
agent visibly acting on the inline nudge, and TC-INBOX-PROMPTED passes with
zero reply --help detours (was 100%). Unit: 10/10 on the touched files.
@ivanmkc

ivanmkc commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

Gemini leg — fix confirmed agent-agnostic

Same eval (CASE_SET=termchart-inbox, CONCURRENCY=1) on tc-gemini against this branch:

Case (Gemini) Before After
TC-INBOX-UNPROMPTED ❌ FAIL (inbox never read, board plain) PASS — saw the inline ACT-nudge, ran termchart inbox, board carries the console-only content (deterministic gate)
TC-INBOX-PROMPTED ✅ (with --help detour) ✅ PASS — zero --help detours

Both agents that exhibited the failure now pass both cases. 4/4 eval cases green across Claude + Gemini.

@ivanmkc ivanmkc marked this pull request as ready for review July 8, 2026 02:15
@ivanmkc ivanmkc merged commit b655ccc into master Jul 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants