agent-compat: bidirectional inbox eval (console, issue #108)

[ivanmkc]

What

Adds an agent-generator eval for the bidirectional console (issue #108) to the agent-compat harness, exercising the human↔agent inbox from both sides, plus a deep-dive on how Claude actually behaves.

Two interactive_simulation cases (case-set termchart-inbox):

• TC-INBOX-PROMPTED — the scenario tells the agent to read the console and reply. Baseline: can it drive human→agent (read) + agent→human (reply) when told?
• TC-INBOX-UNPROMPTED — the scenario says nothing about the console. The only signal is the 📬 nudge the CLI prints on push. Tests the core UX claim: does the agent notice unprompted human input on its own?

The console note is pre-seeded out-of-band (seed_inbox.py → POST /inbox, gated by SEED_INBOX=1) because it doesn't arrive through the agent-generator conversation. It asks for two things a default login screen never has (a maintenance banner + a Beam me in button label) so a good-but-generic login form can't pass by coincidence.

Each case gates on a deterministic pull+grep for the console-only string first, with the tc-judge rubric secondary. judge_board.py gains --rubric-name so both cases share one rubric.

Findings (Claude Code, claude -p)

	Prompted	Unprompted
Reads termchart inbox	✅ (first action)	❌ never (3/3 runs)
Applies console content	✅ banner + relabeled button	❌ board stays a plain login form
Posts termchart reply	✅	n/a
Verdict	PASS (all objectives)	FAIL (deterministic + rubric)

1. Claude ignores the unprompted 📬 nudge (headline). In every unprompted run it ran only push + list, saw 📬 1 console message … in stderr, and finished — never reading the inbox. The "notice unprompted human input" claim doesn't hold once the agent has satisfied the literal task; the passive stderr nudge reads as informational.
2. termchart reply positional-arg friction. Claude's first reply attempt was always termchart reply <scope> "text" (positional), which fails; it recovered via --help → --message. Reply should accept a positional message, or the error should name the required --message flag.
3. Harness false-positives fixed. (a) a "natural" console ask (forgot-password) couldn't distinguish reading-the-console from drawing-a-good-form; (b) tc-judge (LLM) + CONCURRENCY>1 on a shared viewer let a scoped pull cross-read another case's board, passing a real failure. Fixed by login-alien asks, a deterministic pull+grep gate, and running at CONCURRENCY=1.

Run it

SEED_INBOX=1 CASE_SET=termchart-inbox CONCURRENCY=1 bash scripts/experiments/agent-compat/run.sh tc-claude

Draft — this is an eval/harness addition (no product code changes). The product takeaways (findings 1 & 2) are candidates for follow-up.

[ivanmkc]

Cross-agent reproduction + nudge confirmation

Gemini CLI (tc-gemini, same case-set, CONCURRENCY=1) reproduces Claude's pattern exactly:

	Prompted	Unprompted
Claude	✅ PASS (reads inbox, applies banner+button, replies)	❌ FAIL (never applies console content)
Gemini	✅ PASS (reads inbox, applies banner+button, replies)	❌ FAIL (never applies console content)

So finding #1 (unprompted 📬 nudge ignored) is agent-agnostic — a product/UX gap, not a model quirk. Both agents fully drive the console when told to; neither acts on it unprompted.

The nudge does fire — verified directly against a local viewer (seed an unread console message, push as a fresh agent):

$ termchart push --project tc-inbox --agent unprompted --type component ... 2>stderr
$ cat stderr
📬 1 console message from the human — run: termchart inbox --project tc-inbox --agent unprompted

The failure is reception, not emission: the agent sees the 📬 line in its push output and proceeds to termchart list + wraps up, having already satisfied the literal task. The passive stderr nudge doesn't compel a follow-up termchart inbox.

Possible product follow-ups (out of scope for this eval PR): make the nudge imperative in the skill contract (e.g. inbox-watch skill instructs "if you see 📬, you MUST run termchart inbox before finishing"), or fold an inbox check into the end-of-task flow, or return the unread message inline in the push result rather than a separate stderr line.

[ivanmkc]

Coding-standards review pass (dev-coding-standards)

Audited the PR's Python (seed_inbox.py, judge_board.py) against the ADK coding standards; fixed in 1882166:

• Broad except Exception (rule 2) — seed_inbox.py per-event POST now catches (URLError, TimeoutError, ValueError); judge_board.py's retry loop now catches (APIError, httpx.HTTPError, JSONDecodeError, TimeoutError, ConnectionError). The one remaining broad catch is main()'s crash guard, which is the standard's entry-point exemption (logs + distinct exit 3 so infra failures never masquerade as rubric FAILs).
• Error-message substring matching (rule 15) — transient classification now uses the structured APIError.code (429/5xx) instead of sniffing "rate"/"quota"/"429" out of the message string.
• Function-scoped imports (rule 16) — the in-function genai import now carries its justification (optional-dep gating: the module must import cleanly on host python without the agent-generator uv env).
• Drive-by: the seed summary line counted doc keys as scopes ("3 scope(s)" for 2 real) — now counts real scopes only.

Verified at the surface, not just compiled: --help on host python (no genai) exits 0; a real judge call returns correct verdicts; a forced 404 (non-transient) fails fast in 2.1s with zero backoff retries; an unreachable viewer produces per-event FAIL lines + posted 0 event(s) across 2 scope(s).

Conscious deviations from the standards (documented, not oversights):

• No Pydantic models (rule 1): seed_inbox.py runs on host python3 (run.sh calls it outside any uv env), so stdlib-only is a hard constraint; judge_board.py handles ~2 tiny ad-hoc shapes in a 120-line harness script — a model layer would outweigh the script.
• --rubric fallback resolution (rule 9-adjacent): the rubric arg resolves via --rubric → --rubric-name → agent-name default. Intentional — it keeps generated case commands path-free/portable, and the tc-judge wrapper always pins --rubric-dir explicitly.