Problem
The two runners treat WARNING checks differently:
- Client runner (
src/runner/client.ts): overallFailure = failed > 0 || warnings > 0 || ... — any WARNING fails the scenario run.
- Server runner (
src/runner/server.ts): the pass/fail denominator counts only SUCCESS and FAILURE; warnings are reported but never affect the result.
Since tier-check requires pass_rate = 1.0 for Tier 1 on both conformance and client_conformance (src/tier-check/tier-logic.ts), the same severity mapping (SHOULD → WARNING) is a Tier 1 blocker on the client side but cosmetic on the server side. Whether an SDK's tier is affected by a SHOULD-violation currently depends on which side of the wire the scenario tests, which is hard to defend in a tiering dispute.
Related: src/expected-failures.ts counts WARNING as failure in baseline mode, which is a third distinct behavior.
Ask
Pick one semantic and apply it in both runners (and baseline mode). Options:
- Warnings never fail runs or tiers anywhere (server behavior today); SHOULD compliance becomes report-only.
- Warnings fail everywhere (client behavior today); Tier 1 then requires SHOULD compliance on both sides.
- Keep warnings non-failing for the run but surface a separate SHOULD-compliance rate in tier-check so tiering can reference it explicitly.
Context: noticed while landing #322, whose DELETE-status check is a WARNING because the spec pins no success status (#429). No current Tier 1 SDK emits warnings on either side, so any of the above can land without demoting anyone today.
Problem
The two runners treat WARNING checks differently:
src/runner/client.ts):overallFailure = failed > 0 || warnings > 0 || ...— any WARNING fails the scenario run.src/runner/server.ts): the pass/fail denominator counts only SUCCESS and FAILURE; warnings are reported but never affect the result.Since tier-check requires
pass_rate = 1.0for Tier 1 on bothconformanceandclient_conformance(src/tier-check/tier-logic.ts), the same severity mapping (SHOULD → WARNING) is a Tier 1 blocker on the client side but cosmetic on the server side. Whether an SDK's tier is affected by a SHOULD-violation currently depends on which side of the wire the scenario tests, which is hard to defend in a tiering dispute.Related:
src/expected-failures.tscounts WARNING as failure in baseline mode, which is a third distinct behavior.Ask
Pick one semantic and apply it in both runners (and baseline mode). Options:
Context: noticed while landing #322, whose DELETE-status check is a WARNING because the spec pins no success status (#429). No current Tier 1 SDK emits warnings on either side, so any of the above can land without demoting anyone today.