Skip to content

WARNING severity is inconsistent between client and server runners (client warnings can block Tier 1, server warnings cannot) #430

Description

@pcarleton

Problem

The two runners treat WARNING checks differently:

  • Client runner (src/runner/client.ts): overallFailure = failed > 0 || warnings > 0 || ... — any WARNING fails the scenario run.
  • Server runner (src/runner/server.ts): the pass/fail denominator counts only SUCCESS and FAILURE; warnings are reported but never affect the result.

Since tier-check requires pass_rate = 1.0 for Tier 1 on both conformance and client_conformance (src/tier-check/tier-logic.ts), the same severity mapping (SHOULD → WARNING) is a Tier 1 blocker on the client side but cosmetic on the server side. Whether an SDK's tier is affected by a SHOULD-violation currently depends on which side of the wire the scenario tests, which is hard to defend in a tiering dispute.

Related: src/expected-failures.ts counts WARNING as failure in baseline mode, which is a third distinct behavior.

Ask

Pick one semantic and apply it in both runners (and baseline mode). Options:

  1. Warnings never fail runs or tiers anywhere (server behavior today); SHOULD compliance becomes report-only.
  2. Warnings fail everywhere (client behavior today); Tier 1 then requires SHOULD compliance on both sides.
  3. Keep warnings non-failing for the run but surface a separate SHOULD-compliance rate in tier-check so tiering can reference it explicitly.

Context: noticed while landing #322, whose DELETE-status check is a WARNING because the spec pins no success status (#429). No current Tier 1 SDK emits warnings on either side, so any of the above can land without demoting anyone today.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions