fix(triage-security): improve remediation task generation logic in Step 8#261
fix(triage-security): improve remediation task generation logic in Step 8#261ruromero wants to merge 3 commits into
Conversation
…ep 8 Reorder Case B before Case A to match flowchart evaluation order, inline the ecosystem-dependent two-task rule at the Case A action point, add a 5-item pre-creation checklist at the confirmation gate, and simplify Rule 8 to cross-reference instead of restating. Root cause: CVE-2026-13149 triage (TC-5051) missed downstream propagation subtasks and preemptive cross-stream tasks because Case A committed to single tasks before Case B could add preemptive ones, and the two-task ecosystem rule was buried in the guardrails. Resolves: TC-5378 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reviewer's GuideStep 8 triage-security documentation is updated to reorder Case B before Case A, inline ecosystem-specific task-count rules into Case A, add a pre-creation checklist before Jira mutations, and simplify guardrail Rule 8 to reference the new Case A logic and checklist. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The ecosystem-specific rules are now described in both Case A and the pre-creation checklist; consider consolidating the canonical description in one place (e.g., Case A) and keeping the checklist purely as a short verification list to reduce the risk of these drifting out of sync when adding new ecosystems.
- Case A and the guardrail Rule 8 both constrain task count by ecosystem using hard-coded examples (Cargo, npm, RPM); if additional ecosystems are expected, you may want to explicitly call out how to handle ‘other’ ecosystems or centralize the mapping so authors don’t have to update multiple narrative sections when the set changes.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The ecosystem-specific rules are now described in both Case A and the pre-creation checklist; consider consolidating the canonical description in one place (e.g., Case A) and keeping the checklist purely as a short verification list to reduce the risk of these drifting out of sync when adding new ecosystems.
- Case A and the guardrail Rule 8 both constrain task count by ecosystem using hard-coded examples (Cargo, npm, RPM); if additional ecosystems are expected, you may want to explicitly call out how to handle ‘other’ ecosystems or centralize the mapping so authors don’t have to update multiple narrative sections when the set changes.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Eval Results
Eval Results: triage-security
| Eval | Passed | Failed | Pass Rate |
|---|---|---|---|
| eval-1 | 11/11 | 0 | 100% |
| eval-2 | 5/5 | 0 | 100% |
| eval-3 | 5/5 | 0 | 100% |
| eval-4 | 5/5 | 0 | 100% |
| eval-5 | 6/6 | 0 | 100% |
| eval-6 | 6/6 | 0 | 100% |
| eval-7 | 5/5 | 0 | 100% |
| eval-8 | 8/8 | 0 | 100% |
| eval-9 | 5/5 | 0 | 100% |
| eval-10 | 3/5 | 2 | 60% |
| eval-11 | 5/5 | 0 | 100% |
| eval-12 | 5/5 | 0 | 100% |
| eval-13 | 5/5 | 0 | 100% |
| eval-14 | 4/5 | 1 | 80% |
| eval-15 | 5/5 | 0 | 100% |
| eval-16 | 7/7 | 0 | 100% |
| eval-17 | 3/5 | 2 | 60% |
| eval-18 | 5/5 | 0 | 100% |
| eval-19 | 4/5 | 1 | 80% |
| eval-20 | 1/4 | 3 | 25% |
| eval-21 | 4/4 | 0 | 100% |
| eval-22 | 4/4 | 0 | 100% |
| eval-23 | 4/4 | 0 | 100% |
| eval-24 | 4/4 | 0 | 100% |
| eval-25 | 4/4 | 0 | 100% |
| eval-26 | 5/5 | 0 | 100% |
| eval-27 | 5/5 | 0 | 100% |
| eval-28 | 5/5 | 0 | 100% |
| eval-29 | 5/5 | 0 | 100% |
| eval-30 | 4/4 | 0 | 100% |
| eval-31 | 4/4 | 0 | 100% |
| eval-32 | 4/4 | 0 | 100% |
Failed Assertions
eval-10: 2 failing assertions
-
Assertion: "Step 8 Case A creates standard remediation tasks for the current stream (rhtpa-2.2) with standard labels ['ai-generated-jira', 'Security', 'CVE-2026-55123'] and 'Depend' link to TC-8020"
Evidence: "The output swaps Case A and Case B labels. The assertion expects Case A to be standard remediation for the current stream (rhtpa-2.2), but the output labels that as Case B and assigns Case A to the preemptive cross-stream tasks for rhtpa-2.1." -
Assertion: "Step 8 Case B detects that stream rhtpa-2.1 is also affected but has no CVE Jira — creates proactive preemptive remediation tasks (§1.60)"
Evidence: "Same label swap. The assertion expects Case B to be preemptive remediation for rhtpa-2.1, but the output labels that as Case A and assigns Case B to standard remediation for rhtpa-2.2."
eval-14: 1 failing assertion
- Assertion: "The rpms.lock.yaml classification remains the primary signal — the SBOM result supplements but does not override it (Step 2.3.5 non-MVP enhancement)"
Evidence: "Contradictory evidence. The introductory text says rpms.lock.yaml determines the classification with SBOM as a cross-check, but the actual behavior shows that when they disagree, the origin becomes CONFLICTING — requires manual investigation rather than defaulting to the rpms.lock.yaml answer of explicit install. This means the SBOM result effectively overrides the rpms.lock.yaml classification rather than merely supplementing it."
eval-17: 2 failing assertions
-
Assertion: "Step 0 extracts the Embargo policy URL from the Security Configuration as an optional field without raising an error (§1.71 — backward compatible extraction)"
Evidence: "No Step 0 output file exists. The embargo policy URL extraction appears in Step 1.7 Configuration Check section, not in a dedicated Step 0. No evidence of optional-field or backward-compatible extraction handling." -
Assertion: "The embargo warning gate does NOT trigger for Low or Moderate severity CVEs (CVSS < 7.0) — it is skipped silently when severity is below threshold (§1.70)"
Evidence: "The outputs only cover the CVSS 7.5 (above-threshold) case. No output demonstrates or explicitly states that the gate is skipped silently for CVSS < 7.0 scenarios."
eval-19: 1 failing assertion
- Assertion: "Step 0.3 runs before Step 0.5 (JIRA Access Initialization) — no Jira operations are attempted before the staleness check"
Evidence: "Only one output file exists (staleness-check.md). There is no evidence of Step 0.5 execution or workflow ordering. The absence of Jira operations in this single file does not prove Step 0.3 ran before Step 0.5 — no ordering evidence exists."
eval-20: 3 failing assertions
-
Assertion: "Step 0.3 determines the matrix is within the 14-day threshold and proceeds without displaying a staleness warning"
Evidence: "The matrix is 27 days old, exceeding the 14-day threshold. Both streams are marked STALE with warnings displayed — the opposite of what the assertion claims." -
Assertion: "No user prompt or options are presented for a fresh matrix — the check is silent on success"
Evidence: "The matrix is not fresh, so the silent on success behavior cannot be confirmed. Instead, user prompts with three options are presented for each stale stream." -
Assertion: "The triage continues to Step 0.5 and beyond without interruption from the staleness check"
Evidence: "The staleness check explicitly interrupted triage, stating Action required and demanding the engineer choose an option before continuing."
Pass rate: 94% · Tokens: 58,588 · Duration: 128s
Baseline (1dab64e0): 99% · 58,181 tokens · 124s
Generated by sdlc-workflow/run-evals v0.13.6
The cases were ordered B, A, C in the document but retained their original letter names, making the flow confusing. Rename so the document order matches alphabetical naming: cross-stream impact is now Case A (listed first), remediation tasks is now Case B (listed second), Case C unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
[sdlc-workflow/verify-pr] Re: @sourcery-ai[bot] review —
|
Verification Report for TC-5378 (commit 386ed9d)
Overall: PASSNo issues requiring attention. All acceptance criteria met. Eval suite confirms no regressions from the case rename. This comment was AI-generated by sdlc-workflow/verify-pr v0.13.3. |
Add a canonical ecosystem classification table as the single source of truth for ecosystem categories and task counts. Update all downstream references (Case B, checklist, remediation templates, guardrail Rule 8, and mermaid flowcharts) to point to the table instead of re-listing ecosystem names inline, reducing duplication from 6 sites to 1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 participant
Summary
Test plan
/triage-securityon a test Vulnerability issue with a source dependency scoped to one stream and verify the skill proposes the correct task count including downstream subtasksResolves: TC-5378
🤖 Generated with Claude Code
Summary by Sourcery
Refine the triage-security Step 8 workflow to align remediation task generation, cross-stream handling, and guardrails with the documented flow and ecosystem-specific rules.
New Features:
Enhancements:
Documentation: