Surface eval outcomes in `experiments analyze` and align experiment workflows/docs with eval-backed metrics by Copilot · Pull Request #48741 · github/gh-aw

Copilot · 2026-07-28T21:37:51Z

gh aw experiments analyze already resolved eval-backed metric references (eval:<id> / evals.<id>) to question text, but it did not show observed eval outcomes for those metrics. This PR adds eval-result visibility to analysis output and aligns experiment guidance/workflows so experiment metrics can be evaluated consistently.

CLI analysis: eval-backed metric outcomes
- Extends experiment analysis with metric_eval_results for eval-referenced metrics.
- Loads and summarizes evals.jsonl from durable eval branches (local/remote) by eval ID.
- Surfaces YES/NO/UNKNOWN totals plus latest observed answer/run in human-readable and JSON analysis output.
Analysis pipeline wiring
- Threads eval-result summaries into computeExperimentAnalyses / computeExperimentAnalysis.
- Keeps behavior best-effort and non-fatal when eval data is absent.
Workflow coverage updates
- Adds evals: sections to experiment workflows that had experiments: but no eval definitions.
- Regenerates corresponding .lock.yml files to keep compiled workflows in sync.
Guidance/docs updates
- Updates experiment instructions/docs to explicitly recommend eval-backed metrics and note that experiments analyze surfaces resolved eval question + observed eval outcomes.

type MetricEvalResults struct {
    Yes          int    `json:"yes"`
    No           int    `json:"no"`
    Unknown      int    `json:"unknown"`
    Total        int    `json:"total"`
    LatestAnswer string `json:"latest_answer,omitempty"`
    LatestRunID  string `json:"latest_run_id,omitempty"`
}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-07-28T21:52:20Z

@copilot resolve the merge conflicts on this branch.

Copilot

Pull request overview

Adds eval outcome visibility to experiment analysis and aligns workflow guidance with eval-backed metrics.

Changes:

Loads and summarizes durable eval results in CLI analysis.
Adds tests and documentation for eval-backed metrics.
Adds eval definitions and regenerates associated workflows.

Show a summary per file

File	Description
`pkg/cli/experiments_command.go`	Loads and summarizes eval results.
`pkg/cli/experiments_command_test.go`	Tests eval summarization.
`pkg/cli/experiments_analyze_statistics.go`	Exposes eval outcomes in analysis.
`pkg/cli/experiments_analyze_statistics_test.go`	Tests analysis wiring.
`docs/src/content/docs/experimental/experiments.md`	Documents eval outcome display.
`.github/aw/experiments.md`	Updates experiment guidance.
`.github/workflows/typist.md`	Adds an experiment eval.
`.github/workflows/test-quality-sentinel.md`	Adds an experiment eval.
`.github/workflows/smoke-temporary-id.md`	Adds an experiment eval.
`.github/workflows/smoke-temporary-id.lock.yml`	Compiles eval jobs and persistence.
`.github/workflows/smoke-project.md`	Adds an experiment eval.
`.github/workflows/smoke-project.lock.yml`	Compiles eval jobs and persistence.
`.github/workflows/smoke-gemini.md`	Adds an experiment eval.
`.github/workflows/smoke-gemini.lock.yml`	Compiles eval jobs and persistence.
`.github/workflows/smoke-copilot-sub-agents.md`	Adds an experiment eval.
`.github/workflows/smoke-copilot-sub-agents.lock.yml`	Compiles eval jobs and persistence.
`.github/workflows/smoke-copilot-aoai-entra.md`	Adds experiment evals.
`.github/workflows/smoke-copilot-aoai-apikey.md`	Adds experiment evals.
`.github/workflows/smoke-antigravity.md`	Adds an experiment eval.
`.github/workflows/smoke-antigravity.lock.yml`	Compiles eval jobs and persistence.
`.github/workflows/gpclean.md`	Adds an experiment eval.
`.github/workflows/gpclean.lock.yml`	Compiles eval jobs and persistence.
`.github/workflows/deep-report.md`	Adds an experiment eval.
`.github/workflows/dataflow-pr-discussion-dataset.md`	Adds an experiment eval.
`.github/workflows/dataflow-pr-discussion-dataset.lock.yml`	Compiles eval jobs and persistence.
`.github/workflows/daily-agentrx-trace-optimizer.md`	Adds an experiment eval.
`.github/workflows/daily-agentrx-trace-optimizer.lock.yml`	Compiles eval jobs and persistence.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

pkg/cli/experiments_command.go:345

The remote path also fetches only the current evals.jsonl snapshot. Since the durable-state job replaces that file each run, remote analysis reports only the latest run rather than cumulative outcomes. Load and summarize the branch's historical file versions, or change persistence to append/deduplicate records before committing.

Files reviewed: 32/32 changed files
Comments generated: 14
Review effort level: Medium

+		}
+		ref = branchName
+	}
+	cmd := exec.Command("git", "show", ref+":"+constants.EvalsResultFilename)


+  - id: tone_style_goal_met
+    question: Does the agent output show that the objective for experiment tone_style was successfully completed?


+  - id: model_size_goal_met
+    question: Does the agent output show that the objective for experiment model_size was successfully completed?


+  - id: sub_agent_strategy_goal_met
+    question: Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed?


+  - id: prompt_style_test_goal_met
+    question: Does the agent output show that the objective for experiment prompt_style_test was successfully completed?


+  - id: sub_agent_strategy_goal_met
+    question: Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed?


+  - id: tool_verbosity_goal_met
+    question: Does the agent output show that the objective for experiment tool_verbosity was successfully completed?


+  - id: output_format_goal_met
+    question: Does the agent output show that the objective for experiment output_format was successfully completed?


+  - id: caveman_mode_goal_met
+    question: Does the agent output show that the objective for experiment caveman_mode was successfully completed?


+  - id: sub_agent_strategy_goal_met
+    question: Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed?


Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-28T22:12:38Z

@copilot resolve the merge conflicts on this branch.

Resolved in 94de0d4.

pelikhan · 2026-07-29T05:46:08Z

@copilot merge main and recompile

github-actions · 2026-07-29T05:51:59Z

Great work on adding eval-result visibility to experiment analysis! 🎯 This feature enhances the experiments analyze command to surface YES/NO/UNKNOWN totals and latest observed answers from eval-backed metrics, making experiment guidance and workflows consistent with the eval infrastructure.

The implementation is solid:

✅ New MetricEvalResults struct cleanly encapsulates eval outcome summaries (yes/no/unknown counts + latest answer/run ID)
✅ Analysis pipeline properly threads eval-result data from local/remote eval branches into computeExperimentAnalyses
✅ Workflow and documentation updates align experiment definitions with eval backing
✅ Comprehensive test coverage confirms eval result injection and display formatting
✅ Large diff driven by .lock.yml regeneration (expected for workflow maintenance)

This PR looks ready for review and merge. The feature is well-scoped, thoroughly tested, and the large diff is primarily generated workflow artifacts, not custom logic. Ship it! 🚀

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

patchdiff.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "patchdiff.githubusercontent.com"

See Network Configuration for more information.

Generated by ✅ Contribution Check · sonnet50 · 66.4 AIC · ⌖ 11.6 AIC · ⊞ 8.3K · ◷

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-29T06:02:06Z

@copilot merge main and recompile

Merged main and recompiled in d740968.

pelikhan · 2026-07-29T06:23:09Z

@copilot run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-29T06:42:32Z

@copilot run pr-finisher skill

Ran pr-finisher pass and pushed 917bfe2.

github-actions · 2026-07-29T10:47:13Z

🎉 This pull request is included in a new release.

Release: v0.83.5

Add eval metric summaries to experiments analyze

17e78d1

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Jul 28, 2026

Copilot AI changed the title ~~Update experiments analyze to surface eval metric outcomes~~ Surface eval outcomes in experiments analyze and align experiment workflows/docs with eval-backed metrics Jul 28, 2026

Copilot created this pull request from a session on behalf of pelikhan July 28, 2026 21:38 View session

Copilot finished work on behalf of pelikhan July 28, 2026 21:38

Copilot AI requested a review from pelikhan July 28, 2026 21:38

pelikhan marked this pull request as ready for review July 28, 2026 21:52

Copilot AI review requested due to automatic review settings July 28, 2026 21:52

Copilot started reviewing on behalf of pelikhan July 28, 2026 21:52 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Copilot started work on behalf of pelikhan July 28, 2026 21:56 View session

Merge origin/main and resolve gpclean workflow conflicts

94de0d4

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 28, 2026 22:13

github-actions Bot added pr-action:batch_review pr-agent:github-copilot pr-batch:workflow-maintenance pr-priority:medium pr-risk:medium pr-type:feature labels Jul 29, 2026

github-actions Bot mentioned this pull request Jul 29, 2026

[Contribution Check Report] Contribution Check — 2026-07-29 #48756

Open

This comment has been minimized.

Sign in to view

Copilot started work on behalf of pelikhan July 29, 2026 05:46 View session

Merge origin/main and recompile workflows

d740968

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 29, 2026 06:02

Copilot started work on behalf of pelikhan July 29, 2026 06:23 View session

Guard linter shared indexes against nil analysis package in tests

917bfe2

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 29, 2026 06:43

Merge branch 'main' into copilot/update-experiments-command

2856544

pelikhan merged commit 3b1d700 into main Jul 29, 2026
29 of 30 checks passed

pelikhan deleted the copilot/update-experiments-command branch July 29, 2026 07:38

github-actions Bot mentioned this pull request Jul 29, 2026

Smoke Test: MAI-Code-1-Flash - 30432843600 #48808

Closed

github-actions Bot mentioned this pull request Jul 29, 2026

Smoke Test: MAI-Code-1-Flash - 30448364948 #48855

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Surface eval outcomes in `experiments analyze` and align experiment workflows/docs with eval-backed metrics - #48741

Surface eval outcomes in `experiments analyze` and align experiment workflows/docs with eval-backed metrics#48741
pelikhan merged 5 commits into
mainfrom
copilot/update-experiments-command

Copilot AI commented Jul 28, 2026 •

edited

Loading

Uh oh!

pelikhan commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

This comment has been minimized.

pelikhan commented Jul 29, 2026

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

Copilot AI commented Jul 29, 2026

Uh oh!

pelikhan commented Jul 29, 2026

Uh oh!

Copilot AI commented Jul 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		- id: tone_style_goal_met
		question: Does the agent output show that the objective for experiment tone_style was successfully completed?

		- id: model_size_goal_met
		question: Does the agent output show that the objective for experiment model_size was successfully completed?

		- id: sub_agent_strategy_goal_met
		question: Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed?

		- id: prompt_style_test_goal_met
		question: Does the agent output show that the objective for experiment prompt_style_test was successfully completed?

		- id: tool_verbosity_goal_met
		question: Does the agent output show that the objective for experiment tool_verbosity was successfully completed?

		- id: output_format_goal_met
		question: Does the agent output show that the objective for experiment output_format was successfully completed?

		- id: caveman_mode_goal_met
		question: Does the agent output show that the objective for experiment caveman_mode was successfully completed?

Uh oh!

Conversation

Copilot AI commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

This comment has been minimized.

pelikhan commented Jul 29, 2026

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

Copilot AI commented Jul 29, 2026

Uh oh!

pelikhan commented Jul 29, 2026

Uh oh!

Copilot AI commented Jul 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 28, 2026 •

edited

Loading