Add E2E test pipeline for azure.ai.agents extension (Tier 0/1)

[v1212]

Status: DRAFT - pending secrets setup and --allow-tool syntax confirmation

Summary

Adds a Copilot CLI-driven E2E test pipeline for the azure.ai.agents extension. Uses the official copilot -p programmatic mode (per GitHub docs).

Architecture (identical to local testing)

Copilot CLI (npm install -g @github/copilot)
  | MCP protocol (stdio, config: ~/.copilot/mcp-config.json)
cli-interactive-tester (MCP server)
  | tmux sessions
azd ai agent CLI (under test)

Workflow

1. Checkout (ref: trangevi/test-scenarios)
2. Setup: Node.js, Go, Python 3.12
3. Install Copilot CLI (npm install -g @github/copilot)
4. Build azd + install extension
5. Install cli-interactive-tester (git clone + pip install -e)
6. Azure Login + gh auth (Tier 1/2 only)
7. Create profile.local.yaml from secrets
8. Configure MCP (write ~/.copilot/mcp-config.json)
9. Run: copilot -p "" --allow-tool='cli-interactive-tester(*)' --no-ask-user
10. Upload results as artifacts
11. Always-run teardown for Tier 2

Trigger

• workflow_dispatch only (tier selector: 0 / 0+1 / 0+1+2)
• No PR trigger until pipeline is validated

Items to confirm

1. --allow-tool syntax for MCP-registered tools (is it cli-interactive-tester(*)?)
2. COPILOT_PAT - who creates the Fine-grained PAT with "Copilot Requests" permission?
3. cli-interactive-tester repo - public or private? If private, need authenticated clone
4. prompt-ci-run.md - needs to be created in scenarios directory (CI-adapted prompt)

Secrets needed

• COPILOT_PAT (Fine-grained PAT with "Copilot Requests" permission)
• AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID
• FOUNDRY_PROJECT_ENDPOINT
• GH_TOKEN (for gh auth + manifest-url scenario)

Validation

All 44 scenarios validated PASS in manual Copilot-driven local run.

[Copilot]

Pull request overview

Adds a new GitHub Actions workflow to run end-to-end (E2E) scenario-based checks for the azure.ai.agents azd extension on PRs and via manual dispatch, intended to cover Tier 0 (offline) and Tier 1 (Azure-authenticated, no provision) scenarios, with a commented Tier 2 placeholder.

Changes:

• Introduces .github/workflows/e2e-ext-azure-ai-agents.yml with Tier 0 and Tier 1 jobs that build azd, install the extension, and execute scenario YAMLs.
• Adds workflow_dispatch input scaffolding for a future Tier 2 live-Azure run (currently commented out).

[glharper]

Review findings

This is a draft targeting wujia/e2e-pipeline-staging, and several items are already flagged by the author (--allow-tool syntax, COPILOT_PAT, repo visibility, prompt-ci-run.md). The notes below are beyond those.

🔴 Concrete bug — azure/login will fail without id-token: write

The Azure Login step uses OIDC federated credentials (client-id/tenant-id/subscription-id, no client-secret). OIDC login requires permissions: id-token: write, but the workflow only declares contents: read. As written, azure/login@v2 cannot fetch the OIDC token and Tier 1/2 runs will fail at the login step. Add id-token: write.

🟠 Convention — authenticated workloads should prefer ADO pipelines

Per cli/azd/AGENTS.md: "Prefer Azure DevOps pipelines for jobs that need secrets or Azure credentials — the team uses internal ADO pipelines for authenticated workloads in this public repo." No other workflow in .github/workflows/ currently uses azure/login — this PR would be the first. The Tier 1/2 path (Azure creds + COPILOT_PAT + GH_TOKEN + FOUNDRY_PROJECT_ENDPOINT) is exactly the kind of authenticated workload the team routes through ADO. Worth confirming with maintainers before this leaves staging. (Tier 0 is offline/no-auth and is a good fit for GitHub Actions.)

🟠 Tier 2 teardown can silently leak Azure resources

cd ~/working/azd-agents-shared/*/ 2>/dev/null && azd down --force --purge || true

• If the glob matches multiple dirs, cd gets multiple args and fails.
• If cd fails (path missing, e.g. setup aborted early), && short-circuits and || true keeps the job green — so no teardown runs and resources keep costing money with no signal. Given the explicit "~$2-5" cost note, consider failing loudly (or emitting a ::warning::) when the expected directory isn't found, and quoting/guarding the glob.

🟡 Minor / hardening

• Secret interpolation in run: scripts. "Create test profile" inlines ${{ secrets.AZURE_SUBSCRIPTION_ID }} / FOUNDRY_PROJECT_ENDPOINT directly into the shell. Prefer passing them via env: (as the Copilot step correctly does for the token) to avoid shell-injection / log-exposure edge cases.
• actions/setup-node@v4 has no node-version — pin one for reproducibility.
• Hardcoded ref: trangevi/test-scenarios (already TODO'd) — must not reach main; it ties the workflow to an unmerged branch.
• PATH across steps — re-exporting cli/azd in each step works, but echo "$DIR" >> $GITHUB_PATH once (per AGENTS.md guidance) is cleaner.
• Trailing blank lines at EOF of the workflow file.
• COPILOT_GITHUB_TOKEN env name — verify the Copilot CLI actually reads this (on the author's confirm list).

✅ Good

• Least-privilege permissions block present (just needs id-token).
• workflow_dispatch-only with tier gating and a Tier 2 cost-confirmation input — appropriately cautious.
• concurrency with cancel-in-progress: false avoids clobbering in-flight Azure resource runs.
• Tier-gated steps (if: contains(inputs.tier, ...)) are consistent and correct.
• prompt-ci-run.md is clear; if: always() artifact upload + teardown are sensible.

Bottom line: Right architecture and appropriately gated, but not mergeable beyond staging yet: fix the id-token: write permission (hard blocker for Tier 1/2), resolve the ADO-vs-Actions convention question for the authenticated tiers, and harden the teardown. The Tier-0-only path is close to ready.

[v1212]

Thanks @glharper, great review! Addressed all items in the latest commit (d08b5c9). Here's the breakdown:

🔴 id-token: write — Fixed. Added to permissions block.

🟠 ADO convention — Totally valid point. I'm aware of the convention in AGENTS.md. This PR's goal is to validate whether this pattern (Copilot driving cli-interactive-tester to exercise azd interactively) works end-to-end in GitHub CI. If the team decides authenticated tiers should move to ADO, I'm fully supportive of that. Happy to discuss in our next sync.

🟠 Teardown glob — Fixed. Now uses explicit guard with ::warning:: annotation instead of silent || true.

🟡 Secret interpolation — Fixed. Secrets now passed via env: block, referenced as ${VAR} in shell.

🟡 Node version — Pinned to node-version: '20'.

🟡 PATH — Switched to >> $GITHUB_PATH in the build step, removed per-step exports.

🟡 Hardcoded ref — Keeping TODO as-is, will update once #8524 merges.

Re: overall direction — This is still a draft targeting a staging branch for all tests with the MCP tool and copilot driven framework. If later it is determined to move tier 1/2 tests to ADO, I would propose to abandon this PR but implement same pipeline in ADO for tier 0/1/2, leaving only tier 0 in GitHub action seems not helpful enough.

[v1212]

https://github.com/v1212/azure-dev/actions/runs/27403167828 validated the pipeline in fork repo, it proved the testing framework working with copilot, mcp tool, and test cases. @glharper

[microsoft-github-policy-service]

Hi @@v1212. Thank you for your interest in helping to improve the Azure Developer CLI experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.