Experiment: code-review with BCQuality plugin#737
Draft
Groenbech96 wants to merge 30 commits into
Draft
Conversation
…gins experiment toggle Adds a design spec and a task-by-task implementation plan for a new experiment toggle that installs agent plugins (a marketplace pinned to a git commit, or a local marketplace) into the running CLI via its own 'plugin marketplace add' + 'plugin install' commands, records them in ExperimentConfiguration, and removes them after the run. Docs only; no code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…iew) Addresses review: execution-based categories run entries as a parallel matrix (max-parallel 64) on a shared self-hosted runner, and the CLI plugin store is user-scope/global, so concurrent installs/teardowns would race. Fix: install into a fresh per-entry config home (COPILOT_HOME / CLAUDE_CONFIG_DIR under .bcbench), applied to both the plugin commands and the agent launch. This removes teardown entirely (isolated home discarded with the checkout) and moots the teardown_plugins signature comment. Verified live on both CLIs (fresh home: install + skill load + headless session, real config untouched). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…r-entry home) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es, dedupe marketplace clone Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the terminal AgentError out of 'case _' to after the match block in _materialize and _home_env_var, matching the existing pattern in types.py (EvaluationCategory). Behavior unchanged; clears the github-code-quality mixed-returns advisory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per PR #736 review: move the shutil.which(...) resolution + not-found check to the top of run_copilot_agent / run_claude_code (before the 'Running...' log and all setup work), so a missing CLI fails fast. Behavior otherwise unchanged; the resolved binary is still passed to setup_plugins_from_config. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI's pre-commit 'ty check' hook failed: add_call[1]/install_call[1] were typed
dict | None (kwargs.get('env')) and thus not subscriptable. Record env as a
non-optional dict (kwargs.get('env') or {}). ty now exits 0.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a self-contained example marketplace (bcbench-example-marketplace + bcbench-example-plugin bundling one no-op skill) under instructions/microsoft-BCApps/plugins/bcbench-example/, referenced by a disabled local entry in config.yaml. Flip enabled: true to smoke-test that a plugin installs into the per-entry isolated home and its skill loads (verified live: marketplace add + install + skill list all succeed). Guarded by a test tying config.yaml -> example dir -> marketplace.json -> plugin.json so the example can't silently rot. EXPERIMENT.md documents it and the built-in marketplace-name collision caveat. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…(revert before merge) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, install) Adds INFO logging around plugin setup so CI logs clearly show each step — 'Installing N plugin marketplace(s) into isolated COPILOT_HOME=...', 'Cloning plugin marketplace repo@commit' / 'Copying local plugin marketplace ...', 'Registering marketplace ... from ...', and 'Installing/Installed plugin ...'. Previously only the post-install line (and clone_at_commit's clone log) showed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Local `plugins` entries resolved their `path` under instructions/<owner>-<repo>/, which broke for dataset entries whose repo (e.g. microsoftInternal/NAV) differs from where the example was stored. Resolve `local` sources from a fixed, repo-agnostic base (src/bcbench/agent/shared/plugins/) instead, and move the bundled bcbench-example there. Update tests, spec, plan, EXPERIMENT.md, and the config.yaml comments/path accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The `marketplace` plugin source now accepts an optional `path` (marketplace root within the clone), mirroring `local`, so repos that host a marketplace in a subdirectory are supported. Enable a self-contained marketplace example that clones this repo at a pinned commit and points `path` at the bundled example, so an evaluation run visibly logs the plugin clone under bcbench.operations.git_operations (the local copy path logged no clone). The local example is kept as a disabled, test-validated template. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a real-world marketplace plugin example (obra/superpowers, marketplace "superpowers-dev", plugin "superpowers") pinned to v6.1.1, enabled for a smoke run so the agent can discover and invoke a plugin skill (e.g. brainstorming) via the `skill` tool in headless -p mode. The prior bundled examples are kept as disabled, test-validated templates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ppet Installing a plugin makes its skills available but not used: on well-specified tasks the agent works directly and invokes nothing. Add an "Encouraging plugin usage" section to EXPERIMENT.md (MCP/hooks are automatic; skills are discretionary and need a nudge via the instructions lever or category prompt), and ship a ready-to-use, subtle nudge snippet (instructions/skill-usage-nudge.md) that experimenters can append to a repo AGENTS.md and toggle on with instructions.enabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the obra/superpowers marketplace example (it was only enabled to demo skill usage). The two bundled examples (local bcbench-example and the BC-Bench self-clone marketplace) remain as disabled templates, so nothing installs by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove the blank line before setup_plugins_from_config in both runners so it groups with the other setup_* calls. - Drop the "what we're not doing" (extraKnownMarketplaces) explanation from the plugin_operations module docstring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per review: keep the local bcbench-example and the obra/superpowers marketplace example (both disabled), remove the BC-Bench self-clone entry, and cut the plugins comment block down to the essentials to match the rest of the config. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reduce the "Encouraging plugin usage" section to the single reliable lever (custom instructions) and inline the subtle nudge as a copy-paste block instead of a standalone file under instructions/ (which is not an established location for loose markdown). Remove the skill-usage-nudge.md file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These docs/superpowers/ spec and plan files were superpowers-workflow artifacts and are not part of this repo's docs. The design rationale is captured in EXPERIMENT.md and the code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dd failure) An earlier commit failed to stage EXPERIMENT.md because a sibling deleted-file pathspec errored, so the "point 1 only" trim + inlined nudge never landed. Commit it now, removing the broken link to the deleted snippet file and the extra points. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enable the BCQuality plugin (microsoft/BCQuality PR #84 head, ships the `bcquality-al-review` skill) and the custom-instructions lever, and tell the BCApps code reviewer it can use `bcquality-al-review` to ground AL findings in BCQuality's curated rules. Validated locally on synthetic__security-001: the agent invokes the skill and its review cites BCQuality knowledge files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment: code-review with the BCQuality plugin
Runs the code-review category with microsoft/BCQuality PR #84 packaged as an installable plugin, and instructs the code reviewer that it may use it.
Config (this branch)
plugins:microsoft/BCQuality @ a1b0ef89c6e22987894aa6d45551d4c88879f4cb→["bcquality"], enabled (ships thebcquality-al-reviewskill). The bundledlocal+superpowersexamples stay disabled.instructions.enabled: true+ a "Reviewing AL code → usebcquality-al-review" note added to the BCAppsAGENTS.md, so the reviewer knows the skill is available (recorded ascustom_instructions=True).Local validation (
synthetic__security-001, claude-haiku-4.5)plugins=['bcquality@a1b0ef8…'].tool_usage={'skill': 1, …},SKILL INVOCATIONS: ['bcquality-al-review'].review.jsonis grounded in BCQuality: caught the expected critical (hardcoded key, line 16) citingsecrettext-for-credentials.md, plus a bonus high citingsecrettext-with-httpclient.md.CI
Triggered the code-review evaluation workflow on this branch (run
28927178335).Base
Targets
feat/marketplace-local-plugins(depends on the plugins toggle in #736); will retarget tomainonce that merges.