test: add model-parameterized onboarding testcase#994
Open
denispetre wants to merge 2 commits into
Open
Conversation
Adds testcases/model-onboarding/, a standalone LangGraph testcase that exercises a single runtime-specified model across the distinct get_chat_model code paths it is expected to support, plus optional file attachments, rolling every path x file cell into one success boolean. Unlike multimodal-invoke (which hardcodes its model matrix), the model is supplied via input.json — onboarding a new model requires no code change. A PATH_REGISTRY maps six code-path keys (azure_responses, azure_chat_completions, vertex, bedrock_converse, bedrock_invoke, anthropic_sdk) to builder closures; files [] runs a text-only reachability check via plain ainvoke. Uses the full-strength assert pattern (nupkg, status, non-empty summary, "Successful execution." in the local run log, and trace assertions via expected_traces.json) rather than multimodal-invoke's minimal assert. Auto-discovered by integration_tests.yml and run across the alpha, staging, and cloud (prod) matrix with per-environment secrets. README documents both the local and CI run mechanisms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds .github/workflows/model_onboarding.yml, a workflow_dispatch workflow that runs the model-onboarding testcase against a model named at dispatch time. Inputs (model_name, paths, files, prompt, agenthub_config, environments) are written into input.json at runtime via jq, so changing the model under test no longer requires editing or committing a file. Environments are selectable and fan out via matrix; each leg uses its own ALPHA_*/STAGING_*/CLOUD_* secrets. The comma-separated path/file lists are normalized in the shell before jq to avoid regex escaping hazards in YAML. README documents this as Mechanism B (on-demand, no commit); the existing push/PR integration matrix becomes Mechanism C (regression gate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
testcases/model-onboarding/— a standalone LangGraph testcase that exercises one runtime-specified model across the distinctget_chat_modelcode paths it is expected to support, plus optional file attachments, rolling everypath × filecell into a singlesuccessboolean.Unlike
multimodal-invoke(which hardcodes its model matrix), the model is supplied viainput.json— onboarding a new model requires no code change.How to use
Edit
input.json:{ "prompt": "Describe the content of this file in one sentence.", "model_spec": { "model_name": "gpt-5.2-2025-12-11", "paths": ["azure_responses", "azure_chat_completions"], "agenthub_config": "agentsplayground", "files": ["image", "pdf"] } }paths— whichget_chat_modelcode paths to exercise. Keys:azure_responses,azure_chat_completions,vertex,bedrock_converse,bedrock_invoke,anthropic_sdk. List only the paths the model actually ships on.files—image/pdf, or[]for a text-only reachability check (plainainvoke).Design notes
PATH_REGISTRYmaps each code-path key to a builder closure.status, non-empty summary,"Successful execution."in the local run log, and trace assertions viaexpected_traces.json) rather thanmultimodal-invoke's minimal assert.settings: ✗ …summary instead of crashing the node.Running it
Auto-discovered by
integration_tests.ymland run across thealpha,staging, andcloud(prod) matrix with per-environment secrets.README.mddocuments both the local and CI run mechanisms.Verified locally: graph compiles,
input.jsonvalidates, all six paths + two files register, and the defensive branches produce legible failures. A live LLM run needs real tenant credentials (uipath auth), so it is exercised in CI.🤖 Generated with Claude Code