feat(integrations): the integration contract, and Harbor as its first frontend - #516
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 48aaf7423b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
48aaf74 to
4fd8456
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4fd84563ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4fd8456 to
658de7a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 658de7a7b4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
658de7a to
15b09fa
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 15b09fa9ae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
6e882a5 to
82cf98c
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82cf98cb1e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
82cf98c to
e488823
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4888234da
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
29f2388 to
a44a51f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a44a51f247
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
a44a51f to
7a5cd86
Compare
There was a problem hiding this comment.
💡 Codex Review
hud-python/hud/integrations/harbor/_export.py
Line 114 in 7a5cd86
When a task argument contains an apostrophe, such as {"prompt": "what's next?"}, embedding its JSON inside single quotes produces an invalid hud_entrypoint.sh (sh -n reports an unterminated quoted string), so the exported container exits before serving and grading. The same interpolation is repeated in test.sh; shell-quote the task ID and serialized arguments when formatting both scripts.
hud-python/hud/integrations/harbor/_export.py
Lines 217 to 219 in 7a5cd86
For an environment whose Dockerfile consumes a root-level JSON file, this removes required build inputs indiscriminately. A common Node context with package.json and COPY package*.json ./, for example, is copied and then has package.json deleted, causing the exported Harbor image build to fail; remove only the specific taskset source file rather than every *.json* file.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
7a5cd86 to
aebe6b4
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aebe6b4d14
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
aebe6b4 to
6f2a3fe
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f2a3fe783
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 736ff58af3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
736ff58 to
4a3fbc8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a3fbc8aa2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4a3fbc8 to
235429d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 235429d4a3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
235429d to
c9fec84
Compare
There was a problem hiding this comment.
💡 Codex Review
hud-python/hud/integrations/harbor/_export.py
Line 215 in c9fec84
When a valid HUD template ID contains a double quote, backslash, or newline, interpolating it directly inside this quoted TOML value produces an invalid or structurally altered task.toml. Harbor then cannot load the exported task even though the ID works through the HUD task model and shell commands are already safely quoted; serialize this value with TOML-compatible escaping.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
c9fec84 to
27b58a3
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 27b58a36ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
27b58a3 to
5d6ebe2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d6ebe277c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
5d6ebe2 to
e32924e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e32924ee6f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
e32924e to
a0eae4f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a0eae4f497
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
a0eae4f to
3c9d70b
Compare
…re factories An Integration is a foreign format's frontend: load(ref) -> Taskset rows and environment(ref) -> the live env those rows join. Two verbs, both universal; where the constructor can execute is the format's nature (in-process for a dataset+scorer, inside its image for a container format). Implementations live outside core — core knows only the interface. load_environment gains the reference form this needs: an importable dotted module whose attribute is an Environment, or a factory called with args. 'hud serve pkg.mod:make_env --arg name=x' now works, so a programmatic env serves itself without a source file. The resolver is one function again — the CLI wrapper is the colon split plus error printing.
…ontainers match hosted Workspace checked whether bwrap exists, not whether it works. An installed bwrap that cannot create namespaces — a container whose seccomp profile blocks unprivileged userns, or whose /proc is masked — failed every session instead of falling back. It now probes once with a session-shaped argv (mounting proc is what fails first) and falls back when unusable; require_isolation=True refuses to serve rather than silently running unisolated, for callers that promised a task no network. DockerRuntime did not grant env containers the security options they get when hosted, so an environment isolated in one place and silently did not in the other. Both now start containers the same way.
3c9d70b to
1a703e6
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1a703e656f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
1a703e6 to
dde4f7f
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit dde4f7f. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dde4f7f36b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
integrations/ becomes hud.integrations because an adapted image imports it: the image's CMD serves hud.integrations.harbor:environment, so the module has to ship in the wheel that image installs. Nothing in the SDK or CLI imports it — a Harbor taskset is loaded and placed explicitly. Harbor implements the contract: - load() carries what a task declares, not just its identity: [metadata] as Task.columns (difficulty/category/tags — the platform's facets), [environment] cpu/memory/gpu as RuntimeConfig.resources, real descriptions on templates, verifier.env for the verifier. Time budgets stay off the row — a budget bounds the rollout, not the substrate — and agent_timeout() exposes them for rollout_timeout. - adapt() packages the environment constructor into one HUD-speaking image per build context, whose CMD serves that constructor; rows then run on any container placement. Images are content-addressed and kept, built once per group under a lock. - environment() is what those images serve: the workspace applies the task's declared network isolation, env, workdir and user, healthchecks gate serving, URL MCP servers are published as capabilities, and tests/test.sh grades in place under the task's verifier timeout. Behaviour that cannot be reproduced faithfully is refused, not dropped: allowlist egress, a verifier with its own environment, stdio MCP servers, non-linux os, TPUs, and multi-step tasks (which load, but do not adapt). Environments are grouped by build context *and* declared policy, so one environment always serves one policy. Container, workspace and grading mechanics were ported from #478. Co-authored-by: Nancy <najilau@ucsc.edu>
dde4f7f to
adfb5f0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: adfb5f0933
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Supersedes #478.
The contract
An integration is a foreign format's frontend.
hud.environment.Integrationis two verbs:Where the constructor can execute is the format's nature — anywhere for an in-process format (a dataset plus a scorer), only inside its own image for a container format like Harbor. Implementations live outside core; core knows only the interface.
hud servelearned the reference form this needs:hud serve pkg.mod:make_env --arg name=ximports a dotted module and calls a factory, so a programmatic environment serves itself with no source file. The path form is unchanged.Harbor
adapt()packages the environment constructor into a HUD-speaking image whose CMD serveshud.integrations.harbor:environment;load()stamps each row with that image, so the same rows also run onModalRuntime,DaytonaRuntime, or a deploy of the generated build contexts. That is why the package moved underhud.— the adapted image installs the wheel and imports it. Nothing in the SDK or CLI imports it; a Harbor taskset is loaded and placed explicitly.Rows now carry what the task declared, not just its identity:
[metadata]asTask.columns(difficulty/category/tags),[environment]cpu/memory/gpu asRuntimeConfig.resources, real task descriptions on templates,verifier.envfor the verifier. Time budgets deliberately stay off the row — a budget bounds the rollout, not the substrate — andharbor.agent_timeout()exposes them forrollout_timeout.At serve time the workspace applies the task's declared network isolation, env, workdir and user;
tests/test.shgrades in place under the task's verifier timeout.Refused rather than silently downgraded — a wrong score is worse than a refused task. Adaptation replaces the container's own boot process with the serving command, so anything depending on that boot process is refused: a Dockerfile
ENTRYPOINT,healthcheck(nothing would start the services it awaits), andmcp_servers(nothing would start the servers they point at) — alongsidenetwork_mode = "allowlist", a verifier with its own environment, non-linuxos, TPUs, and multi-step tasks (which load fine, but do not adapt). Environments are grouped by build context and declared policy, with the invariant that anything the constructor consumes per group is either in the group key or refused.Two core fixes this surfaced
Workspacechecked whether bwrap exists, not whether it works. An installed bwrap that cannot create namespaces failed every session instead of falling back. It now probes once with a session-shaped argv and falls back when unusable;require_isolation=Truerefuses to serve rather than silently running unisolated.DockerRuntimedid not grant env containers the security options they get when hosted, so an environment isolated in one place and silently did not in the other. Both now start containers the same way.Verification
764 unit tests, plus real-docker end-to-end runs at every step: six datasets from the Harbor registry (hello-world, aime, gpqa-diamond, quixbugs-java, compilebench, swebench-verified) all graded 1.0 through
adapt → DockerRuntime, and a task declaringno-networkconfirmed to have its egress genuinely severed (curl: (6) Could not resolve host) while its env vars and workdir applied.Container, workspace and grading mechanics were ported from #478 (co-authored).
Note
Medium Risk
Changes container startup security options and workspace isolation behavior (opt-in nested sandbox, bwrap probing, required isolation), plus a large Harbor adapt/grade path; patterns are fail-closed and covered by extensive tests, but misconfiguration could affect grading fidelity or container containment.
Overview
Introduces
hud.environment.Integration(load→Taskset,environment→Environment) and ships Harbor ashud.integrations.harbor(package moved in-wheel so adapted images canimportit).load_environmentandhud servenow resolve dotted module targets and factories via--arg key=value, matching adapted image CMDs likehud.integrations.harbor:environment.Harbor
adapt()writes/builds per-env-group contexts under.hud-adapt/whose CMD serves the integration constructor;load(..., images=)stamps image refs on rows. Rows also carry[metadata]ascolumnsand[environment]resources asruntime_config; agent timeouts stay off-row (harbor.agent_timeout). Unsupported Harbor declarations (ENTRYPOINT, allowlist network, multi-step adapt, etc.) are refused rather than downgraded.Workspaceprobesusable_bwrap()once and supportsrequire_isolation=True(fail closed).DockerRuntimeadds opt-innested_sandbox=True(seccomp/systempathsunconfined) for images that sandbox internally;harbor.docker_runtime()enables it by default.HUDRuntimewarns once on soft-ignoredruntime_configbut still errors on GPU/explicit limits. Sharedhud.utils.dockerandnormalize_environment_name; export path hardened (shell quoting, dockerignore, non-root USER restore, determinism note in docs).Reviewed by Cursor Bugbot for commit adfb5f0. Bugbot is set up for automated code reviews on this repo. Configure here.