Skip to content

fix(harbor): restore task.toml [metadata] as Task columns on load - #517

Closed
rohitsudhakar1 wants to merge 1 commit into
hud-evals:mainfrom
rohitsudhakar1:fix/harbor-metadata-columns
Closed

fix(harbor): restore task.toml [metadata] as Task columns on load#517
rohitsudhakar1 wants to merge 1 commit into
hud-evals:mainfrom
rohitsudhakar1:fix/harbor-metadata-columns

Conversation

@rohitsudhakar1

@rohitsudhakar1 rohitsudhakar1 commented Jul 27, 2026

Copy link
Copy Markdown

load() documents its rows as carrying task.toml's [metadata] as columns, and _HarborTask still parses and stores the config for it, but the mapping was dropped in 5b7110a ("Simplify the v6 contract surfaces") and nothing has read config since. Every row loaded from a Harbor dataset comes back with columns=None.

Repro

Any Harbor dataset whose tasks carry a [metadata] table:

from integrations.harbor import load

# task.toml:  [metadata] \n category = "networking" \n difficulty = "easy"
[row.columns for row in load("path/to/dataset")]
# -> [None, None]

Impact

The failure is quiet. A converted terminal-bench-style dataset loses the category / difficulty / tags each task ships with, so results can't be filtered or grouped by them. Because the field is merely absent rather than wrong, nothing surfaces the loss — the docstring on load() still promises the behaviour, and _HarborTask.config is still populated for a consumer that no longer exists.

Fix

Restore the mapping behind a small _columns helper. Metadata is left unset rather than emptied when a task.toml has no [metadata] table, so tasks without it don't gain an empty column set.

Tests

Three tests, using the existing conftest fixtures (which already carry [metadata]):

  • test_load_surfaces_task_toml_metadata_as_columns — metadata reaches Task.columns
  • test_load_leaves_columns_unset_without_metadata — no [metadata] table stays None
  • test_load_dataset_carries_per_task_metadata — metadata stays per-task across a dataset whose rows are grouped by build context

The first and third fail on current main:

FAILED integrations/tests/test_harbor.py::test_load_surfaces_task_toml_metadata_as_columns
FAILED integrations/tests/test_harbor.py::test_load_dataset_carries_per_task_metadata
2 failed, 1 passed

and pass with the fix:

uv run pytest integrations/tests/test_harbor.py -q   ->  19 passed
uv run ruff format . --check                         ->  258 files already formatted
uv run ruff check .                                  ->  All checks passed!
uv run pyright                                       ->  0 errors, 21 warnings  (parity with baseline)

The full suite is 725 passed with 6 pre-existing failures in test_workspace.py / test_capability_backing.py around setpriv and shell-uid sandboxing. I verified those fail identically on an unmodified checkout on this machine, so they're unrelated to this change.


Note

Low Risk
Localized Harbor loader fix with no auth, runtime, or export path changes; behavior matches existing Task.columns usage on sync/upload.

Overview
Restores Harbor load() mapping so each row’s Task.columns comes from task.toml’s [metadata] table (e.g. category, difficulty, tags), matching the documented contract that was dropped in a prior simplification.

A new _columns helper copies [metadata] when present and returns None when it’s missing or empty, so tasks without metadata don’t get an empty column dict. Metadata stays per task when rows are grouped by shared environment/ build context.

Adds three tests covering metadata surfacing, unset columns without [metadata], and per-task metadata across grouped datasets.

Reviewed by Cursor Bugbot for commit 3b97097. Bugbot is set up for automated code reviews on this repo. Configure here.

load() documents its rows as carrying `task.toml`'s `[metadata]` as columns, and
`_HarborTask` still parses and stores the config for it, but the mapping was dropped
in 5b7110a and nothing has read `config` since. Every row loaded from a Harbor
dataset comes back with `columns=None`.

The effect is quiet: a converted terminal-bench-style dataset loses the
category/difficulty/tags each task ships with, so results cannot be filtered or
grouped by them on the dashboard - and because the field is merely absent rather
than wrong, nothing surfaces the loss.

Restore the mapping behind a small `_columns` helper. Metadata is left unset rather
than emptied when a task.toml has no [metadata] table, so tasks without it do not
gain an empty column set.

Tests: metadata surfaces on a single task, stays None without a [metadata] table,
and stays per-task across a dataset whose rows are grouped by build context. The
first and third fail on the current code.
@rohitsudhakar1
rohitsudhakar1 force-pushed the fix/harbor-metadata-columns branch from 6976b00 to 3b97097 Compare July 28, 2026 02:01
@jdchawla29

Copy link
Copy Markdown
Collaborator

addressed in #516

@jdchawla29 jdchawla29 closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants