Skip to content

Add Zenodo preservation mirroring for certified releases#457

Merged
MaxGhenis merged 2 commits into
mainfrom
feat/404-zenodo-mirroring
Jul 7, 2026
Merged

Add Zenodo preservation mirroring for certified releases#457
MaxGhenis merged 2 commits into
mainfrom
feat/404-zenodo-mirroring

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Fixes #404
Addresses #439 (code half; DOI recording is per-release ops).

Provenance of the port

Supersedes closed PR #405 (branch zenodo-mirroring, kept for cherry-pick) after its full merits review. The deposit core transfers essentially verbatim — the Zenodo deposition flow, the hardened licence gate (refusing private and gated/unverifiable Hugging Face repos), the metadata builder, and the mocked test harness. The models it depends on (PreservationMirror, preservation_mirrors, preservation_dois, canonical_json_bytes) are all intact on main.

What the bundle-metadata migration (#425/#427/#438) invalidated in #405, and how this PR retargets it:

  • Dead data/release_manifests/ path. _bundled_tro_bytes() now reads the shipped TRO from data/bundle/{country}.trace.tro.jsonld (where scripts/generate_trace_tros.py writes it via BUNDLE_TRO_DIR). That directory was deleted by the migration, so Zenodo mirroring for certified releases #405's module would have raised for every country at runtime.
  • repo_type passthrough. Both _dataset_location_from_uri(...) calls now pass repo_type=country_manifest.data_package.repo_type instead of relying on the removed "model" default. This is load-bearing: the certified US artifact's data_package.repo_type is dataset, so the old default would have produced a wrong (non-/datasets/) Hugging Face URL.
  • Rewritten CLI/trace surfaces. cli.py gained a bundle command group and restructured dispatch since Zenodo mirroring for certified releases #405's June base, so zenodo-mirror is registered in the current _parser() and dispatched in the current main(). The pe:preservationDoi hook in trace.py's build_trace_tro_from_release_bundle (whose signature also changed) is re-placed by hand, emitting DataReleaseManifest.preservation_dois on the performance node. The old release-manifest pathway is not restored (explicit Surface UK Populace Zenodo DOI in bundle provenance #439 requirement).

Tests

All Zenodo/HF interaction is mocked — no live calls, no ZENODO_TOKEN use anywhere.

  • tests/test_zenodo_mirror.py — 15 tests: deposit flow, draft-vs-publish DOI recording, per-mirror sha256 integrity, sandbox targeting, the metadata builder, and the licence gate (private, gated-but-HTTP-200, transient-unverifiable, and public-repo dataset inclusion). The shipped-bytes assertion is fixed to data/bundle/.
  • tests/test_trace_tro.py — 2 added tests for pe:preservationDoi: omitted when the release records no DOI, and surfaced on the performance node when the data release manifest carries one.
  • Local run of the three affected files — 70 passed (test_zenodo_mirror.py 15, test_trace_tro.py 46, test_preservation_mirror.py 9).
  • ruff format --check . and ruff check . clean. mypy src/policyengine/provenance/zenodo.py introduces no new error category — its 4 findings are the same house-style call-arg / import-untyped / no-any-return patterns already present in sibling provenance/ modules (e.g. verify.py, certification.py), and the two retarget edits add none.

Out of scope (per-release ops after merge, tracked on #439)

  • Live deposits and recording the current build's concept DOI into the live data release manifests' preservation_dois are per-release operations performed after merge. The deposited TRO binds the manifest, so the DOI lives on the consuming side, not inside the snapshot.
  • The stale June concept DOIs from Zenodo mirroring for certified releases #405's live deposits (10.5281/zenodo.2067851x) describe builds that main has since superseded; they are not embedded anywhere in this PR. The DOI-surfacing test uses a synthetic placeholder value.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits July 7, 2026 07:14
Port the reviewed-good deposit core from closed PR #405, retargeted to
the bundle-metadata system that main adopted after that branch's base.

- provenance/zenodo.py: deposit a release's certification record (bundle
  manifest, bundle TRACE TRO, data release manifest) on Zenodo, returning
  PreservationMirror entries. The licence gate refuses dataset bytes from
  private or gated source repos. Retargeted vs #405: read the shipped TRO
  from data/bundle/ (the deleted data/release_manifests/ path) and pass
  data_package.repo_type through to _dataset_location_from_uri instead of
  relying on the old "model" default.
- cli.py: register the zenodo-mirror subcommand and its dispatch branch
  in the current parser/main structure.
- trace.py: surface DataReleaseManifest.preservation_dois on the bundle
  TRO performance node as pe:preservationDoi (#439 code half; the old
  release-manifest pathway is not restored).
- Tests: 15 mocked Zenodo/HF deposit tests and 2 TRO DOI-surfacing tests;
  all Zenodo/HF interaction is mocked, no live calls.
- Re-add the preservation-mirroring docs section and a changelog fragment.

Fixes #404

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix six issues found reviewing the deposit flow, each with a regression
test (no live network; all Zenodo/HF interaction mocked).

MAJOR:
- Partial failure no longer orphans an unfindable draft. Steps after the
  deposition is created (upload, metadata, publish — e.g. a
  deposit:actions-scope 403) are wrapped so any ZenodoDepositError
  re-raises with the deposit id and URL and a "delete or resume" note.
- _assert_source_repo_public no longer fails open. It now requires
  positive confirmation: a 200 with a JSON object reporting private=False
  and gated falsey. A non-JSON/non-object 200 (e.g. a Cloudflare page)
  falls through to the unverifiable refusal, and private=true is checked
  explicitly (an authenticated read of a private repo returns 200
  {"private": true}).

MINOR/NIT:
- CLI catches ValueError so an unknown country id exits cleanly instead
  of dumping a traceback (get_release_manifest raises ValueError).
- --include-dataset verifies fetched bytes against the certified sha256
  and refuses on mismatch, naming both hashes, before depositing.
- Draft mirror URLs are no longer fabricated /files/ paths off the edit
  page (they would 404). Drafts carry the deposit URL as a provisional
  handle; published records keep per-file record URLs. Docstring updated.
- _bundled_tro_bytes catches KeyError as well as FileNotFoundError, so a
  zipimport loader's missing-resource error maps to ZenodoDepositError.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis MaxGhenis marked this pull request as ready for review July 7, 2026 14:34
@MaxGhenis MaxGhenis merged commit 0356e43 into main Jul 7, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mirror certified releases to Zenodo

1 participant