Add Zenodo preservation mirroring for certified releases#457
Merged
Conversation
Port the reviewed-good deposit core from closed PR #405, retargeted to the bundle-metadata system that main adopted after that branch's base. - provenance/zenodo.py: deposit a release's certification record (bundle manifest, bundle TRACE TRO, data release manifest) on Zenodo, returning PreservationMirror entries. The licence gate refuses dataset bytes from private or gated source repos. Retargeted vs #405: read the shipped TRO from data/bundle/ (the deleted data/release_manifests/ path) and pass data_package.repo_type through to _dataset_location_from_uri instead of relying on the old "model" default. - cli.py: register the zenodo-mirror subcommand and its dispatch branch in the current parser/main structure. - trace.py: surface DataReleaseManifest.preservation_dois on the bundle TRO performance node as pe:preservationDoi (#439 code half; the old release-manifest pathway is not restored). - Tests: 15 mocked Zenodo/HF deposit tests and 2 TRO DOI-surfacing tests; all Zenodo/HF interaction is mocked, no live calls. - Re-add the preservation-mirroring docs section and a changelog fragment. Fixes #404 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix six issues found reviewing the deposit flow, each with a regression
test (no live network; all Zenodo/HF interaction mocked).
MAJOR:
- Partial failure no longer orphans an unfindable draft. Steps after the
deposition is created (upload, metadata, publish — e.g. a
deposit:actions-scope 403) are wrapped so any ZenodoDepositError
re-raises with the deposit id and URL and a "delete or resume" note.
- _assert_source_repo_public no longer fails open. It now requires
positive confirmation: a 200 with a JSON object reporting private=False
and gated falsey. A non-JSON/non-object 200 (e.g. a Cloudflare page)
falls through to the unverifiable refusal, and private=true is checked
explicitly (an authenticated read of a private repo returns 200
{"private": true}).
MINOR/NIT:
- CLI catches ValueError so an unknown country id exits cleanly instead
of dumping a traceback (get_release_manifest raises ValueError).
- --include-dataset verifies fetched bytes against the certified sha256
and refuses on mismatch, naming both hashes, before depositing.
- Draft mirror URLs are no longer fabricated /files/ paths off the edit
page (they would 404). Drafts carry the deposit URL as a provisional
handle; published records keep per-file record URLs. Docstring updated.
- _bundled_tro_bytes catches KeyError as well as FileNotFoundError, so a
zipimport loader's missing-resource error maps to ZenodoDepositError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #404
Addresses #439 (code half; DOI recording is per-release ops).
Provenance of the port
Supersedes closed PR #405 (branch
zenodo-mirroring, kept for cherry-pick) after its full merits review. The deposit core transfers essentially verbatim — the Zenodo deposition flow, the hardened licence gate (refusing private and gated/unverifiable Hugging Face repos), the metadata builder, and the mocked test harness. The models it depends on (PreservationMirror,preservation_mirrors,preservation_dois,canonical_json_bytes) are all intact onmain.What the bundle-metadata migration (#425/#427/#438) invalidated in #405, and how this PR retargets it:
data/release_manifests/path._bundled_tro_bytes()now reads the shipped TRO fromdata/bundle/{country}.trace.tro.jsonld(wherescripts/generate_trace_tros.pywrites it viaBUNDLE_TRO_DIR). That directory was deleted by the migration, so Zenodo mirroring for certified releases #405's module would have raised for every country at runtime.repo_typepassthrough. Both_dataset_location_from_uri(...)calls now passrepo_type=country_manifest.data_package.repo_typeinstead of relying on the removed"model"default. This is load-bearing: the certified US artifact'sdata_package.repo_typeisdataset, so the old default would have produced a wrong (non-/datasets/) Hugging Face URL.cli.pygained abundlecommand group and restructured dispatch since Zenodo mirroring for certified releases #405's June base, sozenodo-mirroris registered in the current_parser()and dispatched in the currentmain(). Thepe:preservationDoihook intrace.py'sbuild_trace_tro_from_release_bundle(whose signature also changed) is re-placed by hand, emittingDataReleaseManifest.preservation_doison the performance node. The old release-manifest pathway is not restored (explicit Surface UK Populace Zenodo DOI in bundle provenance #439 requirement).Tests
All Zenodo/HF interaction is mocked — no live calls, no
ZENODO_TOKENuse anywhere.tests/test_zenodo_mirror.py— 15 tests: deposit flow, draft-vs-publish DOI recording, per-mirror sha256 integrity, sandbox targeting, the metadata builder, and the licence gate (private, gated-but-HTTP-200, transient-unverifiable, and public-repo dataset inclusion). The shipped-bytes assertion is fixed todata/bundle/.tests/test_trace_tro.py— 2 added tests forpe:preservationDoi: omitted when the release records no DOI, and surfaced on the performance node when the data release manifest carries one.test_zenodo_mirror.py15,test_trace_tro.py46,test_preservation_mirror.py9).ruff format --check .andruff check .clean.mypy src/policyengine/provenance/zenodo.pyintroduces no new error category — its 4 findings are the same house-stylecall-arg/import-untyped/no-any-returnpatterns already present in siblingprovenance/modules (e.g.verify.py,certification.py), and the two retarget edits add none.Out of scope (per-release ops after merge, tracked on #439)
preservation_doisare per-release operations performed after merge. The deposited TRO binds the manifest, so the DOI lives on the consuming side, not inside the snapshot.10.5281/zenodo.2067851x) describe builds thatmainhas since superseded; they are not embedded anywhere in this PR. The DOI-surfacing test uses a synthetic placeholder value.🤖 Generated with Claude Code