Deterministic HTS/ruling-citation enrichment for CROSS-style customs corpora#2153
Deterministic HTS/ruling-citation enrichment for CROSS-style customs corpora#2153JSv4 wants to merge 20 commits into
Conversation
…oms corpora; fix bulk-ingest ext gate and relationship-embedding bottleneck - opencontractserver/enrichment/services/customs_ruling_citation_service.py: detects HTS tariff codes (plain annotations) and CBP ruling-number citations (CorpusReference rows resolved against sibling document titles) from each document's own post-parse text, reusing EnrichmentWriter for persistence. Runnable via manage.py enrich_customs_rulings. Validated at 100% precision/recall against a 96-doc pilot vs. the source dataset's own golden-tested extraction. - ingest_corpus management command: ext_ok now unions get_convertible_extensions() so files eligible for the configured file converter (e.g. Gotenberg -> .doc) are actually accepted, matching the command's own docstring. - embeddings_task.py: calculate_embeddings_for_relationship_batch's explicit-embedder path now batches through embed_texts_batch() (shared _batch_embed_items helper, also used by the annotation path) instead of one HTTP call per relationship -- a real bottleneck for parsers that emit many relationships per document (e.g. Warp-Ingest's heading-hierarchy OC_SUBTREE_GROUP rows). - celery worker start script: ignore .pilot_data so watchfiles doesn't choke on large locally-materialized ingest batches.
A sustained bulk-ingest run drove warp-ingest to ~5.7GB with no per-container memory limit anywhere in local.yml, exhausting host RAM/swap and causing the kernel OOM-killer to take down unrelated desktop processes. Caps + restart policy contain future runaway growth to a single container instead of the whole host.
convert_document_to_pdf, extract_thumbnail, ingest_doc, and set_doc_lock_state previously shared one queue. On a .doc-heavy bulk corpus this let the flood of cheap conversion tasks (enqueued for every document up front) statistically starve the actual parsing/unlock work, observed at a ~190:1 completion ratio even after doubling worker concurrency. extract_thumbnail/ingest_doc/ set_doc_lock_state now route to their own doc_parse queue with dedicated consumer capacity; standard worker startup scripts updated to subscribe to it so normal (non-bulk) uploads keep working.
Capping only the four heaviest services (docling-parser, warp-ingest, vector-embedder, celeryworker) left postgres, redis, django, gotenberg, multimodal-embedder, and docxodus-parser unbounded. Under concurrent load from a parallelized enrichment run plus a large Celery backlog drain, their aggregate uncapped memory usage exhausted host RAM/swap and forced a hard reboot, even though each individually-capped service stayed under its own ceiling. Every Docker-managed service now has a mem_limit.
load_document_text_and_layer's S3 fetch + PAWLS parse dominates per-document cost (~230ms measured) and touches no ORM state, so it's safe to prefetch across a thread pool while writes stay single- threaded on the caller. Also adds --limit for a quick, fully-complete pass over a deterministic subset of a corpus, useful for evaluating output quality/UX without enriching a corpus end-to-end in one sitting; citation resolution still considers every document's title in the corpus, not just the scanned subset.
…lution title_index was keyed on the raw document title, but ~98% of corpus 95's real titles retain a file extension (A83482.doc) that the ruling-citation regex never produces, so every citation into those documents resolved as unresolved regardless of how many candidates were found.
get_or_create's lookup key (source_annotation, reference_type, canonical_key) never included the resolution outcome, so a row written UNRESOLVED before its target existed (or before a resolution bug was fixed) was never upgraded by a later run that DID find the target — the run's in-memory resolved count silently diverged from what persisted. Shared EnrichmentWriter fix, so it also applies to the general authority/law-reference discovery path, not just customs rulings.
Review: Deterministic HTS/ruling-citation enrichment for CROSS-style customs corporaNice, well-documented PR — the changelog fragments and code comments make the "why" behind each change easy to follow, and the writer heal-logic fix (with regression test in Correctness
Both cand = Candidate(..., normalized_data={"ruling_number": number})
...
resolutions.append(Resolution(candidate=cand, source_document_id=doc.id, resolution_status=..., target_document_id=...))
Because the customs service never does this, Fix: pass Test coverage gapThe 15 new tests in Performance / memory (worth a second look given this PR's own motivation)
Minor
Everything else looks solid
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The customs-ruling enrichment service skipped every non-PDF document
before detection ran, so the official CROSS bulk export (text/plain)
produced zero output across a 10,000-document corpus
(docs/benchmarks/pr2153-cross-txt-enrichment-handoff.md).
- Anchoring type is now an input to persistence, not an eligibility
gate: PDF documents keep TOKEN annotations; text documents get
SPAN_LABEL annotations in the canonical {start, end, text} / page=0
shape. HTS dedupe matches by label text across both stored span
representations, so importer sidecar rows are never duplicated.
- EnrichmentWriter's span fallback now emits the canonical span shape
(page=0 + anchored text, was page=1 with no text), heals pre-fix
span rows in place on re-run, and reuses the candidate raw text
instead of re-fetching each document's extract from storage. The
shape is built once in span_projection.span_annotation_payload with
the SPAN_NO_PAGE sentinel in constants/annotations.py.
- Canonical ruling identity is path/external_id-derived (cross:
namespace > active path basename > title stem), shape-validated, and
ambiguous identities are counted in canonical_id_collisions and left
unresolved instead of resolving to the last-indexed document.
- Summary metric documents_skipped_not_pdf renamed to
documents_skipped_unanchorable; HTS rows now record the run's
Analysis; prefetch concurrency is bounded to the storage pool via
CUSTOMS_ENRICHMENT_PREFETCH_WORKERS (default pool size - 1).
- DB-backed regression suite in test_customs_ruling_enrichment.py,
including an official-export-shaped ZIP -> zip-to-corpus ->
enrichment contract test.
Verified live against the 10K CROSS corpus (PK 96): a 200-document
slice produced 686 HTS span annotations with zero skips and re-ran
idempotently with no S3 pool warnings.
| "HQ H900001 rules that the vehicles are classified under " | ||
| "subheading 8703.23.01, HTSUS.", |
The branch's previously-unpushed commits (50a4602..ffe93ea) had never run through CI's pre-commit --all-files pass and carried black/isort/ pyupgrade drift plus 8 mypy errors. All fixes are behavior-neutral: - black/isort/pyupgrade auto-fixes in document_queries.py, file_converter.py, test_redis_settings_urls.py, corpuses/models.py, test_doc_tasks.py. - annotations/signals.py and tasks/doc_tasks.py: replaced default-arg on_commit lambdas (which mypy cannot infer) with typed module-level dispatch helpers bound via functools.partial; same freeze-at-signal semantics and dedup task_ids. - corpuses/models.py: renamed the label-scan loop variable so the later Optional lookup no longer collides with its inferred type. - test_redis_settings_urls.py: load_local_settings returns dict[str, Any] (was dict[str, object], not indexable). - test_embeddings_task.py: annotated the callbacks accumulator. pre-commit --all-files passes end to end; affected suites (test_embeddings_task, test_doc_tasks, test_redis_settings_urls, test_document_queries, test_file_converters, customs enrichment) all pass.
…ases Closes two follow-ups from the CROSS TXT enrichment handoff (docs/benchmarks/pr2153-cross-txt-enrichment-handoff.md): - meta.csv accepts an optional external_id column (opencontractserver/utils/metadata_file_parser.py): a producer-namespaced identifier (e.g. cross:H022844) stored verbatim on DocumentPath.external_id by import_zip_with_folder_structure (new external_ids_applied result counter). Values over the field's 512-char limit are rejected per-row with a warning, never truncated. Customs citation resolution already reads the cross: namespace first, so renamed documents keep resolving; the namespace match is case-insensitive (CROSS:H022844 was stored but silently ignored). - import_document's update branch now inherits ingestion_source / external_id / ingestion_metadata from the previous path version unless the caller supplies fresh values (opencontractserver/documents/ versioning.py) - move/delete/restore already did; the update branch was the only lifecycle op that silently dropped a stamped identity on content re-import. - enrich_customs_rulings summary records per-phase cost separately: load_seconds (storage fetches, summed across the prefetch pool), match_seconds (regex detection + citation resolution), write_seconds (annotation/reference/graph persistence), and a load_failures breakdown of documents_skipped_unanchorable. HTS detection was split from persistence (_find_hts_matches) so the attribution is exact. test_no_metadata_columns_warning's asserted warning wording was updated because the warning now names the third optional column - intentional contract extension, not a behavior regression. New tests: external_id parser coverage, an opaque-filename ZIP -> import -> enrichment resolution test, upversion lineage inheritance/override, the case-variant namespace, and timing/load_failures assertions.
Full rerun over corpus 96 after the fix: 10,000 scanned / 0 skipped, 37,172 HTS span annotations, 300 citation mentions -> 299 unresolved references (the slice contains none of its citation targets, as the handoff predicted), 0 identity collisions, 0 connection-pool warnings with the bounded prefetch. Wall time 206 minutes, persistence-dominated.
Reference-web analysis of the 10K official-export benchmark (corpus 96)
found 74% of the true document graph missing: ground truth from the
source database is 6,623 in-corpus edges; the export captured 1,727.
Two causes, two fixes (the exporter-side cross-batch fix lands in the
CROSS-Corpus repo):
- Legacy citation grammar: the legacy HQ/NY slice has bare zero-padded
numeric ruling numbers ("HQ/084665.txt"), cited in text as
"<series token> <6 digits>" ("HRL 087392") - invisible to the
prefixed-only grammar. _LEGACY_RULING_CITE_RE mines exactly that
shape (707 instances per 500 sampled docs, zero false positives
observed; six digits required because 5 digits after "NY" is a ZIP
code in 148/149 sampled instances). Identity and citations meet on
one canonical key (_canonical_ruling_key: prefixed verbatim, bare
digits zero-stripped). Live 500-doc validation: 163 citation
candidates, 39 resolved graph edges (previously ~1 candidate, 0
resolvable).
- relationships.csv import is now idempotent: get_or_create on the
edge's natural identity (source, target, corpus, label, type)
instead of blind create, so relationships-only patch ZIPs and re-run
batches never duplicate edges; duplicates count as
relationships_skipped.
Corpus 96 was patched live with the 4,896 missing metadata edges via a
relationships-only ZIP: the reference web now shows 6,676 resolved
references across 5,081 connected documents (was 1,741 / 1,996).
ReviewLarge PR (72 files, ~4.4k additions) bundling a deterministic HTS/CBP customs-ruling citation enrichment feature with several fixes discovered while stress-testing it against a real ~115K-document corpus (title/extension matching, Strong points
Issues to look atInfra / memory caps (
Frontend upload UX ( Test coverage gaps (lower severity, but worth noting since this PR explicitly leans on "verified live against real data") Scope noteThis PR bundles quite a few logically-distinct fixes (citation enrichment, writer healing, OOM/queue routing, relationship idempotency, lineage inheritance, ingest extension handling, upload progress refactor). The PR description does a good job narrating why they're coupled (discovered while stress-testing the primary feature against a real corpus), and nothing here reads as unrelated scope creep — but given the size, consider whether the OOM/queue-routing infra work and the upload-progress frontend refactor could land as their own PRs in the future for easier review/bisection. No security concerns identified (no XSS risk in citation rendering — all user content renders as plain JSX text; no new secrets or exposed ports in the compose changes). |
Ground truth vs captured analysis (74% of the graph missing), the two fixes (exporter cross-batch routing in CROSS-Corpus; legacy series-token citation grammar here), and the final corpus-96 numbers: 9,377 citation candidates, 5,007 resolved, 11,644 total graph edges across 5,092 connected documents (was 1,741/1,996).
ReviewReviewed the full diff (72 files, +4413/-267) across three areas: enrichment/citation core logic, infra/ingest plumbing, and frontend upload/intelligence UI. Overall this is a well-structured PR — good changelog fragment discipline, docs updated per the pointer-based convention, and the two headline correctness fixes (title/extension matching, UNRESOLVED→RESOLVED heal) are each backed by a real regression test rather than just a passing-test claim. Correctness
Nice-to-haves / minor
What's solid
Test coverage summaryCoverage is generally strong for the two headline fixes and the settings fix, but has three gaps worth closing before merge: the NOTES-relationship idempotency interaction above, the |
codecov/patch flagged 92 uncovered lines across the branch. Backend (42 lines, all verified hit in a local coverage run): - enrich_customs_rulings management command (was 0%): named-owner run, superuser default + --limit, unknown-owner and no-superuser errors. - customs service edge branches: defensive unknown-anchor-type skip, analysis-FAILED-and-reraise path, explicit CUSTOMS_ENRICHMENT_PREFETCH_WORKERS override (and its floor of 1), third claimant of an already-ambiguous ruling number, HTS matches rejected by normalization, PDF token-projection failure skip. - create_relationships_from_parsed dedupe: re-import counts relationships_skipped, creates nothing. - ensure_labels_and_labelset non-string label_type coercion. - _queue_embeddings_for_unlocked_document: both dispatch-failure except paths swallowed; structural dispatch skipped when no set. - bulkDocumentUploadStatus non-eager success branch (eager fast-path shadows it under test settings; override_settings flips it off). Frontend (50 lines, verified against local CT + unit lcov): - New UploadProgress unit tests: file-count vs streamed-archive progress modes, status texts, statusText override, determinate/ indeterminate variants (file now 11/11 lines, 20/20 branches). - upload-modal.ct.tsx: bulk-upload flow — success with corpus selection and multipart payload assertion, running-job lock state, in-flight progress, failure with backend detail / empty detail fallback, completed-with-errors, failed with empty errors fallback. - bulk-import-modal.ct.tsx: job success summary + Close reset, job failure with errors banner. 32/32 component tests, full unit suite, lint, tsc, pre-commit green.
ReviewLarge PR (20 commits, ~5.5k additions) combining a new deterministic HTS/ruling-citation enrichment feature for CROSS-style customs corpora with several infra hardening fixes (Celery queue starvation, Docker OOM caps, redis socket-timeout regression) and frontend bulk-upload UX work. Reviewed backend enrichment logic, infra/pipeline changes, and frontend upload UI separately given the scope. Overall the PR is unusually well-instrumented — most behavioral changes are tied to a documented real-world incident ( Customs enrichment / citation resolution (
|
|
Superseded by #2154, which reworks this feature as grammar tiers inside the standard enrichment engine and carries every fix from this branch (ported in b8d43af + b073a42): TXT/SPAN support, path/external_id-derived ruling identity, the series-token legacy citation grammar, the external_id import contract, upversion lineage inheritance, relationship-import idempotency, and the writer span-sentinel/healing fixes. Output equivalence on the 10K official-export benchmark is documented at #2154 (comment) — same 9,377 citation set, same 5,007 resolved pairs, same 11,644-edge graph. Closing unmerged; this branch remains the reference for the original standalone-service implementation and the benchmark handoff doc. |
Summary
local.ymland dedicated Celery queue routing for doc-parse stages to prevent host-wide OOM under bulk ingest load.opencontractserver/enrichment/services/customs_ruling_citation_service.pymatched citation targets against raw document titles, so citation resolution silently failed whenever a title carried a file extension (~98% of the corpus); and the sharedEnrichmentWriter(opencontractserver/enrichment/writer.py) never upgraded aCorpusReferencerow fromUNRESOLVEDtoRESOLVEDon a later successful run, which also affects the general authority-discovery path, not just this service.--limitoption toenrich_customs_rulingsfor fast, fully-complete evaluation passes over a bounded document subset, with parallelized per-document text prefetch (I/O-bound, no ORM state touched).Test plan
pytest opencontractserver/tests/test_customs_ruling_citation_service.py(15/15 passing, incl. new title/extension regression tests)pytest opencontractserver/tests/test_enrichment_writer.py(27/27 passing, incl. new UNRESOLVED→RESOLVED heal regression test)pytest opencontractserver/tests/test_enrichment_in_flight.py(26/26 passing)