Skip to content

Deterministic HTS/ruling-citation enrichment for CROSS-style customs corpora#2153

Closed
JSv4 wants to merge 20 commits into
mainfrom
feature/customs-ruling-corpus-enrichment
Closed

Deterministic HTS/ruling-citation enrichment for CROSS-style customs corpora#2153
JSv4 wants to merge 20 commits into
mainfrom
feature/customs-ruling-corpus-enrichment

Conversation

@JSv4

@JSv4 JSv4 commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds deterministic (non-LLM) HTS tariff-code and CBP ruling-number citation extraction for CROSS-style customs ruling corpora, plus per-container memory caps in local.yml and dedicated Celery queue routing for doc-parse stages to prevent host-wide OOM under bulk ingest load.
  • Two correctness fixes found while evaluating enrichment quality on a real ~115K-document corpus: opencontractserver/enrichment/services/customs_ruling_citation_service.py matched citation targets against raw document titles, so citation resolution silently failed whenever a title carried a file extension (~98% of the corpus); and the shared EnrichmentWriter (opencontractserver/enrichment/writer.py) never upgraded a CorpusReference row from UNRESOLVED to RESOLVED on a later successful run, which also affects the general authority-discovery path, not just this service.
  • Adds a --limit option to enrich_customs_rulings for fast, fully-complete evaluation passes over a bounded document subset, with parallelized per-document text prefetch (I/O-bound, no ORM state touched).

Test plan

  • pytest opencontractserver/tests/test_customs_ruling_citation_service.py (15/15 passing, incl. new title/extension regression tests)
  • pytest opencontractserver/tests/test_enrichment_writer.py (27/27 passing, incl. new UNRESOLVED→RESOLVED heal regression test)
  • pytest opencontractserver/tests/test_enrichment_in_flight.py (26/26 passing)
  • pre-commit (black/isort/flake8/mypy) clean on all touched files
  • Verified live against real data: purged stale pre-fix test artifacts on a real corpus, re-ran enrichment, and confirmed in the actual frontend UI that citations resolve correctly and render as a bidirectional reference graph (HTS annotations highlighted in the PDF viewer; References panel shows correct "cites"/"cited by" links)

JSv4 added 7 commits July 10, 2026 23:03
…oms corpora; fix bulk-ingest ext gate and relationship-embedding bottleneck

- opencontractserver/enrichment/services/customs_ruling_citation_service.py: detects HTS tariff
  codes (plain annotations) and CBP ruling-number citations (CorpusReference rows resolved
  against sibling document titles) from each document's own post-parse text, reusing
  EnrichmentWriter for persistence. Runnable via manage.py enrich_customs_rulings.
  Validated at 100% precision/recall against a 96-doc pilot vs. the source dataset's own
  golden-tested extraction.
- ingest_corpus management command: ext_ok now unions get_convertible_extensions() so files
  eligible for the configured file converter (e.g. Gotenberg -> .doc) are actually accepted,
  matching the command's own docstring.
- embeddings_task.py: calculate_embeddings_for_relationship_batch's explicit-embedder path now
  batches through embed_texts_batch() (shared _batch_embed_items helper, also used by the
  annotation path) instead of one HTTP call per relationship -- a real bottleneck for parsers
  that emit many relationships per document (e.g. Warp-Ingest's heading-hierarchy
  OC_SUBTREE_GROUP rows).
- celery worker start script: ignore .pilot_data so watchfiles doesn't choke on large
  locally-materialized ingest batches.
A sustained bulk-ingest run drove warp-ingest to ~5.7GB with no
per-container memory limit anywhere in local.yml, exhausting host
RAM/swap and causing the kernel OOM-killer to take down unrelated
desktop processes. Caps + restart policy contain future runaway growth
to a single container instead of the whole host.
convert_document_to_pdf, extract_thumbnail, ingest_doc, and
set_doc_lock_state previously shared one queue. On a .doc-heavy bulk
corpus this let the flood of cheap conversion tasks (enqueued for
every document up front) statistically starve the actual
parsing/unlock work, observed at a ~190:1 completion ratio even after
doubling worker concurrency. extract_thumbnail/ingest_doc/
set_doc_lock_state now route to their own doc_parse queue with
dedicated consumer capacity; standard worker startup scripts updated
to subscribe to it so normal (non-bulk) uploads keep working.
Capping only the four heaviest services (docling-parser, warp-ingest,
vector-embedder, celeryworker) left postgres, redis, django, gotenberg,
multimodal-embedder, and docxodus-parser unbounded. Under concurrent
load from a parallelized enrichment run plus a large Celery backlog
drain, their aggregate uncapped memory usage exhausted host RAM/swap
and forced a hard reboot, even though each individually-capped service
stayed under its own ceiling. Every Docker-managed service now has a
mem_limit.
load_document_text_and_layer's S3 fetch + PAWLS parse dominates
per-document cost (~230ms measured) and touches no ORM state, so it's
safe to prefetch across a thread pool while writes stay single-
threaded on the caller. Also adds --limit for a quick, fully-complete
pass over a deterministic subset of a corpus, useful for evaluating
output quality/UX without enriching a corpus end-to-end in one sitting;
citation resolution still considers every document's title in the
corpus, not just the scanned subset.
…lution

title_index was keyed on the raw document title, but ~98% of corpus 95's
real titles retain a file extension (A83482.doc) that the ruling-citation
regex never produces, so every citation into those documents resolved as
unresolved regardless of how many candidates were found.
get_or_create's lookup key (source_annotation, reference_type,
canonical_key) never included the resolution outcome, so a row written
UNRESOLVED before its target existed (or before a resolution bug was
fixed) was never upgraded by a later run that DID find the target —
the run's in-memory resolved count silently diverged from what
persisted. Shared EnrichmentWriter fix, so it also applies to the
general authority/law-reference discovery path, not just customs
rulings.
@claude

claude Bot commented Jul 12, 2026

Copy link
Copy Markdown

Review: Deterministic HTS/ruling-citation enrichment for CROSS-style customs corpora

Nice, well-documented PR — the changelog fragments and code comments make the "why" behind each change easy to follow, and the writer heal-logic fix (with regression test in test_enrichment_writer.py) is a good general-purpose correctness fix independent of the customs feature.

Correctness

normalized_data (incl. ruling_number) is silently dropped for every citation this service persistsopencontractserver/enrichment/services/customs_ruling_citation_service.py:390-414 (_build_citation_resolutions)

Both Resolution(...) constructions build the Candidate with normalized_data={"ruling_number": number}, but never forward it onto the Resolution itself:

cand = Candidate(..., normalized_data={"ruling_number": number})
...
resolutions.append(Resolution(candidate=cand, source_document_id=doc.id, resolution_status=..., target_document_id=...))

Resolution.normalized_data (opencontractserver/enrichment/resolver.py:35) is an independent dataclass field that defaults to {} — it is not derived from candidate.normalized_data automatically. Every other resolver in this codebase forwards it explicitly, e.g. ReferenceResolver.resolve_document (resolver.py:73-90) and resolve_law (resolver.py:63-70) both pass normalized_data=dict(cand.normalized_data).

Because the customs service never does this, EnrichmentWriter._ensure_corpus_reference (writer.py:497: normalized = dict(res.normalized_data)) always computes normalized = {}, so every CorpusReference row this service creates is persisted with normalized_data=None (writer.py:518: normalized or None). The extracted ruling_number is discarded on write and never recoverable from the reference row — despite citations_resolved/citation_candidates in the summary correctly counting it in-memory. Any downstream consumer (GraphQL field, UI tooltip, backfill script) that expects normalized_data.get("ruling_number") — mirroring how exhibit_number is available on the general document-reference path — will get nothing.

Fix: pass normalized_data=dict(cand.normalized_data) in both Resolution(...) calls in _build_citation_resolutions.

Test coverage gap

The 15 new tests in test_customs_ruling_citation_service.py are all SimpleTestCase (no DB) and only exercise the regex/normalization helpers (_normalize_hts, _HTS_TEXT_RE, _RULING_CITE_RE, _ruling_number_from_title). Nothing calls CustomsRulingCitationService.enrich_corpus() end-to-end and asserts on the persisted Annotation/CorpusReference rows — which is almost certainly how the normalized_data bug above got through. The writer-heal regression test added in test_enrichment_writer.py exercises EnrichmentService, not this new service, so the citation-specific write path (HTS annotation creation + dedup, Resolution/CorpusReference construction, the thread-pool prefetch) has zero DB-backed coverage. Given the PR description's own emphasis on validating against a real 96-doc pilot, it'd be worth adding at least one TransactionTestCase/TestCase that runs enrich_corpus() against a small fixture corpus and asserts on the resulting CorpusReference.normalized_data, resolution_status, and target_document_id.

Performance / memory (worth a second look given this PR's own motivation)

enrich_corpus() (customs_ruling_citation_service.py:255-262) uses pool.map(cls._safe_load_text_and_layer, scanned_documents) with a 12-worker ThreadPoolExecutor. concurrent.futures.Executor.map() submits all items to the pool immediately (it builds the full [self.submit(...) for ...] list up front) and yields results in submission order; it does not bound how many completed-but-not-yet-consumed results can be buffered. If per-document consumption (regex matching + _write_hts_annotations's queries + writer.write()'s get_or_create calls) is slower than the ~230ms/doc I/O prefetch the comment cites, the producer will outpace the consumer for the whole run, and completed (doc_text, layer) pairs will accumulate in memory rather than staying capped at ~PREFETCH_WORKERS in flight. On a large, --limit-less run (the PR mentions a 220K-document corpus) this could mean unbounded memory growth inside a single container — precisely the failure mode the rest of this PR's mem_limit changes exist to contain. Worth confirming this was actually run end-to-end without --limit against the full 220K corpus; if not, consider bounding in-flight work explicitly (e.g. a fixed-size sliding window via concurrent.futures.wait(..., return_when=FIRST_COMPLETED) instead of eager map()).

Minor

  • title_index (customs_ruling_citation_service.py:206-208) is built via a plain dict comprehension keyed on _ruling_number_from_title(doc.title). If two documents in the same corpus resolve to the same ruling number (duplicate/re-ingested title), the later one silently wins and citations could resolve to the wrong document with no log line. Probably rare in practice, but a logger.warning on collision would make a real occurrence debuggable instead of silent.
  • ingest_corpus.py's from opencontractserver.pipeline.utils import get_convertible_extensions is a local import inside handle() rather than at module scope — no apparent circular-import reason visible from the diff; a top-level import would be more idiomatic if nothing actually requires deferring it.

Everything else looks solid

  • CorpusDocumentService.get_corpus_documents_visible_to_user(...) is the correct choice here per the repo's documented corpus-access semantics (this service persists derived Annotation/CorpusReference rows from document text, so it needs the MIN(document, corpus) variant, not corpus-as-gate).
  • The Celery queue-routing fix and local.yml mem-limit additions are well justified with concrete numbers (190:1 completion ratio, 5.7GB growth) and match the existing worker-start-script wiring.
  • The EnrichmentWriter UNRESOLVED→RESOLVED heal is a legitimate, narrowly-scoped forward-only fix with good regression coverage, and correctly reuses the existing jurisdiction/authority_type heal pattern in the same method.

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.40945% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
.../src/components/widgets/modals/BulkImportModal.tsx 94.73% 2 Missing ⚠️
...ponents/widgets/modals/UploadModal/UploadModal.tsx 97.61% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

JSv4 added 7 commits July 14, 2026 09:29
The customs-ruling enrichment service skipped every non-PDF document
before detection ran, so the official CROSS bulk export (text/plain)
produced zero output across a 10,000-document corpus
(docs/benchmarks/pr2153-cross-txt-enrichment-handoff.md).

- Anchoring type is now an input to persistence, not an eligibility
  gate: PDF documents keep TOKEN annotations; text documents get
  SPAN_LABEL annotations in the canonical {start, end, text} / page=0
  shape. HTS dedupe matches by label text across both stored span
  representations, so importer sidecar rows are never duplicated.
- EnrichmentWriter's span fallback now emits the canonical span shape
  (page=0 + anchored text, was page=1 with no text), heals pre-fix
  span rows in place on re-run, and reuses the candidate raw text
  instead of re-fetching each document's extract from storage. The
  shape is built once in span_projection.span_annotation_payload with
  the SPAN_NO_PAGE sentinel in constants/annotations.py.
- Canonical ruling identity is path/external_id-derived (cross:
  namespace > active path basename > title stem), shape-validated, and
  ambiguous identities are counted in canonical_id_collisions and left
  unresolved instead of resolving to the last-indexed document.
- Summary metric documents_skipped_not_pdf renamed to
  documents_skipped_unanchorable; HTS rows now record the run's
  Analysis; prefetch concurrency is bounded to the storage pool via
  CUSTOMS_ENRICHMENT_PREFETCH_WORKERS (default pool size - 1).
- DB-backed regression suite in test_customs_ruling_enrichment.py,
  including an official-export-shaped ZIP -> zip-to-corpus ->
  enrichment contract test.

Verified live against the 10K CROSS corpus (PK 96): a 200-document
slice produced 686 HTS span annotations with zero skips and re-ran
idempotently with no S3 pool warnings.
Comment on lines +464 to +465
"HQ H900001 rules that the vehicles are classified under "
"subheading 8703.23.01, HTSUS.",
JSv4 added 4 commits July 15, 2026 03:00
The branch's previously-unpushed commits (50a4602..ffe93ea) had never
run through CI's pre-commit --all-files pass and carried black/isort/
pyupgrade drift plus 8 mypy errors. All fixes are behavior-neutral:

- black/isort/pyupgrade auto-fixes in document_queries.py,
  file_converter.py, test_redis_settings_urls.py, corpuses/models.py,
  test_doc_tasks.py.
- annotations/signals.py and tasks/doc_tasks.py: replaced default-arg
  on_commit lambdas (which mypy cannot infer) with typed module-level
  dispatch helpers bound via functools.partial; same freeze-at-signal
  semantics and dedup task_ids.
- corpuses/models.py: renamed the label-scan loop variable so the later
  Optional lookup no longer collides with its inferred type.
- test_redis_settings_urls.py: load_local_settings returns
  dict[str, Any] (was dict[str, object], not indexable).
- test_embeddings_task.py: annotated the callbacks accumulator.

pre-commit --all-files passes end to end; affected suites
(test_embeddings_task, test_doc_tasks, test_redis_settings_urls,
test_document_queries, test_file_converters, customs enrichment) all pass.
…ases

Closes two follow-ups from the CROSS TXT enrichment handoff
(docs/benchmarks/pr2153-cross-txt-enrichment-handoff.md):

- meta.csv accepts an optional external_id column
  (opencontractserver/utils/metadata_file_parser.py): a producer-namespaced
  identifier (e.g. cross:H022844) stored verbatim on
  DocumentPath.external_id by import_zip_with_folder_structure (new
  external_ids_applied result counter). Values over the field's 512-char
  limit are rejected per-row with a warning, never truncated. Customs
  citation resolution already reads the cross: namespace first, so renamed
  documents keep resolving; the namespace match is case-insensitive
  (CROSS:H022844 was stored but silently ignored).
- import_document's update branch now inherits ingestion_source /
  external_id / ingestion_metadata from the previous path version unless
  the caller supplies fresh values (opencontractserver/documents/
  versioning.py) - move/delete/restore already did; the update branch was
  the only lifecycle op that silently dropped a stamped identity on
  content re-import.
- enrich_customs_rulings summary records per-phase cost separately:
  load_seconds (storage fetches, summed across the prefetch pool),
  match_seconds (regex detection + citation resolution), write_seconds
  (annotation/reference/graph persistence), and a load_failures breakdown
  of documents_skipped_unanchorable. HTS detection was split from
  persistence (_find_hts_matches) so the attribution is exact.

test_no_metadata_columns_warning's asserted warning wording was updated
because the warning now names the third optional column - intentional
contract extension, not a behavior regression.

New tests: external_id parser coverage, an opaque-filename ZIP -> import ->
enrichment resolution test, upversion lineage inheritance/override, the
case-variant namespace, and timing/load_failures assertions.
Full rerun over corpus 96 after the fix: 10,000 scanned / 0 skipped,
37,172 HTS span annotations, 300 citation mentions -> 299 unresolved
references (the slice contains none of its citation targets, as the
handoff predicted), 0 identity collisions, 0 connection-pool warnings
with the bounded prefetch. Wall time 206 minutes, persistence-dominated.
Reference-web analysis of the 10K official-export benchmark (corpus 96)
found 74% of the true document graph missing: ground truth from the
source database is 6,623 in-corpus edges; the export captured 1,727.
Two causes, two fixes (the exporter-side cross-batch fix lands in the
CROSS-Corpus repo):

- Legacy citation grammar: the legacy HQ/NY slice has bare zero-padded
  numeric ruling numbers ("HQ/084665.txt"), cited in text as
  "<series token> <6 digits>" ("HRL 087392") - invisible to the
  prefixed-only grammar. _LEGACY_RULING_CITE_RE mines exactly that
  shape (707 instances per 500 sampled docs, zero false positives
  observed; six digits required because 5 digits after "NY" is a ZIP
  code in 148/149 sampled instances). Identity and citations meet on
  one canonical key (_canonical_ruling_key: prefixed verbatim, bare
  digits zero-stripped). Live 500-doc validation: 163 citation
  candidates, 39 resolved graph edges (previously ~1 candidate, 0
  resolvable).

- relationships.csv import is now idempotent: get_or_create on the
  edge's natural identity (source, target, corpus, label, type)
  instead of blind create, so relationships-only patch ZIPs and re-run
  batches never duplicate edges; duplicates count as
  relationships_skipped.

Corpus 96 was patched live with the 4,896 missing metadata edges via a
relationships-only ZIP: the reference web now shows 6,676 resolved
references across 5,081 connected documents (was 1,741 / 1,996).
@claude

claude Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review

Large PR (72 files, ~4.4k additions) bundling a deterministic HTS/CBP customs-ruling citation enrichment feature with several fixes discovered while stress-testing it against a real ~115K-document corpus (title/extension matching, EnrichmentWriter UNRESOLVED→RESOLVED healing, relationship-import idempotency, version-lineage inheritance, ingest convertible-extension handling) plus Celery queue routing / memory-cap work to prevent OOM under bulk ingest. Reviewed by domain: enrichment/citation logic, infra/queue config, frontend upload UI, and the peripheral ingest/versioning/annotation changes.

Strong points

  • The two headline correctness fixes are solid and well-tested:
    • customs_ruling_citation_service.py's title/extension fix (Path(title).stem, used as a last-priority fallback behind external_id/corpus-path derivation) is covered by test_title_stem_is_backward_compatible_fallback and friends.
    • writer.py::_ensure_corpus_reference's heal is correctly forward-only (never downgrades RESOLVED→UNRESOLVED) and idempotency is explicitly tested.
  • New regexes (_HTS_TEXT_RE, _RULING_CITE_RE, etc.) use bounded quantifiers — no ReDoS risk — with good false-positive guards (ZIP codes, bare years/dollar amounts).
  • The parallel per-document text prefetch in enrich_customs_rulings only does storage I/O off the main thread (no ORM writes), so it sidesteps the SynchronousOnlyOperation pitfall the project's conventions warn about.
  • The versioning.py::import_document lineage-inheritance fix correctly lets explicit caller values win over inherited ones (only inherits keys the caller didn't pass).
  • Queue routing itself is correct: the doc_parse queue name matches exactly across CELERY_TASK_ROUTES (config/settings/base.py) and both local/production worker start scripts' -Q flags, and the routed task dotted-paths match real task definitions with no name= overrides.
  • Changelog fragments are well-formed and match CLAUDE.md's <slug>.<type>.md convention.

Issues to look at

Infra / memory caps (local.yml)

  1. Redis has a new mem_limit: 2g but no backing volume. Under exactly the bulk-ingest load this PR targets, if in-flight Celery messages/results push redis past 2g, the cgroup OOM-kills it and compose restarts it empty — silently dropping the whole queue (celery, worker_uploads, doc_parse) with no surfaced error. Worth either adding persistence, raising the cap with headroom, or at least calling out the tradeoff in the comment (the asymmetry with postgres's WAL-durability justification isn't acknowledged).
  2. celeryworker's mem_limit: 6g is shared with ad-hoc bulk workers (per the comment, celeryworker_bulk reuses this same definition with higher --concurrency for exactly the kind of long-running 115K-doc ingest this PR is designed around). Bumping concurrency without a corresponding cap bump risks the new limit OOM-killing the very bulk worker it's meant to protect — the fix could shift from "OOM kills the host" to "OOM kills the bulk worker mid-ingest," which is better but still worth flagging/testing under real bulk concurrency.
  3. No test asserts the new doc_parse queue routing/consistency between task routes and worker -Q flags — currently only manually verifiable. A regression here (e.g., a future task added without a route) would silently starve rather than fail loudly.
  4. Minor: config/settings/base.py comment says "the later three stages" but four tasks are routed to doc_parse — cosmetic miscount.

Frontend upload UX (importHttp.ts, UploadModal.tsx, BulkImportModal.tsx)
5. Progress bar regression for non-chunked (<50MB) zip uploads: fetch has no upload-progress API, so onProgress now only fires once at 100% completion — sitting at 0% then jumping to done — versus the previous fake setInterval animation that at least gave incremental feedback. Worth confirming this UX tradeoff is intentional.
6. GET_BULK_DOCUMENT_UPLOAD_STATUS polling in both modals has no timeout/retry cap; per the backend's IDOR-guard cache-miss path (config/graphql/document_queries.py), a lost/foreign job ID returns completed:false forever, so a client could poll indefinitely with no error surfaced to the user.
7. New completion-handling logic (cache eviction, success/error branching) in both BulkImportModal.tsx and UploadModal.tsx is largely untested — only the "still running" polling state has component-test coverage; UploadModal.tsx has no .ct.tsx at all.
8. Minor DRY nit: the two modals duplicate ~30 lines of near-identical polling boilerplate — could be a shared hook.
9. Stale docstring on uploadZipFile in useUploadMutations.ts (still says "returns true/false" but now returns ImportZipRestResult).

Test coverage gaps (lower severity, but worth noting since this PR explicitly leans on "verified live against real data")
10. No test for the --limit flag or the thread-pool prefetch mechanism in enrich_customs_rulings.py, despite being an explicit deliverable.
11. No test that re-imports the same relationships.csv twice to confirm the new idempotency fix actually prevents duplicate DocumentRelationship rows — existing tests only cover the pre-existing "missing document" skip path.
12. Three real concurrency/race fixes appear bundled in without their own changelog fragments: deferring annotation/note embedding dispatch to transaction.on_commit (annotations/signals.py), unlock-time embedding dispatch (doc_tasks.py), and the corpus label-set creation race fix (corpuses/models.py::ensure_labels_and_labelset). Per CLAUDE.md's changelog policy, production bug fixes should get a fragment each — these currently ride along under the customs-ruling entries.

Scope note

This PR bundles quite a few logically-distinct fixes (citation enrichment, writer healing, OOM/queue routing, relationship idempotency, lineage inheritance, ingest extension handling, upload progress refactor). The PR description does a good job narrating why they're coupled (discovered while stress-testing the primary feature against a real corpus), and nothing here reads as unrelated scope creep — but given the size, consider whether the OOM/queue-routing infra work and the upload-progress frontend refactor could land as their own PRs in the future for easier review/bisection.

No security concerns identified (no XSS risk in citation rendering — all user content renders as plain JSX text; no new secrets or exposed ports in the compose changes).

Ground truth vs captured analysis (74% of the graph missing), the two
fixes (exporter cross-batch routing in CROSS-Corpus; legacy series-token
citation grammar here), and the final corpus-96 numbers: 9,377 citation
candidates, 5,007 resolved, 11,644 total graph edges across 5,092
connected documents (was 1,741/1,996).
@claude

claude Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review

Reviewed the full diff (72 files, +4413/-267) across three areas: enrichment/citation core logic, infra/ingest plumbing, and frontend upload/intelligence UI. Overall this is a well-structured PR — good changelog fragment discipline, docs updated per the pointer-based convention, and the two headline correctness fixes (title/extension matching, UNRESOLVED→RESOLVED heal) are each backed by a real regression test rather than just a passing-test claim.

Correctness

  • opencontractserver/tasks/import_tasks.py:566 — the new idempotent relationship import uses get_or_create keyed on (source_document, target_document, corpus, annotation_label, relationship_type), with data/creator only in defaults. For NOTES-type relationships this collapses distinct notes between the same document pair into a single row — the model's own docstring says "For NOTES types... multiple notes can exist between the same documents" (opencontractserver/documents/models.py:597). A second, different note between the same two documents with the same label now silently no-ops (counted as relationships_skipped instead of created). This looks like a real regression introduced by the idempotency fix, not just an edge case — worth confirming whether NOTES should be excluded from the new natural-key dedup (e.g. keyed differently, or by content hash) rather than sharing the same get_or_create path as structural relationships. No test exercises two NOTES rows between the same document pair (before or after this change), so it slipped through.

  • opencontractserver/enrichment/services/customs_ruling_citation_service.py_ruling_number_from_title strips only one trailing extension. A double-extension title (e.g. A83482.doc.bak, or anything produced by a backup/export tool) still fails to normalize and silently drops that document from the identity index — the same bug class as this PR's headline fix, one level deeper, and untested.

  • opencontractserver/enrichment/writer.py heal logic (~line 604) — the UNRESOLVED→RESOLVED heal is forward-only and won't re-target a row that's already RESOLVED if the correct target later changes (e.g. a duplicate-identity collision gets corrected upstream). Possibly intentional for concurrency safety, but worth a one-line comment confirming the scope is deliberate, since "never upgrades... on a later successful run" (per the PR description) reads more broadly than what's implemented.

  • local.yml — the redis service's mem_limit: 2g doesn't have an observed-peak citation the way the other new memory caps in this PR do, and it now also carries the new doc_parse queue's traffic alongside celery/worker_uploads. Worth confirming 2g is comfortable under the bulk-ingest scenario this PR is designed for, or the OOM problem this PR fixes elsewhere could resurface in Redis itself.

Nice-to-haves / minor

  • useUploadMutations.ts:176 — stale JSDoc still says uploadZipFile "returns true/false"; it now returns a structured ImportZipRestResult.
  • UploadModal.tsx:169 — after a zip-import job completes there's no explicit "Done/Close" affordance the way BulkImportModal.tsx has; a user could re-click "Upload ZIP" against the same file/corpus.
  • importHttp.ts:570 — for zip files under CHUNK_THRESHOLD_BYTES, onProgress fires only once at completion (0%→100% jump), no interim feedback for moderately large non-chunked archives.
  • customs_ruling_citation_service.py:292ThreadPoolExecutor.map() eagerly submits every scanned document's prefetch with no backpressure; on the PR's own 115K-doc example, a fetch-fast/write-slow imbalance could buffer many decoded texts/PAWLs layers in memory at once. Worth bounding (e.g. chunksize or a bounded queue) if this is meant to scale unattended.
  • No test exercises the new --limit option on enrich_customs_rulings (command or service), despite it being called out specifically in the PR description.

What's solid

  • CustomsRulingCitationService.enrich_corpus correctly uses CorpusDocumentService.get_corpus_documents_visible_to_user (the MIN-permission variant), consistent with the CLAUDE.md guidance that authority-enrichment must not use corpus-as-gate semantics (issue Reconcile CorpusObjsService corpus-as-gate vs GraphQL MIN-permission semantic #1682/Fix authority discovery document visibility #2084).
  • The parallelized prefetch only parallelizes the I/O read; all ORM writes happen sequentially on the consuming thread, so there's no cross-thread transaction/session hazard.
  • Celery queue routing (doc_parse) covers all the relevant task names in both local and production worker -Q lists — no orphaned-queue risk.
  • external_id handling in metadata_file_parser.py correctly rejects (not truncates) values over the 512-char DocumentPath.external_id limit, and is stored/read via ORM — no injection risk. Well covered by test_metadata_file_parser.py::TestExternalIdColumn.
  • The CHANNEL_LAYERS/CELERY_BROKER_URL fix (moving off manual REDIS_URL string-splitting) is a genuine bug fix for multi-digit DB numbers/query params, with a real regression test (test_redis_settings_urls.py) via subprocess.
  • documents/versioning.py lineage inheritance correctly defaults from current_path and lets explicit path_kwargs override, mirroring existing move/delete/restore patterns.
  • Frontend: thorough error handling in importHttp.ts (non-2xx, malformed JSON, missing fields, retryable vs. non-retryable chunk failures), no XSS issues, no Playwright CT split-import violations, and the new IntelligencePanel test genuinely exercises corpus-context-aware navigation rather than being a smoke test.

Test coverage summary

Coverage is generally strong for the two headline fixes and the settings fix, but has three gaps worth closing before merge: the NOTES-relationship idempotency interaction above, the --limit option, and the double-extension title edge case.

codecov/patch flagged 92 uncovered lines across the branch. Backend
(42 lines, all verified hit in a local coverage run):

- enrich_customs_rulings management command (was 0%): named-owner run,
  superuser default + --limit, unknown-owner and no-superuser errors.
- customs service edge branches: defensive unknown-anchor-type skip,
  analysis-FAILED-and-reraise path, explicit
  CUSTOMS_ENRICHMENT_PREFETCH_WORKERS override (and its floor of 1),
  third claimant of an already-ambiguous ruling number, HTS matches
  rejected by normalization, PDF token-projection failure skip.
- create_relationships_from_parsed dedupe: re-import counts
  relationships_skipped, creates nothing.
- ensure_labels_and_labelset non-string label_type coercion.
- _queue_embeddings_for_unlocked_document: both dispatch-failure
  except paths swallowed; structural dispatch skipped when no set.
- bulkDocumentUploadStatus non-eager success branch (eager fast-path
  shadows it under test settings; override_settings flips it off).

Frontend (50 lines, verified against local CT + unit lcov):

- New UploadProgress unit tests: file-count vs streamed-archive
  progress modes, status texts, statusText override, determinate/
  indeterminate variants (file now 11/11 lines, 20/20 branches).
- upload-modal.ct.tsx: bulk-upload flow — success with corpus
  selection and multipart payload assertion, running-job lock state,
  in-flight progress, failure with backend detail / empty detail
  fallback, completed-with-errors, failed with empty errors fallback.
- bulk-import-modal.ct.tsx: job success summary + Close reset, job
  failure with errors banner.

32/32 component tests, full unit suite, lint, tsc, pre-commit green.
@claude

claude Bot commented Jul 18, 2026

Copy link
Copy Markdown

Review

Large PR (20 commits, ~5.5k additions) combining a new deterministic HTS/ruling-citation enrichment feature for CROSS-style customs corpora with several infra hardening fixes (Celery queue starvation, Docker OOM caps, redis socket-timeout regression) and frontend bulk-upload UX work. Reviewed backend enrichment logic, infra/pipeline changes, and frontend upload UI separately given the scope. Overall the PR is unusually well-instrumented — most behavioral changes are tied to a documented real-world incident (docs/benchmarks/pr2153-cross-txt-enrichment-handoff.md) and paired with a regression test.

Customs enrichment / citation resolution (customs_ruling_citation_service.py, writer.py, enrich_customs_rulings.py)

  • Minor/plausible — _RULING_CITE_RE may be broader than intended. The pattern r"\b([A-Z]\d{5,6}|[A-Z]{2}\d{6})\b" matches any single-letter+5-6-digit or two-letter+6-digit token, not just the documented CBP prefixes. Unresolved matches aren't dropped — they still get persisted as UNRESOLVED Annotation/CorpusReference rows. The PR's false-positive measurement ("707 instances, zero false positives") was done for the separate legacy series-token regex on a 500-doc sample; I didn't find an equivalent precision check for this broader pattern, and there's no negative test for realistic non-ruling identifiers (invoice/entry/protest numbers) that share this shape. Worth a quick sanity check against the real corpus, or at least a comment on the intentional trade-off if it's already vetted.
  • Minorown_numbers_by_doc_id (self-citation suppression) includes the low-confidence title-stem candidate along with high-confidence identity candidates. If a legacy document's filename-derived title coincidentally matches another document's real ruling number, a genuine cross-reference into that other document could be silently suppressed. Narrow edge case, currently untested.
  • Checked and confirmed correct: no ReDoS-prone regexes, canonical key derivation is provably collision-free between prefixed/legacy namespaces, ambiguous identities are logged via canonical_id_collisions and excluded rather than resolved "last write wins," the UNRESOLVED→RESOLVED heal in writer.py is heal-forward-only using the existing lookup key (no duplicate-row risk), the prefetch ThreadPoolExecutor is properly scoped with no leaked threads and no cross-thread writes, and permission/service-layer usage correctly goes through CorpusDocumentService.get_corpus_documents_visible_to_user (the MIN-permission variant), matching the CLAUDE.md guidance for enrichment paths that persist verbatim excerpts. Test coverage here is thorough (idempotent re-runs, span/token healing, collision handling, CLI edge cases).

Infra / pipeline (Celery routing, local.yml, redis settings, import/versioning)

  • Minor — new embedding-dispatch calls aren't routed to the new doc_parse queue. _queue_embeddings_for_unlocked_document (opencontractserver/tasks/doc_tasks.py ~L458-494) calls calculate_embedding_for_doc_text.delay(...) and ensure_embeddings_for_corpus.delay(...), but CELERY_TASK_ROUTES in config/settings/base.py only routes extract_thumbnail/ingest_doc/remap_pending_annotations/set_doc_lock_state to doc_parse. These embedding tasks stay on the default queue right next to the convert_document_to_pdf flood this PR is specifically fixing, so the same starvation pattern (190:1 completion ratio observed on the bulk run) could recur for embedding dispatch. Not a correctness bug — failures here are already logged/swallowed — but seems like a gap in an otherwise carefully-reasoned fix and worth a follow-up route entry.
  • Verified correct: the CHANNEL_LAYERS redis config now uses {"address": REDIS_URL, "socket_timeout": None} rather than a bare tuple/URL, matching the documented fix for the issue WebSocket 1011 reconnect churn: channels-redis 4.3.0 incompatible with redis-py 8.0 default socket_timeout=5 #1886 WebSocket 1011 regression, with a solid new regression test (test_redis_settings_urls.py). Queue routing is otherwise consistent across task routes, worker -Q startup flags, and tests. The idempotent relationship-CSV import correctly keys get_or_create on (source, target, corpus, label, type). versioning.py's external_id/ingestion_source inheritance is key-presence-based (not truthiness-based), so an explicit empty string from the caller still overrides correctly — no silent-overwrite bug. metadata_file_parser.py's 512-char external_id cap matches the model field exactly and rejects (doesn't truncate) oversized values per-row.
  • Low-confidence note: local.yml's new mem_limit + restart: unless-stopped pairing means a container that exceeds its cap under load heavier than what was measured will restart-loop rather than fail loudly once — probably an acceptable tradeoff given the caps were derived from real observed peaks with margin, just flagging it as a design choice worth being aware of operationally.

Frontend (upload/import UI)

  • Worth confirming intent — non-chunked ZIP uploads show static 0% progress, then jump to 100%. In frontend/src/utils/importHttp.ts, the chunked upload path (importZipToCorpusChunked, line ~328) reports real incremental progress via completed / totalChunks, but the two non-chunked multipart paths (importDocumentsZipMultipart L571, importZipToCorpusMultipart L627) call onProgress exactly once, with 1, only after fetch() fully resolves. Since chunking only kicks in above UPLOAD.CHUNK_THRESHOLD_BYTES (50MB) while archives up to MAX_IMPORT_ZIP_BYTES (500MB) are accepted, most real-world uploads will show a progress bar pinned at 0% for the entire transfer before jumping to 100% — which reads as "stuck," and contradicts the new prop's own doc comment in UploadProgress.tsx ("Actual byte-transfer progress for a single streamed archive"). Functionally harmless (the job-status polling that follows is correct), but likely worth either using XMLHttpRequest/a readable-stream reader for real byte progress on the non-chunked path too, or falling back to the indeterminate spinner instead of a determinate bar stuck at 0%.
  • Everything else in this area checked out well: the switch from "toast success + close after 1.5s" to real HTTP-202-then-poll-bulkDocumentUploadStatus job tracking is a solid, well-tested correctness fix (this was previously lying about completion); the IntelligencePanel corpus-navigation-context fix correctly guards on activeCorpus?.id === corpusId before using corpus-scoped navigation, avoiding stale-context leaks across corpus switches; no new state that belongs in a Jotai atom, no XSS-risk rendering, no leftover debug code.

Test coverage

Overall very strong for a PR this size — new tests for regex edge cases, idempotent re-runs, redis settings, queue routing, and the async upload-job UI flow. The main coverage gaps worth calling out are the two "minor/plausible" findings above (no negative test for the broad ruling-citation regex shape, no test asserting intermediate progress values during a slow non-chunked upload) — both are cheap to add and would lock in the intended behavior either way.

Summary

No blockers found. The two most actionable items are the missing doc_parse route for the new embedding-dispatch calls (quick fix, keeps the queue-starvation fix complete) and confirming whether the non-chunked upload progress bar behavior is intentional. Nice work overall — the level of production validation (10K/115K-document live runs, documented ground-truth comparisons) backing the enrichment logic is well above what's typical for a PR of this size.

@JSv4

JSv4 commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #2154, which reworks this feature as grammar tiers inside the standard enrichment engine and carries every fix from this branch (ported in b8d43af + b073a42): TXT/SPAN support, path/external_id-derived ruling identity, the series-token legacy citation grammar, the external_id import contract, upversion lineage inheritance, relationship-import idempotency, and the writer span-sentinel/healing fixes. Output equivalence on the 10K official-export benchmark is documented at #2154 (comment) — same 9,377 citation set, same 5,007 resolved pairs, same 11,644-edge graph. Closing unmerged; this branch remains the reference for the original standalone-service implementation and the benchmark handoff doc.

@JSv4 JSv4 closed this Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant