Skip to content

🧪 Declarative fixture + snapshot extraction tests#83

Merged
ubmarco merged 4 commits into
mainfrom
test/declarative-extraction-fixtures
Jun 30, 2026
Merged

🧪 Declarative fixture + snapshot extraction tests#83
ubmarco merged 4 commits into
mainfrom
test/declarative-extraction-fixtures

Conversation

@ubmarco

@ubmarco ubmarco commented Jun 28, 2026

Copy link
Copy Markdown
Member

What

Adds a declarative fixture + snapshot test layer for C/C++/… marker
extraction. Each case is a YAML fixture (input: lang + config + source)
plus a captured JSON snapshot (expected: the normalized extraction output), so
the whole language matrix is covered as data rather than as bespoke test
functions.

Details

  • Harness tests/test_extraction_fixtures.py — parametrizes over
    tests/data/extraction/*.yaml, runs SourceAnalyse on a temp source file,
    normalizes the result to {needs, need_refs, marked_rst, warnings} (1-indexed
    lines), and asserts a single-file JSON snapshot per case (via a new
    ExtractionSnapshotExtension in conftest.py).
  • Fixtures (tests/data/extraction/): default one-liners across all eight
    languages (cpp, c, python, csharp, rust, yaml, go, jsonc); escape-sequence
    resolution; custom one-line configs (bracketed + space-separated); need-id
    references (default + custom marker); four parser warning kinds; and
    @rst/@endrst blocks.
  • Optional per-case extract toggle (subset of oneline/need_refs/rst)
    keeps a case focused so, e.g., a @need-ids: marker isn't also parsed as a
    one-line need.
  • Fixture format + how-to-update are documented in
    tests/data/extraction/README.md.

Scope

Test-only; no library changes. Covers the marker-extraction surface; existing
parser / pipeline / discovery tests are untouched. (newline_in_field is
reachable only at the parser level — noted in warnings.yaml.)

Running

python -m pytest tests/test_extraction_fixtures.py
python -m pytest tests/test_extraction_fixtures.py --snapshot-update   # review + accept changes

@codecov-commenter

codecov-commenter commented Jun 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.14286% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.40%. Comparing base (a441c17) to head (83d6cd1).

Files with missing lines Patch % Lines
tests/conftest.py 80.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #83      +/-   ##
==========================================
+ Coverage   90.82%   91.40%   +0.57%     
==========================================
  Files          33       34       +1     
  Lines        2942     3012      +70     
  Branches      315      322       +7     
==========================================
+ Hits         2672     2753      +81     
+ Misses        168      160       -8     
+ Partials      102       99       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ubmarco added 4 commits June 28, 2026 21:06
Add tests/data/extraction/ with a README describing the declarative
extraction-test format: per-case YAML fixtures (lang/config/source) plus a
normalized JSON snapshot of the extraction output (needs/need_refs/marked_rst/
warnings). Harness + fixtures + snapshots follow in subsequent commits.
Add a parametrized harness that loads tests/data/extraction/*.yaml cases
(lang/config/source), runs SourceAnalyse on a temp source file, normalizes the
output to {needs,need_refs,marked_rst,warnings} (1-indexed lines) and asserts a
single-file JSON snapshot per case (ExtractionSnapshotExtension).

Fixtures: default one-liners across all eight languages (cpp,c,python,csharp,
rust,yaml in oneline.yaml; go,jsonc in extra_languages.yaml).
…ixtures

Add declarative fixtures + JSON snapshots for: escape-sequence resolution
(escapes.yaml), custom one-line configs incl bracketed + space-separated
(custom_config.yaml), need-id references with default/custom markers
(need_refs.yaml), four parser warning kinds (warnings.yaml), and @rst/@endrst
blocks (marked_rst.yaml).

Add an 'extract' fixture toggle (subset of oneline/need_refs/rst, default all) so
a focused case isn't polluted by the '@' one-line start also matching
'@need-ids:'/'@rst'. 25 cases total; all snapshots captured.
@ubmarco ubmarco force-pushed the test/declarative-extraction-fixtures branch from f6e0be6 to 83d6cd1 Compare June 28, 2026 19:06

@ubmarco ubmarco left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — test-only, no library changes, and the data-as-tests approach is a good fit for the extraction surface.

Went through:

  • the harness (tests/test_extraction_fixtures.py): parametrization over tests/data/extraction/*.yaml, the SourceAnalyse run on a temp source file, and the _normalize reduction to {needs, need_refs, marked_rst, warnings} with 1-indexed lines and stable sorting;
  • the single-file JSON snapshots via ExtractionSnapshotExtension in conftest.py;
  • all eight fixtures + their snapshots: the language matrix (cpp / c / python / csharp / rust / yaml / go / jsonc), escape-sequence resolution, the bracketed and space-separated custom one-line configs, need-id refs (default + custom marker), the four parser warning kinds, and the @rst/@endrst blocks;
  • the README.md format + how-to-update docs.

The per-case extract toggle is a nice touch — scoping a @need-ids: case to [need_refs] so the @ start isn't also picked up as a one-line need avoids exactly the kind of cross-talk that would otherwise make these fixtures flaky. The note that newline_in_field is only reachable at the parser level is helpful context too.

One optional, non-blocking thought: _build_oneline_style (and the need_id_markers lookup in the test body) silently ignore unknown keys, so a misspelled config key in a fixture — start_seq: for start_sequence:, say — would quietly no-op and the snapshot would capture default behavior instead. Since the fixtures are the source of truth here, it might be worth rejecting unknown config keys so a typo fails loudly.

Nothing blocking from me.

@ubmarco ubmarco marked this pull request as ready for review June 29, 2026 22:52
@ubmarco ubmarco requested a review from chrisjsewell June 29, 2026 23:27
@ubmarco ubmarco changed the title test: declarative fixture+snapshot extraction tests 🧪 Declarative fixture + snapshot extraction tests Jun 29, 2026
@ubmarco ubmarco merged commit 79656f4 into main Jun 30, 2026
13 checks passed
@ubmarco ubmarco deleted the test/declarative-extraction-fixtures branch June 30, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants