perf: speed up fresh Bitmap.Or sorted interleaved source-only high-key inserts

[perfloop-agent]

Description

Speed up the measured fresh-receiver, sorted interleaved/source-only high-key insertion shapes in Bitmap.Or, without changing the public API or intended bitmap semantics. No general latency improvement is claimed for the measured fresh-overlap, single-interior, or tail-adjacent shapes.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Test improvements
• Build/CI changes

Changes Made

What was changed?

• Added a planned backward metadata merge for sorted Bitmap.Or high-key suffixes.
• Added focused merge, ownership, malformed-order fallback, and fresh-receiver benchmark coverage.

Why was it changed?

• On the measured dense interleaved/source-only insertion shapes, repeated interior source-only high keys previously caused one suffix shift of the aligned keys, containers, and copy-on-write marker slices per key.

How was it changed?

• Count source-only keys, grow the three aligned metadata slices once, and place the suffix backward.
• Preserve the existing equal-key getUnionedWritableContainer path and source-container COW/clone ownership rules.
• Decline the bulk path for unordered decoded metadata and continue through the existing forward merge.

Testing

Recorded checks passed: pairing, environment_consistent, baseline_valid, benchmark_passed, bulk-merge-semantics-and-ownership, bitmap-or-profile-boundary, all-go-packages, gofmt-simplify, unconvert, bulk-merge-tail-ownership-and-unordered-fallback, malformed-metadata-forward-fallback, measurements_sane, metric_present, sample_sufficiency, and sample_independence.

The package-suite check was go test -count=1 ./.... The focused semantic check was go test -count=1 -run '^TestBitmapOrBulkMerge' ..

Formatting

The recorded gofmt-simplify and unconvert checks passed.

Fuzzing

Not run in this case; no fuzzing result is claimed as PR evidence.

Performance Impact

For the primary measured fresh-receiver 4,096-container sorted interleaved/source-only workload, ns/op improved from 6,864,674 to 999,185.5: 85.4445% lower, n=10 per arm, p=1.08251e-05.

The benchmark fixture was authored for this case. The baseline was re-measured interleaved with candidate 8d782a0b121115c2b10a286024120b0e53ba71b7 against 44559cd5c0e01050eb7ee87feb73a1b6ee90cf43 on linux/amd64, go1.26.4, -cpu=1. Every entry below is a labelled baseline → candidate pair, with n=10 per arm.

Selector and outcome	ns/op, baseline → candidate	B/op, baseline → candidate	allocs/op, baseline → candidate
BenchmarkBitmapOrBulkMerge/fresh-append-only-4096 — guardrail held	1228823.5 → 1225280 (0.288365%; p=0.578742)	733136 → 733136 (0%; p=1)	16397 → 16397 (0%; p=1)
BenchmarkBitmapOrBulkMerge/fresh-copy-on-write-interleaved-4096 — guardrail held	5371004 → 657424 (87.7598%; p=1.08251e-05)	626614 → 340022 (45.7366%; p=1.08251e-05)	8203 → 8197 (0.073144%; p=1.08251e-05)
BenchmarkBitmapOrBulkMerge/fresh-interleaved-1024 — guardrail held	745517 → 249378.5 (66.5496%; p=1.08251e-05)	152144 → 113744 (25.2392%; p=1.08251e-05)	4106 → 4103 (0.0730638%; p=1.08251e-05)
BenchmarkBitmapOrBulkMerge/fresh-interleaved-4096 — primary ns/op won; allocation guardrails held	6864674 → 999185.5 (85.4445%; p=1.08251e-05)	733137 → 446544 (39.0913%; p=1.08251e-05)	16397 → 16391 (0.0365921%; p=1.08251e-05)
BenchmarkBitmapOrBulkMerge/fresh-interleaved-64 — guardrail held	28418 → 16489.5 (41.9752%; p=1.08251e-05)	7440 → 7440 (0%; p=1)	263 → 263 (0%; p=1)
BenchmarkBitmapOrBulkMerge/fresh-interleaved-65 — guardrail held	29118 → 16630 (42.8876%; p=1.08251e-05)	7572 → 7572 (0%; p=1)	267 → 267 (0%; p=1)
BenchmarkBitmapOrBulkMerge/fresh-overlap-4096 — guardrail held	647421 → 647360.5 (0.00934477%; p=0.795936)	217168 → 217168 (0%; p=1)	12292 → 12292 (0%; p=1)
BenchmarkBitmapOrBulkMerge/fresh-single-interior-4096 — guardrail held	554201.5 → 545615 (1.54935%; p=0.0630128)	290768 → 262224 (9.81676%; p=1.08251e-05)	8199 → 8199 (0%; p=1)
BenchmarkBitmapOrBulkMergeTailAdjacent/fresh-single-interior-4096 — ns/op unconfirmed; B/op and allocs/op guardrails held	392028 → 392595.5 (-0.14476%; p=0.970512)	290768 → 262224 (9.81676%; p=1.08251e-05)	8199 → 8199 (0%; p=1)

Seven selector-level latency guardrails held. The tail-adjacent latency guard is unconfirmed and not statistically significant, rather than a claimed win or held guardrail. The primary workload's B/op and allocs/op series are guardrails; their before/after values are reported above, not presented as additional primary wins.

Trade-off: the sorted fast path performs a sortedness validation before planned placement. If validation declines, the operation uses the existing forward path.

From a clean checkout with ordinary Go tooling, reproduce the primary native benchmark at each revision as follows:

git clone https://github.com/perfloop/roaring.git roaring-repro
cd roaring-repro
git worktree add --detach ../roaring-baseline 44559cd5c0e01050eb7ee87feb73a1b6ee90cf43
git worktree add --detach ../roaring-candidate 8d782a0b121115c2b10a286024120b0e53ba71b7
for worktree in ../roaring-baseline ../roaring-candidate; do
  (cd "$worktree" && go test -run='^$' -bench='^BenchmarkBitmapOrBulkMerge$/^fresh-interleaved-4096$' -count=1 -benchmem -cpu=1 .)
done

Run the command ten times per worktree for the reported sample count.

Breaking Changes

• None.

Related Issues

• None.

Additional Notes

Implementation changes are in roaring.go and roaringarray.go; focused semantic coverage and benchmark fixtures are in bitmap_or_bulk_merge_test.go, bitmap_or_bulk_merge_regression_test.go, and bitmap_or_bulk_merge_tail_adjacent_bench_test.go.

Prior proof revisions cand_e0zxpjxk2v (2f2d608433f840ff255aa009ce4d50534a4a6366) and cand_mee63wmm3f (a44478d8630cc6ded6d1eaa952a38237dda5b9d7) were inconclusive. cand_qnfnmn9d4z (ea59f145a1bdb72dfddc91aa210e027842eb60f1) and cand_9664d2j0de (c88ff333991cc76ae6072cc11cac5cd74a5e0c10) were validated proof-hardening revisions. The final candidate only clarifies the nested tail-adjacent benchmark leaf name while retaining the sealed compatibility selector.

Full proof: https://app.perfloop.ai/t/oss/case_jyvmgdg1pq

---

This is an automated, human-reviewed Perfloop contribution opened by perfloop-agent. The body links to reproducible proof for the change. The change was benchmarked on perfloop/roaring, an automation fork of RoaringBitmap/roaring; the commit on this PR's head branch carries the proof.

Assisted-by: PerfloopAgent:gpt-5.6-terra