Use ls-hpack's fast Huffman decoder for HPACK/QPACK strings#13259
Open
phongn wants to merge 3 commits into
Open
Use ls-hpack's fast Huffman decoder for HPACK/QPACK strings#13259phongn wants to merge 3 commits into
phongn wants to merge 3 commits into
Conversation
The vendored ls-hpack code only ported the conservative 4-bit FSM decoder (lshpack_dec_huff_decode_full), although the 16-bit decode table it needs (hdecs) has been in huff-tables.h all along. Port the fast decoder, which emits up to 3 bytes per table lookup and falls back to the full decoder for codes longer than 16 bits. Decoding typical header values is about 2x faster (see the new tools/benchmark/benchmark_HuffmanDecode.cc). One deliberate divergence from upstream ls-hpack: the tail check also rejects padding of 8 or more bits, per RFC 7541 section 5.2, keeping the strictness of the FSM decoder. New differential tests assert the two decoders agree on validity and content for exhaustive short inputs and seeded fuzz. The only permitted difference is an exactly-sized destination, where the FSM decoder can spuriously report LSHPACK_ERR_MORE_BUF; ATS callers always allocate twice the encoded length, so they are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Document the RFC 7541 padding divergence in the vendored README and pin it with a deterministic test (the fuzz coverage of that branch was seed dependent). Document huffman_decode's buffer sizing contract in the header. Export LSHPACK_ERR_MORE_BUF as upstream does instead of a test literal. Scope the lib/ include path to the consumers that need it rather than exporting it from the lshpack target. Consolidate the per-byte sentinel assertions, cutting the parity tests' assertion count from 10.7M to 0.4M. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jun 11, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR ports LiteSpeed ls-hpack’s 16-bit table (“fast”) Huffman decoder into ATS’s vendored lib/ls-hpack and switches ATS’s huffman_decode() wrapper (used by both HPACK and QPACK via xpack_decode_string()) to use it, aiming to reduce header decompression CPU cost while preserving ATS’s stricter RFC 7541 padding validation.
Changes:
- Add
lshpack_dec_huff_decode()(fast decoder) to the vendored ls-hpack code and wirehuffman_decode()to use it instead of the 4-bit FSM decoder. - Add extensive parity / fuzz / strict-padding regression tests to ensure the fast decoder matches the existing behavior (including ATS’s deliberate stricter padding rule).
- Add a Catch2 benchmark target to compare full vs fast decoder performance.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tools/benchmark/CMakeLists.txt | Adds the new Huffman decode micro-benchmark executable target. |
| tools/benchmark/benchmark_HuffmanDecode.cc | New Catch2 benchmark comparing lshpack_dec_huff_decode_full vs lshpack_dec_huff_decode on representative inputs. |
| src/proxy/hdrs/unit_tests/test_Huffmancode.cc | Adds known-vector decode coverage, exhaustive/parity tests, fuzzing, and strict padding regression coverage for the fast decoder. |
| src/proxy/hdrs/HuffmanCodec.cc | Switches ATS huffman_decode() wrapper to call the fast ls-hpack decoder. |
| src/proxy/hdrs/CMakeLists.txt | Ensures unit tests can include vendored ls-hpack headers (lib/). |
| lib/ls-hpack/README.md | Updates referenced upstream version and documents ATS’s deliberate strict-padding divergence. |
| lib/ls-hpack/lshpack.h | Exposes LSHPACK_ERR_MORE_BUF and declares lshpack_dec_huff_decode(). |
| lib/ls-hpack/lshpack.cc | Implements the fast decoder and adds ATS-specific guard rejecting padding ≥ 8 bits. |
| include/proxy/hdrs/HuffmanCodec.h | Documents huffman_decode() sizing expectations (dst must be strictly larger than decoded output). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the home-grown Huffman codec was replaced with vendored LiteSpeed ls-hpack code (#12357), only the conservative 4-bit FSM decoder (
lshpack_dec_huff_decode_full) was ported — althoughhuff-tables.hhas carried the 64K-entry table for upstream's fast decoder all along. This PR ports the fast decoder (lshpack_dec_huff_decode, from ls-hpack v2.3.5) and switcheshuffman_decode()to it.The fast decoder consumes 16 bits of input per table lookup and emits up to 3 bytes, falling back to the FSM decoder for the rare codes longer than 16 bits. HPACK and QPACK share the wrapper through
xpack_decode_string(), so both HTTP/2 and HTTP/3 header decoding benefit. No new memory footprint: thehdecstable has been compiled into the binary since the original vendoring.Performance
tools/benchmark/benchmark_HuffmanDecode.cc(new; build with-DENABLE_BENCHMARKS=ON), release build, Ice Lake:text/css)RFC 7541 strictness (deliberate divergence from upstream ls-hpack)
Differential testing revealed that upstream's fast decoder accepts padding of 8–10 bits when it follows the final symbol near the end of the input. RFC 7541 §5.2 requires that "a padding strictly longer than 7 bits MUST be treated as a decoding error", and the FSM decoder has always enforced this. The ported decoder adds a guard in the tail check to keep the strict behavior, so this PR does not loosen what ATS accepts. The divergence is documented in
lib/ls-hpack/README.mdand pinned by the deterministicdecode_overlong_paddingtest, which fails if a future re-sync drops the guard.Semantics and compatibility
dst_len ==decoded length), where either decoder may reportLSHPACK_ERR_MORE_BUFdepending on how trailing padding falls on nibble boundaries. One byte of headroom guarantees success; ATS sizes destinations at 2× the encoded length (strictly larger than any decoded result, since Huffman expansion is at most 8/5), so callers are unaffected. The sizing contract is now documented onhuffman_decode().xpack_decode_string()checkslen < 0).