Skip to content

Def 51195 truncate request body#18

Open
quirky4 wants to merge 2 commits into
cloudlinux:mainfrom
quirky4:def-51195-truncate-request-body
Open

Def 51195 truncate request body#18
quirky4 wants to merge 2 commits into
cloudlinux:mainfrom
quirky4:def-51195-truncate-request-body

Conversation

@quirky4

@quirky4 quirky4 commented Jul 23, 2026

Copy link
Copy Markdown

Add Transaction.TruncateRequestBody: release request-body memory after analysis

Problem

Connectors that proxy to an upstream keep a transaction open between the request phase and the logging phase — for a slow backend that's easily dozens of seconds. For that entire window the transaction pins the full buffered request body, although phases 3–5 never re-read it:

  • the body buffer holds the complete body (with the default config fully in memory — SecRequestBodyLimit defaults to 128 MiB and SecRequestBodyInMemoryLimit defaults to the same value, so it never spills to disk);
  • for urlencoded/raw bodies, the REQUEST_BODY variable holds a second full copy as a string.

In practice a parked transaction with an 8 MB body costs ~12 MB of RSS (buffer growth overshoot + variable copy). Under many concurrent slow-backend requests this adds up to the dominant memory consumer, and SecRequestBodyInMemoryLimit doesn't help: the body processor rebuilds the full REQUEST_BODY string from disk anyway.

Proposal

// on types.Transaction
TruncateRequestBody(limit int64) error

Releases the buffered request body once request-phase analysis is done, keeping at most limit bytes as a preview for audit log part C. limit=0 discards the body entirely. Connectors call it right after ProcessRequestBody(); a parked transaction then costs ~90 KB + limit regardless of body size.

Design notes

  • The in-memory branch actually frees memory: the prefix is copied into a fresh buffer instead of using bytes.Buffer.Truncate, which retains the full backing array. The file-backed branch truncates the temp file in place, or closes and removes it for limit=0.
  • REQUEST_BODY is cut to the same prefix via strings.Clone (a plain substring would pin the original backing string). REQUEST_BODY_LENGTH keeps the original received length — it reports what arrived, and audit consumers may want it next to the preview. Parsed derivatives (ARGS_POST, JSON/XML matches) stay intact: they are small and phase 3–5 rules may legitimately use them.
  • Audit part C emits at most limit bytes — a raw prefix; it may cut mid-token.
  • Readers obtained via RequestBodyReader() before truncation fail with types.ErrBodyTruncated: their positions may point beyond the shrunk buffer, and quietly serving the prefix or EOF would turn a contract violation (holding a reader across truncation) into silent data loss.
  • Calling before ProcessRequestBody is an error, except when the rule engine is off or the transaction is already interrupted — states in which lastPhase never advances and request-body analysis will never run, so truncation cannot hide anything from rules.
  • Failure ordering: a failing os.File.Truncate leaves length and readers untouched; a failing Close in the discard path still removes the temp file (at that point nothing else — including Reset() — would ever clean it up).

Also included

  • fix: repair fork-only tests failing under TinyGo and -race — makes the branch's full go test -race ./... green, including the pre-existing TestExpirevar race.

Verification

  • 11 dedicated tests: limit=0 / limit>0 for in-memory and file-backed buffers, backing-allocation actually dropped (capacity assertions), audit part C equals the preview, REQUEST_BODY/REQUEST_BODY_LENGTH/ARGS_POST state, double-call idempotency, early-call error, engine-off/interrupted acceptance, stale-reader invalidation, Close/Truncate failure paths
  • go test -race ./... green, golangci-lint v2.6.2 clean

Artem Murashkin and others added 2 commits July 23, 2026 13:23
Two fork-only tests (both from cloudlinux/coraza 7e054cf) fail in CI but
pass under plain `go test`, so the sync merge did not surface them.

initcol_test.go: gate with `//go:build !tinygo`. The test lives in
package actions_test and uses the `md` mock, which is only defined in
setvar_persistence_test.go — a file already gated `!tinygo`. Under TinyGo
that file drops out and `md` is undefined, breaking compilation. The test
exercises the SESSION persistent collection, the same feature family the
fork already excludes from TinyGo, so gating it matches existing intent.

expirevar_test.go: give each parallel subtest its own action instance.
The test got one expirevarFn via Get("expirevar") and shared it across
eight t.Parallel() subtests, each calling Init(), which writes the
receiver's collection/key/ttl fields — a data race caught by `go test
-race`. It is a test-only defect: in production each rule gets its own
action, Init runs once at compile time, and Evaluate only reads. Also
renamed the three duplicate "missing ttl" subtests for readable output.

Verified: `go vet -tags tinygo ./internal/...` clean; `go test -race`
on internal/actions clean under both CI tag sets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Connectors that keep a transaction open between the request and logging
phases (waiting for the upstream response) pin the full buffered request
body for the lifetime of the parked transaction, although those phases
never re-read it. TruncateRequestBody(limit) releases the body right
after request-phase analysis, keeping at most limit bytes as a preview
for audit log part C.

The in-memory branch copies the prefix into a fresh buffer instead of
bytes.Buffer.Truncate, which retains the backing array (and transaction
pooling would keep that capacity alive past Close). The file-backed
branch truncates the temp file in place, or closes and removes it for
limit=0. The REQUEST_BODY variable is cut to the same prefix via
strings.Clone so the original backing string is released too;
REQUEST_BODY_LENGTH keeps the original received length and parsed
derivatives (ARGS_POST, JSON/XML matches) stay intact.

Readers obtained via RequestBodyReader before truncation are
invalidated and fail with types.ErrBodyTruncated: their positions may
point beyond the shrunk buffer, and quietly serving the prefix or EOF
would turn a contract violation (holding a reader across truncation)
into silent data loss.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@quirky4
quirky4 marked this pull request as ready for review July 23, 2026 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant