Skip to content

fix: stream ES signed UploadParts to prevent OOM#85

Merged
ServerSideHannes merged 2 commits into
mainfrom
fix/es-signed-upload-oom
Jun 22, 2026
Merged

fix: stream ES signed UploadParts to prevent OOM#85
ServerSideHannes merged 2 commits into
mainfrom
fix/es-signed-upload-oom

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Summary

  • Route signed, known-length, non-chunked UploadParts (e.g. Elasticsearch 16MB snapshot parts) through the framed O(frame)-memory path instead of buffering the whole body
  • Stop pre-reading request bodies in request_handler when x-amz-content-sha256 is present — the SigV4 verifier uses the header hash directly
  • Add classify_upload() unit tests guarding the ES regression

Prod logs showed upload_path: "buffered" with is_large_signed: false for 16MB ES parts, reserving only 8MB while actual peak was ~50MB/request → OOMKilled at 512Mi.

Test plan

  • uv run pytest tests/unit/test_upload_path.py tests/unit/test_routing.py -q
  • uv run pytest tests/unit/ -q (431 passed)
  • Deploy to staging and confirm ES snapshot uploads log upload_path: "framed"
  • Monitor s3proxy-python pods for OOMKilled restarts during ES backup window

Made with Cursor

ServerSideHannes and others added 2 commits June 22, 2026 10:30
Elasticsearch snapshots send 16MB parts with a real x-amz-content-sha256
hash (not UNSIGNED/STREAMING/aws-chunked). Those fell below the 32MB framed
threshold and were fully buffered in both request_handler and UploadPart,
causing OOM under concurrent snapshot load.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ServerSideHannes ServerSideHannes merged commit 2cf8dd4 into main Jun 22, 2026
4 checks passed
@ServerSideHannes ServerSideHannes deleted the fix/es-signed-upload-oom branch June 22, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant