Stop non-reusable hackney_conn processes when a sync request completes (#902)#903
Open
lucianoengel wants to merge 1 commit into
Open
Stop non-reusable hackney_conn processes when a sync request completes (#902)#903lucianoengel wants to merge 1 commit into
lucianoengel wants to merge 1 commit into
Conversation
The synchronous body-completion path (`receiving({call,From}, body, ...)` and
`stream_body`) returned `{next_state, connected, ...}` unconditionally after the
body was read, never consulting `no_reuse` or `should_close_connection/1`. A
connection that must not be reused — flagged `no_reuse` (proxy tunnels, SSL
upgrades, pool disabled) or carrying `Connection: close` — therefore parked in
`connected` forever, pinned alive by its long-lived owner's link, leaking one
`hackney_conn` gen_statem process per request until the node OOMs.
The async streaming path already handles this in `finish_async_streaming/1`;
the sync path (used by e.g. httpoison's blocking requests) did not.
Add `finish_sync_request/3`, mirroring the async logic: for a non-reusable
connection, close the socket and stop the process; otherwise return to
`connected` for keep-alive reuse. Wired into both the `body` and `stream_body`
completion clauses.
Observed on a production node making a few plain-HTTP/1.1 JSON-RPC calls per
second: `hackney_conn` processes climbed 1714 -> 2539 (~1.5/s) with only ~34
live OS ports (thousands of socketless zombies stuck in `connected`). After the
fix the count oscillates near zero, returning to 0 between request bursts and
tracking live OS ports 1:1.
Fixes benoitc#902
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #902.
Problem
The synchronous body-completion path never closes a non-reusable connection when the request completes. After the body is fully read,
receiving({call, From}, body, ...)andreceiving({call, From}, stream_body, ...)both return{next_state, connected, ...}unconditionally — they never consultno_reuseorshould_close_connection/1:{ok, Body, NewData} -> %% Socket still valid - return to connected state {next_state, connected, NewData, [{reply, From, {ok, Body}}]};So a connection flagged
no_reuse(proxy tunnels, SSL upgrades, pool disabled) or carryingConnection: closereturns toconnectedand stays there. When the calling owner is long-lived (a pool worker, a persistent client process), no owner-DOWNever fires, the idle socket produces no{tcp_closed, _}, and there is no path to termination — onehackney_conngen_statem process leaks per request until the node OOMs.The async path already handles this correctly in
finish_async_streaming/1(it stops non-reusable / non-pooled connections). This PR brings the sync path to parity.Fix
Add
finish_sync_request/3, mirroring the async logic, and call it from both thebodyandstream_bodycompletion clauses:Reusable keep-alive connections still return to
connected.Note on the discriminator: I key on
no_reuse orelse should_close_connection/1rather than copyingfinish_async_streaming/1'sshould_close orelse pool_pid == undefined. In practice the leaking connections wereno_reuse: trueand pooled (pool_pidset,should_close: false), so the async condition would not have caught them —no_reuseis the correct signal for "this connection must not be reused." (Arguablyfinish_async_streaming/1should checkno_reusetoo, but I've kept this PR scoped to the observed leak.)Evidence
Observed on a production node making a few plain-HTTP/1.1 JSON-RPC calls per second.
Before —
hackney_connprocess count climbing ~1.5/s (1714 → 2539) while only ~34 OS ports exist; every process:sys.get_state→connected,checkin_info/1→no_reuse => true, sole link ishackney_conn_sup. Socketless zombies.After — sampling the process count every 15s:
The count oscillates near zero, returns to 0 between request bursts, and tracks live OS ports 1:1 (
ports = 32 baseline + conns) — every surviving process is a genuine in-flight request, not an orphan. Spot-checking survivor states returned%{connected: 1}(one in-flight request) vs%{connected: <thousands>}before.Verified against
master(also reproduces on the 4.5.2 release; the sync path is identical). Module compiles clean.