Skip to content

Stop non-reusable hackney_conn processes when a sync request completes (#902)#903

Open
lucianoengel wants to merge 1 commit into
benoitc:masterfrom
lucianoengel:fix-hackney-conn-sync-leak
Open

Stop non-reusable hackney_conn processes when a sync request completes (#902)#903
lucianoengel wants to merge 1 commit into
benoitc:masterfrom
lucianoengel:fix-hackney-conn-sync-leak

Conversation

@lucianoengel

Copy link
Copy Markdown

Fixes #902.

Problem

The synchronous body-completion path never closes a non-reusable connection when the request completes. After the body is fully read, receiving({call, From}, body, ...) and receiving({call, From}, stream_body, ...) both return {next_state, connected, ...} unconditionally — they never consult no_reuse or should_close_connection/1:

{ok, Body, NewData} ->
    %% Socket still valid - return to connected state
    {next_state, connected, NewData, [{reply, From, {ok, Body}}]};

So a connection flagged no_reuse (proxy tunnels, SSL upgrades, pool disabled) or carrying Connection: close returns to connected and stays there. When the calling owner is long-lived (a pool worker, a persistent client process), no owner-DOWN ever fires, the idle socket produces no {tcp_closed, _}, and there is no path to termination — one hackney_conn gen_statem process leaks per request until the node OOMs.

The async path already handles this correctly in finish_async_streaming/1 (it stops non-reusable / non-pooled connections). This PR brings the sync path to parity.

Fix

Add finish_sync_request/3, mirroring the async logic, and call it from both the body and stream_body completion clauses:

finish_sync_request(From, Reply, #conn_data{transport = Transport, socket = Socket,
                                            no_reuse = NoReuse} = Data) ->
    case NoReuse orelse should_close_connection(Data) of
        true ->
            case Socket of
                undefined -> ok;
                _ -> Transport:close(Socket)
            end,
            {stop_and_reply, normal, [{reply, From, Reply}],
             Data#conn_data{socket = undefined}};
        false ->
            {next_state, connected, Data, [{reply, From, Reply}]}
    end.

Reusable keep-alive connections still return to connected.

Note on the discriminator: I key on no_reuse orelse should_close_connection/1 rather than copying finish_async_streaming/1's should_close orelse pool_pid == undefined. In practice the leaking connections were no_reuse: true and pooled (pool_pid set, should_close: false), so the async condition would not have caught them — no_reuse is the correct signal for "this connection must not be reused." (Arguably finish_async_streaming/1 should check no_reuse too, but I've kept this PR scoped to the observed leak.)

Evidence

Observed on a production node making a few plain-HTTP/1.1 JSON-RPC calls per second.

Beforehackney_conn process count climbing ~1.5/s (1714 → 2539) while only ~34 OS ports exist; every process :sys.get_stateconnected, checkin_info/1no_reuse => true, sole link is hackney_conn_sup. Socketless zombies.

After — sampling the process count every 15s:

conns=0  ports=32
conns=4  ports=36
conns=6  ports=38
conns=0  ports=32
conns=0  ports=32
conns=0  ports=32
conns=5  ports=37
conns=5  ports=37
conns=0  ports=32
conns=0  ports=32
conns=1  ports=33
conns=2  ports=34

The count oscillates near zero, returns to 0 between request bursts, and tracks live OS ports 1:1 (ports = 32 baseline + conns) — every surviving process is a genuine in-flight request, not an orphan. Spot-checking survivor states returned %{connected: 1} (one in-flight request) vs %{connected: <thousands>} before.

Verified against master (also reproduces on the 4.5.2 release; the sync path is identical). Module compiles clean.

The synchronous body-completion path (`receiving({call,From}, body, ...)` and
`stream_body`) returned `{next_state, connected, ...}` unconditionally after the
body was read, never consulting `no_reuse` or `should_close_connection/1`. A
connection that must not be reused — flagged `no_reuse` (proxy tunnels, SSL
upgrades, pool disabled) or carrying `Connection: close` — therefore parked in
`connected` forever, pinned alive by its long-lived owner's link, leaking one
`hackney_conn` gen_statem process per request until the node OOMs.

The async streaming path already handles this in `finish_async_streaming/1`;
the sync path (used by e.g. httpoison's blocking requests) did not.

Add `finish_sync_request/3`, mirroring the async logic: for a non-reusable
connection, close the socket and stop the process; otherwise return to
`connected` for keep-alive reuse. Wired into both the `body` and `stream_body`
completion clauses.

Observed on a production node making a few plain-HTTP/1.1 JSON-RPC calls per
second: `hackney_conn` processes climbed 1714 -> 2539 (~1.5/s) with only ~34
live OS ports (thousands of socketless zombies stuck in `connected`). After the
fix the count oscillates near zero, returning to 0 between request bursts and
tracking live OS ports 1:1.

Fixes benoitc#902

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant