fix: segfault when too many confurrent request were sent on macos#260
Draft
imor wants to merge 3 commits into
Draft
fix: segfault when too many confurrent request were sent on macos#260imor wants to merge 3 commits into
imor wants to merge 3 commits into
Conversation
Member
|
@imor Can we prefix this PR as
|
Contributor
Author
|
@steve-chavez yeah I intend to clean this PR up. I added this as a draft so that I don't forget. I only needed this change to run the tests on my local setup. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Bug fix.
What is the current behavior?
The background worker segfaults under normal concurrent HTTP traffic on macOS/BSD (kqueue). Reproduced with as few as 30 concurrent
net.http_getcalls — no timeouts or exotic load required, so this is hittable in production under moderate traffic, not just in stress tests.Because a background worker crash is treated as a backend crash, Postgres kills all other backends and restarts the whole cluster when this happens, so the blast radius extends well beyond the worker itself.
Root cause: in
src/event.c's kqueue backend,multi_socket_cb()tracks which kqueue filters (EVFILT_READ/EVFILT_WRITE) are registered for a socket viaSocketInfo.action, so it knows what toEV_DELETEonce curl signals it's done with that socket (CURL_POLL_REMOVE). That field was only ever set once, at first registration. When curl calls back for the same socket with a different interest — the normal lifecycle of basically every HTTP request (CURL_POLL_OUTwhile sending, thenCURL_POLL_INwhile reading the response) — the code took the "socket already known" branch and never updatedaction, even though it still went ahead and registered the new filter.The result: on cleanup, only the filter matching the stale
actiongets removed from the kqueue; the other one leaks, still pointing (viaudata) at aSocketInfostruct that getspfree()'d anyway. When that dangling kqueue registration later fires,get_socket_fd()dereferences the freed memory to get a socket fd, and that garbage value is passed straight intocurl_multi_socket_action(), corrupting/crashing inside libcurl's own per-socket hash table.Curl_hash_pickis libcurl's internal sockhash lookup used bycurl_multi_socket_action— exactly what you'd expect from handing curl a corrupted/stale socket reference.What is the new behavior?
multi_socket_cb()now computes the actual add/remove delta between the previous and newwhatbitmask on every call, not just at first registration, and keepssock_info->actioncontinuously in sync with what's actually registered in the kqueue.This also correctly handles the "downgrade" case (e.g.
CURL_POLL_INOUT→CURL_POLL_IN), which a naive "just always overwriteaction" fix would still get wrong — it would stop tracking the dropped filter without ever issuing the matchingEV_DELETE, leaking it the same way.The Linux/epoll backend in the same file is unaffected by this bug: it doesn't store a real heap pointer as curl's per-socket userdata (it passes the address of a local stack
boolused only as a non-NULL existence marker, never dereferenced) and always callsEPOLL_CTL_MODwith the currentwhatbits directly, so there's no stale cached bitmask to go wrong. No behavior change on Linux is expected or intended from this PR.