curl-curl/src
Stefan Eissing c9b95c0bb3
lib: graceful connection shutdown
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
  involved (each transfer might carry its own) into a temporary list.
  Let each connection cache on the list contribute sockets and
  POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
  them peform their maintenance. This will make another non-blocking
  attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
  their poll events via the callback. When `multi_socket()` is invoked
  for a socket not known by an active transfer, forward this to the
  multi's cache for processing. On closing a connection, remove its
  socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
  which happen without a graceful shutdown and should
  no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
  on FTP up- and downloads.

Closes #13976
2024-06-26 08:33:17 +02:00
..
.checksrc tool: use our own stderr variable 2023-09-28 10:50:56 +00:00
.gitignore copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
CMakeLists.txt cmake: whitespace, formatting/tidy-up in comments 2024-05-27 18:07:10 +02:00
curl.rc (lib)curl.rc: set debug flag also for CURLDEBUG and UNITTESTS 2024-05-22 08:38:12 +02:00
Makefile.am build: untangle CURLDEBUG and DEBUGBUILD macros 2024-05-28 08:12:00 +02:00
Makefile.inc curl: (on linux) add MPTCP support 2024-06-07 10:54:19 +02:00
Makefile.mk Makefile.mk: drop Windows support 2023-12-16 13:12:22 +00:00
mkhelp.pl managen: output tabs for each 8 leading spaces 2024-06-25 16:13:30 +02:00
slist_wc.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
slist_wc.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
terminal.c windows: fix UWP builds, add GHA job 2024-06-05 00:52:24 +02:00
terminal.h curl: make warnings and other messages aware of terminal width 2024-05-28 23:12:32 +02:00
tool_binmode.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_binmode.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_bname.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_bname.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_cb_dbg.c build: enable missing OpenSSF-recommended warnings, with fixes 2023-12-16 13:12:37 +00:00
tool_cb_dbg.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_cb_hdr.c tool_cb_hdr: allow etag and content-disposition for 3xx reply 2024-06-14 09:28:42 +02:00
tool_cb_hdr.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_cb_prg.c curl: make warnings and other messages aware of terminal width 2024-05-28 23:12:32 +02:00
tool_cb_prg.h curl: make warnings and other messages aware of terminal width 2024-05-28 23:12:32 +02:00
tool_cb_rea.c tool_cb_rea: limit rate unpause for -T . uploads 2024-05-02 09:43:30 +02:00
tool_cb_rea.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_cb_see.c tool: move tool_ftruncate64 to tool_util.c 2024-04-25 09:20:38 +02:00
tool_cb_see.h tool: move tool_ftruncate64 to tool_util.c 2024-04-25 09:20:38 +02:00
tool_cb_soc.c curl: (on linux) add MPTCP support 2024-06-07 10:54:19 +02:00
tool_cb_soc.h curl: (on linux) add MPTCP support 2024-06-07 10:54:19 +02:00
tool_cb_wrt.c src: tidy up types, add necessary casts 2024-05-17 12:32:04 +02:00
tool_cb_wrt.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_cfgable.c tool_cfgable: free {proxy_}cipher13_list on exit 2024-05-05 11:13:06 +02:00
tool_cfgable.h tcpkeepalive: add CURLOPT_TCP_KEEPCNT and --keepalive-cnt 2024-06-12 09:31:17 +02:00
tool_dirhie.c windows: use built-in _WIN32 macro to detect Windows 2023-11-22 15:42:25 +00:00
tool_dirhie.h tool: use errorf() for error output 2023-06-01 08:19:11 +02:00
tool_doswin.c windows: fix UWP builds, add GHA job 2024-06-05 00:52:24 +02:00
tool_doswin.h windows: use built-in _WIN32 macro to detect Windows 2023-11-22 15:42:25 +00:00
tool_easysrc.c build: delete/replace clang warning pragmas 2024-01-27 21:19:41 +00:00
tool_easysrc.h build: enable missing OpenSSF-recommended warnings, with fixes 2023-12-16 13:12:37 +00:00
tool_filetime.c windows: fix UWP builds, add GHA job 2024-06-05 00:52:24 +02:00
tool_filetime.h windows: use built-in _WIN32 macro to detect Windows 2023-11-22 15:42:25 +00:00
tool_findfile.c windows: use built-in _WIN32 macro to detect Windows 2023-11-22 15:42:25 +00:00
tool_findfile.h windows: use built-in _WIN32 macro to detect Windows 2023-11-22 15:42:25 +00:00
tool_formparse.c build: enable missing OpenSSF-recommended warnings, with fixes 2023-12-16 13:12:37 +00:00
tool_formparse.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_getparam.c tool_getparam: fix the bsearch call for ip-tos names 2024-06-13 09:45:51 +02:00
tool_getparam.h tool: make parser reject blank arguments if not supported 2024-01-03 23:21:29 +01:00
tool_getpass.c src: tidy up types, add necessary casts 2024-05-17 12:32:04 +02:00
tool_getpass.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_help.c curl: make warnings and other messages aware of terminal width 2024-05-28 23:12:32 +02:00
tool_help.h TLS: add support for ECH (Encrypted Client Hello) 2024-04-16 08:10:53 +02:00
tool_helpers.c src: tidy up types, add necessary casts 2024-05-17 12:32:04 +02:00
tool_helpers.h src: tidy up types, add necessary casts 2024-05-17 12:32:04 +02:00
tool_hugehelp.c.cvs copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_hugehelp.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_ipfs.c tool: follow-up getenv fix 2024-03-31 11:04:00 +02:00
tool_ipfs.h curl: move IPFS code into src/tool_ipfs.[ch] 2023-11-07 07:42:28 +01:00
tool_libinfo.c curl: add support for the IPFS protocols: 2023-09-23 11:02:10 +02:00
tool_libinfo.h curl: add support for the IPFS protocols: 2023-09-23 11:02:10 +02:00
tool_listhelp.c tcpkeepalive: add CURLOPT_TCP_KEEPCNT and --keepalive-cnt 2024-06-12 09:31:17 +02:00
tool_main.c src: tidy up types, add necessary casts 2024-05-17 12:32:04 +02:00
tool_main.h tool: use our own stderr variable 2023-09-28 10:50:56 +00:00
tool_msgs.c curl: make warnings and other messages aware of terminal width 2024-05-28 23:12:32 +02:00
tool_msgs.h build: enable missing OpenSSF-recommended warnings, with fixes 2023-12-16 13:12:37 +00:00
tool_operate.c lib: graceful connection shutdown 2024-06-26 08:33:17 +02:00
tool_operate.h curl: support -w '%{num_retries} 2024-06-11 09:07:43 +02:00
tool_operhlp.c curl: use curl_getenv instead of the curlx_ version 2024-03-30 22:45:47 +01:00
tool_operhlp.h tool_operate: Fix error codes on bad URL & OOM 2023-02-07 08:20:04 +01:00
tool_paramhlp.c tool_paramhlp: remove duplicate assign 2024-04-21 10:51:12 +02:00
tool_paramhlp.h tool_getparam: replace malloc + copy by dynbuf for --data 2024-01-08 22:38:22 +01:00
tool_parsecfg.c src: tidy up types, add necessary casts 2024-05-17 12:32:04 +02:00
tool_parsecfg.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_progress.c tool: use our own stderr variable 2023-09-28 10:50:56 +00:00
tool_progress.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_sdecls.h windows: use built-in _WIN32 macro to detect Windows 2023-11-22 15:42:25 +00:00
tool_setopt.c NTLM_WB: drop support 2024-04-08 13:58:58 +02:00
tool_setopt.h curl: make --libcurl output better CURLOPT_*SSLVERSION 2024-03-14 17:59:22 +01:00
tool_setup.h tool: move tool_ftruncate64 to tool_util.c 2024-04-25 09:20:38 +02:00
tool_sleep.c src: tidy up types, add necessary casts 2024-05-17 12:32:04 +02:00
tool_sleep.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_stderr.c tool: use our own stderr variable 2023-09-28 10:50:56 +00:00
tool_stderr.h tool: use errorf() for error output 2023-06-01 08:19:11 +02:00
tool_strdup.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_strdup.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_urlglob.c build: enable missing OpenSSF-recommended warnings, with fixes 2023-12-16 13:12:37 +00:00
tool_urlglob.h tool_urlglob: use curl_off_t instead of longs 2023-05-31 08:14:41 +02:00
tool_util.c tool: move tool_ftruncate64 to tool_util.c 2024-04-25 09:20:38 +02:00
tool_util.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_version.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_vms.c copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_vms.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
tool_writeout_json.c tool_writeout: bsearch the variable name 2024-06-10 23:12:05 +02:00
tool_writeout_json.h tool_writeout: bsearch the variable name 2024-06-10 23:12:05 +02:00
tool_writeout.c curl: support -w '%{num_retries} 2024-06-11 09:07:43 +02:00
tool_writeout.h curl: support -w '%{num_retries} 2024-06-11 09:07:43 +02:00
tool_xattr.c tool_xattr: "guess" URL scheme if none is provided 2024-03-28 16:28:23 +01:00
tool_xattr.h copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
var.c curl: when allocating variables, add the name into the struct 2024-02-07 23:11:40 +01:00
var.h curl: when allocating variables, add the name into the struct 2024-02-07 23:11:40 +01:00