To simplify the directory layout.
- OS400 and vms support move from `packages` to `projects`.
- Windows README and `generate.bat` files move from `projects`
to `projects/Windows`.
Closes#20271
Add infrastructure to colled and dispatch notifications for transfers
and the multi handle in general. Applications can register a callback
and en-/disable notification type the are interested in.
Without a callback installed, notifications are not collected. Same when
a notification type has not been enabled.
Memory allocation failures on adding notifications lead to a general
multi failure state and result in CURLM_OUT_OF_MEMORY returned from
curl_multi_perform() and curl_multi_socket*() invocations.
Closes#18432
Tweaks around handling of --limit-rate:
* tracing: trace outstanding timeouts by name
* multi: do not mark transfer as dirty that have
an EXPIRE_TOOFAST set
* multi: have one static function to asses speed limits
* multi: when setting EXPIRE_TOOFAST remove the transfers
from the dirty set
* progress: rename vars and comment on how speed limit
timeouts are calculated, for clarity
* transfer: when speed limiting, exit the receive loop
after a quarter of the limit has been received, not
on the first chunk received.
* cf-ip-happy.c: clear EXPIRE_HAPPY_EYEBALLS on connect
* scorecard: add --limit-rate parameter to test with
speed limits in effect
Closes#18454
Changed strategy to start up and terminate resolver thread.
When starting up:
Start the thread with mutex acquired, wait for signal from thread that
it started and has incremented the ref counter. Thread set
pthread_cancel() to disabled before that and only enables cancelling
during resolving itself. This assure that the ref counter is correct and
the unlinking of the resolve context always happens.
When shutting down resolving:
If ref counting shows thread has finished, join it, free everything. If
thread has not finished, try pthread_cancel() (non Windows), but keep
the thread handle around.
When destroying resolving:
Shutdown first, then, if the thread is still there and 'quick_exit' is
not set, join it and free everything. This might occur a delay if
getaddrinfo() hangs and cannot be interrupted by pthread_cancel().
Destroying resolving happens when another resolve is started on an
easy handle or when the easy handle is closed.
Add test795 to check that connect timeout triggers correctly
when resolving is delayed. Add debug env var `CURL_DNS_DELAY_MS`
to simulate delays in resolving.
Fix test1557 to set `quick_exit` and use `xxx.invalid` as domain
instead of `nothing` that was leading to hangers in CI.
Closes#18263
Add support for CURLOPT_READFUNCTION with WebSocket urls when *not* in
connect-only mode, e.g. when using curl_multi_perform.
Install the callback function and set CURLOPT_UPLOAD. Return
CURL_READFUNC_PAUSE when having nothing more to send and unpause the
transfer when more data is ready.
This will send the read bytes in a WebSocket BINARY frame.
Add support for this mode in the pytest "ws_data" client and have all
tests run in 'curl_ws_send/recv' and 'peform' mode as well.
Add `curl_ws_start_frame()`. Document, cover in libcurl-ws.md and
explain the READFUNCTION mode for websockets.
Add example `websocket-updown` for this.
Closes#17683
Adds `curl_off_t curl_multi_get_offt(CURLM *multi_handle, CURLMinfo_offt
info)` to the multi interface with enums:
* CURLMINFO_XFERS_CURRENT: current number of transfers
* CURLMINFO_XFERS_RUNNING: number of running transfers
* CURLMINFO_XFERS_PENDING: number of pending transfers
* CURLMINFO_XFERS_DONE: number of finished transfers to read
* CURLMINFO_XFERS_ADDED: total number of transfers added, ever
Add documentation for functions and info enums.
Add use in the curl command line tool to replace two static
variables counting the same "from the outside".
refs #17870Closes#17992
- add 'use warnings' and 'use strict' where missing from Perl scripts.
- fix 'Use of uninitialized value'.
- fix missing declarations.
- test1140.pl: fix 'Possible precedence issue with control flow operator'.
- fix other misc issues.
Most actual errors found during this PR were fixed and merged via
separate PRs.
Likely there are remaining warnings not found and fixed in this PR.
Closes#17877
- vtls: fix unused variable and symbols.
- ftp: fix unused variable.
- http: fix unused variables.
- smtp: fix unsued variable.
- wolfssl: fix unused variable with !proxy.
- libssh: fix unused argument.
- curl_trc: sync guards between declaration and definition.
- curl_trc: add missing guard for `Curl_trc_ssls` when !verbose.
- curl_trc: fix errors with !http + http3.
- curl_trc: fix missing function with !http + nghttp2.
- cf-h2-proxy: disable when !http + nghttp2, to avoid calling undeclared
functions.
- sha256: fix missing declaration in rare configs.
- md4: fix symbol conflict when building GnuTLS together with AWS-LC or
wolfSSL. By prioritizing the latter two. AWS-LC has no option
to disable the clashing symbol. wolfSSL does, but the most seamless is
to skip including GnuTLS's standalone `md4.h` to avoid the clash.
- build: fix errors with !http + nghttp2.
- build: catch !ssl + ssls-export combination in source. Convert
build-level errors to warnings.
- build: fix errors with !http + http3.
- build: fix building curl tool and unit1302 in rare combinations.
By always compiling base64 curlx functions.
- cmake: add `_CURL_SKIP_BUILD_CERTS` internal option.
To disable automatically building certs with the testdeps target.
To improve performance when testing builds.
(used locally to find the failing builds fixed in this PR.)
Closes#17962
Adds a "meta_hash" to each easy handle for keeping special data during
operations. All meta data set needs to add its destructor callback, so
that meta data gets destroyed properly when the easy handle is cleaned
up or reset.
Add data->master_mid for "sub" transfers that belong to a "master" easy
handle. When a "sub" transfer is done, the corresponding "master" can
add a callback to be invoked. Used in DoH name resolution.
DoH: use easy meta hash to add internal structs for DoH name resolution.
One in each in each probe easy handle. When probes are done, response
data is copied from the probe to the initiating easy.
This allows DoH using transfers and their probes to be cleaned up in any
sequence correctly.
Fold DoH cleanup into the Curl_async_shutdown() and Curl_async_destroy()
functions.
Closes#16384
- fold DoH and async HTTPS-RR handling into common code.
have common cleanups, etc. Have a CURLcode result in async
handling to allow HTTPS RR parsing to fail.
- keep target, ipv4hints, ipv6hints, port and echconfig also
when resolving via cares. We need to know `target` and `port`
when evaluating possible ALPN candidates to not go astray.
- add CURL_TRC_DNS for tracing DNS operations
- replace DoH specific tracing with DNS, use doh as alias
for dns in curl_global_tracea()
Closes#16132
Adds the experimental feature `ssls-export` to libcurl and curl for
importing and exporting SSL sessions from/to a file.
* add functions to libcurl API
* add command line option `--ssl-sessions <filename>` to curl
* add documenation
* add support in configure
* add support in cmake
+ add pytest case
Closes#15924
And drop the prefix. This function was not use elsewhere and it should
certainly not be present in libcurl code when not used in the library.
Closes#15796
- avoid regexp in grep to make it run faster.
- add support for parsing Apple `nm` output:
- skip leading underscore from function names.
- pick object name from output.
Closes#15070
Just a tidy up to contain 'ifdef' pollution of common
code parts with implementation specifics.
- remove the ifdef hyper unpausing in easy.c
- add hyper client reader for CURL_CR_PROTOCOL phase
that implements the unpause method for calling
the hyper waker if it is set
Closes#13075
- update client reader documentation
- client reader, add rewind capabilities
- tell creader to rewind on next start
- Curl_client_reset() will keep reader for future rewind if requested
- add Curl_client_cleanup() for freeing all resources independent of
rewinds
- add Curl_client_start() to trigger rewinds
- move rewind code from multi.c to sendf.c and make part of
"cr-in"'s implementation
- http, move the "resume_from" handling into the client readers
- the setup of a HTTP request is reshuffled to follow:
* determine method, target, auth negotiation
* install the client reader(s) for the request, including crlf
conversions and "chunked" encoding
* apply ranges to client reader
* concat request headers, upgrades, cookies, etc.
* complete request by determining Content-Length of installed
readers in combination with method
* send
- add methods for client readers to
* return the overall length they will generate (or -1 when unknown)
* return the amount of data on the CLIENT level, so that
expect-100 can decide if it want to apply itself
* set a "resume_from" offset or fail if unsupported
- struct HTTP has become largely empty now
- rename `Client_reader_*` to `Curl_creader_*`
Closes#13026
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
clarify when and at what level they operate
- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer
setup of `conn->sockfd` and `conn->writesockfd` on which
connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index
as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
naming consistency
- clarify that the special CURLE_AGAIN hangling to return
`CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
when it arrived in more than a single chunk (to be made
into a sperate PR, also)
Added as documented [in
CLIENT-READER.md](5b1f31dfba/docs/CLIENT-READERS.md).
- old `Curl_buffer_send()` completely replaced by new `Curl_req_send()`
- old `Curl_fillreadbuffer()` replaced with `Curl_client_read()`
- HTTP chunked uploads are now formatted in a client reader added when
needed.
- FTP line-end conversions are done in a client reader added when
needed.
- when sending requests headers, remaining buffer space is filled with
body data for sending in "one go". This is independent of the request
body size. Resolves#12938 as now small and large requests have the
same code path.
Changes done to test cases:
- test513: now fails before sending request headers as this initial
"client read" triggers the setup fault. Behaves now the same as in
hyper build
- test547, test555, test1620: fix the length check in the lib code to
only fail for reads *smaller* than expected. This was a bug in the
test code that never triggered in the old implementation.
Closes#12969
This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.
The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.
Curl's protocol handler `readwrite()` method been changed:
```diff
- CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
- const char *buf, size_t blen,
- size_t *pconsumed, bool *readmore);
+ CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+ bool is_eos, bool *done);
```
The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:
* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
finished
This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.
The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.
HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.
The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:
```
static CURLcode readwrite_data(data, ...)
{
do {
Curl_xfer_recv_resp(data, buf)
...
Curl_xfer_write_resp(data, buf)
...
} while(interested);
...
}
```
All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.
All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.
Closes#12480
This should reduce false-positive to almost zero. Checks for presence in
unit tests if --unit is specified, which is intended for debug builds
where unit testing is enabled.
Closes#11932
- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING
checksrc is updated to allow non-year using copyright statements
Closes#10205
Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.
This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.
Closes#8869
Add curl_url_strerror() to convert CURLUcode into readable string and
facilitate easier troubleshooting in programs using URL API.
Extend CURLUcode with CURLU_LAST for iteration in unit tests.
Update man pages with a mention of new function.
Update example code and tests with new functionality where it fits.
Closes#7605
That is functions that are declared global but are not used from outside
of the file in which it is declared. Such functions should be made
static or even at times be removed.
It also verifies that all used curl_ prefixed functions are "blessed"
Closes#3538
