To sync names for the same macro logic between lib and src, and to move
it to the curlx namespace, to match `curlx_free()` that it's calling.
Closes#21151
Add test1715 to check proper handling of chunked transfer
encoding in CONNECT responses. Change proxy error code from
56 (RECV_ERROR) for everything to 7 (COULDNT_CONNECT) when
the server response could be read successfully, but establishing
the connection is not possible (http status code wrong).
Adapt several test expectations from 56 to 7.
Closes#21084
Improve the name, type and handling of `data->req.keepon`:
- Rename `keepon` to `io_flags`
- make `io_flags` and `uint8_t` and reposition in struct
- Rename `KEEP_*` defines to `REQ_IO_*`, move to request.h
- Replace all direct bit tests to `CURL_REQ_WANT_*` use
- Replace all direct bit manipulations with new macros
Closes#20905
Convert more `int port` to `uint16_t` port types. Reshuffle ports in
connectdata to save some bytes. Change `conn->destination` format to
- make it more readable and thus usable in tracing
- add the IPv6 scope_id only when not default (global)
and make it resemble more the textual format for IPv6
(e.g. suffix '%<scope_id>')
Closes#20918
Also:
- support per-directory and per-upper-directory whitelist entries.
- convert badlist input grep tweak into the above format.
(except for 'And' which had just a few hits.)
- fix many code exceptions, but do not enforce.
(there also remain about 350 'will' uses in lib)
- fix badwords in example code, drop exceptions.
- badwords-all: convert to Perl.
To make it usable from CMake.
- FAQ: reword to not use 'will'. Drop exception.
Closes#20886
For HTTP/2, add error code description to close failures.
For HTTP/3, add special handling like in HTTP/2 when streams
have been rejected or an error comes during the response body
and we are not interested in the body.
Closes#20207
When the compiler supports C99.
- map logging functions to macro stubs when verbose logging is disabled
and the compiler is C99. Make sure these stubs silence unused variable
warnings for non-variadic arguments.
Before this patch they mapped to function stubs, the same codepath
used for C89 compiler in this configuration.
- introduce new macros to tell the compiler which code to include
when verbose code is active, or inactive:
- `CURLVERBOSE`: defined when verbose code is active.
To enclose blocks of code only used for verbose logging.
- `VERBOSE(statement);`:
compile statement when verbose code is active.
To mark code lines only used for verbose logging.
- `NOVERBOSE(statement);`:
compile statement when verbose code is inactive.
To suppress warnings for arguments passed to logging functions via
printf masks, e.g. `NOVERBOSE((void)ipaddress);`, yet keeping
the warning in verbose builds.
Note these macros are not the same as `CURL_DISABLE_VERBOSE_STRINGS`.
Verbose code is always active in C89 mode (without variadic macro
support).
- drop existing uses of `CURL_DISABLE_VERBOSE_STRINGS` where redundant,
or replace with the above macros. Ending up reducing the number of
`#ifdef`s, and also the number of lines.
Assisted-by: Daniel Stenberg
Assisted-by: Jay Satiro
Reported-by: Dan Fandrich
Fixes#20341
Refs: #12105#12167Closes#20353
When checking a transfer for being expired via `Curl_timeleft_ms()`,
eleminate the `bool connecting` parameter and have the function check
the `mstate` of the transfer instead.
Advantages:
* eleminate the caller needing awareness if the transfer is
connecting or in a later state
* fix pingpong timeout handling to check the correct timeout
during "proto_connect" phases
* avoid using "connecting" timeouts during establishing a secondary
connection (e.g. FTP) since this would use the timestamp from
the original, primary connect and thus be wrong
Reported-by: Wyuer on github
Fixes#20347Closes#20354
Most by moving functions around. Also delete unused ones.
Reducing their number from 83 to 33.
Remaining ones due to:
- circular dependencies.
- H3 code, that I did not attempt to update and likely the above applies.
- static declarations with attributes (`CURL_PRINTF`, `WARN_UNUSED_RESULT`).
- OS400 code.
Closes#20321
- asyn-thrdd.c: scope an include.
- apply more clang-format suggestions.
- tidy-up PP guard comments.
- delete empty line from the top of headers.
- add empty line after `curl_setup.h` include where missing.
- fix indent.
- CODE_STYLE.md: add `strcpy`.
Follow-up to 8636ad55df#20088
- lib1901.c: drop unnecessary line.
Follow-up to 436e67f65b#20076Closes#20070
- drop unused `http.h` includes.
- drop unused `http1.h` include.
- drop unused `http2.h` includes.
- vssh/ssh.h: drop unused `vssh.h` include.
- urldata.h: drop unused protocol includes.
- url: include `smtp.h` directly.
- rtsp.h: include directly where used.
- imap, smtp: drop redundant include, move another from .h to .c.
Verified with an all non-unity CI run.
Closes#20093
To make it available for all files. Drop includes from individual
sources. This header was already included from most sources and not
specific to any internal subsystem.
Also to ensure that two system symbol redefines on Windows (`read()` and
`write()`) get applied to all sources. Move them to `curl_setup.h`.
Closes#20056
- curl_range: replace `sendf.h` with direct header dependency
`curl_trc.h`.
- drop `curl/curl.h` includes from internal sourcees in favor of the
include made from `curl_setup.h`. Replace it with the latter where
it's the only include.
- include `curl_setup.h` before using macros, where missing.
- drop redundant `stdlib.h`, `string.h` includes, in favor of
`curl_setup_once.h` including them.
- drop redundant `limits.h` in favor of `curl_setup.h` including it.
- fake_addrinfo.h: fix typo in comment.
- curl_setup_once.h: drop `stdio.h` in favor of earlier include in
`curl_setup.h`.
- drop stray, unused, `stddef.h` includes.
- memdebug.h: add missing `stddef.h` include. (relying on accidental
includes via other headers before this patch.)
- stddef.h: document why it's included.
- strerr: drop `curl/mprintf.h` in favor of `curl/curl.h` including it
via `curl_setup.h`.
Closes#20027
Use `data->progress.now` as the timestamp of proecssing a transfer.
Update it on significant events and refrain from calling `curlx_now()`
in many places.
The problem this addresses is
a) calling curlx_now() has costs, depending on platform. Calling it
every time results in 25% increase `./runtest` duration on macOS.
b) we used to pass a `struct curltime *` around to save on calls, but
when some method directly use `curx_now()` and some use the passed
pointer, the transfer experienes non-linear time. This results in
timeline checks to report events in the wrong order.
By keeping a timestamp in the easy handle and updating it there, no
longer invoking `curlx_now()` in the "lower" methods, the transfer
can observer a steady clock progression.
Add documentation in docs/internals/TIME-KEEPING.md
Reported-by: Viktor Szakats
Fixes#19935Closes#19961
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
Change the send parameter from `const void *` to `const uint8_t *` and
adapt calling code. Several had already unsigned chars and were casting.
Closes#19729
Add protocol handler flag `PROTOPT_CONN_REUSE` to indicate that the
protocol allows reusing connections for other tranfers. Add that
to all handlers that support it.
Create connections with `conn->bits.close = FALSE` and remove all
the `connkeep()` calls in protocol handlers setup/connect implementations.
`PROTOPT_CONN_REUSE` assures that the default behaviour applies
at the end of a transfer without need to juggle the close bit.
`conn->bits.close` now serves as an additional indication that a
connection cannot be reused. Only protocol handles that allow
reuse need to set it to override the default behaviour.
Remove all `connclose()` and `connkeep()` calls from connection
filters. Filters should not modify connection flags. They are
supposed to run in eyeballing situations where a filter is just
one of many determining the outcome.
Fix http response header handling to only honour `Connection: close`
for HTTP/1.x versions.
Closes#19333
- badwords.pl: add `-a` option to check all lines in source code files.
Before this patch indented lines were skipped (to avoid Markdown code
fences.)
- GHA/checksrc: use `-a` when verifying the source code.
- GHA/checksrc: disable `So` and `But` rules for source code.
- GHA/checksrc: add docs/examples to the verified sources.
- badwords.txt: delete 4 duplicates.
- badwords.txt: group and sort contractions.
- badwords.txt: allow ` url = `, `DIR`, `<file name`.
Closes#19536
Add new functions in `curlx/warnless.h` for controlled type
conversions:
* curlx_uitouz, convert unsigned into to size_t (should always work)
* curlx_uztoso, convert size_t to curl_off_t, capping at CURL_OFF_T_MAX
* curlx_sztouz, convert ssize_t to size_t, return TRUE when ok
* curlx_sotouz_range, convert curl_off_t to size_t interval, capping
values to interval bounds
Remove some unnecesary casts, convert some internal recv functions
to the "return result, have size_t* arg" pattern.
Closes#19495
Rename `Curl_timeleft()` to `Curl_timeleft_ms()` to make the units in
the returned `timediff_t` clear. (We used to always have ms there, but
with QUIC started to sometimes calc ns as well).
Rename some assigned vars without `_ms` suffix for clarity as well.
Closes#19486
Some internal functions always return CURLE_OK.
- Curl_http_proxy_get_destination() does that from bb4032a, (2 years
ago) And the original inline code does not need to check the status.
- Curl_wildcard_init() does that from e60fe20. (8 years ago)
- Curl_initinfo() does that from a very beginning.
- Curl_pgrsSetDownloadCounter() did not have a return before 914e49b,
ad051e1 recovered its content (2 years ago) but did not completely
recovered the changes related to it.
- auth_digest_get_qop_values() does that from 676de7f.
This directly changes their type to void and cleaned the remaining
checks for their return value.
Closes#19386
After this patch, the codebase no longer overrides system printf
functions. Instead it explicitly calls either the curl printf functions
`curl_m*printf()` or the system ones using their original names.
Also:
- drop unused `curl_printf.h` includes.
- checksrc: ban system printf functions, allow where necessary.
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814Closes#18866
test_10_08, uploading larger files for a h2 proxy, sporadically fails
with a decrpytion error on received data in AWS-LC. The frequency can
be increased by simulated network receive blocks.
Not setting a 4 * TLS record sized buffer, leaving AWS-LC at its
default buffer size seems to mitigate this problem.
Closes#18434
`getsock()` calls operated on a global limit that could
not be configure beyond 16 sockets. This is no longer adequate
with the new happy eyeballing strategy.
Instead, do the following:
- make `struct easy_pollset` dynamic. Starting with
a minimal room for two sockets, the very common case,
allow it to grow on demand.
- replace all protocol handler getsock() calls with pollsets
and a CURLcode to return failures
- add CURLcode return for all connection filter `adjust_pollset()`
callbacks, since they too can now fail.
- use appropriately in multi.c and multi_ev.c
- fix unit2600 to trigger pollset growth
Closes#18164
Add a connection filter query to obtained the negotiated ALPN
protocol to check in setup/protocols how the connection needs
to behave.
Remove the members `alpn` and `proxy_alpn` from `connectdata`.
Closes#17947
- vtls: fix unused variable and symbols.
- ftp: fix unused variable.
- http: fix unused variables.
- smtp: fix unsued variable.
- wolfssl: fix unused variable with !proxy.
- libssh: fix unused argument.
- curl_trc: sync guards between declaration and definition.
- curl_trc: add missing guard for `Curl_trc_ssls` when !verbose.
- curl_trc: fix errors with !http + http3.
- curl_trc: fix missing function with !http + nghttp2.
- cf-h2-proxy: disable when !http + nghttp2, to avoid calling undeclared
functions.
- sha256: fix missing declaration in rare configs.
- md4: fix symbol conflict when building GnuTLS together with AWS-LC or
wolfSSL. By prioritizing the latter two. AWS-LC has no option
to disable the clashing symbol. wolfSSL does, but the most seamless is
to skip including GnuTLS's standalone `md4.h` to avoid the clash.
- build: fix errors with !http + nghttp2.
- build: catch !ssl + ssls-export combination in source. Convert
build-level errors to warnings.
- build: fix errors with !http + http3.
- build: fix building curl tool and unit1302 in rare combinations.
By always compiling base64 curlx functions.
- cmake: add `_CURL_SKIP_BUILD_CERTS` internal option.
To disable automatically building certs with the testdeps target.
To improve performance when testing builds.
(used locally to find the failing builds fixed in this PR.)
Closes#17962
Change the signature of `bufq` functions from
* `ssize_t Curl_bufq_*(..., CURLcode *err)` to
* `CURLcode Curl_bufq_*(..., size_t *pn)`
This allows us to write slightly less code and avoids the ssize_t/size_t
conversions in many cases. Also, it gets the function in line with all
the other send/recv signatures.
Added helper functions in `cfilters.h` for sending from/receving into
a bufq.
Fuzzer now fails to build due to these changes and its testing of
the bufq API.
Closes#17396
Connection filters had a method `get_host()` which had not really been
documented. Since then, the cf had the `query()` method added. Replace
the separate get_host with query.
Add `CF_QUERY_HOST_PORT` as query to connection filters to retrieve
which remote hostname and port the filter (or its sub-filter) is talking
to. The query is implemented by HTTP and SOCKS filters, all others pass
it through.
Add `Curl_conn_get_current_host()` to retrieve the remote host and port
for a connection. During connect, this will return the host the
connection is talking to right now. Before/After connect, this will
return `conn->host.name`.
This is used by SASL authentication.
Closes#17419
Add a bitset `dirty` to the multi handle. The presence of a transfer int
he "dirty" set means: this transfer has something to do ASAP.
"dirty" is set by multiplexing protocols like HTTP/2 and 3 when
encountering response data for another transfer than the current one.
"dirty" is set by protocols that want to be called.
Implementation:
* just an additional `uint_bset` in the multi handle
* `Curl_multi_mark_dirty()` to add a transfer to the dirty set.
* `multi_runsingle()` clears the dirty bit of the transfer at
start. Without new dirty marks, this empties the set after
al dirty transfers have been run.
* `multi_timeout()` immediately gives the current time and
timeout_ms == 0 when dirty transfers are present.
* multi_event: marks all transfers tracked for a socket as dirty.
Then marks all expired transfers as dirty. Then it runs
all dirty transfers.
With this mechanism:
* Most uses of `EXPIRE_RUN_NOW` are replaced by `Curl_multi_mark_dirty()`
* `Curl_multi_mark_dirty()` is cheaper than querying if a transfer is
already dirty or set for timeout. There is no need to check, just do it.
* `data->state.select_bits` is eliminated. We need no longer to
simulate a poll event to make a transfer run.
Closes#17662
cfilter/conn: change send/recv function signatures. Unify the
calling/return conventions in our send/receive handling.
Curl_conn_recv(), adjust pnread type
Parameter `pnread` was a `ssize_t *`, but `size_t *` is better since the
function returns any error in its `CURLcode` return value.
Closes#17546
Move curlx_ functions into its own subdir.
The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.
The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.
The strcase defines are not curlx_ functions and should not be used by
tool or server code.
dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.
When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.
Assisted-by: Jay Satiro
Closes#17253
Remove `blocking` argument from cfilter's connect method.
Implement blocking behaviour in Curl_conn_connect() instead for all
filter chains.
Update filters implementations. Several of which did never use the
paramter (QUIC for example). Simplifies connect handling in TLS filters
that no longer need to loop
Fixed a blocking connect call in FTP when waiting on a socket accept()
which only worked because the filter did not implement it.
Closes#16397
