- allow to specify when they are wanted on starting a resolve
- match dns cache entries accordingly. An entry which never
tried to get HTTPS-RRs is no answer for a resolve that wants
it.
- fix late arrivals of resolve answers to match the "async"
records that started them - if it still exists.
- provide for multiple "async" resolves in a transfer at the
same time. We may need to resolve an IP interface while the
main connection resolve has not finished yet.
- allow lookup of HTTPS-RR information as soon as it is
available, even if A/AAAA queries are still ongoing.
For this, the "async" infrastructure is changed:
- Defined bits for DNS queries `CURL_DNSQ_A`, `CURL_DNSQ_AAAA`
and `CURL_DNSQ_HTTPS`. These replace `ip_version` which says
nothing about HTTPS.
Use them in dns cache entries for matching.
- enhance the `async->id` to be a unique `uint32_t` for
resolves inside one multi. This is weak, as the id may
wrap around. However it is combined with the `mid` of
the easy handle, making collisions highly unlikely.
`data->state.async` is only accessed in few places where
the mid/async-id match is performed.
- vtls: for ECH supporting TLS backends (openssl, rustls, wolfssl),
retrieve the HTTPS-RR information from the dns connection filter.
Delay the connect if the HTTPS-RR is needed, but has not
been resolved yet.
The implementation of all this is complete for the threaded
resolver. c-ares resolver and DoH do not take advantage of
all new async features yet. To be done in separate PRs.
Details:
c-ares: cleanup settings and initialisation. Any ares channel
is only being created on starting a resolve and propagating
operations in setopt.c to the channel are not helpful.
Changed threaded+ares pollset handling so that they do not
overwrite each others `ASYNC_NAME` timeouts.
Add trace name 'threads' for tracing thread queue and
pool used by threaded resolver.
Closes#21175
- Rename `Curl_resolv_unlink()` to `Curl_dns_entry_unlink()`.
- Change `Curl_dnscache_get()` to return CURLcode result. Returns
now `CURLE_COULDNT_RESOLVE_HOST` for "negative" cache entries.
- Add `Curl_dnscache_add_negative()` to put a "negative" entry
into the cache.
Closes#20864
Also:
- support per-directory and per-upper-directory whitelist entries.
- convert badlist input grep tweak into the above format.
(except for 'And' which had just a few hits.)
- fix many code exceptions, but do not enforce.
(there also remain about 350 'will' uses in lib)
- fix badwords in example code, drop exceptions.
- badwords-all: convert to Perl.
To make it usable from CMake.
- FAQ: reword to not use 'will'. Drop exception.
Closes#20886
Backtrack on previous change that aimed to solve the wrong `share.h`
being included. It turns out it did not fix this issue. At the same time
it introduced relative header filenames and the need to include the same
headers differently depending on the source files' location, reducing
readability and editability.
Replace this method by re-adding curl's lib source directory to the
header path and addressing headers by the their full, relative name to
that base directory. Aligning with this method already used in src and
tests.
With these advantages:
- makes includes easier to read, recognize, grep, sort, write, and copy
between sources,
- syncs the way these headers are included across curl components,
- avoids the ambiguity between system `schannel.h`, `rustls.h` vs.
local headers using the same names in `lib/vtls`,
- silences clang-tidy `readability-duplicate-include` checker, which
detects the above issue,
Ref: https://clang.llvm.org/extra/clang-tidy/checks/readability/duplicate-include.html
- possibly silences TIOBE coding standard warnings:
`6.10.2.a: Don't use relative paths in #include statements.`
- long shot: it works well with concatenated test sources, for
clang-tidy-friendly custom unity builds. Ref: #20667
Slight downside: it's not enforced.
If there happens to be a collision between a local `lib/*.h` header and
a system one, the solution is to rename (possibly with its `.c`
counterpart) into the `curl_` namespace. This is also the method used by
curl in the past.
Also:
- curlx/inet_pton: reduce scope of an include.
- toolx/tool_time: apply this to an include, and update VS project
files accordingly. Also dropping unnecessary lib/curlx header path.
- clang-tidy: enable `readability-duplicate-include`.
Follow-up to 3887069c66#19676
Follow-up to 625f2c1644#16991#16949Closes#20623
Since 9d8998c994, the setopt code changes input DEFAULT to an
actual more specific TLS version (1.2) for the backends to use and check
for.
This means that the default value (0L) cannot and should not actually be
used when the TLS backends run. This change adds asserts to verify that
and removes code that accepts the DEFAULT value as a valid version with
the TLS version functions' logic.
Applications can still set a specific lower version if they want (1, 1.0
or 1.1).
Closes#20453
Add casts to `bool`, or use `bit` type in local variables, where
neccessary to avoid MSVC compiler warnings C4242.
Note: There may remain places needing the above updates, where not
tested in CI, and missed in manual review.
Also:
- urldata: convert struct field `connect_only` to bitfield to match its
counterpart in another struct.
- rename curl-specific `bit` type to `curl_bit`.
Closes#20142
- asyn-thrdd.c: scope an include.
- apply more clang-format suggestions.
- tidy-up PP guard comments.
- delete empty line from the top of headers.
- add empty line after `curl_setup.h` include where missing.
- fix indent.
- CODE_STYLE.md: add `strcpy`.
Follow-up to 8636ad55df#20088
- lib1901.c: drop unnecessary line.
Follow-up to 436e67f65b#20076Closes#20070
- replace `sendf.h` with `curl_trc.h` where it was included just for it.
- drop unused `curl_trc.h` includes.
- easy: delete obsolete comment about `send.h` include reason.
Also:
- move out `curl_trc.h` include from `sendf.h` and include it directly
in users, where not done already. To flatten the include tree and
to less rely on indirect includes.
- stop including `sendf.h` from other headers, replace it with forward
declaration of `Curl_easy`, as done already elsewhere.
Verified with an all non-unity CI run.
Closes#20061
Since this function returns allocated resources there is probably at
least a theoretical risk this can return NULL.
Pointed out by ZeroPath
Closes#19756
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
Change the send parameter from `const void *` to `const uint8_t *` and
adapt calling code. Several had already unsigned chars and were casting.
Closes#19729
The mistake is harmless because it is still a size of a pointer, but
this is the correct pointer.
Acked-by: Daniel McCarney
Reported-by: pelioro on hackerone
Bug: https://hackerone.com/reports/3427460Closes#19545
In init_config_builder_verifier() the call to
rustls_root_cert_store_builder_build() set result on failure but did not
return.
Pointed out by ZeroPath
Closes#19125
For files with sizes using an exact multiple of 256 bytes, the final
successful read(s) filled the buffer(s) and the subsequent fread
returned 0 for EOF, which caused read_file_into to fail.
Now, it needs to return 0 and not be EOF to be an error.
Follow-up to dd95a49d49
Pointed out by ZeroPath
Closes#19104
After this patch, the codebase no longer overrides system printf
functions. Instead it explicitly calls either the curl printf functions
`curl_m*printf()` or the system ones using their original names.
Also:
- drop unused `curl_printf.h` includes.
- checksrc: ban system printf functions, allow where necessary.
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814Closes#18866
Also:
- tests/server: replace local `sstrerror()` with `curlx_strerror()`.
- tests/server: show the error code next to the string, where missing.
- curlx: use `curl_msnprintf()` when building for src and tests.
(units was already using it.)
- lib: drop unused includes found along the way.
- curlx_strerror(): avoid compiler warning (and another similar one):
```
In file included from servers.c:14:
../../lib/../../lib/curlx/strerr.c: In function ‘curlx_strerror’:
../../lib/../../lib/curlx/strerr.c:328:32: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=]
328 | SNPRINTF(buf, buflen, "%s", msg);
| ^
../../lib/../../lib/curlx/strerr.c:47:18: note: ‘snprintf’ output 1 or more bytes (assuming 2) into a destination of size 1
47 | #define SNPRINTF snprintf
| ^
../../lib/../../lib/curlx/strerr.c:328:7: note: in expansion of macro ‘SNPRINTF’
328 | SNPRINTF(buf, buflen, "%s", msg);
| ^~~~~~~~
```
Follow-up to 45438c8d6f#18823Closes#18840
By introducing wrappers for them in the curlx namespace:
`curlx_fopen()`, `curlx_fdopen()`, `curlx_fclose()`.
The undefine/redefine/`(function)()` methods broke on systems
implementing these functions as macros. E.g. AIX 32-bit's `fopen()`.
Also:
- rename `lib/fopen.*` to `lib/curl_fopen.*` (for `Curl_fopen()`)
to make room for the newly added `curlx/fopen.h`.
- curlx: move file-related functions from `multibyte.c` to `fopen.c`.
- tests/server: stop using the curl-specific `fopen()` implementation
on Windows. Unicode isn't used by runtests, and it isn't critical to
run tests on longs path. It can be re-enabled if this becomes
necessary, or if the wrapper receives a feature that's critical for
test servers.
Reported-by: Andrew Kirillov
Bug: https://github.com/curl/curl/issues/18510#issuecomment-3274393640
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#18634
The comments on `cf_recv()` function were outdated and described
calling conventions that no longer are true.
Reported in Joshua's sarif data
Closes#18728
Seen with v21.1.1, non-debug-enabled build:
```
lib/vtls/rustls.c:415:23: error: File position of the stream might be 'indeterminate'
after a failed operation. Can cause undefined behavior [clang-analyzer-unix.Stream,-warnings-as-errors]
415 | const size_t rr = fread(buf, 1, sizeof(buf), f);
| ^
```
Ref: https://github.com/curl/curl/actions/runs/17898248031/job/50887746633?pr=18660#step:11:174
Cherry-picked from #18660Closes#18670
This is a variadic function call with a mismatched argument type; on
platforms where uintptr_t and size_t differ, this invokes undefined
behavior.
Reported in Joshua's sarif data
Closes#18628
`-Weverything` is not enabled by curl, and not recommended by LLVM,
because it may enable experimental options, and will result in new
fallouts after toolchain upgrades. This patch aims to fix/silence as much
as possible as found with llvm/clang 21.1.0. It also permanently enables
warnings that were fixed in source and deemed manageable in the future.
`-Wformat` warnings are addressed separately via #18343.
Fix/silence warnings in the source:
- typecheck-gcc.h: fix `-Wreserved-identifier`.
- lib: silence `-Wcast-function-type-strict`.
For llvm 16+ or Apple clang 16+.
- asyn-ares: limit `HAPPY_EYEBALLS_DNS_TIMEOUT` to old c-ares versions.
- curl_trc: fix `-Wc++-hidden-decl`.
- doh: fix `-Wc++-keyword`.
- ftp: fix `-Wreserved-identifier`.
- ldap: fix `-Wreserved-identifier`.
- mqtt: comment unused macro to avoid warning.
- multi_ev: drop unused macros to avoid warnings.
- setopt: fix useless `break;` after `return;`.
- gtls, mbedtls, rustls: silence `-Wconditional-uninitialized`.
- socks_sspi, schannel, x509asn1: fix `-Wimplicit-int-enum-cast`.
- x509asn1: fix `-Wc++-keyword`.
- openssl: scope `OSSL_UI_METHOD_CAST` to avoid unused macro warning.
- libssh2, wolfssl: drop unused macros.
- curl_ngtcp2, curl_quiche, httpsrr, urlapi: drop/limit unused macros.
- tool_getparam: fix useless `break;` after `return;` or `break;`.
Not normally enabled because it doesn't work with unity.
https://github.com/llvm/llvm-project/issues/71046
- tool_operate: fix `-Wc++-keyword`.
- curlinfo: fix a `-Wunsafe-buffer-usage`.
- tests: silence `-Wformat-non-iso`.
- lib557: fix `-Wreserved-identifier`.
- lib1565: silence `-Wconditional-uninitialized`.
Enable the above clang warnings permanently in picky mode:
- `-Wc++-hidden-decl`
- `-Wc++-keyword` (except for Windows, where it collides with `wchar_t`)
- `-Wcast-function-type-strict`
- `-Wcast-function-type`
- `-Wconditional-uninitialized`
- `-Wformat-non-iso` (except for clang-cl)
- `-Wreserved-identifier`
- `-Wtentative-definition-compat`
Silence problematic `-Weverything` warnings globally (in picky mode):
- `-Wused-but-marked-unused` (88000+ hits) and
`-Wdisabled-macro-expansion` (2600+ hits).
Triggered by `typecheck-gcc.h` when building with clang 14+.
Maybe there exists a way to fix within that header?
Ref: https://discourse.llvm.org/t/removing-wused-but-marked-unused/55310
- `-Wunsafe-buffer-usage`. clang 16+. 7000+ hits.
May be useful in theory, but such high volume of hits makes it
impractical to review and possibly address. Meant for C++.
Ref: https://clang.llvm.org/docs/SafeBuffers.html
Ref: https://stackoverflow.com/questions/77017567/how-to-fix-code-to-avoid-warning-wunsafe-buffer-usage
Ref: https://discourse.llvm.org/t/rfc-c-buffer-hardening/65734
Ref: https://github.com/llvm/llvm-project/pull/111624
- `-Wimplicit-void-ptr-cast`. clang 21+. 1700+ hits.
C++ warning, deemed pure noise.
Ref: https://github.com/curl/curl/issues/18470#issuecomment-3253506266
- `-Wswitch-default` (180+ hits), `-Wswitch-enum` (190+ hits),
`-Wcovered-switch-default` (20+ hits).
Next to impossible to fix cleanly, esp. when the covered `case`
branches depend on compile-time options.
- `-Wdocumentation-unknown-command` (8+ hits).
Triggered in a few sources. Seems arbitrary and bogus.
- `-Wpadded` (550+ hits).
- `-Wc++-keyword` on Windows, where it collides with `wchar_t`.
(100+ hits)
Ref: https://github.com/llvm/llvm-project/issues/155988
- `-Wreserved-macro-identifier`. clang 13+. 5+ hits.
Sometimes it's necessary to set external macros that use
the reserved namespace. E.g. `_CRT_NONSTDC_NO_DEPRECATE`,
`__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__`, `__NO_NET_API`,
possibly `_REENTRANT`, and more.
It's not worth trying to silence them individually.
- `-Wnonportable-system-include-path` with `clang-cl`.
It'd be broken by doing what the warning suggests.
- `-Wformat-non-iso` for clang-cl.
CMake `PICKY_COMPILER=ON` (the default) or `./configure`
`--enable-warnings` (not the default) is required to enable these
silencing rules.
Also:
- autotools, cmake: fix Apple clang and mainline llvm version translations.
Ref: https://en.wikipedia.org/wiki/Xcode#Toolchain_versions
- autotools, cmake: enable `-Warray-compare` for clang 20+.
Follow-up to 4b7accda5a#17196
- cmake: fix to enable `-Wmissing-variable-declarations` at an earlier
clang version.
- cmake: update internal logic to handle warning options with `+` in
them.
- cmake: fix internal logic to match the whole option when looking
into `CMAKE_C_FLAGS` for custom-disabled warnings.
Follow-up to b85cb8cb4e#18485Closes#18477
Added in 2011, but has seen little use in the code. The necessary
compiler feature is missing in some compilers (e.g. MSVC), thus in most
places the portable `(void)` cast is used in addition.
Also:
- vtls/rustls: silence unused argument warning with `(void)`.
Necessary for MSVC, for example.
Ref: ee4ed46128Closes#18455
Add tests of CURLINFO_TLS_SSL_PTR and its returned value in test client
'hx-download'. Use obtained pointer to look up the negotiated TLS
version.
Update manpage of CURLINFO_TLS_SSL_PTR to also describe the behaviour of
wolfSSL similar to OpenSSL. Fix the wolfSSl implementation for TCP to
behave like that. Update the QUIC queries.
Fix rustls `get_internals()` to return the rustls_connection* and not
the address of the pointer.
Assisted-by: Viktor Szakats
Closes#18066
Fixing:
- a raw `free()` in ECH code that's malloced in lib code, causing
an invalid free, also reported by valgrind (in non-unity builds).
And in unity builds adjusted to behave like non-unity via #17827:
Ref: https://github.com/curl/curl/actions/runs/16093372427/job/45421778472?pr=17827#step:39:3321
- a local pair of `malloc()`/`free()` to use curl's memory allocators,
and participate in memory tracking when enabled.
Cherry-picked from #17827Closes#17830
Caught by gcc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0:
```
vtls/rustls.c: In function ‘cr_connect’:
vtls/rustls.c:857:61: warning: ‘%s’ directive argument is null [-Wformat-overflow=]
857 | failf(data, "rustls: must provide certificate with key '%s'",
| ^~
```
Closes#17704
The secure transport vTLS backend was the only Curl_ssl struct instance
that populated the false_start field. Since its removed, we can now
remove that field entirely. This was a protocol feature specific to TLS
1.2 that has been replaced by the more widely adopted TLS 1.3 early data
mechanisms.
--false-start is now deprecated
Closes#17595
