To sync names for the same macro logic between lib and src, and to move
it to the curlx namespace, to match `curlx_free()` that it's calling.
Closes#21151
- allow to specify when they are wanted on starting a resolve
- match dns cache entries accordingly. An entry which never
tried to get HTTPS-RRs is no answer for a resolve that wants
it.
- fix late arrivals of resolve answers to match the "async"
records that started them - if it still exists.
- provide for multiple "async" resolves in a transfer at the
same time. We may need to resolve an IP interface while the
main connection resolve has not finished yet.
- allow lookup of HTTPS-RR information as soon as it is
available, even if A/AAAA queries are still ongoing.
For this, the "async" infrastructure is changed:
- Defined bits for DNS queries `CURL_DNSQ_A`, `CURL_DNSQ_AAAA`
and `CURL_DNSQ_HTTPS`. These replace `ip_version` which says
nothing about HTTPS.
Use them in dns cache entries for matching.
- enhance the `async->id` to be a unique `uint32_t` for
resolves inside one multi. This is weak, as the id may
wrap around. However it is combined with the `mid` of
the easy handle, making collisions highly unlikely.
`data->state.async` is only accessed in few places where
the mid/async-id match is performed.
- vtls: for ECH supporting TLS backends (openssl, rustls, wolfssl),
retrieve the HTTPS-RR information from the dns connection filter.
Delay the connect if the HTTPS-RR is needed, but has not
been resolved yet.
The implementation of all this is complete for the threaded
resolver. c-ares resolver and DoH do not take advantage of
all new async features yet. To be done in separate PRs.
Details:
c-ares: cleanup settings and initialisation. Any ares channel
is only being created on starting a resolve and propagating
operations in setopt.c to the channel are not helpful.
Changed threaded+ares pollset handling so that they do not
overwrite each others `ASYNC_NAME` timeouts.
Add trace name 'threads' for tracing thread queue and
pool used by threaded resolver.
Closes#21175
Avoid never-ending growth.
When adding more entries, it now deletes the first entry in the list,
which is the oldest added entry still held in memory. I decided to avoid
a Least Recently Used concept as I suspect with a list with this many
entries most entries have not been used, and we don't save the timestamp
of recent use anyway.
The net effect might (no matter what) be that the removed entry might
feel a bit "random" in the eyes of the user.
Verify with test 1674
Ref #21183Closes#21190
Avoid never-ending growth.
When adding more entries, it now deletes the first entry in the list,
which is the oldest added entry still held in memory. I decided to avoid
a Least Recently Used concept as I suspect with a list with this many
entries most entries have not been used, and we don't save the timestamp
of recent use anyway.
The net effect might (no matter what) be that the removed entry might
feel a bit "random" in the eyes of the user.
Verify with test 1669.
Reported-by: Geeknik Labs
Fixes#21183Closes#21189
A previous refactor changed the TAB check so that the octet could be
accepted in the 'path', which would cause an invalid line in the saved
cookie file so not possible to read the cookie back. Not terrible
because the path cannot contain a raw tab anyway so it would never match
anyway.
Add test 1685 to verify
Reported-by: Izan on hackerone
Closes#21185
Prior to this patch code used either `HAVE_PTHREAD_H`, or
`HAVE_THREADS_POSIX`, or both, to decide if POSIX Threads support is
present. In effect requiring both to be defined for a consistent build.
Drop detecting and guarding for `pthread.h`, and assume it present when
`HAVE_THREADS_POSIX` is set.
OS/400 had `HAVE_PTHREAD_H` set, but not `HAVE_THREADS_POSIX`, which
possibly left threading disabled in most sources.
Ref: https://pubs.opengroup.org/onlinepubs/009695399/basedefs/pthread.h.html
Ref: 930f2e8227#21144Closes#21158
When creating a temp file in order to later replace an original, copying
over the existing permissions can not be considered safe when the user
running libcurl is not the owner of the existing file.
Closes#21092
socks5_req0_init() rejects hostnames longer than 255 bytes, but the
later cast to unsigned char in socks5_req1_init() has no local
indication that it is safe. Add a DEBUGASSERT and comment to document
the invariant and guard against future refactoring.
Closes#21157
A "broken" SOCKS5 proxy can send an invalid length of the encryption
token, which could cause malloc(0) to be called, which is a "platform
can do what it wants" potential problem.
Resolve this by explicitly checking the length and rejecting the invalid
token before ever attempting to allocate any memory.
Closes#21159
When used together with `USE_OPENSSL`. `USE_QUICHE` always implies
`USE_OPENSSL`.
Also: merge two (now identical) `#if` blocks in `openssl.c`.
Closes#21135
Introduce `toolx_ftruncate()` macro and map it to existing replacements
for non-mingw-w64 Windows and DJGPP, or to `ftruncate` otherwise.
Follow-up to 6041b9b11b#21109Closes#21130
New connection filter `cf-dns` that manages DNS queries. If hands
out addresses and HTTPS-RR records to anyone interested. Used by
HTTPS and IP happy eyeballing.
Information may become available *before* the libcurl "dns entry"
is complete, e.g. all queries have been answered. The cf-ip-happy
filter uses this information to start connection attempts as soon
as the first address is available.
The multi MSTATE_RESOLVING was removed. A new connection always
goes to MSTATE_CONNECTING. The connectdata bit `dns_resolved`
indicates when DNS information is complete. This is used for
error reporting and starting the progress meter.
Removed dns entries `data->state.dns[i]`, as the `cf-dns` filter
now keeps the reference now.
Many minor tweaks for making this work and pass address information
around safely.
Closes#21027
By moving the coexist workaround from vtls/openssl.c to vtls/openssl.h.
This way it also applies to vtls.c (and possibly other sources including
`vtls/openssl.h`), which may need it in unity builds before BoringSSL
header `openssl/ssl.h` pulling in the conflicting symbols and causing
conflicts otherwise.
Seen with build config:
```
-DCURL_USE_SCHANNEL=ON -DCURL_USE_OPENSSL=ON
-DCMAKE_UNITY_BUILD=ON -DCMAKE_UNITY_BUILD_BATCH_SIZE=30
```
Fixing:
```
In file included from _cm-win-boringssl/lib/CMakeFiles/libcurl_object.dir/Unity/unity_5_c.c:7:
In file included from lib/vtls/vtls.c:54:
In file included from lib/vtls/openssl.h:33:
In file included from /path/to/boringssl/_x64-win-ucrt/usr/include/openssl/opensslv.h:18:
In file included from /path/to/boringssl/_x64-win-ucrt/usr/include/openssl/crypto.h:18:
/path/to/boringssl/_x64-win-ucrt/usr/include/openssl/base.h:293:29: error: expected ')'
293 | typedef struct X509_name_st X509_NAME;
| ^
/path/to/llvm-mingw/x86_64-w64-mingw32/include/wincrypt.h:1515:29: note: expanded from macro 'X509_NAME'
1515 | #define X509_NAME ((LPCSTR) 7)
| ^
[...]
```
Ref: 2a92c39a21#20567Closes#21136
The myssh_in_SFTP_READDIR_BOTTOM() function would not store the error
code correctly thus it could be ignored and missed when an error was
returned at that particular moment.
Follow-up to 3c26e6a896
Found by Codex Security
Closes#21122
non-HTTP protocols no longer retry after connection reuse failures
because multi_follow() now requires a handler->follow callback that is
NULL for those protocols. Provide a fallback for plain retries.
Follow-up to 1213c31272
Spotted by Codex Security
Closes#21121
When assembling the URL of a pushed resource over a not-secured
connection, require the scheme to be known and not secure.
Reported-by: xkilua on hackerone
Closes#21113
```
lib/vtls/openssl.c:4238:22: error: ‘SSL_R_SSLV3_ALERT_CERTIFICATE_EXPIRED’ undeclared (first use in this function); did you mean ‘SSL_R_TLS_ALERT_CERTIFICATE_EXPIRED’?
4238 | (reason == SSL_R_SSLV3_ALERT_CERTIFICATE_EXPIRED))) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| SSL_R_TLS_ALERT_CERTIFICATE_EXPIRED
lib/vtls/openssl.c:4238:22: note: each undeclared identifier is reported only once for each function it appears in
```
Ref: https://github.com/curl/curl/actions/runs/23641366299/job/68863072427#step:24:189
Cherry-picked from #21118Closes#21119
memory allocated by libcurl must be freed with curl_free() and vice versa,
memory allocated by the tool itself must be freed with curlx_free().
- dynbuf: free libcurl data with curl_free()
- tool_operate: make sure we get URL using the right memory
- tool_operhlp: free libcurl memory with curl_free()
- tool_operate: free curl_maprintf() pointer with curl_free
- var: data from curlx_base64_decode needs curlx_free
- tool_operate: fix memory juggling in etag handling
- tool_cb_hdr: fix memory area mixups
- tool_operate: another mixup in etag management
- tool_cb_hdr: more memory mixup fixes
- tool_cfgable.c: document some details
- tool_help: show global-mem-debug in -V output
Closes#21099
Replacing the OpenSSL-like compatibility interface, and syncing with other
hashes, when building with wolfSSL.
Also: stop stomping on OpenSSL public MD4 symbols. This makes the wolfSSL
coexist workaround unnecessary, while also syncing up with MD5 sources.
After this patch the only remaining reference to wolfSSL's OpenSSL
compatibility layer is in `lib/curl_ntlm_core.c`.
Closes#21093
The refactoring in #20832 introduced some inconsistencies between
windows and posix handling, pointed out by reviews. Fix them:
- rename `wait_on_nop` back to `extrawait` as it was called before
- use multi_timeout() to shorten the user supplied timeout for
both windows/posix in the same way
- remove the extra multi_timeout() check in the posix function
- Add the multi's wakeup socket for monitoring only when there
are other sockets to poll on or when the caller wants the
extra waiting time.
Closes#21072
Add test1715 to check proper handling of chunked transfer
encoding in CONNECT responses. Change proxy error code from
56 (RECV_ERROR) for everything to 7 (COULDNT_CONNECT) when
the server response could be read successfully, but establishing
the connection is not possible (http status code wrong).
Adapt several test expectations from 56 to 7.
Closes#21084
Use a thread queue and pool for asnyc threaded DNS resolves.
Add pytest test_21_* for verification.
Add `CURLMOPT_RESOLVE_THREADS_MAX` to allow applications to
resize the thread pool used.
Add `CURLMOPT_QUICK_EXIT` to allow applications to skip thread
joins when cleaning up a multi handle. Multi handles in
`curl_easy_perform()` inherit this from `CURLOPT_QUICK_EXIT`.
Add several debug environment variables for testing.
Closes#20936
Verify if lines are not longer than 192 characters. Also verify if lines
have less than 79 repeat spaces (and fix one fallout).
To improve readability by avoiding long lines and to prevent adding
overly long lines with text that may go unnoticed in an editor or diff
viewer.
In addition to pre-existing line length limits: 79 for C, 132 for CMake
sources.
Also:
- spacecheck: fix/harden allowlist regexes.
- spacecheck: tidy-up quotes and simplify escaping.
- spacecheck: allow folding strings with repeat spaces.
- GHA: fix a suppressed shellcheck warning.
- GHA/macos: simplify by dropping brew bundle.
- test1119.pl: precompile a regex.
- FAQ.md: delete very long link to a Windows 7/2008 support article
that's lost it relevance.
Closes#21087
On any `Curl_rlimit_start()` the rate limit needs to reset its
values before calculating the effective step duration and adjust
the tokens/burst per step.
Add two fields to the struct to remember the original values.
Closes#21086
The previously documented version was based on version checks made in
the source. In practice though, curl doesn't build with <5.0.0.
Also:
- bump main internal version check.
- drop superfluous internal version checks.
Closes#21080
