Now the logic for handling name duplicates and picking the longest
expiry and strictest subdomain is the same for the callback as for when
reading from file.
Also strip trailing dots from the hostname added by the callback.
A minor side-effect is that the hostname provided by the callback can
now enable subdomains by starting the name with a dot, but we discourage
using such hostnames in documentation.
Amended test 1915 to verify.
Closes#21201
To sync names for the same macro logic between lib and src, and to move
it to the curlx namespace, to match `curlx_free()` that it's calling.
Closes#21151
- allow to specify when they are wanted on starting a resolve
- match dns cache entries accordingly. An entry which never
tried to get HTTPS-RRs is no answer for a resolve that wants
it.
- fix late arrivals of resolve answers to match the "async"
records that started them - if it still exists.
- provide for multiple "async" resolves in a transfer at the
same time. We may need to resolve an IP interface while the
main connection resolve has not finished yet.
- allow lookup of HTTPS-RR information as soon as it is
available, even if A/AAAA queries are still ongoing.
For this, the "async" infrastructure is changed:
- Defined bits for DNS queries `CURL_DNSQ_A`, `CURL_DNSQ_AAAA`
and `CURL_DNSQ_HTTPS`. These replace `ip_version` which says
nothing about HTTPS.
Use them in dns cache entries for matching.
- enhance the `async->id` to be a unique `uint32_t` for
resolves inside one multi. This is weak, as the id may
wrap around. However it is combined with the `mid` of
the easy handle, making collisions highly unlikely.
`data->state.async` is only accessed in few places where
the mid/async-id match is performed.
- vtls: for ECH supporting TLS backends (openssl, rustls, wolfssl),
retrieve the HTTPS-RR information from the dns connection filter.
Delay the connect if the HTTPS-RR is needed, but has not
been resolved yet.
The implementation of all this is complete for the threaded
resolver. c-ares resolver and DoH do not take advantage of
all new async features yet. To be done in separate PRs.
Details:
c-ares: cleanup settings and initialisation. Any ares channel
is only being created on starting a resolve and propagating
operations in setopt.c to the channel are not helpful.
Changed threaded+ares pollset handling so that they do not
overwrite each others `ASYNC_NAME` timeouts.
Add trace name 'threads' for tracing thread queue and
pool used by threaded resolver.
Closes#21175
Prior to this patch code used either `HAVE_PTHREAD_H`, or
`HAVE_THREADS_POSIX`, or both, to decide if POSIX Threads support is
present. In effect requiring both to be defined for a consistent build.
Drop detecting and guarding for `pthread.h`, and assume it present when
`HAVE_THREADS_POSIX` is set.
OS/400 had `HAVE_PTHREAD_H` set, but not `HAVE_THREADS_POSIX`, which
possibly left threading disabled in most sources.
Ref: https://pubs.opengroup.org/onlinepubs/009695399/basedefs/pthread.h.html
Ref: 930f2e8227#21144Closes#21158
When used together with `USE_OPENSSL`. `USE_QUICHE` always implies
`USE_OPENSSL`.
Also: merge two (now identical) `#if` blocks in `openssl.c`.
Closes#21135
New connection filter `cf-dns` that manages DNS queries. If hands
out addresses and HTTPS-RR records to anyone interested. Used by
HTTPS and IP happy eyeballing.
Information may become available *before* the libcurl "dns entry"
is complete, e.g. all queries have been answered. The cf-ip-happy
filter uses this information to start connection attempts as soon
as the first address is available.
The multi MSTATE_RESOLVING was removed. A new connection always
goes to MSTATE_CONNECTING. The connectdata bit `dns_resolved`
indicates when DNS information is complete. This is used for
error reporting and starting the progress meter.
Removed dns entries `data->state.dns[i]`, as the `cf-dns` filter
now keeps the reference now.
Many minor tweaks for making this work and pass address information
around safely.
Closes#21027
Verify if lines are not longer than 192 characters. Also verify if lines
have less than 79 repeat spaces (and fix one fallout).
To improve readability by avoiding long lines and to prevent adding
overly long lines with text that may go unnoticed in an editor or diff
viewer.
In addition to pre-existing line length limits: 79 for C, 132 for CMake
sources.
Also:
- spacecheck: fix/harden allowlist regexes.
- spacecheck: tidy-up quotes and simplify escaping.
- spacecheck: allow folding strings with repeat spaces.
- GHA: fix a suppressed shellcheck warning.
- GHA/macos: simplify by dropping brew bundle.
- test1119.pl: precompile a regex.
- FAQ.md: delete very long link to a Windows 7/2008 support article
that's lost it relevance.
Closes#21087
Replace the `volatile int dirty` with a reference counter
protected by a mutex when available.
Solve the problem of when to call application's lock function
by adding a volatile flag that indicates a share has been added
to easy handles in its lifetime. That flag ever goes from
FALSE to TRUE, so volatile might work (in the absence of a mutex).
(The problem is that the lock/unlock functions need 2-3
`curl_share_setopt()` invocations to become usable and there
is no way of telling if the third will ever happen. Calling
the lock function before the 3rd setopt may crash the
application.)
When removing a share from an easy handle (or replacing it with
another share), detach the easy connection on a share with a
connection pool.
When cleaning up a share, allow this even if it is still used in
easy handles. It will be destroyed when the reference count
drops to 0.
Closes#20870
- Rename `Curl_resolv_unlink()` to `Curl_dns_entry_unlink()`.
- Change `Curl_dnscache_get()` to return CURLcode result. Returns
now `CURLE_COULDNT_RESOLVE_HOST` for "negative" cache entries.
- Add `Curl_dnscache_add_negative()` to put a "negative" entry
into the cache.
Closes#20864
- Split WINSOCK and POSIX code in `multi_wait()` as the ifdef'ery
was becoming unreadable
- define `ENABLE_WAKEUP` to mean the wakeup socketpair is enabled,
no additional USE_WINSOCK check needed. Under WINSOCK
`ENABLE_WAKEUP` is not defined, so it's availability is as before
under the double defined() checks
- When the multi handle has "alive" transfers, the admin handle's
pollset include the wakeup receive socket. This results in the
admin handle running when someone uses `curl_multi_wakeup()`.
- Without any "alive" transfers, the wakeup socket is removed from
the pollset. Otherwise, event based processing would never finish,
eg. leave the event loop.
- The wakeup socket was never registered for event processing before,
e.g. `curl_multi_wakeup()` never worked in that mode.
- Adjust test exepectations on socket callback invocations and
number of sockets appearing in waitfds sets.
Closes#20832
- make extract-unit-protos handle multi-line prototypes - but they need
to be above the implementation
- Prototypes for static functions we use in unit tests should not be in
header files. We generate lib/unitprotos.h for this purpose
- Removed some function wrappers written for unit tests and make them
use UNITTEST function directly.
- Renamed time2str() in the tool to timebuf() since we have the same
name in lib/ and in unit tests they can both be used non-static in a
build.
This reverts commit f95fadd116.
Follow-up to #21010Closes#21014
Silencing these warnings (seen on Solaris 11 SPARC GCC 4.9.2 + OpenCSW):
```
lib1521.c: In function 'test_lib1521':
/include/curl/typecheck-gcc.h:93:13: warning: call to 'Wcurl_easy_setopt_err_conv_cb' declared with attribute warning: curl_easy_setopt expects a curl_conv_callback argument
curl_easy_setopt(curl, CURLOPT_CONV_FROM_NETWORK_FUNCTION,
curl_easy_setopt(curl, CURLOPT_CONV_TO_NETWORK_FUNCTION,
curl_easy_setopt(curl, CURLOPT_CONV_FROM_UTF8_FUNCTION,
/include/curl/typecheck-gcc.h:123:13: warning: call to 'Wcurl_easy_setopt_err_interleave_cb' declared with attribute warning: curl_easy_setopt expects a curl_interleave_callback argument
curl_easy_setopt(curl, CURLOPT_INTERLEAVEFUNCTION,
```
Ref: https://curl.se/dev/log.cgi?id=20260319160651-1785427#prob2
Follow-up to de0adda78cCloses#21017
Extend `#pragma diagnostic push`/`pop` guards to the whole codebase
(from tests and examples only) to disable it for GCC <4.6. Rename guard
to `CURL_HAVE_DIAG` and make it include llvm/clang to be interchangeable
with `__GNUC__ || __clang__` in this context.
The above means no longer disabling certain warnings locally, so pair
this with disabling all picky warnings for GCC <4.6.
Also:
- drop global workarounds for misbehaving GCC <4.6 compiler warnings.
Not needed with picky warnings disabled.
Reported-by: fds242 on github
Reported-by: Sergey Fedorov
Thanks-to: Orgad Shaneh
Follow-up to f07a98ae11#20366Fixes#20892Fixes#20924Closes#20902Closes#20907
To include what's actually used.
Also:
- drop unused includes.
- scope includes where possible.
- drop `curlx/curlx.h` umbrella header.
- config2setopts: include `netinet/in.h` for Cygwin/MSYS2.
Previously included by chance via an unused curlx include.
Closes#20776
A logic error made the function not check the last character, which thus
could make it accept invalid schemes.
Added test 1965 to verify
Reported-by: Otis Cui Lei
Closes#20893
It's mostly a filler word. I've read through each use of it in the code
base and did minor rephrasings when "simply" carried some meaning. The
overwhelming majority of cases, removing it improved the text
significantly. Inspired by #20793.
Closes#20822
To allow building test binaries without test certs, replacing
`_CURL_SKIP_BUILD_CERTS` internal option with a build target that is
similar to `testdeps`, but without building the test certificates.
To make building test binaries a little bit faster, with less noisy
output, and without having to reconfigure the build.
Closes#20708
Prefer `sizeof()` over `strlen()`, static const variables over macros.
Add a couple of `NOLINT`s to silence false positives.
Also sync similar code patterns between libtests.
Cherry-picked from #20720Closes#20723
Tests are build in "unity"-style, by including sources into an umbrella
C files (similar to how CMake unity works). This does not play well with
clang-tidy, which seems to unconditionally ignore C sources included
like this. To fix it, curl's CMake implements a manual clang-tidy
support for tests, which compiles sources one-by-one, while also making
sure sources compile cleanly standalone (e.g. all sources need to
include `first.h`). The manual clang-tidy implementation is fragile, and
performance, in particular when targeting Windows, is abysmal.
This patch introduces an alternate solution, enabled by the
`_CURL_TESTS_CONCAT=ON` option. In this mode, umbrella sources include
the actual sources instead of `#including` them. Allowing to use CMake's
built-in clang-tidy support to compile them, with clang-tidy actually
checking the sources. Making the manual clang-tidy support unnecessary.
In the Windows CI job it results in a 4x performance improvement (4m ->
1m), making it practical to run clang-tidy on tests on Windows, in CI.
The main downside is that clang-tidy doesn't understand the `#line`
directive. Meaning issues found show the wrong filename and line number
next to them. It's not impossible to locate errors this way, but also
not convenient.
Minor/potential downside is that the concatenated source needs to be
reassembled each time an original source is updated. This may result in
more copying on the disk when used in local development. The largest
source is 1.4MB, so probably not a show-stopper on most machines.
Another is the complexity of maintaining two methods in parallel, which
may be necessary till clang-tidy understands `#line`:
https://github.com/llvm/llvm-project/issues/62405
This solution may in theory also enable adding clang-tidy support for
tests in autotools, though I haven't tried.
Targeted for curl CI for now, and used in a GHA/windows job. 100%
experimental, not recommended outside these.
Closes#20667
- vms/curlmsg_vms.h: delete unused/commented code.
- vtls/schannel_verify: sort includes.
- typecheck-gcc.h: fix indent and alignment.
- lib/config-win32.h: drop idle `#undef`.
- spacecheck: check for stray empty lines before after curly braces.
- make literals more readable: 1048576 -> 1024 * 1024
- scope variables.
- use ISO date in a comment.
- drop redundant parentheses.
- drop empty comments.
- unfold lines.
- duplicate/stray spaces in comments.
- fix indent, whitespace, minor typos.
Closes#20690
Examples:
```
lib/vtls/openssl.c:2585:18: warning: 'signed char' to 'int' conversion; consider casting to 'unsigned char' first. [bugprone-signed-char-misuse]
2585 | msg_type = *(const char *)buf;
lib/vtls/openssl.c:2593:18: warning: 'signed char' to 'int' conversion; consider casting to 'unsigned char' first. [bugprone-signed-char-misuse]
2593 | msg_type = *(const char *)buf;
tests/server/mqttd.c:514:10: warning: comparison between 'signed char' and 'unsigned char' [bugprone-signed-char-misuse]
514 | if(passwd_flag == (char)(conn_flags & passwd_flag)) {
tests/server/tftpd.c:362:13: warning: 'signed char' to 'int' conversion; consider casting to 'unsigned char' first. [bugprone-signed-char-misuse]
362 | c = test->rptr[0];
tests/server/tftpd.c:454:9: warning: 'signed char' to 'int' conversion; consider casting to 'unsigned char' first. [bugprone-signed-char-misuse]
454 | c = *p++; /* pick up a character */
src/tool_urlglob.c:272:46: warning: 'signed char' to 'int' conversion; consider casting to 'unsigned char' first. [bugprone-signed-char-misuse]
272 | pat->c.ascii.letter = pat->c.ascii.min = min_c;
src/tool_urlglob.c:273:24: warning: 'signed char' to 'int' conversion; consider casting to 'unsigned char' first. [bugprone-signed-char-misuse]
273 | pat->c.ascii.max = max_c;
tests/libtest/cli_h2_pausing.c:164:23: warning: suspicious usage of 'sizeof()' on an expression of pointer type [bugprone-sizeof-expression]
164 | memset(&resolve, 0, sizeof(resolve));
tests/libtest/cli_upload_pausing.c:158:23: warning: suspicious usage of 'sizeof()' on an expression of pointer type [bugprone-sizeof-expression]
158 | memset(&resolve, 0, sizeof(resolve));
tests/libtest/first.c:86:15: warning: 'signed char' to 'int' conversion; consider casting to 'unsigned char' first. [bugprone-signed-char-misuse]
86 | coptopt = arg[optpos];
```
Also:
- tests/server/mqttd: drop a redundant and a wrongly signed cast.
Ref: https://clang.llvm.org/extra/clang-tidy/checks/bugprone/signed-char-misuse.htmlCloses#20654
Reported by clang-tidy `bugprone-sizeof-expression`.
Silencing:
```
tests/libtest/cli_h2_pausing.c:164:23: warning: suspicious usage of 'sizeof()' on an expression of pointer type [bugprone-sizeof-expression]
164 | memset(&resolve, 0, sizeof(resolve));
| ^
tests/libtest/cli_upload_pausing.c:158:23: warning: suspicious usage of 'sizeof()' on an expression of pointer type [bugprone-sizeof-expression]
158 | memset(&resolve, 0, sizeof(resolve));
| ^
```
Ref: https://clang.llvm.org/extra/clang-tidy/checks/bugprone/sizeof-expression.htmlCloses#20649
- move macro to `curl_setup.h` (from curlx), and rename.
It's required by src, test servers, libtests. Also used by unit/tunit,
(which is fixable but this patch doesn't touch it.)
- special-case it for Windows/Cygwin/MS-DOS.
- build: drop `setmode()`/`_setmode()` detection.
This also avoids detecting the different `setmode()` on BSDs,
and a lot of complexity and overhead.
- use `CURL_O_BINARY`.
Follow-up to 250d613763#15787
Follow-up to 5e70566094#15169Closes#20539
- de-dupe lib/src strdup/memdup functions into curlx.
- introduce `CURLX_STRDUP_LOW()` for mapping `strdup()`, and to do it at
one place within the code, in `curl_setup.h`.
- tests/server: use `curlx_strdup()`. (Also to fix building without
a system `strdup()`.)
- curlx/curlx.h: shorten and tidy up.
- adjust Windows build path to not need `HAVE_STRDUP`.
- build: stop detecting `HAVE_STRDUP` on Windows.
Closes#20497
To avoid redefining the `fstat` system symbol, and to clarify
`struct_stat` is a curl symbol.
- introduce `curlx_fstat()` macro and use it.
- rename `struct_stat` to `curl_struct_stat`.
Also:
- tests: replace direct `curlx_win32_stat()` call with `curlx_stat()`.
- checksrc: disallow direct `_fstati64` and `fstat()` calls, except in
examples.
Closes#20496
