Commit Graph

274 Commits

Author SHA1 Message Date
Stefan Eissing
335dc0e3c5
cf-dns: connection filter for DNS queries
New connection filter `cf-dns` that manages DNS queries. If hands
out addresses and HTTPS-RR records to anyone interested. Used by
HTTPS and IP happy eyeballing.

Information may become available *before* the libcurl "dns entry"
is complete, e.g. all queries have been answered. The cf-ip-happy
filter uses this information to start connection attempts as soon
as the first address is available.

The multi MSTATE_RESOLVING was removed. A new connection always
goes to MSTATE_CONNECTING. The connectdata bit `dns_resolved`
indicates when DNS information is complete. This is used for
error reporting and starting the progress meter.

Removed dns entries `data->state.dns[i]`, as the `cf-dns` filter
now keeps the reference now.

Many minor tweaks for making this work and pass address information
around safely.

Closes #21027
2026-03-30 13:28:38 +02:00
Stefan Eissing
6f9f4b3cb7
lib: add thread pool and queue
- new source files for thread pool and queue
- test cases 3217 and 3218 for them
- internal documentation

Closes #20916
2026-03-23 23:03:58 +01:00
Daniel Stenberg
ceae02db04
rtmp: drop support
- librtmp has no test cases, makes no proper releases and has not had a
  single commit within the last year

- librtmp parses the URL itself and requires non-compliant URLs for this

- we have no RTMP tests

- RTMP was used by 2.2% of curl users (self-identified in the 2025
  survey)

Closes #20673
2026-03-21 14:56:06 +01:00
Stefan Eissing
96d5b5c688
dnscache: own source file, improvements
- Rename `Curl_resolv_unlink()` to `Curl_dns_entry_unlink()`.
- Change `Curl_dnscache_get()` to return CURLcode result. Returns
  now `CURLE_COULDNT_RESOLVE_HOST` for "negative" cache entries.
- Add `Curl_dnscache_add_negative()` to put a "negative" entry
  into the cache.

Closes #20864
2026-03-21 12:04:36 +01:00
Viktor Szakats
030d6aeaf3
build: include curlx headers directly in src and tests
To include what's actually used.

Also:
- drop unused includes.
- scope includes where possible.
- drop `curlx/curlx.h` umbrella header.
- config2setopts: include `netinet/in.h` for Cygwin/MSYS2.
  Previously included by chance via an unused curlx include.

Closes #20776
2026-03-16 12:05:39 +01:00
Stefan Eissing
eb14705280
protocol source, all about protocols and uri schemes
Add protocol.h and protocol.c containing all about libcurl's
known URI schemes and their protocol handlers (so they exist).

Moves the scheme definitions from the various sources files into
protocol.c. Schemes are known and used, even of the protocol
handler is not build or just not implemented at all.

Closes #20906
2026-03-16 08:39:02 +01:00
Viktor Szakats
64f28b8f88
curlx: add local snprintf() helper that always nul-terminates (Windows)
Make the helper use `vsnprintf()` internally on all supported Windows
toolchains (dropping `_snprintf()` and `snprintf()`), ensure to
nul-terminate. Omit the return value to avoid complexity.

Use the helper from `mprintf.c` / `out_double()`, from tests/server code
and the tests/server-specific build of `curlx_inet_ntop()`,
`curlx_strerror()` functions. In the single call (in tests) where the
returned length was used previously, determine it with `strlen()`.

Refs:
https://github.com/libssh2/libssh2/blob/libssh2-1.11.1/src/misc.c#L57-L79
https://learn.microsoft.com/cpp/c-runtime-library/reference/snprintf-snprintf-snprintf-l-snwprintf-snwprintf-l
https://learn.microsoft.com/cpp/c-runtime-library/reference/vsnprintf-vsnprintf-vsnprintf-l-vsnwprintf-vsnwprintf-l

Assisted-by: Jay Satiro
Follow-up to fa8bd1cc09 #20761
Follow-up to 8ab468c8aa #15997

Closes #20765
2026-03-02 21:30:36 +01:00
Viktor Szakats
c05cd2a10e
curl_setup.h: merge back single-use internal header curl_setup_once.h
Originally split in 2006, but the issues cited are no longer present in
current code. As of now both `curl_setup.h` and `curl_setup_once.h` are
included once per compiler invocation, without recursion. The latter is
a sub-header of the former with no clear distinction in their contents.

Merge them to avoid having to decide where to put new global PP logic.
Also to make it easier to overview what gets defined/included globally
and in what order. (Perhaps even allowing some tidying up here.)

Follow-up to 77b3bc239d

Closes #20555
2026-02-11 14:26:30 +01:00
Viktor Szakats
cdfc8dc7ad
build: tidy up and simplify setmode() detection and use
- move macro to `curl_setup.h` (from curlx), and rename.
  It's required by src, test servers, libtests. Also used by unit/tunit,
  (which is fixable but this patch doesn't touch it.)
- special-case it for Windows/Cygwin/MS-DOS.
- build: drop `setmode()`/`_setmode()` detection.
  This also avoids detecting the different `setmode()` on BSDs,
  and a lot of complexity and overhead.
- use `CURL_O_BINARY`.

Follow-up to 250d613763 #15787
Follow-up to 5e70566094 #15169

Closes #20539
2026-02-08 15:47:47 +01:00
Viktor Szakats
31a4f415af
build: tidy up and dedupe strdup functions
- de-dupe lib/src strdup/memdup functions into curlx.
- introduce `CURLX_STRDUP_LOW()` for mapping `strdup()`, and to do it at
  one place within the code, in `curl_setup.h`.
- tests/server: use `curlx_strdup()`. (Also to fix building without
  a system `strdup()`.)
- curlx/curlx.h: shorten and tidy up.
- adjust Windows build path to not need `HAVE_STRDUP`.
- build: stop detecting `HAVE_STRDUP` on Windows.

Closes #20497
2026-02-03 14:02:30 +01:00
Viktor Szakats
6974bd7cc8
curlx: dedupe basename copies into curlx_basename()
Also stop redefining system `basename()` symbol. Call `curlx_basename()`
instead, and map that to `basename()` if available.

Closes #20424
2026-01-25 12:21:54 +01:00
Viktor Szakats
814b54d83e
tidy-up: miscellaneous
- whitespace, indent, comments, clang-format.
- openssl: move feature guards within function blocks.
- tunit: drop redundant blocks.

Closes #20361
2026-01-20 12:37:56 +01:00
Daniel Stenberg
6aaac9dd38
vquic: drop support for OpenSSL-QUIC
- It is slower and uses more memory than the alternatives and is only
  experimental in curl.
- We disable a few tests for OpenSSL-QUIC because of flakiness
- It gets little attention from OpenSSL and we have no expectation of the
  major flaws getting corrected anytime soon.
- No one has spoken up for keeping it
- curl users building with vanilla OpenSSL can still use QUIC through the
  means of ngtcp2

Closes #20226
2026-01-17 22:49:34 +01:00
Daniel Stenberg
a535be4ea0
curlx: curlx_strcopy() instead of strcpy()
This function REQUIRES the size of the target buffer as well as the
length of the source string. Meant to make it harder to do a bad
strcpy().

Removes 23 calls to strcpy().

Closes #20067
2025-12-22 23:01:05 +01:00
Wyatt O'Day
1dc6ddde06
mbedTLS: cleanup insecure/deprecated code
1. With `MBEDTLS_SSL_PROTO_TLS1_2` not enabled, the mbedTLS code was not
able to connect to any server due to broken logic in curl's
`mbed_set_ssl_version_min_max()`. Now it correctly sets the minimum
supported TLS version based on what is compiled in the library.

2. If debugging is enabled, move the debugging enabling earlier in the
`mbed_connect_step1()` so that verbose errors are actually displayed if
failures happen (see the previous point -- it would've made debugging
that issue easier).

3. Remove the constant `mbedtls_x509_crt_profile_fr` and instead use
mbedTLS-included profile `mbedtls_x509_crt_profile_next` with
`mbedtls_ssl_conf_cert_profile()`. This will follow the latest standards
as new mbedTLS versions are released (rather than being stuck-in-time
until someone comes along to fix what was hard-coded here). This has the
immediate benefit of no longer supporting SHA1 certs and insecure RSA
key-lengths (1024). This fix immediately prevents previously possible
MITM attacks (SHA1 hashes and RSA-1024 keys can be forged relatively
easily by nation-state actors and criminal organizations with
deep-pockets).

4. Added [predictive
resistance](https://mbed-tls.readthedocs.io/en/latest/kb/how-to/add-a-random-generator/#enabling-prediction-resistance)
to the random number generator (adding more entropy to the RNG).

5. Split the random number generator into initialization, the actual
random generation, and the "freeing" of the resources. This
significantly reduces the overhead of using the RNG.

6. Removed the separate RNG function in the TLS connect stage (instead
use the "main" one) and remove the ad-hoc threading support. Instead
properly document how to enable threading in mbedTLS. As it was, other
internals of mbedTLS could have race conditions (in the RSA module in
particular) if `MBEDTLS_THREADING_C` was *not* enabled. And if it is
enabled, then these race-conditions cannot happen. And also, if
MBEDTLS_THREADING_C is enabled then the RNG functions [are fully
thread-safe](https://mbed-tls.readthedocs.io/en/latest/kb/development/thread-safety-and-multi-threading/).

   So, the previous ad-hoc threading support was both partial and broken.

7. Enable support for disabling `MBEDTLS_PEM_PARSE_C`.

8. Add support for `CURLOPT_SSLCERTTYPE` so user can specify `PEM` or
`DER` and get faster execution.

Closes #19983
2025-12-20 17:33:57 +01:00
Viktor Szakats
6d0ee7b17b
curlx: add curlx_rename(), fix to support long filenames on Windows
Move existing `Curl_rename()` `rename()` wrapper from lib to
curlx/fopen, and make it a curlx macro/function. To allow using
the local worker function to fixup long filenames on Windows.

Then fix the Windows-specific rename implementation to support long
filenames. This operation may happen when using a cookie jar, HSTS cache
or alt-svc cache, via libcurl or the curl tool.

Before this patch, when passing a long filename to the above options,
a `<random>.tmp` file was left on the disk without renaming it to the
filename passed to curl. There was also 1 second delay for each
attempted rename operation.

Also:
- checksrc: ban raw `rename()` and `MoveFileEx*()` functions.
- Note: `Curl_rename()` returned 1 on failure before this patch, while
  `curlx_rename()` returns -1 after, to match POSIX `rename()`.

Refs:
https://learn.microsoft.com/windows/win32/api/winbase/nf-winbase-movefileexa
https://learn.microsoft.com/windows/win32/fileio/maximum-file-path-limitation

Ref: #20040

Closes #20042
2025-12-20 16:03:11 +01:00
Viktor Szakats
193cb00ce9
build: stop overriding standard memory allocation functions
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.

To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.

This concludes the long journey to avoid redefining standard functions
in the curl codebase.

Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.

This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
  libcurl allocators. After, it maps to standard allocators, like
  the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
  maps to libcurl allocators.

Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
  and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
  allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
  To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
  (was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.

Follow-up to b12da22db1 #18866
Follow-up to db98daab05 #18844
Follow-up to 4deea9396b #18814
Follow-up to 9678ff5b1b #18776
Follow-up to 10bac43b87 #18774
Follow-up to 20142f5d06 #18634
Follow-up to bf7375ecc5 #18503
Follow-up to 9863599d69 #18502
Follow-up to 3bb5e58c10 #17827

Closes #19626
2025-11-28 10:44:26 +01:00
Viktor Szakats
62683ad3f4
curlx: replace sprintf with snprintf
To avoid using a deprecated function on Windows.

Also: de-dupe `SNPRINTF` definition in curlx.

Closes #19681
2025-11-25 11:11:06 +01:00
Viktor Szakats
3887069c66
lib: rename internal header share.h to curl_share.h to avoid collision
Windows CRTs have a `share.h`. Before this patch when trying to
`#include <share.h>` it, the compiler picked up curl's internal
`lib/share.h` instead. Rename it to avoid this issue.

CRT `share.h` has constants necessary for using safe open CRT functions.

Also rename `lib/share.c` to keep matching the header.

Ref: https://learn.microsoft.com/cpp/c-runtime-library/sharing-constants
Ref: 625f2c1644 #16949 #16991
Cherry-picked from #19643
Closes #19676
2025-11-25 00:26:50 +01:00
Stefan Eissing
24b36fdd15
ratelimit: redesign
Description of how this works in `docs/internal/RATELIMITS.ms`.

Notable implementation changes:
- KEEP_SEND_PAUSE/KEEP_SEND_HOLD and KEEP_RECV_PAUSE/KEEP_RECV_HOLD
  no longer exist. Pausing is down via blocked the new rlimits.
- KEEP_SEND_TIMED no longer exists. Pausing "100-continue" transfers
  is done in the new `Curl_http_perform_pollset()` method.
- HTTP/2 rate limiting implemented via window updates. When
  transfer initiaiting connection has a ratelimit, adjust the
  initial window size
- HTTP/3 ngtcp2 rate limitin implemnented via ack updates
- HTTP/3 quiche does not seem to support this via its API
- the default progress-meter has been improved for accuracy
  in "current speed" results.

pytest speed tests have been improved.

Closes #19384
2025-11-24 23:34:05 +01:00
Daniel Stenberg
c545e10fa7
sftp: fix range downloads in both SSH backends
When asking for the last N bytes of a file, and that size was larger
than the file size, it would miss the first byte due to a logic error.

The fixed range parser is now made a common function in the file now
renamed to vssh.c (from curl_path.c) - used by both backends.

Unit test 2605 verifies the parser.

Reported-by: Stanislav Fort (Aisle Research)
Closes #19460
2025-11-11 14:51:22 +01:00
Viktor Szakats
c96b7c4636
des: merge curl_des into curl_ntlm_core.c
`curl_des.c` contained a single, short, function
`Curl_des_set_odd_parity()`, called from `curl_ntlm_core.c` alone.

Move it there, and define it only when needed.

Follow-up to 300876a7a6
Follow-up to 8cc70db2db

Closes #19209
2025-10-24 12:39:27 +02:00
Stefan Eissing
357808f4ad
multi: add notifications API
Add infrastructure to colled and dispatch notifications for transfers
and the multi handle in general. Applications can register a callback
and en-/disable notification type the are interested in.

Without a callback installed, notifications are not collected. Same when
a notification type has not been enabled.

Memory allocation failures on adding notifications lead to a general
multi failure state and result in CURLM_OUT_OF_MEMORY returned from
curl_multi_perform() and curl_multi_socket*() invocations.

Closes #18432
2025-10-07 10:55:31 +02:00
Viktor Szakats
34ad78da89
curlx: move Curl_strerror, use in src and tests, ban strerror globally
Also:
- tests/server: replace local `sstrerror()` with `curlx_strerror()`.
- tests/server: show the error code next to the string, where missing.
- curlx: use `curl_msnprintf()` when building for src and tests.
  (units was already using it.)
- lib: drop unused includes found along the way.
- curlx_strerror(): avoid compiler warning (and another similar one):
  ```
  In file included from servers.c:14:
  ../../lib/../../lib/curlx/strerr.c: In function ‘curlx_strerror’:
  ../../lib/../../lib/curlx/strerr.c:328:32: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=]
    328 |       SNPRINTF(buf, buflen, "%s", msg);
        |                                ^
  ../../lib/../../lib/curlx/strerr.c:47:18: note: ‘snprintf’ output 1 or more bytes (assuming 2) into a destination of size 1
     47 | #define SNPRINTF snprintf
        |                  ^
  ../../lib/../../lib/curlx/strerr.c:328:7: note: in expansion of macro ‘SNPRINTF’
    328 |       SNPRINTF(buf, buflen, "%s", msg);
        |       ^~~~~~~~
  ```

Follow-up to 45438c8d6f #18823

Closes #18840
2025-10-06 09:44:23 +02:00
Stefan Eissing
eefd03c572
ssl: support Apple SecTrust configurations
- configure/cmake support for enabling the option
- supported in OpenSSL and GnuTLS backends
- when configured, Apple SecTrust is the default trust store
  for peer verification. When one of the CURLOPT_* for adding
  certificates is used, that default does not apply.
- add documentation of build options and SSL use

Closes #18703
2025-10-03 12:02:23 +02:00
Viktor Szakats
20142f5d06
build: avoid overriding system symbols for fopen functions
By introducing wrappers for them in the curlx namespace:
`curlx_fopen()`, `curlx_fdopen()`, `curlx_fclose()`.

The undefine/redefine/`(function)()` methods broke on systems
implementing these functions as macros. E.g. AIX 32-bit's `fopen()`.

Also:
- rename `lib/fopen.*` to `lib/curl_fopen.*` (for `Curl_fopen()`)
  to make room for the newly added `curlx/fopen.h`.
- curlx: move file-related functions from `multibyte.c` to `fopen.c`.
- tests/server: stop using the curl-specific `fopen()` implementation
  on Windows. Unicode isn't used by runtests, and it isn't critical to
  run tests on longs path. It can be re-enabled if this becomes
  necessary, or if the wrapper receives a feature that's critical for
  test servers.

Reported-by: Andrew Kirillov
Bug: https://github.com/curl/curl/issues/18510#issuecomment-3274393640

Follow-up to bf7375ecc5 #18503
Follow-up to 9863599d69 #18502
Follow-up to 3bb5e58c10 #17827

Closes #18634
2025-09-30 01:10:36 +02:00
Daniel Stenberg
b011e3fcfb
vssh: drop support for wolfSSH
The implementation was incomplete and lesser than the other backends. No
one ever reported a bug or requested enhancements for this, indicating
that this backend was never used.

Closes #18700
2025-09-24 22:59:33 +02:00
Daniel Stenberg
5ab120bc4e
krb5: drop support for Kerberos FTP
It was accidentally broken in commit 0f4c439fc7, shipped since
8.8.0 (May 2024) and yet not a single person has noticed or reported,
indicating that we might as well drop support for FTP Kerberos.

Krb5 support was added in 54967d2a3a (July 2007), and we have
been carrying the extra license information around since then for this
code. This commit removes the last traces of that code and thus we can
remove the extra copyright notices along with it.

Reported-by: Joshua Rogers
Closes #18577
2025-09-20 23:58:28 +02:00
Stefan Eissing
af69c9d636
ip happy eyeballing: keep attempts running
When `CURLOPT_HAPPY_EYEBALLS_TIMEOUT_MS` expires, start the next ip
connect attempt, but keep all ongoing attempts alive.

Separate happy-eyeballs connection filter into own source files.

Closes #18105
2025-08-01 09:30:59 +02:00
Viktor Szakats
3bb5e58c10
memory: make function overrides work reliably in unity builds
Fixing:
- HTTPS-RR builds with c-ares and Linux MUSL.
- curl-for-win minimal builds with Linux MUSL.

It should fix all other kinds of entaglement between curl's redefintions
of system symbols and system (or 3rd-party) headers sensitive to that.

It also syncs memory override behavior between unity & non-unity builds,
thus reducing build variations.

The idea is to define and declare everything once in `curl_setup.h`,
without overriding any system symbols with curl ones yet. Then, like
before this patch, override them, if necessary, in each source file via
`curl_memory.h` and `memdebug.h`, after including system headers.
To ensure a clean slate with no overrides at the beginning of each
source file, reset all of them unconditionally at the end of
`curl_setup.h`, by including `curl_mem_undef.h`. (This assumes
`curl_setup.h` is always included first, which is already the case
throughout the codebase.)

`curl_mem_undef.h` can also be included explicitly wherever overrides
are causing problems. E.g. in tests which use unity-style builds and
a previously included `curl_memory.h`/`memdebug.h` can be spilling into
other source files.

The simplified role of the two override headers:
- `curl_memory.h`: overrides system memory allocator functions to
  libcurl ones, when memory tracing (aka `CURLDEBUG`) is disabled.
- `memdebug.h`: overrides system memory allocator and some other
  functions to curl debug functions, when memory tracing is enabled.

Changed made in this patch, step-by-step:
- curl_memory.h: move allocator typedefs and protos to `curl_setup.h`.
- memdebug.h: move `ALLOC_*` macros to `curl_setup.h`.
- memdebug.h: move allocator protos to `curl_setup.h`.
- memdebug.h: move `Curl_safefree()` macro to `curl_setup.h`.
  (it's a regular macro, with a one-time, global, definition.)
- curl_memory.h: move system symbol undefs to a new, separate header:
  `curl_mem_undef.h`.
- curl_setup.h: include `curl_mem_undef.h` at the end, unconditionally,
  to reset system symbol macros after each inclusion.
- handle `sclose()` and `fake_sclose()` in `curl_setup.h`. They are not
  system symbols, a one-time definition does the job.

Also:
- GHA/linux: enable unity mode for the HTTP-RR c-ares MUSL job.
  Follow-up to 17ab4d62e6 #16413

That said, I'd still find it better to avoid redefining system macros.
To communicate clearly the fact that they are not the original system
calls and they do behave differently. And, it would allow dropping the
undef/redef dance in each source file, and maintaining the logic with
it. The "last #include files should be in this order" comments in each
source would also become unnecessary. Also the trick of using
`(func)` (or interim macros) to call the non-overridden function where
required. This method works for printf and most everything else already.
For `_tcsdup`, socket and fopen functions this could work without
disturbing the codebase much.

Ref: #16428 (clean reboot of)

Closes #17827
2025-07-28 17:45:04 +02:00
Daniel Stenberg
91138b014d
vquic: drop msh3
It has never been properly functional in curl while there are several
alternatives that are.

Closes #17729
2025-07-27 17:57:17 +02:00
Viktor Szakats
916f241f2f
lib: make CURLX_SET_BINMODE() and use it
Use it from libtests' `first.c` and thus also from units, and tunits.

Also:
- cmake: drop stray `curltool` lib dependency for units.
- units: stop depending on `src` headers.
- tests/server: drop depending on `src` headers.
  (the remaining one listed in the comments, `tool_xattr.h`, was not
  actually used from servers.)
- tests/server: drop duplicate curlx headers.
  (Except `warnless.h`, which is tricky on Windows.)

Closes #17672
2025-06-19 15:57:37 +02:00
Viktor Szakats
35d0c047ce
lib: make curlx_wait_ms() and use it
Move function to curlx/, change all callers.

Also:
- src: replace local implementation.
- tests/client: replace local ad-hoc sleep code.
- tests/libtest: replace local `wait_ms()` implementation.
- lib1531: replace local ad-hoc sleep code.
- tests/server: replace local, simplified copy.
- tests/server: formatting, drop some unused headers.

Closes #17641
2025-06-19 15:57:37 +02:00
Viktor Szakats
2c27a67daa
tests: always make bundles, adapt build and tests
Make test bundles the default. Drop non-bundle build mode.
Also do all the optimizations and tidy-ups this allows, simpler builds,
less bundle exceptions, streamlined build mechanics.

Also rework the init/deinit macro magic for unit tests. The new method
allows using unique init/deinit function names, and calling them with
arguments. This is in turn makes it possible to reduce the use of global
variables.

Note this drop existing build options `-DCURL_TEST_BUNDLES=` from cmake
and `--enable-test-bundles` / `--disable-test-bundles` from autotools.

Also:
- rename test entry functions to have unique names: `test_<testname>`
  This removes the last exception that was handled in the generator.
- fix `make dist` to not miss test sources with test bundles enabled.
- sync and merge `tests/mk-bundle.pl` into `scripts/mk-unity.pl`.
- mk-unity.pl: add `--embed` option and use it when `CURL_CLANG_TIDY=ON`
  to ensure that `clang-tidy` does not miss external test C sources.
  (because `clang-tidy` ignores code that's #included.)
- tests/unit: drop no-op setup/stop functions.
- tests: reduce symbol scopes, global macros, other fixes and tidy-ups.
- tool1621: fix to run, also fix it to pass.
- sockfilt: fix Windows compiler warning in certain unity include order,
  by explicitly including `warnless.h`.

Follow-up to 6897aeb105 #17468

Closes #17590
2025-06-14 21:08:23 +02:00
Daniel Stenberg
08a3e8e19a
TLS: remove support for Secure Transport and BearSSL
These libraries do not support TLS 1.3 and have been marked for removal
for over a year. We want to help users select a TLS dependency that is
future-proof and reliable, and not supporting TLS 1.3 in 2025 does not
infer confidence. Users who build libcurl are likely to be served better
and get something more future-proof with a TLS library that supports
1.3.

Closes #16677
2025-06-11 07:54:19 +02:00
Daniel Stenberg
1886260a95
lib: make curlx_inet_ntop()
move function to curlx/, change all callers

Closes #17560
2025-06-09 13:16:01 +02:00
Daniel Stenberg
d4dd43b20d
curlx: move curlx_inet_pton
Used by test server code.

Closes #17300
2025-05-09 13:45:24 +02:00
Daniel Stenberg
c74d3e10d2
curlx: add curlx_winapi_ functions
Split them out from lib/strerror. Used by test code.

Closes #17299
2025-05-09 10:45:36 +02:00
Daniel Stenberg
16db059a93
curlx: move version_win32.[ch] to curlx/
For curlx_verify_windows_version

Closes #17290
2025-05-08 17:09:32 +02:00
Daniel Stenberg
4190c73094
curlx: move nonblock.[ch] into curlx/
Closes #17288
2025-05-08 13:22:02 +02:00
Daniel Stenberg
2d528898f7
lib: move multibyte.[ch] to curlx/
This file provides functions in the curlx set.

Closes #17285
2025-05-08 10:19:19 +02:00
Daniel Stenberg
255aac56f9
curlx: move into to curlx/
Move curlx_ functions into its own subdir.

The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.

The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.

The strcase defines are not curlx_ functions and should not be used by
tool or server code.

dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.

When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.

Assisted-by: Jay Satiro

Closes #17253
2025-05-07 11:01:15 +02:00
Daniel Stenberg
e0ebc3ff13
lib: provide a getaddrinfo wrapper
This uses c-ares under the hood and supports the CURL_DNS_SERVER
environment variable - for debug builds only. The getaddrinfo()
replacement function is only used if CURL_DNS_SERVER is set to make a
debug build work more like a release version without the variable set.

'override-dns' is a new feature for the test suite when curl can be told
to use a dedicated DNS server, and test 2102 is the first to require
this.

Requires c-ares 1.26.0 or later.

Closes #17134
2025-04-28 23:48:02 +02:00
Stefan Eissing
657aae79c0
lib: add meta_hash to connection, eliminate hash_offt
With a meta_hash at each connection (similar to easy handle, let
multi_ev.c store its pollsets as meta data, no longer needing its own
hashes.

This eliminates the last use of Curl_hash_offt. Remove it.

Closes #17095
2025-04-22 15:57:18 +02:00
Stefan Eissing
909af1a43b
multi: do transfer book keeping using mid
Change multi's book keeping of transfers to no longer use lists, but a
special table and bitsets for unsigned int values.

`multi-xfers` is the `uint_tbl` where `multi_add_handle()` inserts a new
transfer which assigns it a unique identifier `mid`. Use bitsets to keep
track of transfers that are in state "process" or "pending" or
"msgsent".

Use sparse bitsets to replace `conn->easyq` and event handlings tracking
of transfers per socket. Instead of pointers, keep the mids involved.

Provide base data structures and document them in docs/internal:
* `uint_tbl`: a table of transfers with `mid` as lookup key,
   handing out a mid for adds between 0 - capacity.
* `uint_bset`: a bitset keeping unsigned ints from 0 - capacity.
* `uint_spbset`: a sparse bitset for keeping a small number of
  unsigned int values
* `uint_hash`: for associating `mid`s with a pointer.

This makes the `mid` the recommended way to refer to transfers inside
the same multi without risk of running into a UAF.

Modifying table and bitsets is safe while iterating over them. Overall
memory requirements are lower as with the double linked list apprach.

Closes #16761
2025-04-17 17:28:38 +02:00
Stefan Eissing
56e40ae6a5
asyn resolver code improvements
"asyn" is the internal name under which both c-ares and threaded
resolver operate. Make the naming more consistent. Implement the c-ares
resolver in `asyn-ares.*` and the threaded resolver in `asyn-thrdd.*`.
The common functions are in `asyn-base.c`.

When `CURLRES_ASYNCH` is defined, either of the two is used and
`data->state.async` exists. Members of that struct vary for the selected
implementation, but have the fields `hostname`, `port` and `ip_version`
always present. This are populated when the async resolving starts and
eliminate the need to pass them again when checking on the status and
processing the results of the resolving.

Add a `Curl_resolv_blocking()` to `hostip.h` that relieves FTP and SOCKS
from having to repeat the same code.

`Curl_resolv_check()` remains the function to check for status of
ongoing resolving. Now it also performs internally the check if the
needed DNS entry exists in the dnscache and if so, aborts the asnyc
operation. (libcurl right now does not check for duplicate resolve
attempts. an area for future improvements).

The number of functions in `asyn.h` has been reduced. There were subtle
difference in "cancel()" and "kill()" calls, both replaced by
`Curl_async_shutdown()` now. This changes behaviour for threaded
resolver insofar as the resolving thread is now always joined unless
`data->set.quick_exit` is set. Before this was only done on some code
paths. A future improvement would be a thread pool that keeps a limit
and also could handle joins more gracefully.

DoH, not previously tagged under "asny", has its struct `doh_probes` now
also in `data->state.async`, moved there from `data->req` because it
makes more sense. Further integration of DoH underneath the "asyn"
umbrella seems like a good idea.

Closes #16963
2025-04-16 09:34:20 +02:00
Viktor Szakats
c799f608f2
autotools: fix dllmain.c in unity builds
Sync it with cmake to:
- exclude it from all builds except Windows and Cygwin.
- exclude it from unity builds for Cygwin to avoid the included
  `windows.h` header interfere with the rest of the code.

Also:
- fix to trim ending spaces from `CSOURCES` for the `tidy` target.
  The solution requires a non-POSIX `-E` `sed` option. Supported by BSD
  and GNU implementations.
  Follow-up to 37523c91bc #16480

Follow-up to 60c3d04465 #14815
Follow-up to 7860f575fe #12408

Closes #16712
2025-03-14 10:38:28 +01:00
Viktor Szakats
07f984a776
ntlm: merge ntlm.h into ntlm.c
It's the only user since dropping NTLM_WB support.

Follow-up to 50def7c881 #13249

Closes #16690
2025-03-13 00:03:15 +01:00
Daniel Stenberg
09a5b2f2de
lib: rename curlx_strtoofft to Curl_str_numblanks()
The function is no longer used via the curlx shortcut.

Remove the strtoofft.[ch] files.

Closes #16642
2025-03-10 10:39:20 +01:00
Stefan Eissing
df672695e5
shutdowns: split shutdown handling from connection pool
Further testing with timeouts in event based processing revealed that
our current shutdown handling in the connection pool was not clear
enough. Graceful shutdowns can only happen inside a multi handle and it
was confusing to track in the code which situation actually applies. It
seems better to split the shutdown handling off and have that code
always be part of a multi handle.

Add `cshutdn.[ch]` with its own struct to maintain connections being
shut down. A `cshutdn` always belongs to a multi handle and uses that
for socket/timeout monitoring.

The `cpool`, which can be part of a multi or share, either passes
connections to a `cshutdn` or terminates them with a one-time, best
effort.

Add an `admin` easy handle to each multi and share. This is used to
perform all maintenance operations where no "real" easy handle is
available. This solves the problem that the multi admin handle requires
some additional initialisation (e.g. timeout list).

The share needs its admin handle as it is often cleaned up when no other
transfer or multi handle exists any more. But we need a `data` in almost
every call.

Fix file:// handling of errors when adding a new connection to the pool.

Changes in `curl` itself:

- for parallel transfers, do not set a connection pool in the share,
  rely on the multi's connection pool instead. While not a requirement
  for the new `cshutdn` to work, this is

  a) helpful in testing to trigger graceful shutdowns
  b) a broader code coverage of libcurl via the curl tool

- on test_event with uv, cleanup the multi handle before returning from
  parallel_event(). The uv struct is on the stack, cleanup of the multi
  later will crash when it tries to register sockets. This is a "eat
  your own dogfood" related fix.

Closes #16508
2025-03-02 11:13:34 +01:00