To reduce complexity.
- is_finished() checks if the individual transfer is done
- handle_completed() is the logic that runs for a completed
transfer
Closes#20573
Drop detecting it at configure time, along with the interim macro
`HAVE_MSG_NOSIGNAL`. There is no longer a reason for this workaround,
and allows to save the work at configure time and simplify.
Also say in a comment that `sys/socket.h` is defining this macro.
Follow-up to 77b3bc239dCloses#20559
When removing an easy handle from a multi, there was an optimization
to update the timer only when the removed handle had any timers.
With the introduction of the "dirty" bitset, easy handles can now cause
a timeout of 0 to be set without having anything in their timer list.
Removing such a handle needs to update the timer now always, so that
it may get cleared when there is nothing more to wait for.
The previous "not clearing a 0 timer" should not have any effect on
application's logic. Without clearing, the timer will fire and then
adjust itself to the proper value. But it would cause one more timer
fire than necessary.
Reported-by: Jan Macku
Fixes https://github.com/curl/curl/issues/20498
Closes https://github.com/curl/curl/pull/20502
- schannel: fix mixed-up declaration. (originally fenced infof for
verbose, then changed to failf with the fence kept, then fence
removed and variable marked as verbose, when in fact it's not, but
not tested and caught in CI.
- fix two other fallouts.
- GHA/windows: disable verbose strings in a mingw job.
- appveyor: disable verbose strings in an MSVC job.
- appveyor: add way to pass any CMake option per-job.
Cherry-picked from #20387
Follow-up to 61093e2a81#20353Closes#20388
When the compiler supports C99.
- map logging functions to macro stubs when verbose logging is disabled
and the compiler is C99. Make sure these stubs silence unused variable
warnings for non-variadic arguments.
Before this patch they mapped to function stubs, the same codepath
used for C89 compiler in this configuration.
- introduce new macros to tell the compiler which code to include
when verbose code is active, or inactive:
- `CURLVERBOSE`: defined when verbose code is active.
To enclose blocks of code only used for verbose logging.
- `VERBOSE(statement);`:
compile statement when verbose code is active.
To mark code lines only used for verbose logging.
- `NOVERBOSE(statement);`:
compile statement when verbose code is inactive.
To suppress warnings for arguments passed to logging functions via
printf masks, e.g. `NOVERBOSE((void)ipaddress);`, yet keeping
the warning in verbose builds.
Note these macros are not the same as `CURL_DISABLE_VERBOSE_STRINGS`.
Verbose code is always active in C89 mode (without variadic macro
support).
- drop existing uses of `CURL_DISABLE_VERBOSE_STRINGS` where redundant,
or replace with the above macros. Ending up reducing the number of
`#ifdef`s, and also the number of lines.
Assisted-by: Daniel Stenberg
Assisted-by: Jay Satiro
Reported-by: Dan Fandrich
Fixes#20341
Refs: #12105#12167Closes#20353
In some legacy systems IPv6 might dynamically work/not work and thus
curl needs to check/probe to see if it should indeed be used.
This change moves the probe that checks for working IPv6 to the multi
handle setup function instead of delaying it to when the first name
resolve is performed. This avoids a later tricky error path if the
socket cannot be created due to OOM.
Closes#20383
When checking a transfer for being expired via `Curl_timeleft_ms()`,
eleminate the `bool connecting` parameter and have the function check
the `mstate` of the transfer instead.
Advantages:
* eleminate the caller needing awareness if the transfer is
connecting or in a later state
* fix pingpong timeout handling to check the correct timeout
during "proto_connect" phases
* avoid using "connecting" timeouts during establishing a secondary
connection (e.g. FTP) since this would use the timestamp from
the original, primary connect and thus be wrong
Reported-by: Wyuer on github
Fixes#20347Closes#20354
This allows builds know about all schemes - but only have the protocol
implementations for those actually built-in.
It further allows multiple protocols to reuse the same protocol setup
and functions for both TLS and non-TLS implementations instead of
needing two (or more) structs.
The scheme information is now in 'struct Curl_scheme' and all the
function pointers for each scheme/protocol implementation are in struct
Curl_protocol.
The URL API now always work with all known protocols.
Closes#20351
Define `struct Curl_sigpipe_ctx` that can be passed as argunent
to "lower" functions so that applying a transfers 'no_signal'
setting can be delayed as much as possible and sometimes avoided
alltogether.
Fixes#20326Closes#20329
Reported-by: Dag Haavi Finstad
Add casts to `bool`, or use `bit` type in local variables, where
neccessary to avoid MSVC compiler warnings C4242.
Note: There may remain places needing the above updates, where not
tested in CI, and missed in manual review.
Also:
- urldata: convert struct field `connect_only` to bitfield to match its
counterpart in another struct.
- rename curl-specific `bit` type to `curl_bit`.
Closes#20142
Declutter the ifdefs in socketpair.h. Introduce Curl_wakeup_*()
function that encapsulate the details about how the socketpair
is implemented.
This moves the EVENTFD specials from the using code into socketpair
implemenatation, avoiding duplications in three places.
Closes#20340
When a download size is known and rate limiting is in effect, adjust the
duration of each measurement step and its rate for maximum precision.
Since it is unpredictable how long the last bytes of a download will
take, download speed can be thrown off if the "last bytes" are a
significant amount of the total download. Make the "last bytes" small in
comparision to the rest and "stretch" the rate limit intervals to
accommodate the difference.
Fix ngtcp2 receive data acknowldgements to be based on a local window
size tracking. This allows window updates controlled by rate limits.
Fix ratelimit wait time calculation to accomodate negative tokens.
h3 rate limit, update timeers
Make download rate limits work correctly in ngtcp2. Fix multi handling
of rate limits to set a timer for when limits will update again.
Without running the transfer on limit updates, protocols like h2/h3 may
stall if the server does not send due to stream windows being too small.
scorecard: measure download speedlimits
When running scorecard with --limit-rate=n, show the reported download
speed averages plus percentage deviation from the limit.
Closes#20228
Avoid using PRIu32 and PRId32 in product source code. We don't need it.
It reduces readability. It is also inconsistent since unsigned int has
the same size and does not require the define.
DJGPP warns about using %u for uint32_t by default because it seems to
typedef it to unsigned long instead of unsigned int. Which even that is
annoying since long and int are both 32 bit on this platform.
We use our own *printf() implementation and we know this is safe.
This work-around defines uint32_t for DJGPP into unsigned int to avoid
the warnings and thus the need to use PRIu32 and PRId32.
Closes#20215
Also adjust a printf mask for signedness.
Fixing with MS-DOS DJGPP gcc 12.2.0:
```
lib/conncache.c:612:22: error: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:394:22: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:520:20: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:520:20: error: format '%u' expects argument of type 'unsigned int', but argument 5 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:611:20: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:614:22: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:887:20: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:887:20: error: format '%u' expects argument of type 'unsigned int', but argument 5 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:2719:26: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:2725:30: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:2729:28: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:3126:34: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:3348:34: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi.c:3991:28: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi_ev.c:343:24: error: format '%u' expects argument of type 'unsigned int', but argument 6 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi_ev.c:413:24: error: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi_ev.c:584:36: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi_ntfy.c:113:34: error: format '%d' expects argument of type 'int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi_ntfy.c:113:34: error: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'}
lib/multi_ntfy.c:171:22: error: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'}
lib/url.c:883:22: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
lib/url.c:889:22: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'uint32_t' {aka 'long unsigned int'}
```
Bug: https://github.com/curl/curl/pull/20199#discussion_r2666363334
Follow-up to 4c9e4e99c1#20208Closes#20200
Note: This patch doesn't aim to add `timeval.h` includes missing from
local headers using `curltime` type. They remain relying on `urldata.h`
being included first. This patch also doesn't delete existing, used
includes already present in local headers (as internal users may rely
on them).
Ref: #20106Closes#20126
Fix the pollset in perform state to not add sockets for directions
that are blocked. This otherwise will lead to busy loops for a
transfer that cannot be progressed.
Reported-by: Fizn-Ahmd on github
Fixes#20091Closes#20109
- asyn-thrdd.c: scope an include.
- apply more clang-format suggestions.
- tidy-up PP guard comments.
- delete empty line from the top of headers.
- add empty line after `curl_setup.h` include where missing.
- fix indent.
- CODE_STYLE.md: add `strcpy`.
Follow-up to 8636ad55df#20088
- lib1901.c: drop unnecessary line.
Follow-up to 436e67f65b#20076Closes#20070
- they rarely catch any problems
- we have other ways to test different send/recv problems
- the number of such calls vary much more per invoke than others, making
memdebugging harder
- reducing the total number of fallible functions per test is good
- they were not used as intended anyway
Closes#20097
- replace `sendf.h` with `curl_trc.h` where it was included just for it.
- drop unused `curl_trc.h` includes.
- easy: delete obsolete comment about `send.h` include reason.
Also:
- move out `curl_trc.h` include from `sendf.h` and include it directly
in users, where not done already. To flatten the include tree and
to less rely on indirect includes.
- stop including `sendf.h` from other headers, replace it with forward
declaration of `Curl_easy`, as done already elsewhere.
Verified with an all non-unity CI run.
Closes#20061
To make it available for all files. Drop includes from individual
sources. This header was already included from most sources and not
specific to any internal subsystem.
Also to ensure that two system symbol redefines on Windows (`read()` and
`write()`) get applied to all sources. Move them to `curl_setup.h`.
Closes#20056
Since Curl_mntfy_dispatch_all() is called with high frequency and
mostly unnecessary, add a check macro to avoid the call when not
needed.
Closes#20034
- curl_range: replace `sendf.h` with direct header dependency
`curl_trc.h`.
- drop `curl/curl.h` includes from internal sourcees in favor of the
include made from `curl_setup.h`. Replace it with the latter where
it's the only include.
- include `curl_setup.h` before using macros, where missing.
- drop redundant `stdlib.h`, `string.h` includes, in favor of
`curl_setup_once.h` including them.
- drop redundant `limits.h` in favor of `curl_setup.h` including it.
- fake_addrinfo.h: fix typo in comment.
- curl_setup_once.h: drop `stdio.h` in favor of earlier include in
`curl_setup.h`.
- drop stray, unused, `stddef.h` includes.
- memdebug.h: add missing `stddef.h` include. (relying on accidental
includes via other headers before this patch.)
- stddef.h: document why it's included.
- strerr: drop `curl/mprintf.h` in favor of `curl/curl.h` including it
via `curl_setup.h`.
Closes#20027
Always use curlx_now() when calling Curl_pgrs_now(data). Tests with the
"manual" updates to now proved differ more then 100ms in parallel testing.
Add `curlx_nowp()` to set current time into a struct curltime.
Add `curlx_ptimediff_ms() and friends, passing pointers.
Update documentation.
Closes#19998
Use `data->progress.now` as the timestamp of proecssing a transfer.
Update it on significant events and refrain from calling `curlx_now()`
in many places.
The problem this addresses is
a) calling curlx_now() has costs, depending on platform. Calling it
every time results in 25% increase `./runtest` duration on macOS.
b) we used to pass a `struct curltime *` around to save on calls, but
when some method directly use `curx_now()` and some use the passed
pointer, the transfer experienes non-linear time. This results in
timeline checks to report events in the wrong order.
By keeping a timestamp in the easy handle and updating it there, no
longer invoking `curlx_now()` in the "lower" methods, the transfer
can observer a steady clock progression.
Add documentation in docs/internals/TIME-KEEPING.md
Reported-by: Viktor Szakats
Fixes#19935Closes#19961
MSTATE_TUNNELING is no longer in use now that we have proxy connection
filters. Remove the state.
Remove the http handler `connect_it` method as it was merely a NOP.
Closes#19894
Since we no longer traverse the transfers attached to a connection,
change the sparse bitset to just a `uint32_t` counter.
This makes multi_ev the single user of sparse bitsets for transfers
using a socket and allocation failures are handled there correctly.
Refs #19818Closes#19836
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
Add protocol handler flag `PROTOPT_CONN_REUSE` to indicate that the
protocol allows reusing connections for other tranfers. Add that
to all handlers that support it.
Create connections with `conn->bits.close = FALSE` and remove all
the `connkeep()` calls in protocol handlers setup/connect implementations.
`PROTOPT_CONN_REUSE` assures that the default behaviour applies
at the end of a transfer without need to juggle the close bit.
`conn->bits.close` now serves as an additional indication that a
connection cannot be reused. Only protocol handles that allow
reuse need to set it to override the default behaviour.
Remove all `connclose()` and `connkeep()` calls from connection
filters. Filters should not modify connection flags. They are
supposed to run in eyeballing situations where a filter is just
one of many determining the outcome.
Fix http response header handling to only honour `Connection: close`
for HTTP/1.x versions.
Closes#19333
Fold the special connection pool shutdown handling in multi the things
the admin handle cares about. Add the admin handle to the 'process'
bitset, deduce it from the 'running' count.
The admin handle is the processed like any other transfer, but has a
special case in `multi_runsingle()`. Simplifies all other multi
processing parts.
Closes#19604
* move the TIMER_POSTQUEUE to the time a connection is chosen,
so that TIMER_NAMELOOKUP always happens afterwards
* client writer: do not trigger TIMER_STARTTRANSFER on CLIENTWRITE_INFO
as ftp and other pingpong protocols write that before starting anything
that is the tranfer itself
* Elimnating debug trancing of "closed stream/connection - bailing"
as confusing, as connection is not really closed on most cases.
* Setting 'data->req.upload_done` correctly, so that no "abort upload"
is happening at the end of a perfectly fine download.
* Adding test cases with up-/download of 0-length files.
* pytest: add a "timeline" of timer value checks to Resulst in curl.py,
so that this can be used in several test cases, replacing the local
stuff in test_16
* add timeline checks to ftp test cases
Closes#19269
Windows CRTs have a `share.h`. Before this patch when trying to
`#include <share.h>` it, the compiler picked up curl's internal
`lib/share.h` instead. Rename it to avoid this issue.
CRT `share.h` has constants necessary for using safe open CRT functions.
Also rename `lib/share.c` to keep matching the header.
Ref: https://learn.microsoft.com/cpp/c-runtime-library/sharing-constants
Ref: 625f2c1644#16949#16991
Cherry-picked from #19643Closes#19676
Description of how this works in `docs/internal/RATELIMITS.ms`.
Notable implementation changes:
- KEEP_SEND_PAUSE/KEEP_SEND_HOLD and KEEP_RECV_PAUSE/KEEP_RECV_HOLD
no longer exist. Pausing is down via blocked the new rlimits.
- KEEP_SEND_TIMED no longer exists. Pausing "100-continue" transfers
is done in the new `Curl_http_perform_pollset()` method.
- HTTP/2 rate limiting implemented via window updates. When
transfer initiaiting connection has a ratelimit, adjust the
initial window size
- HTTP/3 ngtcp2 rate limitin implemnented via ack updates
- HTTP/3 quiche does not seem to support this via its API
- the default progress-meter has been improved for accuracy
in "current speed" results.
pytest speed tests have been improved.
Closes#19384
- badwords.pl: add `-a` option to check all lines in source code files.
Before this patch indented lines were skipped (to avoid Markdown code
fences.)
- GHA/checksrc: use `-a` when verifying the source code.
- GHA/checksrc: disable `So` and `But` rules for source code.
- GHA/checksrc: add docs/examples to the verified sources.
- badwords.txt: delete 4 duplicates.
- badwords.txt: group and sort contractions.
- badwords.txt: allow ` url = `, `DIR`, `<file name`.
Closes#19536
Add new functions in `curlx/warnless.h` for controlled type
conversions:
* curlx_uitouz, convert unsigned into to size_t (should always work)
* curlx_uztoso, convert size_t to curl_off_t, capping at CURL_OFF_T_MAX
* curlx_sztouz, convert ssize_t to size_t, return TRUE when ok
* curlx_sotouz_range, convert curl_off_t to size_t interval, capping
values to interval bounds
Remove some unnecesary casts, convert some internal recv functions
to the "return result, have size_t* arg" pattern.
Closes#19495
