If the source line had square brackets before the match, the stripping
of the banned function left the original line intact, and repeated the
check on it forever. E.g. with banned function `open` in `lib518.c`:
```c
t518_testfd[0] = open(DEV_NULL, O_RDONLY);
```
Closes#18775
By introducing wrappers for them in the curlx namespace:
`curlx_fopen()`, `curlx_fdopen()`, `curlx_fclose()`.
The undefine/redefine/`(function)()` methods broke on systems
implementing these functions as macros. E.g. AIX 32-bit's `fopen()`.
Also:
- rename `lib/fopen.*` to `lib/curl_fopen.*` (for `Curl_fopen()`)
to make room for the newly added `curlx/fopen.h`.
- curlx: move file-related functions from `multibyte.c` to `fopen.c`.
- tests/server: stop using the curl-specific `fopen()` implementation
on Windows. Unicode isn't used by runtests, and it isn't critical to
run tests on longs path. It can be re-enabled if this becomes
necessary, or if the wrapper receives a feature that's critical for
test servers.
Reported-by: Andrew Kirillov
Bug: https://github.com/curl/curl/issues/18510#issuecomment-3274393640
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#18634
Add script to run all Perl sources through `perl -c` to ensure no
issues, and run this script via GHA/checksrc in CI.
Fallouts:
- fix two repeated declarations.
- move `shell_quote()` from `testutil.pm` to `pathhelp.pm`, to
avoid circular dependency in `globalconfig.pm`.
Closes#18745
The implementation was incomplete and lesser than the other backends. No
one ever reported a bug or requested enhancements for this, indicating
that this backend was never used.
Closes#18700
Replace `WSASocketW()` with `CURL_SOCKET()`. Also replace a call
to `socketclose()` with `sclose()`. According to a comment,
`socketclose()` was chosen to silence test 1498 (and 2300) reporting
`MEMORY FAILURE`. These reports were accurate, and were caused by
calling `WSASocketW()` instead of `socket()` (now `CURL_SOCKET()`).
This also fixes the curl `sclose()` call on an error branch, which is
now correctly paired with a curl socket open. The mismatched open/close
calls caused an issue in TrackMemory-enabled (aka `CURLDEBUG`) builds.
Docs confirm that `socket()` is defaulting to overlapped I/O, matching
the replaced `WSASocketW()` call:
https://learn.microsoft.com/windows/win32/api/winsock2/nf-winsock2-socket#remarks
Also:
- checksrc: ban `WSASocket*()` functions.
- report `SOCKERRNO` instead of `GetLastError()` for socket calls,
to match the rest of the codebase.
Follow-up to 9a2663322c#17572Closes#18633
- protocols MUST match one in the accept-list
- protocols are typically all uppercase
- drop All
- use SCP and SFTP instead of SSH
- add Protocols: to some options previously missing one
Closes#18675
Before this patch `accept4()`, `socket()`, `socketpair()`, `send()` and
`recv()` system symbols were remapped via macros, using the same name,
to local curl debug wrappers. This patch replaces these overrides by
introducing curl-namespaced macros that map either to the system symbols
or to their curl debug wrappers in `CURLDEBUG` (TrackMemory) builds.
This follows a patch that implemented the same for `accept()`.
The old method required tricks to make these redefines work in unity
builds, and avoid them interfering with system headers. These tricks
did not work for system symbols implemented as macros.
The new method allows to setup these mappings once, without interfering
with system headers, upstream macros, or unity builds. It makes builds
more robust.
Also:
- checksrc: ban all mapped functions.
- docs/examples: tidy up checksrc rules.
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#18503
- update Microsoft documentation links.
(also drop language designator where present.)
- checksrc: allow longer than 78 character lines if they
contain a https URL. To make these links easier to use and parse.
- merge links that were split into two lines.
Closes#18626
To avoid hitting `-Wreserved-macro-identifier` where possible.
- amigaos: introduce local macro instead of reusing `__request()`.
- easy_lock: avoid redefining `__has_builtin()`.
Follow-up to 33fd57b8ff#9062
- rand: drop interim macro `_random()`.
- windows: rename local macro `_tcsdup()` to `Curl_tcsdup()`.
To avoid using the reserved macro namespace and to avoid
colliding with `_tcsdup()` as defined by Windows headers.
- checksrc: ban `_tcsdup()` in favor of `Curl_tcsdup()`.
- tool_doswin: avoid redefining `_use_lfn()` (MS-DOS).
- tool_findfile: limit `__NO_NET_API` hack to AmigaOS.
Syncing this pattern with `lib/netrc.c`.
Follow-up to 784a8ec2c1#16279
- examples/http2-upload: avoid reserved namespace for local macro.
More cases will be removed when dropping WinCE support via #17927.
Cases remain when defining external macros out of curl's control.
Ref: #18477Closes#18482
- When an option name is used in text, this script no longer outputs the
short plus long version in the manpage output. It makes the text much
more readable.
This always showing both verions was previously done primarily to make
sure roffit would linkify it correctly, but since roffit 0.17 it
should link both long or short names correctly.
- When managen outputs generic text about options at the end of the
description it now highlights them properly so that they too get
linkified correctly in the HTML version. For consistency.
Closes#18580
Tweaks around handling of --limit-rate:
* tracing: trace outstanding timeouts by name
* multi: do not mark transfer as dirty that have
an EXPIRE_TOOFAST set
* multi: have one static function to asses speed limits
* multi: when setting EXPIRE_TOOFAST remove the transfers
from the dirty set
* progress: rename vars and comment on how speed limit
timeouts are calculated, for clarity
* transfer: when speed limiting, exit the receive loop
after a quarter of the limit has been received, not
on the first chunk received.
* cf-ip-happy.c: clear EXPIRE_HAPPY_EYEBALLS on connect
* scorecard: add --limit-rate parameter to test with
speed limits in effect
Closes#18454
Changed strategy to start up and terminate resolver thread.
When starting up:
Start the thread with mutex acquired, wait for signal from thread that
it started and has incremented the ref counter. Thread set
pthread_cancel() to disabled before that and only enables cancelling
during resolving itself. This assure that the ref counter is correct and
the unlinking of the resolve context always happens.
When shutting down resolving:
If ref counting shows thread has finished, join it, free everything. If
thread has not finished, try pthread_cancel() (non Windows), but keep
the thread handle around.
When destroying resolving:
Shutdown first, then, if the thread is still there and 'quick_exit' is
not set, join it and free everything. This might occur a delay if
getaddrinfo() hangs and cannot be interrupted by pthread_cancel().
Destroying resolving happens when another resolve is started on an
easy handle or when the easy handle is closed.
Add test795 to check that connect timeout triggers correctly
when resolving is delayed. Add debug env var `CURL_DNS_DELAY_MS`
to simulate delays in resolving.
Fix test1557 to set `quick_exit` and use `xxx.invalid` as domain
instead of `nothing` that was leading to hangers in CI.
Closes#18263
Add support for CURLOPT_READFUNCTION with WebSocket urls when *not* in
connect-only mode, e.g. when using curl_multi_perform.
Install the callback function and set CURLOPT_UPLOAD. Return
CURL_READFUNC_PAUSE when having nothing more to send and unpause the
transfer when more data is ready.
This will send the read bytes in a WebSocket BINARY frame.
Add support for this mode in the pytest "ws_data" client and have all
tests run in 'curl_ws_send/recv' and 'peform' mode as well.
Add `curl_ws_start_frame()`. Document, cover in libcurl-ws.md and
explain the READFUNCTION mode for websockets.
Add example `websocket-updown` for this.
Closes#17683
Adds `curl_off_t curl_multi_get_offt(CURLM *multi_handle, CURLMinfo_offt
info)` to the multi interface with enums:
* CURLMINFO_XFERS_CURRENT: current number of transfers
* CURLMINFO_XFERS_RUNNING: number of running transfers
* CURLMINFO_XFERS_PENDING: number of pending transfers
* CURLMINFO_XFERS_DONE: number of finished transfers to read
* CURLMINFO_XFERS_ADDED: total number of transfers added, ever
Add documentation for functions and info enums.
Add use in the curl command line tool to replace two static
variables counting the same "from the outside".
refs #17870Closes#17992
Also:
- GHA/linux, macos: test `install` with autotools too.
- GHA/linux, macos: enable fish and zsh completion in an autotools job.
- scripts: drop dynamic perl checks redundant after this patch.
- scripts: drop two interim variables.
- autotools: make `build-certs` target check perl first.
- autotools: replace `$(PERL)` with `@PERL@` to match the pattern used
in most automake scripts. For consistency. This makes `PERL` be
defined at configure-time, as opposed to make-time, for these cases
now.
Closes#18141
Fixing on macOS, and possibly other BSDs:
```
sed: 83: ./docs/THANKS-filter: RE error: illegal byte sequence
```
Where line 83 contains `\xED`.
Switch to raw encoding to avoid `sed` evaluating the stream of bytes.
Ref: #18061Closes#18062
- add 'use warnings' and 'use strict' where missing from Perl scripts.
- fix 'Use of uninitialized value'.
- fix missing declarations.
- test1140.pl: fix 'Possible precedence issue with control flow operator'.
- fix other misc issues.
Most actual errors found during this PR were fixed and merged via
separate PRs.
Likely there are remaining warnings not found and fixed in this PR.
Closes#17877
- vtls: fix unused variable and symbols.
- ftp: fix unused variable.
- http: fix unused variables.
- smtp: fix unsued variable.
- wolfssl: fix unused variable with !proxy.
- libssh: fix unused argument.
- curl_trc: sync guards between declaration and definition.
- curl_trc: add missing guard for `Curl_trc_ssls` when !verbose.
- curl_trc: fix errors with !http + http3.
- curl_trc: fix missing function with !http + nghttp2.
- cf-h2-proxy: disable when !http + nghttp2, to avoid calling undeclared
functions.
- sha256: fix missing declaration in rare configs.
- md4: fix symbol conflict when building GnuTLS together with AWS-LC or
wolfSSL. By prioritizing the latter two. AWS-LC has no option
to disable the clashing symbol. wolfSSL does, but the most seamless is
to skip including GnuTLS's standalone `md4.h` to avoid the clash.
- build: fix errors with !http + nghttp2.
- build: catch !ssl + ssls-export combination in source. Convert
build-level errors to warnings.
- build: fix errors with !http + http3.
- build: fix building curl tool and unit1302 in rare combinations.
By always compiling base64 curlx functions.
- cmake: add `_CURL_SKIP_BUILD_CERTS` internal option.
To disable automatically building certs with the testdeps target.
To improve performance when testing builds.
(used locally to find the failing builds fixed in this PR.)
Closes#17962
Re-running this script now makes it track the already mentioned
changelog entries and not add them again even if the git log contains
them.
This makes the script better handle reruns in a release branch after
rebasing on a later version of master.
Closes#17937
- GHA/checksrc: make a step name more accurate.
- cmakelint.sh: make sure to run from project root.
To make it easier to run locally.
- cmakelint.sh: use `set -eu`.
Closes#17915
`checksrc.bat` was outdated and required Perl for `checksrc.pl` anyway.
Rewrite `checksrc-all.sh` in Perl, making it usable in envs without
a POSIX shell.
Closes#17882
In order to do unit tests for private functions, functions that are
marked UNITTEST but without a global scope in the library, functions
that do not have prototypes in their corresponding header file, unit
tests previously brought their own private prototype *copy* into the
unit test.
This was error-prone when the internal function changes but the change
might be missed in the unit test which then uses an outdated prototype
copy for testing.
This change removes the private prototypes from unit tests and instead
introduces a C file parser that parses the specific C files and extracts
the necessary unit test prototypes into a generated header file for unit
tests to use. This geneated lib/unitprotos.h header is then included by
unit tests that need private prototypes.
Assisted-by: Viktor Szakats
Closes#17750
- tests/libtest: move exception to `stub_gssapi.h`.
- tests/libtest: move remaining exception to `testtrace.c`.
- tests/server: drop obsolete exception.
- docs/examples: move `BANNEDFUNC` exceptions to local files (3 lines).
- docs/examples: move `ERRNOVAR` exception to `ephiperfifo.c`.
- docs/examples: drop `typedef struct` (8 files).
- lib/curlx: add `.checksrc` with banned funcs copied from lib.
- checksrc: ban `strncpy`, `strtok_r`, `strtoul` by default.
Drop local bans. Add exception for `strtoul` to `tests/server'.
- lib, src: sync banned funcs.
Also:
- REUSE: drop `stunnel.pem`, it no longer exists.
- docs/examples: formatting.
- docs/examples: simplify some `sizeof()`s.
Closes#17764
Replace existing `mk-unity.pl` `--embed` workaround with running
`clang-tidy` manually on individual test source instead. This aligns
with how clang-tidy works and removes `mk-unity.pl` from the solution.
Also:
- mqttd: fix potentially uninitialized buffer by zero filling it.
```
tests/server/mqttd.c:484:41: error: The left operand of '<<' is a garbage value
[clang-analyzer-core.UndefinedBinaryOperatorResult,-warnings-as-errors]
484 | payload_len = (size_t)(buffer[10] << 8) | buffer[11];
| ^
[...]
tests/server/mqttd.c:606:45: error: The left operand of '<<' is a garbage value
[clang-analyzer-core.UndefinedBinaryOperatorResult,-warnings-as-errors]
606 | topiclen = (size_t)(buffer[1 + bytes] << 8) | buffer[2 + bytes];
| ^
```
- sockfilt: fix potential out-of-bound pointer:
```
tests/server/sockfilt.c:1128:33: error: The 2nd argument to 'send' is a buffer
with size 17010 but should be a buffer with size equal to or greater than
the value of the 3rd argument (which is 18446744073709551615)
[clang-analyzer-unix.StdCLibraryFunctions,-warnings-as-errors]
1128 | ssize_t bytes_written = swrite(sockfd, buffer, buffer_len);
| ^
```
- clang-tidy: suppress bogus `bzero()` warnings that happens
inside the notorious `FD_ZERO()` macros, on macOS.
Ref: https://github.com/curl/curl/pull/17680#issuecomment-2991730158Closes#17705
Tidy up headers and includes to ensure all individual test source
compile cleanly (but not link). To allow running clang-tidy (and
possibly other static analyzers) on them. It also improves readability
and allows to verify them locally, without the bundle logic.
clang-tidy ignores #included C files, so it's blind to bundle C files
the include these tests. The current workaround of embedding has
a couple of downsides:. meaningless filenames and line numbers,
missing issues, messing up self header paths. Thus, running it on
individual sources would be beneficial.
Also:
- de-duplicate includes.
- untangle some includes.
- formatting/indentation fixes.
- merge `getpart.h` into `first.h`.
Ref: https://github.com/curl/curl/pull/17680#issuecomment-2991730158Closes#17703
To make building the http client tests faster, with no duplication, by
using the build method that other test binaries already use.
The difference compared to other tests is that these don't use internal
libcurl headers or code. With the exception of `curl_config.h`, for
a feature macro.
Before this patch, these tests were built like examples.
Also:
- de-duplicate code and give unique names to colliding symbols.
- add local getopt implementation and enable all code for MSVC.
Adapted for curl via Public Domain source:
4e618ef782/getopt.h
Credits-to: Christopher Wellons
Thanks!
Closes#17627
Before this patch `memdebug.c` was compiled as a separate source in
unity builds. This was necessary because `memdebug.c` failed to compile
if `memdebug.h` was included before it, in `CURLDEBUG` mode. This patch
fixes this issue and allows to compile `memdebug.c` as part of the unity
source batch. This removes an exception and makes builds perform a notch
better.
- introduce `CURL_SCLOSE()` macro as an immutable synonym of `sclose()`.
- memdebug: replace `sclose()` reference with `CURL_SCLOSE()` to compile
as expected when `sclose()` is overridden by `memdebug.h`.
- memdebug: make it not break when including `memdebug.h` before it in
`CURLDEBUG` mode. Do this by calling low-level functions as
`(function)`.
- autotools, cmake: drop memdebug exception, include it like any other
source file. This is now possible because `memdebug.c` doesn't break
if `memdebug.h` was included before it, in `CURLDEBUG` builds.
- mk-unity: drop `--exclude` option. No longer used after this patch.
- drop `MEMDEBUG_NODEFINES` macro hack. No longer necessary.
Ref: #16747Closes#16746Closes#16738Closes#17631
