With default invocation, make generated file timestamps deterministic
by looking up (via the GitHub API) the last commit that modified
`certdata.txt`, along with its commit timestamp.
Also:
- show the URL used to download `certdata.txt` from.
- make `ca-bundle.crt` timestamp match `certdata.txt`'s.
Closes#20528
- when scanning source code, this now only checks source code comments
and double-quote strings. No more finding bad words as part of code
- this allows the full scan to be done in a single invocation
- detects source code or markdown by file name extension
- moved the whitelist words config into the single `badwords.txt` file,
no more having them separately (see top of file for syntax)
- all whitelisted words are checked case insensitively now
- removed support for whitelisting words on a specific line number. We
did not use it and it is too fragile
Removing the actual code from getting scanned made the script take an
additional 0.5 seconds on my machine.
Scanning 1525 files now takes a little under 1.7 seconds for me.
Closes#20909
Also:
- support per-directory and per-upper-directory whitelist entries.
- convert badlist input grep tweak into the above format.
(except for 'And' which had just a few hits.)
- fix many code exceptions, but do not enforce.
(there also remain about 350 'will' uses in lib)
- fix badwords in example code, drop exceptions.
- badwords-all: convert to Perl.
To make it usable from CMake.
- FAQ: reword to not use 'will'. Drop exception.
Closes#20886
It's mostly a filler word. I've read through each use of it in the code
base and did minor rephrasings when "simply" carried some meaning. The
overwhelming majority of cases, removing it improved the text
significantly. Inspired by #20793.
Closes#20822
- 'badwords' is now a target in Makefile.am
- change badwords.txt to specify plain "words" instead of regexes so the
script can build single regexes when scanning, which makes the script
perform much faster (~6 times faster)
Closes#20869
- also query `INTERFACE_COMPILE_DEFINITIONS` and
`INTERFACE_COMPILE_OPTIONS`.
To further sync clang-tidy manual command-lines with the C compiler
command-lines generated by CMake. In practice this adds `-I` options
(via `INTERFACE_COMPILE_OPTIONS`) to the end of the command-line for
dependency header directories. It does not change the outcome of
clang-tidy runs.
- limit querying `COMPILE_DEFINITIONS` and `COMPILE_OPTIONS` to the top
target, the test itself. To not include options such as
`-DCURL_HIDDEN_SYMBOLS` and `-fvisibility=hidden` (when set) via
the libcurl shared lib.
To sync with the actual C compiler command-line.
- drop redundant `unset()`.
- rename local function to be more specific.
Follow-up to d9386a2f8e#20759Closes#20829
To avoid a system include masking a custom directory, and e.g. picking
up system OpenSSL headers from `/usr/include` on Linux, instead of the
correct ones from a custom header directory, move system include
directories to the back of the header path list. Also to match what
CMake seems to be doing for the C compiler command-lines it generates.
CMake seems to use `-I`, while for these invocations we stick with
`-isystem` just in case.
This area remains fragile and likely not the final issue.
Fixing (seen in GHA/linux H3 c-ares):
```
Error while processing bld/tests/libtest/lib1521.c.
/usr/include/openssl/macros.h:147:4: error: "OPENSSL_API_COMPAT expresses an impossible API compatibility level" [clang-diagnostic-error]
Found compiler error(s).
147 | # error "OPENSSL_API_COMPAT expresses an impossible API compatibility level"
| ^
FAILED: [code=1] tests/libtest/CMakeFiles/libtests-clang-tidy
```
Ref: https://github.com/curl/curl/actions/runs/22468472670/job/65079885471?pr=20751
Bug: https://github.com/curl/curl/pull/20751#issuecomment-3970180687
Cherry-picked from #20751Closes#20759
Tests are build in "unity"-style, by including sources into an umbrella
C files (similar to how CMake unity works). This does not play well with
clang-tidy, which seems to unconditionally ignore C sources included
like this. To fix it, curl's CMake implements a manual clang-tidy
support for tests, which compiles sources one-by-one, while also making
sure sources compile cleanly standalone (e.g. all sources need to
include `first.h`). The manual clang-tidy implementation is fragile, and
performance, in particular when targeting Windows, is abysmal.
This patch introduces an alternate solution, enabled by the
`_CURL_TESTS_CONCAT=ON` option. In this mode, umbrella sources include
the actual sources instead of `#including` them. Allowing to use CMake's
built-in clang-tidy support to compile them, with clang-tidy actually
checking the sources. Making the manual clang-tidy support unnecessary.
In the Windows CI job it results in a 4x performance improvement (4m ->
1m), making it practical to run clang-tidy on tests on Windows, in CI.
The main downside is that clang-tidy doesn't understand the `#line`
directive. Meaning issues found show the wrong filename and line number
next to them. It's not impossible to locate errors this way, but also
not convenient.
Minor/potential downside is that the concatenated source needs to be
reassembled each time an original source is updated. This may result in
more copying on the disk when used in local development. The largest
source is 1.4MB, so probably not a show-stopper on most machines.
Another is the complexity of maintaining two methods in parallel, which
may be necessary till clang-tidy understands `#line`:
https://github.com/llvm/llvm-project/issues/62405
This solution may in theory also enable adding clang-tidy support for
tests in autotools, though I haven't tried.
Targeted for curl CI for now, and used in a GHA/windows job. 100%
experimental, not recommended outside these.
Closes#20667
- vms/curlmsg_vms.h: delete unused/commented code.
- vtls/schannel_verify: sort includes.
- typecheck-gcc.h: fix indent and alignment.
- lib/config-win32.h: drop idle `#undef`.
- spacecheck: check for stray empty lines before after curly braces.
- make literals more readable: 1048576 -> 1024 * 1024
- scope variables.
- use ISO date in a comment.
- drop redundant parentheses.
- drop empty comments.
- unfold lines.
- duplicate/stray spaces in comments.
- fix indent, whitespace, minor typos.
Closes#20690
By passing to clang-tidy the C compiler with `--target` and sysroot
options, if any.
Fixing (GHA/windows, linux-mingw, CM clang-tidy):
```
lib/curl_setup.h:841:10: error: 'io.h' file not found [clang-diagnostic-error]
841 | #include <io.h>
| ^~~~~~
Found compiler error(s).
FAILED: [code=1] tests/server/CMakeFiles/servers-clang-tidy bld/tests/server/CMakeFiles/servers-clang-tidy
cd tests/server && /usr/bin/clang-tidy --config-file=.clang-tidy.yml
--warnings-as-errors=* --checks=-clang-diagnostic-unused-function first.c getpart.c util.c dnsd.c [...]
-- <-D-options> <-I-options> <cflags>
```
For reference, this is CMake's built-in clang-tidy invocation:
```
/usr/local/bin/cmake -E __run_co_compile --tidy="/usr/bin/clang-tidy;--config-file=.clang-tidy.yml;
--warnings-as-errors=*;--extra-arg-before=--driver-mode=gcc" --source=lib/curl_fopen.c
-- /usr/bin/clang --target=x86_64-w64-mingw32 <-D-options> <-I-options> <cflags>
```
Also:
- bump cmakelint `--max-statements`. Needs 59 after this patch.
- use undocumented CMake variables:
- `CMAKE_C_COMPILE_OPTIONS_TARGET` for `--target=`
- `CMAKE_C_COMPILE_OPTIONS_SYSROOT` for `--sysroot=`
Cherry-picked from #20631Closes#20640
To avoid redefining the `fstat` system symbol, and to clarify
`struct_stat` is a curl symbol.
- introduce `curlx_fstat()` macro and use it.
- rename `struct_stat` to `curl_struct_stat`.
Also:
- tests: replace direct `curlx_win32_stat()` call with `curlx_stat()`.
- checksrc: disallow direct `_fstati64` and `fstat()` calls, except in
examples.
Closes#20496
- stop redefining system symbol `lseek`, by introducing `curl_lseek()`.
- handle AmigaOS quirk within the macro mapping.
- add missing parenthesis to `LSEEK_ERROR` values.
- tool_util: use curl `lseek` macros in `tool_ftruncate64()`.
- move `LSEEK_ERROR` to right-hand side of if expressions.
- checksrc: disallow direct uses of `_lseeki64`, `llseek`, `lseek`.
Closes#20488
To avoid noise due to local C files when using automatic local checksrc
checks (e.g. via CMake `-DCURL_LINT=ON` option, or `curl-lint` target).
Also replace single-quote with double-quote in external git command, for
portability.
Follow-up to 88ff396549#17882
Follow-up to e785e898a6#17376Closes#20439
- drop leading indent from Markdown.
- switch to Markdown section markers where missing.
- move `&&` and `||` to the end of the line (C, Perl).
- openssl: add parenthesis to an if sub-expression.
- misc clang-format nits.
- unfold Markdown links.
- SSL-PROBLEMS.md: drop stray half code-fence.
Closes#20402
Use non-deprecated CRT function variants on Windows.
- introduce `curlx_fdopen()`, `curlx_close()` and use them. Map them to
non-deprecated, underscored, CRT functions on Windows.
- replace `close()` uses with either `sclose()` (for sockets) or
`curlx_close()` (for files).
- map `fileno`, `unlink`, `isatty` to their non-deprecated, underscored,
versions on Windows.
- tool_dirhie: map `mkdir` to `_mkdir` on Windows.
- easy: use `_strdup()` on Windows, regardless of how `HAVE_STRDUP` is
set.
- cmake: assume `HAVE_STRDUP` on Windows. To allow dropping a detection
hack using `_CRT_NONSTDC_NO_DEPRECATE` with MSVC. Windows always has
`_strdup()` which the code uses, but also needs `HAVE_STRDUP` defined
to disable curl's own `strdup()` implementation.
- curl_setup.h: drop `_CRT_NONSTDC_NO_DEPRECATE` as no longer necessary.
Closes#20212
This allows builds know about all schemes - but only have the protocol
implementations for those actually built-in.
It further allows multiple protocols to reuse the same protocol setup
and functions for both TLS and non-TLS implementations instead of
needing two (or more) structs.
The scheme information is now in 'struct Curl_scheme' and all the
function pointers for each scheme/protocol implementation are in struct
Curl_protocol.
The URL API now always work with all known protocols.
Closes#20351
Accept argument as a custom amount. Previously it showed all functions
with a complexity score above 57. This way it adapts better as we
gradually decrease complexity in functions.
Closes#20273
To simplify the directory layout.
- OS400 and vms support move from `packages` to `projects`.
- Windows README and `generate.bat` files move from `projects`
to `projects/Windows`.
Closes#20271
To run checksrc and spacecheck on the source tree. Also for cmake
to sync up with autotools' `checksrc` target.
- cmake: `curl-lint`
With `-DCURL_LINT=ON`, checks run automatically for all targets.
- autotools: `lint`
Closes#20175
- asyn-thrdd.c: scope an include.
- apply more clang-format suggestions.
- tidy-up PP guard comments.
- delete empty line from the top of headers.
- add empty line after `curl_setup.h` include where missing.
- fix indent.
- CODE_STYLE.md: add `strcpy`.
Follow-up to 8636ad55df#20088
- lib1901.c: drop unnecessary line.
Follow-up to 436e67f65b#20076Closes#20070
- they rarely catch any problems
- we have other ways to test different send/recv problems
- the number of such calls vary much more per invoke than others, making
memdebugging harder
- reducing the total number of fallible functions per test is good
- they were not used as intended anyway
Closes#20097
Move existing `Curl_rename()` `rename()` wrapper from lib to
curlx/fopen, and make it a curlx macro/function. To allow using
the local worker function to fixup long filenames on Windows.
Then fix the Windows-specific rename implementation to support long
filenames. This operation may happen when using a cookie jar, HSTS cache
or alt-svc cache, via libcurl or the curl tool.
Before this patch, when passing a long filename to the above options,
a `<random>.tmp` file was left on the disk without renaming it to the
filename passed to curl. There was also 1 second delay for each
attempted rename operation.
Also:
- checksrc: ban raw `rename()` and `MoveFileEx*()` functions.
- Note: `Curl_rename()` returned 1 on failure before this patch, while
`curlx_rename()` returns -1 after, to match POSIX `rename()`.
Refs:
https://learn.microsoft.com/windows/win32/api/winbase/nf-winbase-movefileexahttps://learn.microsoft.com/windows/win32/fileio/maximum-file-path-limitation
Ref: #20040Closes#20042
It makes them work in Schannel's CA bundle loader, and curl tool's
set/get file timestamp operations (e.g. in `-R`/`--remote-time`). Also
to match file open operations, that already support long filenames.
E.g. when using `--remote-time`, fixing:
```
Warning: Failed to set filetime 1741363917 on outfile: CreateFile failed:
Warning: GetLastError 0x00000003
```
The special long filename logic is necessary to support Windows releases
prior to Windows 10 v1607. With the latter, it's possible to opt-in to
this behavior via a manifest setting. Note that Windows itself also needs
to opt-in to support this. Finally note that curl itself needs passing
`--globoff` to let long filenames through, pending #20044 and #20046.
Refs:
https://learn.microsoft.com/windows/win32/api/fileapi/nf-fileapi-createfileahttps://learn.microsoft.com/windows/win32/fileio/maximum-file-path-limitation
Ref: #8361
Inspired by: #19286
Inspired-by: Mathesh V
Closes#19286Closes#20040
