- make extract-unit-protos handle multi-line prototypes - but they need
to be above the implementation
- Prototypes for static functions we use in unit tests should not be in
header files. We generate lib/unitprotos.h for this purpose
- Removed some function wrappers written for unit tests and make them
use UNITTEST function directly.
- Renamed time2str() in the tool to timebuf() since we have the same
name in lib/ and in unit tests they can both be used non-static in a
build.
This reverts commit f95fadd116.
Follow-up to #21010Closes#21014
To allow building test binaries without test certs, replacing
`_CURL_SKIP_BUILD_CERTS` internal option with a build target that is
similar to `testdeps`, but without building the test certificates.
To make building test binaries a little bit faster, with less noisy
output, and without having to reconfigure the build.
Closes#20708
Tests are build in "unity"-style, by including sources into an umbrella
C files (similar to how CMake unity works). This does not play well with
clang-tidy, which seems to unconditionally ignore C sources included
like this. To fix it, curl's CMake implements a manual clang-tidy
support for tests, which compiles sources one-by-one, while also making
sure sources compile cleanly standalone (e.g. all sources need to
include `first.h`). The manual clang-tidy implementation is fragile, and
performance, in particular when targeting Windows, is abysmal.
This patch introduces an alternate solution, enabled by the
`_CURL_TESTS_CONCAT=ON` option. In this mode, umbrella sources include
the actual sources instead of `#including` them. Allowing to use CMake's
built-in clang-tidy support to compile them, with clang-tidy actually
checking the sources. Making the manual clang-tidy support unnecessary.
In the Windows CI job it results in a 4x performance improvement (4m ->
1m), making it practical to run clang-tidy on tests on Windows, in CI.
The main downside is that clang-tidy doesn't understand the `#line`
directive. Meaning issues found show the wrong filename and line number
next to them. It's not impossible to locate errors this way, but also
not convenient.
Minor/potential downside is that the concatenated source needs to be
reassembled each time an original source is updated. This may result in
more copying on the disk when used in local development. The largest
source is 1.4MB, so probably not a show-stopper on most machines.
Another is the complexity of maintaining two methods in parallel, which
may be necessary till clang-tidy understands `#line`:
https://github.com/llvm/llvm-project/issues/62405
This solution may in theory also enable adding clang-tidy support for
tests in autotools, though I haven't tried.
Targeted for curl CI for now, and used in a GHA/windows job. 100%
experimental, not recommended outside these.
Closes#20667
Drop separate `TrackMemory` (aka `CURLDEBUG`) debug feature.
After recent changes (thread-safety,
193cb00ce9, and updates leading up to
it), `TrackMemory` is unlikely to cause build or runtime issues.
To simplify builds and debug options, enable `TrackMemory`
unconditionally for debug-enabled (aka `DEBUGBUILD`) builds. Before
this patch, this was already the default, with an option to disable
it, or enable it in non-debug-enabled builds.
Note, in practice these two debug options already went hand in hand. It
was not possible to toggle them separately for a long time due to bugs,
before 59dc9f7e69 (2024-05-28) fixed it.
This patch also removes/deprecates separate knobs and feature flags for
`TrackMemory`:
- autotools: `--enable-curldebug`/`--disable-curldebug`
- cmake: `-DENABLE_CURLDEBUG=ON`/`OFF`
- C macro: `CURLDEBUG`
- libcurl: `CURL_VERSION_CURLDEBUG` symbol deprecated in favor
of `CURL_VERSION_DEBUG`. They always return the same value after this
patch.
Also:
- drop `TrackMemory` from `curl -V` output.
- rename internal `CURLDEBUG` macro to `CURL_MEMDEBUG` internally.
To avoid confusion with `DEBUGBUILD`, but to keep guarding
`TrackMemory`-related internals for readability.
- runtests: bind `TrackMemory` to debug feature. Keep it a separate
test feature requirement, for clarity.
- CI: drop test builds for combinations of the two options.
- GHA/linux: no longer disable TrackMemory in the TSAN job.
Ref: https://github.com/curl/curl/pull/20328#issuecomment-3754528407Closes#20331
A too long name is likely to cause a problem later anyway and get
reported there. We don't enforce file name lengths for any other
systems.
Assisted-by: Jay Satiro
Assisted-by: Viktor Szakats
Closes#20143
Time output:
- was broken for really large times
- now uses spaces instead of --:--:--
- >99999y is now the largest shown time
- show HH:MM:SS as long as hours are fewer than 100
Size:
- made the decimal output always only use a single decimal
Test:
- Add test 1622 to verify these functions
Closes#20177
- Use 32767-1 instead of PATH_MAX-1 (260-1) as the maximum allowable
length of a path in Windows.
Prior to this change the path sanitizer in Windows used 32767-1 as the
maximum length only for paths that had the "\\" prefix like
"\\?\longpath". Since then we added some workarounds to open longer
paths without "\\?\" prefix by normalizing the path and adding that
prefix, and the sanitizer is called before the prefix is added.
Bug: https://github.com/curl/curl/issues/20044
Reported-by: Viktor Szakats
Closes https://github.com/curl/curl/pull/20046
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
Passing the option as-is to libcurl is fine, but checking that the file
exists allows the tool to better provide a helpful message.
This now done for the following options:
--cacert, --crlfile, --knownhosts, --netrc-file, --proxy-cacert amd
--proxy-crlfile
Bonus: bail out properly on OOM errors in the --cert parser.
Reported-by: Wesley Moore
Fixes#19583Closes#19585
After this patch, the codebase no longer overrides system printf
functions. Instead it explicitly calls either the curl printf functions
`curl_m*printf()` or the system ones using their original names.
Also:
- drop unused `curl_printf.h` includes.
- checksrc: ban system printf functions, allow where necessary.
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814Closes#18866
To make the source code match the functions called at runtime.
And to avoid the preprocessor trick that may introduces build issues.
Before this patch, libtests, tunits and units were calling a mixture
of curl and system printf calls, then transformed them all to curl
printf calls by including `curl_printf.h`.
Changes made:
- tests: stop including `curl_printf.h`.
- libtest: switch a couple of outlier system printf calls to curl
printf.
- unit: use more curl printf to avoid casts and show whole values.
- unit: switch remaining calls to curl printf explicitly.
- tunit: switch to call curl printf explicitly.
- libtest, tunit, unit: ban system printf.
- unit1307, unit1607, unit1609, unit1652, unit1655, unit3214: bump
types/masks to avoid casts.
After this patch:
- libtests, tunits, units: use exclusively curl printf.
(as before, but explicitly, without relying on redefinitions.)
- servers: is unchanged (it can only use system printf).
Closes#18814
- tests: merge cmake commands.
- tests: use `target_compile_definitions()`.
- tests/server: use generator expression for platform-specific macro.
- tests/unit: sync `Makefile.am` comment with cmake.
- tests/unit: merge two `AM_CPPFLAGS` lines to keep synced with cmake.
- tests: move macro definitions to `first.h` headers from build level.
`CURL_NO_OLDIES`, `CURL_DISABLE_DEPRECATION`, `WITHOUT_LIBCURL`,
`CURL_STATICLIB` (for servers).
To share more logic.
Pass `CURL_STATICLIB` in server on all platforms for simplicity.
(On non-Windows, it's a no-op. It's already done like this with curlu
and libcurltool.)
Also for lib:
- lib: merge commands.
- lib: sync macro order with tests (also in `Makefile.am`).
Closes#17768
To simplify dependencies, and sync tunits and units builds further.
`curlcheck.h` already depended on logic implemented within libtests:
it referenced a global variable (`unitfail`) defined in `first.c` and
declared in `test.h`.
Also:
- rename to `unitcheck.h` to indicate it's meant for unit tests.
- make `unitcheck.h` include `first.h` instead of `test.h`.
This brings header use closer to libtests. It also includes
`curlx/curlx.h` for all unit tests by default now.
- move `unitfail` declaration from `test.h` to `first.h`.
To match its definition in `first.c`.
- drop now redundant per-test curlx header includes.
Closes#17868
To make `ninja units-clang-tidy` target work without manually building
core components first.
Also rename the clang-tidy test target generator macro to align its name
with the built-in `add_custom_target()` function.
Follow-up to c9bb9cd165#17750Closes#17810
It allows to pass `.h` files only where they are really needed, which is
`EXTRA_DIST` (or `*SOURCES`) for autotools.
Also:
- rename variables to be shorter and consistent.
- drop references to non-local headers. We don't need to pass them as
dist sources.
- drop empty variables and references, after the above.
- server: add placeholder `UTIL_H`.
Closes#17745
Sync how libcurltool is built in different modes and build systems.
cmake:
- build libcurltool with curlx when building shared libcurl.
To make it possible to use standard libcurl when linking tunits.
Also syncing this with autotools.
The remaining difference is that cmake allows to select shared or
static for curl tool and tests/examples independently.
- fix to link with libcurl instead of libcurlu.
To sync with autotools and to link with the standard libcurl for
tool unit tests.
- fix `source_group()` to always include curlx sources.
- add missing 'curlx header files' source group.
autotools:
- build libcurltool without curlx when building static libcurl in
non-unity builds.
To avoid double compilation, just to be thrown away at link time.
Also to sync with unity builds.
both:
- sync source order between autotools and cmake.
- make sure to pass all headers with both autotools and cmake.
This is a no-op with cmake. Maybe a future patch should make sure
to not pass those to remove that noise.
Ref: #17696Closes#17727
To make all src and test code refer to curlx headers the same way.
Also:
- src: move `curlx.h` include to `tool_setup.h`.
- src/tool_setup.h: drop stray `curlx/timeval.h`.
- servers: de-duplicate `curlx.h` and `curl_setup.h` includes.
- libtests, units: drop stray curlx sub-headers in favor of
`<curlx/curlx.h>`.
- tests: include `curlx.h` with `<>` instead of `""`. To match
other parts of the codebase.
Closes#17680
Replace existing `mk-unity.pl` `--embed` workaround with running
`clang-tidy` manually on individual test source instead. This aligns
with how clang-tidy works and removes `mk-unity.pl` from the solution.
Also:
- mqttd: fix potentially uninitialized buffer by zero filling it.
```
tests/server/mqttd.c:484:41: error: The left operand of '<<' is a garbage value
[clang-analyzer-core.UndefinedBinaryOperatorResult,-warnings-as-errors]
484 | payload_len = (size_t)(buffer[10] << 8) | buffer[11];
| ^
[...]
tests/server/mqttd.c:606:45: error: The left operand of '<<' is a garbage value
[clang-analyzer-core.UndefinedBinaryOperatorResult,-warnings-as-errors]
606 | topiclen = (size_t)(buffer[1 + bytes] << 8) | buffer[2 + bytes];
| ^
```
- sockfilt: fix potential out-of-bound pointer:
```
tests/server/sockfilt.c:1128:33: error: The 2nd argument to 'send' is a buffer
with size 17010 but should be a buffer with size equal to or greater than
the value of the 3rd argument (which is 18446744073709551615)
[clang-analyzer-unix.StdCLibraryFunctions,-warnings-as-errors]
1128 | ssize_t bytes_written = swrite(sockfd, buffer, buffer_len);
| ^
```
- clang-tidy: suppress bogus `bzero()` warnings that happens
inside the notorious `FD_ZERO()` macros, on macOS.
Ref: https://github.com/curl/curl/pull/17680#issuecomment-2991730158Closes#17705
Derive it from `$BUNDLE` instead. autotools seems to be already relying
on `$BUNDLE_SRC` being equal to `$BUNDLE.c`. (I haven't realized this
before aaebb45f58b3f62876a68c17c71ac37d98f1b3bb.)
Also drop redundant `nodist_<target>_SOURCE` lines in tunits and units.
Follow-up to aaebb45f58#17688
Follow-up to 2c27a67daa#17590Closes#17692
To reduce the diff between tests CMakeFiles.txt, and syncing with
autotools, which already used the `BUNDLE` variable like this.
Also:
- fold lines that went over 132 chars after this change.
- autotools: sync order of macros with cmake.
Closes#17667
- GHA/windows: make a mingw autotools build static only.
- GHA/windows: fix a CI script issue with the build above.
- src: fix to pass `LIBCURL_PC_LIBS_PRIVATE` instead of `LINKFLAGS`.
This makes the libs propagate to tunits, making the local hack there
unnecessary. `LINKFLAGS` had this single use in the repo, and it was
empty in local tests.
- tests: drop passing redundant `LIBCURL_PC_LDFLAGS_PRIVATE`.
- tests: drop redundant target name from config variables.
- examples, tests/client: drop `LIBDIR` temp variables with single uses.
- examples, tests: formatting to sync `Makefile.am` scripts with each
other.
Closes#17661
Instead of relying on CMake's built-in unity feature, use `mk-unity.pl`,
as already done with autotools. It simplified the build, shortens logs
and makes debugging easier because of the fewer build variations.
It also allows testing / fixing with cmake and those automatically apply
to autotools builds too. cmake builds can be much-much faster, esp.
when working the builds themselves.
It also enables "unity" in old cmake versions. Basically every test
target is a single generated .c source.
Also:
- drop a `lib` unity workaround for libtests with autotools after fixing
the issue in libtests itself. It drops a few exceptions and makes
libcurl build faster (in autotools unity).
- fix another `lib` autotools unity issue and drop the workaround for it
from `mk-unity.pl`. `srcdir` was missing from the header path.
- simplify `mk-unity.pl` command-lines, drop exclusions.
Follow-up to 2c27a67daa#17590Closes#17628
Make test bundles the default. Drop non-bundle build mode.
Also do all the optimizations and tidy-ups this allows, simpler builds,
less bundle exceptions, streamlined build mechanics.
Also rework the init/deinit macro magic for unit tests. The new method
allows using unique init/deinit function names, and calling them with
arguments. This is in turn makes it possible to reduce the use of global
variables.
Note this drop existing build options `-DCURL_TEST_BUNDLES=` from cmake
and `--enable-test-bundles` / `--disable-test-bundles` from autotools.
Also:
- rename test entry functions to have unique names: `test_<testname>`
This removes the last exception that was handled in the generator.
- fix `make dist` to not miss test sources with test bundles enabled.
- sync and merge `tests/mk-bundle.pl` into `scripts/mk-unity.pl`.
- mk-unity.pl: add `--embed` option and use it when `CURL_CLANG_TIDY=ON`
to ensure that `clang-tidy` does not miss external test C sources.
(because `clang-tidy` ignores code that's #included.)
- tests/unit: drop no-op setup/stop functions.
- tests: reduce symbol scopes, global macros, other fixes and tidy-ups.
- tool1621: fix to run, also fix it to pass.
- sockfilt: fix Windows compiler warning in certain unity include order,
by explicitly including `warnless.h`.
Follow-up to 6897aeb105#17468Closes#17590
To avoid redundant work in CI and to avoid a single checksrc issue make
all autotools jobs fail. After this patch checksrc issues make fail
the checksrc job, the `dist / verify-out-of-tree-autotools-debug`,
`dist / maketgz-and-verify-in-tree` jobs and the fuzzer job (if run).
Of these, the `dist` jobs replicate local builds, also testing the build
logic.
Also add a script to check the complete local repository, optionally
with the build tree to verify generated C files.
Also:
- automatically run checksrc in subdirectories having a `checksrc`
target. (examples, OS400, tests http/client, unit and tunit)
- tests/libtest: make sure to run `checksrc` on generated `lib1521.c`.
(requires in-tree autotools build.)
- tests: run `checksrc` on targets also for non-`DEBUGBUILD`
builds. It ensures to check `lib1521.c` in CI via job
`dist / maketgz-and-verify-in-tree`.
- src: drop redundant `$(builddir)` in autotools builds.
- scripts: add `checksrc-all.sh` script to check all C sources and
the build directory as an option.
- use the above from CI, also make it verify all generated sources.
- silence `checksrc` issues in generated C sources.
- checksrc: add `-v` option to enable verbose mode.
- checksrc: make verbose mode show checked filename and fix to only
return error on failure.
- make sure that generated C files pass `checksrc`.
Assisted-by: Daniel Stenberg
Closes#17376
- add more unusual input cases
- add a valid non-http protocol
- fix tests so an input that should be stripped but isn't is a failure
- fix detection of when stripcredentials() would be available to test
- avoid using a NULL pointer
Closes#17304
Move curlx_ functions into its own subdir.
The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.
The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.
The strcase defines are not curlx_ functions and should not be used by
tool or server code.
dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.
When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.
Assisted-by: Jay Satiro
Closes#17253
- unit tests need no tool code as they are libcurl unit tests
- unit test 1621 is now tunit test 1621 instead, as it tests tool code
- build unit tests with BUILDING_LIBCURL as they pretent to be libcurl
Closes#17259
