To reduce log noise in local test runs:
- move the `buildinfo.txt` dump and header info lines
`OS:`, `Perl:`, `diff:` behind a `--ci` `runtests.pl` option.
- enable this option for the CI test targets.
- hide `Env:` header info line if empty.
- merge `Env:` output into a single `logmsg()` call.
Closes#18147
Also:
- GHA/linux, macos: test `install` with autotools too.
- GHA/linux, macos: enable fish and zsh completion in an autotools job.
- scripts: drop dynamic perl checks redundant after this patch.
- scripts: drop two interim variables.
- autotools: make `build-certs` target check perl first.
- autotools: replace `$(PERL)` with `@PERL@` to match the pattern used
in most automake scripts. For consistency. This makes `PERL` be
defined at configure-time, as opposed to make-time, for these cases
now.
Closes#18141
Before this patch there was no (easy) way to run tests without various
verbose options, hindering readability of the output due its length.
Unfortunately CMake reserves `test` as a target name, which was
the reason for its removal earlier. Re-add it with the name `tests`,
to have the functionality without the name collision.
Also add a `tests` target as an alias to `test` for autotools.
Ref: https://cmake.org/cmake/help/v4.1/policy/CMP0037.html
Follow-up to cfea4f2f4d#6258Closes#18145
libtests and clients were built the same way after recent overhauls.
libtests are used by runtests, clients by pytests.
Merge clients into libtests, aligning their entry function signature,
dropping common utility functions, and simplifying the build.
Note: After this patch `CURLDEBUG` applies to cli tests, when enabled.
Also:
- lib552: drop local copy-paste debug callback in favor of testtrace.
- lib552: drop local copy-paste dump function in favor of testtrace.
- clients: use `long` for HTTP version, drop casts.
- clients: replace local dump function in favor of testrace clone.
- sync cli test entry function prototype with libtests'.
- h2_serverpush: replace local trace callback with testtrace.
- de-duplicate 3 websocket close, ping, ping, functions. Kept the pong
iteration from `ws_pingpong`. Note: the pong clone in `lib2304` was
returning an error when `curl_ws_recv()` returned non-zero and
the payload matched the expected one anyway. After this patch, this
case returns success, as it does in `ws_pingpong`.
`lib2304` keeps passing, but I'm not sure if the previous behavior
was intentional.
- display full value in websocket close, ping, pong, drop casts.
Closes#18079
To have all the tests binaries directly under the tests directory.
There seems to be no issue adding non-http test clients to this subdir.
Closes#17637
Make test bundles the default. Drop non-bundle build mode.
Also do all the optimizations and tidy-ups this allows, simpler builds,
less bundle exceptions, streamlined build mechanics.
Also rework the init/deinit macro magic for unit tests. The new method
allows using unique init/deinit function names, and calling them with
arguments. This is in turn makes it possible to reduce the use of global
variables.
Note this drop existing build options `-DCURL_TEST_BUNDLES=` from cmake
and `--enable-test-bundles` / `--disable-test-bundles` from autotools.
Also:
- rename test entry functions to have unique names: `test_<testname>`
This removes the last exception that was handled in the generator.
- fix `make dist` to not miss test sources with test bundles enabled.
- sync and merge `tests/mk-bundle.pl` into `scripts/mk-unity.pl`.
- mk-unity.pl: add `--embed` option and use it when `CURL_CLANG_TIDY=ON`
to ensure that `clang-tidy` does not miss external test C sources.
(because `clang-tidy` ignores code that's #included.)
- tests/unit: drop no-op setup/stop functions.
- tests: reduce symbol scopes, global macros, other fixes and tidy-ups.
- tool1621: fix to run, also fix it to pass.
- sockfilt: fix Windows compiler warning in certain unity include order,
by explicitly including `warnless.h`.
Follow-up to 6897aeb105#17468Closes#17590
Using a mixture of techniques to avoid symbols collisions:
- reduce scope.
- add `t*_` / `T*_` prefix.
- move shared functions to `testutil.c`.
(`suburl()`, `rlim2str()`)
- clone re-used lib*.c sources.
(lib587, lib645)
- include shared symbols just once in re-used `lib*.c` sources.
(using `LIB*_C` guards.)
- drop re-used `lib*.c` sources where they were identical or
unused.
- make macros global.
- #undef macros before use.
What remain is the entry functions `test`, and `unit_setup`,
`unit_stop` in unit tests.
Also:
- fix formatting and other minor things along the way.
- add `const` where possible.
- sync some symbol names between tests.
- drop `mk-bundle-hints.sh` that's no longer necessary.
Closes#17468
In an attempt to make them easier to find.
The man pages runtests.md and testcurl.md are in /docs
The rest of the test documentation is in /docs/tests
Closes#17463
Require now pytest-xdist from tests/http/requirements.txt and
run pytest in 'auto' parallel mode (counts cpu cores).
For CI runs, set the worker count to 4, overriding the
core count of 2 exposed in the images.
- use Filelock to generate allocated ports at start for all
workers and have subsequent workers just read the file and
take the ports for their slot
- make httpd config clearing a function fixture so every test
starts with a clean httpd config
- have fixture `configures_httpd` as parameter of test cases
that configure httpd anyway, saving one reload
- add pytest-xdist and filelock to required pyhton modules
- add installs to ruff CI
- give live checks waiting for a server to start up longer time
- add fixtures to tests that rely on a server
- do not stop servers unnecessarily. failures may not start them
properly again, leading to unexpected fails in whatever follows
- add a https: port to httpd that is *not* back by QUIC to allow
failover tests without stopping the QUIC server
Closes#17295
Refreshed, cleaned up, improved and now checks *all* options.
This must have stopped working at some point. gcc-14 least shows these
warnings with this change, not without.
Add test 745 to verify that all options listed in curl.h is also checked
by the typechecker.
This improved checker found almost 30 mistakes in the curl git
repository.
Closes#17143
Add `--retry=<num>` option to tell runtests to retry the first `<num>`
tests that failed. Retries aren't run right away, but added to the end
of the test queue. Once all retry slots are used, test fail as normal.
In CI, typically a single test fails for flakiness, and rarely over 5.
Make the `ci-test` targets default to `--retry=5`.
Closes#17091
This document now lists all previous releases.
This allows us to verify that documentation refers to actual release
versions.
Test 971 now verifies options-in-versions and all command line options
documentation individually. Fixed a few discrepancies.
Test 1488 verifies libcurl options "Added-in" to exist. Fixed a few
discrepancies there as well.
Closes#16907
Before this patch the curl repository and source tarball distribution
contained test certificates as binary blobs. Used by runtests.
Drop these certificates in favor of generating them dynamically as
part of the build process. Both via autotools and CMake.
As part of this, improve certificates, the generator script and process,
file layout, and fix any issue to make it work fast and smooth both in
CI and local builds.
Note, cert generator scripts require OpenSSL >=1.0.2
(or LibreSSL >=3.1.0). Generation requires POSIX shell, also with CMake.
Without a POSIX shell tests relying on TLS (and stunnel) will fail.
Details:
- build: generate certs as part of the test run process.
- build, tests: generate certs in the build directory.
- binarycheck: drop concept of known binary files with hashes.
- binarycheck: move binary check logic into spacecheck and drop this
separate checker tool.
- build: fix to clean all cert files.
- autotools: fix to not run leaf cert generators in parallel. To avoid
confusion when updating the revocation database and counter.
- scripts: drop `scripts` subdir, merge two scripts into one,
auto-generate root cert, allow generating multiple leafs at once.
- scripts: switch to EC-256 keys (was: RSA-2048). For key size and perf.
- scripts: drop `-x` echo, text dumps, most other output. To avoid log
noise and make it quicker in CI.
- scripts: make it non-RSA-specific.
- scripts: delete unused code.
- scripts: use POSIX shell shebang. Some envs don't have bash (Alpine).
- scripts: pass test pseudo-secrets via the command-line. To avoid:
```
+ openssl genrsa -out test-ca.key -passout fd:0 2048
Invalid password argument, starting with "fd:"
```
- cmake: fix to launch generator scripts via the detected POSIX shell.
- cmake: fix `build-certs` rule to not depend on `SRPFILES`
(`srp-verifier-*`).
- cmake: drop `EXCLUDE_FROM_ALL` for the cert subdir. It makes
the Visual Studio generator miss to create the `clean-certs`,
`build-certs` targets. No target depend on them, so they don't execute
implicitly anyway. Fixes:
```
MSBUILD : error MSB1009: Project file does not exist.
Switch: clean-certs.vcxproj
```
- cmake: add `VERBATIM USES_TERMINAL` to `build-certs` target.
- GHA/linux: install openssl on Alpine, for the cert generator scripts.
Follow-up to 556f722fe3#16593
Follow-up to fa461b4eff#14486Closes#16824
In the cases observed throughout the last year, `handle64` run once per
test run, but with no action (match or task kill). It did not help with
flakiness and seems redundant.
runtests launched it (if present) in Cygwin/MSYS jobs too, where it
probably shouldn't have, because we have seen no flakiness there. In CI
the tool was present and launched in MSYS2 jobs, but not in Cygwin.
After this patch the "clearlocks" warning remain in the log. They are
consistently appearing once in every MSVC CI log, early in the tests:
```
test 3207 SKIPPED: curl lacks OpenSSL support
[...START-OF-TESTS...]
test 0003...[HTTP POST with auth and contents but with content-length set to 0]
--pd---e--- OK (3 out of 1596, remaining: 17:50, took 1.423s, duration: 00:02)
test 0007...[HTTP with cookie parser and header recording]
--pd--oe--- OK (7 out of 1596, remaining: 07:51, took 1.485s, duration: 00:02)
test 0006...[HTTP with simple cookie send]
--pd---e--- OK (6 out of 1596, remaining: 09:11, took 1.488s, duration: 00:02)
test 0005...[HTTP over proxy]
--pd---e--- OK (5 out of 1596, remaining: 11:03, took 1.491s, duration: 00:02)
CUSTOMBUILD : error : 169: cleardir(log/8/lock) failed [D:\a\curl\curl\bld\tests\test-ci.vcxproj]
test 0001...[HTTP GET]
--pd---e--- OK (1 out of 1596, remaining: 55:34, took 1.466s, duration: 00:02)
test 0004...[Replaced internal and added custom HTTP headers]
```
Ref: https://github.com/curl/curl/actions/runs/13546192228/job/37858323380?pr=16484#step:14:167
Ref: e53523fef0#14859
Ref: 311c31ec8e#6179
Follow-up to 3a8920e5ed#16600Closes#16484
If libtests, units and servers binaries are all present, auto-enable
bundle mode.
Drop manual runtests option.
Note: Make sure to "make clean" before changing the test bundle build
setting.
Also fix to append executable extension to all libtest and unit test
executables when launching them. This should make it a tiny bit faster
on Windows.
Follow-up to f4f25505df#15000
Follow-up to 71cf0d1fca#14772Closes#16750
Keep only the generated files needed for tests. Place generated
intermediaries in `tests/certs/gen` where they are ignored by git. No
longer generated `*.dhp` files.
Have a shorter naming scheme: `test-ca` instead of `EdelCurlRoot-ca` and
`test-localhost` instead of `Server-localhost-sv`, etc.
Remove the `stunnel` certificate as it was nearly a duplicate of
`test-localhost`.
No longer copy a generated certificates to `tests/stunnel.pem`. Let test
server default to `certs/test-localhost.pem` instead.
Closes#16593
- build: fix to exclude 'documentation' tests when building
the documentation is explicitly disabled. Both for cmake
and `./configure`.
Reported-by: Daniel Engberg
Fixes#15703
- test481, test482: fix for builds with no manual.
Reported-by: Daniel Engberg
Fixes#15703
- configure: fix to always detect Perl. Running tests require
it when run. Before this patch Perl wasn't detected when
documentation/manual/embedded-CA were all disabled,
making tests fail to start.
- test1177: add keyword `documentation`. It depends no
`curl_version_info.3`.
- GHA/linux: test cmake and `./configure` with docs and
manual disabled and tests run.
Closes#15704
They complete in 4 and 7 minutes, and do not hold back the main Linux
workflow.
Also:
- bump default parallelism for `test-torture` target to `-j20`
(was: `-j2`).
- drop redundant package install from `rustls` jobs.
Closes#15360
- CI default: raise parallelism to 20
- CI valgind: set parallelism to 6
- CI non-native: adapt parallelism for OS builds
- CI Windows: no longer ignore FTP, TFTP, MQTT and SMTP
- CI Windows: restrict test timeout to 10 minutes
- CI Windows: do not run tests for msh3 build
- tests, various: restrict curl invocation to ipv4 to avoid talking to
any ipv6 test server running in parallel
- tests: requiring http/2 server no longer needs to mention http server
- test 190: use a fixed timeout of 10 seconds instead of %FTPTIME2, as
that value rises under parallel load beyond what the server is waiting
and then produces different results
- test 1540: add debug logging
- testrunner: add verify checks for http/2 and http/3 server
Closes#15040
It enables running pytests in cmake jobs, regardless of underlying build
tool choice (= makes it work with ninja.)
Also:
- drop pytest logic launching `make` and exiting in case of failure.
Maybe there is a better way and keep this functionality somehow, bind
it to a command-line option? make it fail softly?
- GHA/linux: invoke pytest via the build, not directly.
- autotools: add missing dummy runtests targets when cross-compiling.
Closes#15034
- perform torture tests with '-j2' for shorter runtime
- when waiting on test results overly long, log the tests
waited for and eventually log the test log directories
for easier analysis what is wrong in CI jobs.
- sockfilt.c: treat the windows errno 109 (ERROR_BROKEN_PIPE)
as a socket closed by the client and do not exit.
- when verifying https server, do not in addition check
the http server behind it also
- when tearing down the stunnel of a non-responsive https
server, tear down the http server with it
Closes#14960
Running the http pytest had to be done from tests directory or above,
because the repeat argument fixture was defined in tests/conftest.py.
However, the repeat argument is not needed because its functionality
can be provided by pytest-repeat as documented in the test's
README.md. So, removed the pytest_addoption function for the repeat
argument and the pytest_report_header function is moved to
tests/http/conftest.py.
TODO: Remove repeat argument from all tests. As a stopgap, a
one-element list is defined for it for now.
Closes#14611
Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It then first searches for
"\nALL OPTIONS".
Then, it looks for the start to display at "\n -[option]" and stops
again at "\n -". Except for the last option in the man page, which
ends at "\nFILES" - the subtitle for the section following all options
in the manpage.
Test 1707 to 1710 verify
Closes#13997
The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on
results of a number of test runs. Some CI services use super-
oversubscribed build machines that can barely run the curl tests
already with no parallelism without frequently failing with
timing-induced failures. These continue to be run without parallelism.
Other services provide two fast, unloaded cores and these run with 14
processes, which is a good default for this kind of environment.
Here's a summary of the number of test processes by CI service:
Appveyor - 2 (Windows MSVC), 1 (others)
Azure - 2
Circle CI - 14
Cirrus - 28 (macOS), 14 (Linux), 7 (FreeBSD), 5 (macOS torture), 2 (Windows)
GitHub Actions - 3 (macOS), 2 (Linux)
Some of these are a bit conservative to keep timing-induced flakiness down.
The net result is that the first test results should arrive only
3 minutes after a commit submission.
Changes merged via separate commits:
- 2a7c8b27fd#14171
- 72341068a2
- efce544418#14244
- c6cf411bac
Ref: #10818Closes#11510
As runtests.md and testcurl.md. Very few people actually need these as
manpages anyway.
With this, we have no more nroff formatted documents in git.
Closes#14324
- generate AVAILABILITY manpage sections automatically - for consistent
wording
- allows us to double-check against other documumentation (symbols-in-versions
etc)
- enables proper automation/scripting based on this data
- lots of them were wrong or missing in the manpages
- several of them repeated (sometimes mismatching) backend support info
Add test 1488 to verify "added-in" version numbers against
symbols-in-versions.
Closes#14217
`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be guarded with
`DEBUGBUILD`.
Replace these uses with `DEBUGBUILD`.
This leaves `CURLDEBUG` uses solely for its intended purpose: to enable
the memory tracking debug feature.
Also:
- autotools: rely on `DEBUGBUILD` to enable `checksrc`.
Instead of `CURLDEBUG`, which worked in most cases because debug
builds enable `CURLDEBUG` by default, but it's not accurate.
- include `lib/easyif.h` instead of keeping a copy of a declaration.
- add CI test jobs for the build issues discovered.
Ref: https://github.com/curl/curl/pull/13694#issuecomment-2120311894Closes#13718
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:
- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)
tools:
- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown
This setup generates .3 versions of all the curldown versions at build time.
CI:
Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...
Closes#12730
It is hard to name the scripts sensibly. Lots of them are similarly
named and the name did not tell which test that used them.
The new approach is rather to name them based on the test number that
runs them. Also helps us see which scripts are for individual tests
rather than for general test infra.
- badsymbols.pl -> test1167.pl
- check-deprecated.pl -> test1222.pl
- check-translatable-options.pl -> test1544.pl
- disable-scan.pl -> test1165.pl
- error-codes.pl -> test1175.pl
- errorcodes.pl -> test1477.pl
- extern-scan.pl -> test1135.pl
- manpage-scan.pl -> test1139.pl
- manpage-syntax.pl -> test1173.pl
- markdown-uppercase.pl -> test1275.pl
- mem-include-scan.pl -> test1132.pl
- nroff-scan.pl -> test1140.pl
- option-check.pl -> test1276.pl
- options-scan.pl -> test971.pl
- symbol-scan.pl -> test1119.pl
- version-scan.pl -> test1177.pl
Closes#12487
CI builds will now run these tests, but will ignore the results if they
fail. The relevant tests are ones that are sensitive to timing or
have edge conditions that make them more likely to fail on CI servers,
which are often heavily overloaded and slow.
This change only adds two additional tests to be ignored, since the
others already had the flaky keyword.
Closes#11865
At least FreeBSD preserves cwd across makefile lines, so rules
consisting of more than one "cd X; do_something" must be explicitly run
in a subshell to avoid this. This problem caused the Cirrus FreeBSD
build to fail when parallel make jobs were enabled.
- Makefile support for building test specific clients in tests/http/clients
- auto-make of clients when invoking pytest
- added test_09_02 for server PUSH_PROMISEs using clients/h2-serverpush
- added test_02_21 for lib based downloads and pausing/unpausing transfers
curl url parser:
- added internal method `curl_url_set_authority()` for setting the
authority part of a url (used for PUSH_PROMISE)
http2:
- made logging of PUSH_PROMISE handling nicer
Placing python test requirements in requirements.txt files
- separate files to base test suite and http tests since use
and module lists differ
- using the files in the gh workflows
websocket test cases, fixes for we and bufq
- bufq: account for spare chunks in space calculation
- bufq: reset chunks that are skipped empty
- ws: correctly encode frames with 126 bytes payload
- ws: update frame meta information on first call of collect
callback that fills user buffer
- test client ws-data: some test/reporting improvements
Closes#11006
This is currently useful for starting a test server on its own without
an associated test, which can be used for interactive curl testing or
for validating parts of the test harness itself. More commands can be
added to perform additional functions in the future.
Ref: #10818Closes#11008
testutil.pm now contains a few miscellaneous functions that are used in
several places but have no better place to live. subvariables moves to
servers.pm since most variables that it substitutes relate to servers,
so this is the most appropriate place. Rename a few functions for better
naming consistency.
Ref: #10818Closes#10995
This is code that is directly responsible for running a single test.
This will eventually run in a separate process as part of the parallel
testing project.
Ref: #10818
This helps enforce more modularization and encapsulation. Enable and fix
warnings on a few packages. Also, rename ftp.pm to processhelp.pm since
there's really nothing ftp-specific in it.
Ref: #10818
