- explicitly install `libldap-dev` to not rely on test-specific packages
installing it implicitly, to have the same `curl -V` output for each
TLS backend build pair.
Follow-up to 0455d8772a#18509
- install `libev-dev` for tests. It's a runtime dependency for
the local build of `nghttpx`. Missing it made pytest skip 178 tests.
Also skewing the 'Gain' time. I estimate it to account for 3 minutes,
making the total gain ~20 minutes.
Follow-up to 0455d8772a#18509
(It may be a better solution to disable libev for the local nghttp2
build, to avoid this hidden dependency.)
- fix quiche jobs to use the local build of `libnghttp2`.
- stop installing the `clang` package for Windows-cross. `clang` and
`clang-tidy` tools are preinstalled on the Ubuntu 24.04 runner.
Closes#18519
In the last couple of months some jobs started taking a lot of time and
often timing out due to slow `apt install` from the Azure Ubuntu mirror.
The jobs affected were those that installed large packages:
GHA/http3-linux and the 3 cross-build jobs in GHA/windows.
This patch reduces the installed packaged to the minimum required
to complete the jobs. Saving a minute+ for each http3-linux job (a total
of 20+ minutes for the workflow.) Also saving bandwidth and reducing
the chance for long downloads or timeouts with slow Azure repos.
Details:
- http3: delete redundant packages from the `build-cache` job.
- http3: install gnutls dependencies for gnutls jobs only.
- http3: do not install test dependencies in jobs not running tests.
- http3: drop redundant packages from the curl jobs.
- Windows-cross: replace `mingw-w64` with `gcc-mingw-w64-x86-64-win32`
for the 3 Windows cross-build job. Dropping C++, 32-bit, and 64-bit
POSIX-threaded parts. Saving time and significant bandwidth for each
of the 3 jobs:
Download size: 277 MB -> 65 MB (installed: 1300 MB -> 400 MB)
- Windows-cross: restore previous job time limit of 15m (from 45m)
Follow-up to ff5140a25f#18163
Before:
https://github.com/curl/curl/actions/runs/17611514207 (http3)
https://github.com/curl/curl/actions/runs/17611514185/job/50034354923 (Windows cross)
After:
https://github.com/curl/curl/actions/runs/17628406362?pr=18509 (http3)
https://github.com/curl/curl/actions/runs/17627562551/job/50088055529?pr=18509 (Windows cross)
http3 job | Bef. | Aft. |
:------------------ | ------: | ------: |
Build caches (hot) | 10s | 12s |
AM awslc | 3m 0s | 1m 54s |
CM awslc | 4m 32s | 3m 4s |
AM boringssl | 3m 9s | 1m 48s |
CM boringssl | 3m 43s | 3m 2s |
AM gnutls | 3m 9s | 2m 18s |
CM gnutls | 4m 19s | 2m 55s |
AM libressl | 2m 14s | 1m 24s |
CM libressl | 5m 30s | 2m 57s |
AM openssl | 5m 16s | 4m 17s |
CM openssl | 1m 50s | 1m 47s |
AM openssl-quic | 2m 58s | 1m 7s |
CM openssl-quic | 4m 16s | 2m 43s |
AM quiche | 2m 54s | 1m 34s |
CM quiche | 5m 0s | 3m 15s |
AM quictls | 2m 34s | 1m 13s |
CM quictls | 4m 20s | 3m 17s |
AM wolfssl | 2m 48s | 1m 30s |
CM wolfssl | 4m 49s | 3m 22s |
Total: | 66m 21s | 43m 27s |
Gain: | | 22m 54s |
Out of curiousity, build times as seen in the http3 build-cache job:
- TLS backends:
- openssl: 2m25s
- libressl: 27s
- aws-lc: 41s
- boringssl: 1m8s
- quictls: 1m46s
- gnutls: 6m30s
- wolfssl: 51s
- quiche + boringssl: 1m9s
- ng* libs (not yet optimized for build speed):
- nghttp3: 13s
- ngtcp2: 52s (with 6 backends, 3 runs)
- ngtcp2: 19s (boringssl)
- nghttp2: 21s
Ref: https://github.com/curl/curl/actions/runs/17626120054/job/50083344805
A similar effort in curl-for-win, affecting 2 GHA/curl-for-win Windows
jobs (though they use the default Debian repo, with no issues):
- with llvm/clang:
Download size: 648 MB -> 430 MB (installed: 3344 MB -> 2333 MB)
- with gcc:
Download size: 550 MB -> 328 MB (installed: 2815 MB -> 1804 MB)
Ref: e19665d948
Ref: 6b14c3946a
Bug: https://github.com/curl/curl/pull/18502#issuecomment-3270259744Closes#18509
Perl got bumped from 5.38.4 to 5.40.3. The new version crashes when
loading the `Win32::Process*` modules built and cached in CI. The build
job uses Perl 5.38.4.
To avoid the crash, include the Perl version (hashed) in the cache key,
so that it's only loaded when the Perl version matches.
This solution is imperfect, because some of the jobs will not use the
Perl modules in transition periods, when different jobs use different
Perl versions. Anyway, can't think of a better one for now. Another
option is to drop the effort with these modules. After all they did not
help with crashes and hangs, nor with performance. While adding quite
a bit of CI complexity.
Also:
- test early if the modules load and log the result.
Follow-up to 52775a7fb4#18296Closes#18425
It's causing false-positives with clang-tidy v21, in cases in system
headers (seen in `FD_ISSET()` with macOS SDK). In some cases in
tests/server, there was no distinct source line that was triggering it.
Example:
```
/Applications/Xcode_16.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.5.sdk/usr/include/sys/_types/_fd_def.h:83:10: error: Potential out of bound access to 'fds_read.fds_bits' with tainted index [clang-analyzer-security.ArrayBound,-warnings-as-errors]
83 | return _p->fds_bits[(unsigned long)_fd / __DARWIN_NFDBITS] & ((__int32_t)(((unsigned long)1) << ((unsigned long)_fd % __DARWIN_NFDBITS)));
| ^
[...]
/Users/runner/work/curl/curl/tests/server/socksd.c:679:5: note: Taking false branch
679 | if(rc < 0) {
| ^
```
Closes#18422
To make the CI jobs use native Win32 API calls instead of calling
external tools to look up and kill PIDs of native Windows test server
processes.
Follow-up to 2388b0e587#18308Closes#18296
It's somewhat flaky, slow (6-8 minutes), needs maintenance, and did not
turn up real issues to justify keeping.
Notably it did not help catch a regression seen on Solaris OS: #16915
Follow-up to 90e644f944#13583Closes#18314
This patch fixes flakiness caused by MSBuild scanning the runtests.pl
output for regex patterns. When finding a hit, it returns an error code
to cmake, making the build test CI step fail. This happens rarely after
an earlier mitigation tweaking outputs, but, as expected, it did not
resolve it completely.
MSBuild doesn't have an option to disable this behavior. To fix, this
patch migrates the two affected jobs from MSBuild to Ninja. To align
with existing multi-config logic, it uses the `Ninja Multi-Config`
generator, which hasn't been tested before in CI.
Switching to Ninja was not trivial. Visual Studio to this day relies on
an MS-DOS batch file stored at an unstable location (containing spaces
and parenthesis), to initialize its environment. Without this env,
`cl.exe` is unable to find its own components. GHA does not initialize
it (even if it did, it could only default to a single specific target).
CMake helps with this when using a Visual Studio generator, but doesn't
when using Ninja. (On local machines the VS installer adds a couple
of Start menu items for launching pre-configured command prompts.)
Ref: https://learn.microsoft.com/cpp/build/building-on-the-command-line
The MS-DOS batches don't integrate well with CI envs and even less so
with shell scripts. To avoid it, this patch uses manual configuration.
Also without using environment variables, to make it easy to use and
easy to debug and trace in logs. Configuring Visual Studio is relatively
stable across releases and hasn't changed a whole lot in the last 2
decades, but still may need more maintenance compared to llvm, or pretty
much any other toolchain out there. On the upside, it allows to manually
select compiler version, SDK version, cross-combinations, and allows
choosing clang-cl. The configuration aims to find the latest of these
automatically.
Some traps that had to be avoided:
- need to switch to MS-DOS short names to avoid spaces in the VS
component paths.
- need to switch to forward slashes to avoid confusing downstream tools
with backslashes.
- need to pass either MSYS2 for Windows-style path depending on setting.
- need to use a trick to retrieve the oddly named `ProgramFiles(x86)`
Windows env from shell script.
- need to match VS version (2022) and edition (Enterprise), found on GHA
runners.
- need to pass the CMake generator via env so that the space in the name
doesn't trip the shell when passed via a variable.
- trash and unexpected dirs when detecting SDK/toolchain versions.
- need to pass `-external:W0` to the C compiler to avoid MSVC warning:
`D9007: '/external:I' requires '/external:W'; option ignored`
- using cmake options only, to make it run without relying on envs and
work out-of-the-box when running subsequent cmake sessions.
- some others discovered while making work clang-cl locally in
cross-builds.
Ninja also improves performance in most cases (though wasn't a goal
here). After this patch configure is significantly faster (1.5-2x),
builds are a tiny bit faster, except examples which was twice as fast
with MSBuild. Disk space use is 10% lower.
MSBuild builds remain tested in AppVeyor CI and the UWP job.
Before: https://github.com/curl/curl/actions/runs/17025737223/job/48260856051
After: https://github.com/curl/curl/actions/runs/17027981486/job/48266133301
Fixes:
```
=== Start of file stderr1635
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 4 100 4 0 0 449 0 --:--:-- --:--:-- --:--:-- 500
curl : (22) The requested URL returned error : 429 [D:\a\curl\curl\bld\tests\test-ci.vcxproj]
CUSTOMBUILD : warning : Problem : HTTP error. Will retry in 1 second. 1 retry left. [D:\a\curl\curl\bld\tests\test-ci.vcxproj]
[...]
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for 'D:\a\curl\curl\bld\CMakeFiles\621f80ddbb0fa48179f056ca77842ff0\test-ci.rule;D:\a\curl\curl\tests\CMakeLists.txt' exited with code -1. [D:\a\curl\curl\bld\tests\test-ci.vcxproj]
Error: Process completed with exit code 1.
```
Ref: https://github.com/curl/curl/actions/runs/16966304797/job/48091058271?pr=18287#step:13:3471
Bug: https://github.com/curl/curl/discussions/14854#discussioncomment-14104166
Ref: a19bd43210#18307
Follow-up to 9463769f2e#16583Closes#18301
Make the:
- mbedTLS valgrind job finish under 14m, vs 15m before.
- OpenSSL -O3 valgrind job finish in 14m30, vs 16m17.
- OpenSSL libssh2 valgrind job finish in 16m, vs 17m30.
- long valgrind rustls job finish 1 minute earlier, in return
for spending 30s more on the other rustls job.
Keep using autotools for the less slow valgrind job to test this combo.
Closes#18290
Replace autotools with cmake to avoid libtool wrappers that are changing
`LD_LIBRARY_PATH` in a way incompatible with the thread sanitizer.
To fix the output when the sanitizier is finding something:
```
==51718==WARNING: Can't write to symbolizer at fd 7
/usr/bin/llvm-symbolizer-18: /home/runner/work/curl/curl/bld/lib/.libs/libcurl.so.4: no version information available (required by /usr/bin/llvm-symbolizer-18)
/usr/bin/llvm-symbolizer-18: symbol lookup error: /home/runner/openssl/lib/libcrypto.so.3: undefined symbol: __tsan_func_entry
```
Ref: https://github.com/curl/curl/actions/runs/16911402500/job/47913783729#step:39:4466
After:
```
13:50:04.117885 == Info:ThreadSanitizer: thread T1 finished with ignores enabled, created at:
closing connection #0#0 pthread_create <null> (libtests+0x6bc0f) (BuildId: 4fe889446291259934205ac03931c397aa0210d3)
#1 Curl_thread_create /home/runner/work/curl/curl/lib/curl_threads.c:73:6 (libcurl.so.4+0x55a76) (BuildId: cb0f14ba2ad68c9cab0c980d9a5d7a53cc0782da)
#2 async_thrdd_init /home/runner/work/curl/curl/lib/asyn-thrdd.c:500:26 (libcurl.so.4+0x1c153) (BuildId: cb0f14ba2ad68c9cab0c980d9a5d7a53cc0782da)
[...]
```
Ref: https://github.com/curl/curl/actions/runs/16939193922/job/48003405272?pr=18274#step:39:4018
Also:
- disable memory tracker which turned out to be incompatible with
the thread sanitizer and detaching threads.
Ref: #18263 and #curl IRC.
- the job is ~30 seconds faster after this patch.
Reported-by: Stefan Eissing
Bug: https://github.com/curl/curl/pull/18263#issuecomment-3179279440
Follow-up to a2bcec0ee0#14751Closes#18274
actions/checkout@v5 requires a newer libstdc++ than the container
contains. Update it to a backwards-compatible version just like we
already do for libc6.
Ref: #18250Closes#18255
In CI we want to ensure that examples build cleanly, but we don't want
to actually run them there. Meaning it's enough to just compile, but not
link them in CI. Saving time up to 2-4x (MSVC), and disk space up
to 1.2GB (or 8-70x).
Add a new cmake target that compiles all examples without linking them
into runnable binaries. Keep a full build for a single example to test
if it links correctly.
Also:
- CI: switch over all `curl-examples` targets to `curl-examples-build`
- GHA/linux-old: build examples in one of the cmake builds.
Result highlights:
Job | Bef. | Bef. | Aft. | Aft. |
:------------------ | ---: | ----: | ---: |----: |
cygwin | 15s | 9MB | 10s | 1MB |
msys | 13s | 8MB | 7s | 1MB |
dl-mingw 15 | 39s | 113M | 34s | 2MB |
dl-mingw 9.5.0 | 49s | 115MB | 42s | 2MB |
dl-mingw 7.3.0 | 19s | 113MB | 14s | 2MB |
dl-mingw 6.4.0 | 9s | 12MB | 7s | 4MB |
Linux cross | 19s | 28MB | 19s | 2MB |
MSVC UWP | 65s | 374MB | 9s | 17MB |
MSVC x64 | 22s | 846MB | 9s | 17MB |
VS2010 | 48s | 105MB | 15s | 9MB |
VS2022 clang-cl | 195s | 1.2GB | 51s | 20MB |
iOS Xcode | 8s | | 5s | |
macOS LibreSSL | 16s | | 11s | |
Linux aws-lc | 3s | | 1s | |
Follow-up to dda251ef10#18232Closes#18209
By micromanaging the project dependency and its inclusion into the test
project. It feels like an awkward construct, but perhaps better than
nothing.
It's also fragile because it's a static build with no assistance from
the external project (curl in this case). Mitigated in test by disabling
all dependencies and some features.
Since there is no special core cmake logic to be tested here, in CI
the test is tested really. To keep CI jobs at minimum, only add 3 of
them, taking 42s in total. (All 6 would take 270s.)
Follow-up to e2a23d5d0d#17203Closes#18208
