- update `actions/cache` from 5.0.3 to 5.0.4
- update `actions/download-artifact` from 7.0.0 to 8.0.1
- update `actions/upload-artifact` from 6.0.0 to 7.0.0
- update `msys2/setup-msys2` from 2.30.0 to 2.31.0
- update pip `ruff` from 0.15.7 to 0.15.8
Closes#21195Closes#21197
- fail if all retries failed.
- run `dpkg --configure -a` after an aborted slow attempt.
```
Selecting previously unselected package libsys-hostname-long-perl.
Error: slow server, retry
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.
[...]
```
Bug: https://github.com/curl/curl/pull/21107#issuecomment-4163506100
Follow-up to 5172ba5475#21107Closes#21181
It's been going on for almost a year, chances seem slim to have this
fixed upstream.
The observed issue is that on GitHub-supplied CI runner images,
sometimes, `apt-get install` using the stock distro server
`http://azure.archive.ubuntu.com/ubuntu` sees extreme slowness while
downloading data, then reaching the job timeout and aborting, making CI
red and necessitating a manual restart of the failed job(s).
In majority of cases the install step takes no longer than 8-40 seconds
(download + install), while in the failing case it takes 10-20+ minutes
just to download.
Earlier I slimmed down installs to avoid unnecessary packages, which
mitigated, but did not completely fix the issue.
Example:
https://github.com/curl/curl/actions/runs/23599596928/job/68725576899?pr=21105
Follow-up to 0455d8772a#18509Closes#21107
Verify if lines are not longer than 192 characters. Also verify if lines
have less than 79 repeat spaces (and fix one fallout).
To improve readability by avoiding long lines and to prevent adding
overly long lines with text that may go unnoticed in an editor or diff
viewer.
In addition to pre-existing line length limits: 79 for C, 132 for CMake
sources.
Also:
- spacecheck: fix/harden allowlist regexes.
- spacecheck: tidy-up quotes and simplify escaping.
- spacecheck: allow folding strings with repeat spaces.
- GHA: fix a suppressed shellcheck warning.
- GHA/macos: simplify by dropping brew bundle.
- test1119.pl: precompile a regex.
- FAQ.md: delete very long link to a Windows 7/2008 support article
that's lost it relevance.
Closes#21087
ngtcp2 1.14.0 added module dependencies to `ngtcp2_crypto_*.pc` files.
This broke certain build cases in curl, because configure was is
querying pkg-config modules by setting `PKG_CONFIG_LIBDIR` to the
directory specified via `--with-*=` options, including `--with-ngtcp2=`.
Setting `PKG_CONFIG_LIBDIR` tells pkg-config (and pkgconf) to ignore
system locations. This caused that `ngtcp2_crypto_gnutls.pc` could no
longer find its indirect dependencies, if those were present at system
locations (where they typically are). Another fallout was BoringSSL,
because it does not provide `openssl.pc` on its own, and successful
detection relied on finding a non-BoringSSL copy, typically at a system
location (also fixed in ngtcp2 main branch).
Fix `ngtcp2_crypto_*` detections to not touch `PKG_CONFIG_LIBDIR` and
instead prepend `<path>` set via `--with-ngtcp2=` to `PKG_CONFIG_PATH`.
This ensures to pick up any dependent modules from system locations.
Note the side-effect is that potentially undesired modules may be
detected this way from system locations, i.e. it makes this particular
detection less "hermetic" than the rest used in curl configure.
(Configurations using a bare `--with-ngtcp2` with no path were not
affected, and served as a workaround before this patch. It remains a
valid way of configuration after.)
Both `pkgconf` and `pkg-config` use this logic to calculate their search
directory list:
```pseudo
search = {}
if PKG_CONFIG_PATH is set
search += PKG_CONFIG_PATH
endif
if PKG_CONFIG_LIBDIR is set (even if empty)
search += PKG_CONFIG_LIBDIR
else
search += built-in-pkg-config-dirs
endif
```
Refs:
https://github.com/curl/curl/pull/18028/commits (earlier attempt. Failed due to using `PKG_CONFIG_DIR` instead of the correct `PKG_CONFIG_PATH`)
c0874ce824https://man.archlinux.org/man/pkgconf.1.enhttps://manpages.debian.org/unstable/pkgconf/pkgconf.1.en.htmlhttps://manpages.debian.org/unstable/pkg-config/pkg-config.1.en.html10e27fd63c
Bug: https://github.com/curl/curl/pull/18022#issuecomment-3120587041
Bug: https://github.com/ngtcp2/ngtcp2/pull/1689#issuecomment-3120593664
Follow-up to 04d90b5deb#20931
Follow-up to 3c64ffaff4#18415#18188
Follow-up to 99500660af#18028#18022Closes#20920
Configure LibreSSL autotools job with `--with-ngtcp=<path>` instead of
adding ngtcp2 to `PKG_CONFIG_PATH`. To test this way of configuration in
CI and test for the regression reported in #20889.
Turns out this way of configuration isn't affected by the detection
issue in this particular case.
It also works for other backends except for these two, subject to
separate fixes:
- BoringSSL fix: https://github.com/ngtcp2/ngtcp2/pull/2070
- GnuTLS fix and BoringSSL workaround: #20920
Follow-up to 666db80196#20891
Follow-up to 8db0e286b3#18189
Follow-up to 99500660af#18028#18022Closes#20926
To aid debugging cases when dependency detection acts unexpectedly.
Sprung from spending days trying to figure out behavior of ngtcp2 crypto
modules and their dependencies.
You can enable by setting env `CURL_TRACE_PKG_CONFIG` to a non-empty
value. When enabled, details are logged for both successful and
unsuccessful detections. Logging of unsuccessful ones is automatically
enabled when `CURL_CI` env is set, which is the case for all CI jobs.
It works by asking for `--debug` output and grepping for lines that seem
useful for this purpose. Output is different for classic pkg-config and
pkgconf, and may depending on tool version. Also append `--print-errors`
output if any.
Examples (with pkgconf):
Fail, before:
```
checking for libngtcp2_crypto_boringssl options with pkg-config... no
configure: error: --with-ngtcp2 was specified but could not find ngtcp2_crypto_boringssl pkg-config file.
```
Fail, after:
```
checking for libngtcp2_crypto_boringssl options with pkg-config... no
configure: pkg-config --exists libngtcp2_crypto_boringssl trace:
---- begin
trying path: /home/runner/nghttp3/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/ngtcp2-boringssl/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/nghttp3/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/ngtcp2-boringssl/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/nghttp3/build/lib/pkgconfig for openssl
trying path: /home/runner/ngtcp2-boringssl/build/lib/pkgconfig for openssl
trying path: /home/runner/nghttp2/build/lib/pkgconfig for openssl
==== error:
Package openssl was not found in the pkg-config search path.
Perhaps you should add the directory containing `openssl.pc'
to the PKG_CONFIG_PATH environment variable
Package 'openssl', required by 'libngtcp2_crypto_boringssl', not found
---- end
configure: error: --with-ngtcp2 was specified but could not find ngtcp2_crypto_boringssl pkg-config file.
```
Success, after:
```
checking for libngtcp2_crypto_boringssl options with pkg-config... found
configure: pkg-config --exists libngtcp2_crypto_boringssl trace:
---- begin
trying path: /home/runner/awslc/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/nghttp3/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/nghttp2/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/ngtcp2/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/awslc/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/nghttp3/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/nghttp2/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/ngtcp2/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/awslc/build/lib/pkgconfig for openssl
trying path: /home/runner/awslc/build/lib/pkgconfig for libssl
trying path: /home/runner/awslc/build/lib/pkgconfig for libcrypto
---- end
```
More examples:
https://github.com/curl/curl/pull/20926#issuecomment-4064259935
If there is an externally enablable, built-in feature like this in
classic pkg-config or pkgconf, I could not find it.
Also:
- GHA/http3-linux: set `CURL_TRACE_PKG_CONFIG` to log detection details.
H3 builds are prone to hard-to-debug dependency issues.
Ref: #20920
Follow-up to 3c64ffaff4#18415#18188
Follow-up to 99500660af#18028#18022
Cherry-picked from #20926Closes#20931
- update action `actions/cache` from 5.0.1 to 5.0.3
- update action `github/codeql-action` from 4.31.9 to 4.32.4
- update pip `filelock` from 3.20.3 to 3.24.3
- update pip `ruff` from 0.14.14 to 0.15.2
Closes#20782Closes#20783
To simplify setting BoringSSL version, using:
`-DBORINGSSL_VERSION=0.20260211.0`
or
`-DBORINGSSL_VERSION=${boringssl_version}`
Previously it could be set via C flags, using complicated shell quotes:
`-DCMAKE_C_FLAGS="-DCURL_BORINGSSL_VERSION=\\\"${boringssl_version}\\\""`
(the C flags method remains, also for autotools)
It'd be nice if BoringSSL published its version not just via
`MODULE.bazel` in its source tree, but from its public headers, to make
these workarounds unnecessary.
Also:
- GHA/http3-linux: test both options.
Closes#20571
- update `actions/checkout` from 6.0.1 to 6.0.2
- update `ruff` from 0.14.11 to 0.14.14
- update `cryptography` from 46.0.3 to 46.0.4
- update `psutil` from 7.2.1 to 7.2.2
- update `websockets` from 15.0.1 to 16.0
Closes#20490Closes#20491
To enable known linker options dropping unused, dead, code and data from
the executables built.
Useful to reduce binary sizes for curl, libcurl shared lib and apps
linking static libcurl. It's effective on both "unity" and non-unity
builds. Aligning "unity" build sizes with default, non-unity ones.
Supported platforms: Apple, MSVC, llvm/clang and GCC on all tested
platforms: Linux, BSDs, Windows, MSYS2/Cygwin, Android, MS-DOS.
Notes:
- Static libraries grow 20-30% with non-Apple toolchains.
This effect is controlled by separate, optional compiler flags on
non-Apple. This patch enables them automatically for public binaries
(libcurl and curl tool), and leaves them off for internal/test ones.
- MSVC enables this option by default for 'Release' configurations.
The curl build option has no effect on it.
- Observed effect on VS2010 is negligible. VS2012+ is recommended.
- Works with LTO, Fil-C.
- No observed/conclusive effect on build speed.
- On Windows with clang/gcc (mingw-w64/MSYS2/Cygwin) it also enables
`-fno-asynchronous-unwind-tables` as a workaround to make
the toolchain options actually work.
Ref: https://sourceware.org/bugzilla/show_bug.cgi?id=11539
Thanks-to: Andarwinux
Also:
- GHA: enable in Linux and MinGW jobs to test it. Size changes:
- linux aws-lc H3:
curl: 2000000 -> 1937152, libcurl.a: 2065724 -> 2716532 bytes
- macos clang HTTP-only:
curl: 1364376 -> 128799 bytes, libcurl.a: unchanged
- macos llvm MultiSSL:
curl: 410056 -> 405720, libcurl.dylib: 1350336 -> 1348480 bytes
- mingw schannel c-ares U:
curl: 1588736 -> 1507328, libcurl-d.a: 3322040 -> 3884746 bytes
bld: 34 -> 35MB
- GHA: enable in MSVC and Apple jobs to reduce disk footprint, with no
obvious downside. Size changes:
- AppVeyor CI VS2019:
curl: 2339840 -> 1295872, libcurl-d.dll: 3155968 -> 1900544 bytes
bld: 161 -> 97MB
- AppVeyor CI VS2022 clang-cl:
curl: 2933248 -> 2332160, libcurl-d.lib: 4762688 -> 5511330 bytes
bld: 133 -> 121MB
- AppVeyor CI VS2022 HTTP-only:
curl: 3514368 -> 2177024, libcurl-d.lib: 2538420 -> 3151740 bytes
bld: 137 -> 83MB
- GHA intel:
curl: 2629120 -> 2023424, libcurl-d.lib: 4366652 -> 5350670 bytes
bld: 86 -> 69MB
- GHA arm64:
curl: 2832896 -> 2063872, libcurl-d.lib: 4690616 -> 5597250 bytes
bld: 82 -> 66MB
Refs:
https://maskray.me/blog/2021-02-28-linker-garbage-collectionhttps://web.archive.org/web/20110811230637/msdn.microsoft.com/en-us/library/bxwfs976.aspx (VS2010)
https://learn.microsoft.com/cpp/build/reference/opt-optimizationshttps://learn.microsoft.com/cpp/build/reference/gy-enable-function-level-linkingCloses#20357
- It is slower and uses more memory than the alternatives and is only
experimental in curl.
- We disable a few tests for OpenSSL-QUIC because of flakiness
- It gets little attention from OpenSSL and we have no expectation of the
major flaws getting corrected anytime soon.
- No one has spoken up for keeping it
- curl users building with vanilla OpenSSL can still use QUIC through the
means of ngtcp2
Closes#20226
To simplify the directory layout.
- OS400 and vms support move from `packages` to `projects`.
- Windows README and `generate.bat` files move from `projects`
to `projects/Windows`.
Closes#20271
Turns out in practice the internal cache name contains the arch where it
matters (arm or dual-arch local builds), which is part of the cache key
string. Drop `runner.arch` to avoid two arch strings in the key.
Cache keys without an explicit arch designator mean intel.
Revert ff78af5752#20234Closes#20249
To avoid building libcurl in both static and shared flavor by default.
It results in 1.5-2.x speed-up for the curl build step in most jobs.
Saving a total of 6-7 minutes. In the Cygwin job alone it saves 1-1.5m.
Also:
- enable static + shared in a Windows job to keep testing this combo.
Follow-up to ff958fc4b2#20159Closes#20167
