configure: add option to trace pkg-config detection details

To aid debugging cases when dependency detection acts unexpectedly.
Sprung from spending days trying to figure out behavior of ngtcp2 crypto
modules and their dependencies.

You can enable by setting env `CURL_TRACE_PKG_CONFIG` to a non-empty
value. When enabled, details are logged for both successful and
unsuccessful detections. Logging of unsuccessful ones is automatically
enabled when `CURL_CI` env is set, which is the case for all CI jobs.

It works by asking for `--debug` output and grepping for lines that seem
useful for this purpose. Output is different for classic pkg-config and
pkgconf, and may depending on tool version. Also append `--print-errors`
output if any.

Examples (with pkgconf):

Fail, before:
```
checking for libngtcp2_crypto_boringssl options with pkg-config... no
configure: error: --with-ngtcp2 was specified but could not find ngtcp2_crypto_boringssl pkg-config file.
```

Fail, after:
```
checking for libngtcp2_crypto_boringssl options with pkg-config... no
configure: pkg-config --exists libngtcp2_crypto_boringssl trace:
---- begin
trying path: /home/runner/nghttp3/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/ngtcp2-boringssl/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/nghttp3/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/ngtcp2-boringssl/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/nghttp3/build/lib/pkgconfig for openssl
trying path: /home/runner/ngtcp2-boringssl/build/lib/pkgconfig for openssl
trying path: /home/runner/nghttp2/build/lib/pkgconfig for openssl
==== error:
Package openssl was not found in the pkg-config search path.
Perhaps you should add the directory containing `openssl.pc'
to the PKG_CONFIG_PATH environment variable
Package 'openssl', required by 'libngtcp2_crypto_boringssl', not found
---- end
configure: error: --with-ngtcp2 was specified but could not find ngtcp2_crypto_boringssl pkg-config file.
```

Success, after:
```
checking for libngtcp2_crypto_boringssl options with pkg-config... found
configure: pkg-config --exists libngtcp2_crypto_boringssl trace:
---- begin
trying path: /home/runner/awslc/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/nghttp3/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/nghttp2/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/ngtcp2/build/lib/pkgconfig for libngtcp2_crypto_boringssl
trying path: /home/runner/awslc/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/nghttp3/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/nghttp2/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/ngtcp2/build/lib/pkgconfig for libngtcp2
trying path: /home/runner/awslc/build/lib/pkgconfig for openssl
trying path: /home/runner/awslc/build/lib/pkgconfig for libssl
trying path: /home/runner/awslc/build/lib/pkgconfig for libcrypto
---- end
```

More examples:
https://github.com/curl/curl/pull/20926#issuecomment-4064259935

If there is an externally enablable, built-in feature like this in
classic pkg-config or pkgconf, I could not find it.

Also:
- GHA/http3-linux: set `CURL_TRACE_PKG_CONFIG` to log detection details.
  H3 builds are prone to hard-to-debug dependency issues.

Ref: #20920
Follow-up to 3c64ffaff4 #18415 #18188
Follow-up to 99500660af #18028 #18022

Cherry-picked from #20926

Closes #20931

.github/workflows/http3-linux.yml

@@ -346,6 +346,7 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 10
     env:
+      CURL_TRACE_PKG_CONFIG: '1'
       MATRIX_BUILD: ${{ matrix.build.generate && 'cmake' || 'autotools' }}
       MATRIX_INSTALL_PACKAGES: '${{ matrix.build.install_packages }}'
     strategy:

acinclude.m4

@@ -1326,6 +1326,29 @@ AC_DEFUN([CURL_EXPORT_PCDIR], [
   fi
 ])
 
+dnl CURL_TRACE_PCDIR ($module, [$pcdir])
+dnl ------------------------
+dnl show pkg-config module lookup details, along with a detailed errors
+dnl message in case of failure. Supports both pkg-config and pkgconf.
+dnl
+
+AC_DEFUN([CURL_TRACE_PCDIR], [
+  dnl Example pkgconf line:
+  dnl   libpkgconf/pkg.c:746 [pkgconf_pkg_t *pkgconf_pkg_try_specific_path(pkgconf_client_t *, [...]*)]: trying path: /usr/local/lib/pkgconfig for libngtcp2_crypto_gnutls
+  dnl Rest of strings are for catching classic pkg-config lines.
+  trc=`CURL_EXPORT_PCDIR([$2]) dnl
+    $PKGCONFIG --exists --debug $1 2>&1 | $EGREP '(trying path:|Adding directory|Looking for|Scanning directory|Cannot open directory)' | $SED 's/^.*trying path:/trying path:/'`
+  msg=`CURL_EXPORT_PCDIR([$2]) dnl
+    $PKGCONFIG --exists --print-errors $1 2>&1`
+  if test -n "$msg"; then
+    trc=`echo "$trc"; echo '==== error:'; echo "$msg"`
+  fi
+  AC_MSG_NOTICE([pkg-config --exists $1 trace:
+---- begin
+${trc}
+---- end])
+])
+
 dnl CURL_CHECK_PKGCONFIG ($module, [$pcdir])
 dnl ------------------------
 dnl search for the pkg-config tool. Set the PKGCONFIG variable to hold the
@@ -1354,10 +1377,16 @@ AC_DEFUN([CURL_CHECK_PKGCONFIG], [
     if test -z "$itexists"; then
       dnl pkg-config does not have info about the given module! set the
       dnl variable to 'no'
-      PKGCONFIG="no"
       AC_MSG_RESULT([no])
+      if test -n "$CURL_TRACE_PKG_CONFIG$CURL_CI"; then
+        CURL_TRACE_PCDIR([$1], [$2])
+      fi
+      PKGCONFIG="no"
     else
       AC_MSG_RESULT([found])
+      if test -n "$CURL_TRACE_PKG_CONFIG"; then
+        CURL_TRACE_PCDIR([$1], [$2])
+      fi
     fi
   fi
 ])

docs/INSTALL.md

@@ -135,6 +135,10 @@ curl can be built to use a whole range of libraries to provide various useful
 services, and configure tries to auto-detect a decent default. If you want to
 alter it, you can select how to deal with each individual library.
 
+To debug the build itself, you can set the environment variable
+`CURL_TRACE_PKG_CONFIG` to a non-empty value to enable detailed trace
+information and verbose errors from `pkg-config` module detection invocations.
+
 ## Select TLS backend
 
 These options are provided to select the TLS backend to use.