Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
There remain some false positives, hits in test data, and `dir` use,
around 100 issues in total.
There is no plan to enforce badwords on tests.
Also:
- badwords.txt: let a few `manpage[s]` occurrences through
(in Perl code).
Closes#19541
By making them defaults, then fixing and/or reshuffling remaining
exceptions as necessary.
- checksrc: ban by default: `snprintf`, `vsnprintf`, `sscanf`, `strtol`.
- examples: replace `strtol` with `atoi` to avoid a checksrc exception.
- tests/libtest: replace `strtol` with `atol`.
- tests/server: replace most `strtol` with `atol`.
- tests/server: replace most `strtoul` with `atol`/`atoi`.
- tests/server: drop no longer used `util_ultous`.
- fix typo in checksrc rules: `vsnprint` -> `vsnprintf`.
- update local exceptions.
Also:
- examples: ban curl printf functions. They're discouraged in user code.
- examples: replace curl printf with system printf.
Add `snprintf` workaround for <VS2015.
- examples/synctime: fix `-Wfloat-equal`.
- examples/synctime: exclude for non-Windows and non-UWP Windows.
- examples/synctime: build by default.
Closes#18823
Setting the path component to contain the letters:
! $ & ' ( ) { } [ ] * + , ; = : @
now leaves them un-encoded when CURLU_URLENCODE is used.
Amended test 1560 to verify.
Reported-by: Jeroen Ooms
Fixes#17977Closes#18024
- bump format strings to show the full value, drop casts.
- drop redundant casts (enum -> `%d`).
- drop some single-use macros.
- replace `int` with `bool` in testtrace.
Closes#18106
The issue is missed in CI, because valgrind jobs all run with UTF-8
support.
Fixing:
```
test 1560...[URL API]
valgrind ERROR ==13362== 104 bytes in 1 blocks are definitely lost in loss record 1 of 1
==13362== at 0x484D953: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==13362== by 0x48E1302: curl_dbg_calloc (in /curl/bld/lib/libcurl.so.4.8.0)
==13362== by 0x4931D12: curl_url (in /curl/bld/lib/libcurl.so.4.8.0)
==13362== by 0x14F658: get_parts (in /curl/bld/tests/libtest/libtests)
==13362== by 0x150AC6: test_lib1560 (in /curl/bld/tests/libtest/libtests)
==13362== by 0x17F5D5: main (in /curl/bld/tests/libtest/libtests)
```
Ref: https://github.com/curl/curl/actions/runs/16446352985/job/46479298080?pr=17988#step:41:3007
Follow-up to 7d1ca2e7e1#17933Closes#17998
- codespell: break logic out into its own runnable script. Allowing
to run it on local machines.
- codespell: install via `pip`, bump to latest version.
- codespell: show version number in CI log.
- codespell: drop no longer needed word exception: `msdos`.
- codespell: include all curl source tree, except `packages` and
`winbuild`. Drop an obsolete file exclusion.
- add new spellchecker job using the `typos` tool. It includes
the codespell dictionary and a couple more. Use linuxbrew to install
it. This takes 10 seconds, while installing via `cargo` from source
would take over a minute.
- codespell: introduce an inline ignore filter compatible with `cspell`
Make `typos` recognize it, too. Move single exceptions inline.
Fix new typos found. Also rename variables and words to keep
spellchecking exceptions at minumum. This involves touching some tests.
Also switch base64 strings to `%b64[]` to avoid false positives.
Ref: https://github.com/crate-ci/typos/blob/master/docs/reference.md
Ref: https://github.com/codespell-project/codespell?tab=readme-ov-file#inline-ignore
Ref: https://github.com/codespell-project/codespell/issues/1212#issuecomment-1721152455
Ref: https://cspell.org/docs/Configuration/document-settingsCloses#17905
Debian CI found that `lib1560` implements tests that will fail when
UTF-8 isn't supported. We can detect that with `nl_langinfo` and skip
the specific URLs that fail (i.e., those whose `getflags` are either
`CURLU_PUNYCODE` or `CURLU_PUNY2IDN`).
Co-authored-by: Viktor Szakats
Closes#17933
For consistency. RFC 3986 section 2.1 says:
"URI producers and normalizers should use uppercase hexadecimal digits
for all percent-encodings."
Reported-by: Jeroen Ooms
Fixes#17685Closes#17739
Tidy up headers and includes to ensure all individual test source
compile cleanly (but not link). To allow running clang-tidy (and
possibly other static analyzers) on them. It also improves readability
and allows to verify them locally, without the bundle logic.
clang-tidy ignores #included C files, so it's blind to bundle C files
the include these tests. The current workaround of embedding has
a couple of downsides:. meaningless filenames and line numbers,
missing issues, messing up self header paths. Thus, running it on
individual sources would be beneficial.
Also:
- de-duplicate includes.
- untangle some includes.
- formatting/indentation fixes.
- merge `getpart.h` into `first.h`.
Ref: https://github.com/curl/curl/pull/17680#issuecomment-2991730158Closes#17703
The `#undef` hack is no longer necessary after changing the redifitions
to not map back to the original symbols.
This makes it unnecessary to repeat the redefinitions after compiling
`warnless.c` itself (in unity mode).
Which in turns makes it unnecessary to include `warnless.h` again, to
trigger such redefinition.
This also means that `read`/`write` are now redefined on Windows from
the first inclusion of `warnless.h`.
Also:
- tests/server: drop a repeat `warnless.h` include, that is unnecessary
after this patch.
- tests/unit: drop repeat `warnless.h` include.
- tests/libtest: drop repeat `warnless.h` includes.
- tests/libtest: formatting.
Follow-up to 2f312a14da#17619
Follow-up to 84338c4de2#12331
Follow-up to 6239146e93Closes#17673
Make test bundles the default. Drop non-bundle build mode.
Also do all the optimizations and tidy-ups this allows, simpler builds,
less bundle exceptions, streamlined build mechanics.
Also rework the init/deinit macro magic for unit tests. The new method
allows using unique init/deinit function names, and calling them with
arguments. This is in turn makes it possible to reduce the use of global
variables.
Note this drop existing build options `-DCURL_TEST_BUNDLES=` from cmake
and `--enable-test-bundles` / `--disable-test-bundles` from autotools.
Also:
- rename test entry functions to have unique names: `test_<testname>`
This removes the last exception that was handled in the generator.
- fix `make dist` to not miss test sources with test bundles enabled.
- sync and merge `tests/mk-bundle.pl` into `scripts/mk-unity.pl`.
- mk-unity.pl: add `--embed` option and use it when `CURL_CLANG_TIDY=ON`
to ensure that `clang-tidy` does not miss external test C sources.
(because `clang-tidy` ignores code that's #included.)
- tests/unit: drop no-op setup/stop functions.
- tests: reduce symbol scopes, global macros, other fixes and tidy-ups.
- tool1621: fix to run, also fix it to pass.
- sockfilt: fix Windows compiler warning in certain unity include order,
by explicitly including `warnless.h`.
Follow-up to 6897aeb105#17468Closes#17590
Using a mixture of techniques to avoid symbols collisions:
- reduce scope.
- add `t*_` / `T*_` prefix.
- move shared functions to `testutil.c`.
(`suburl()`, `rlim2str()`)
- clone re-used lib*.c sources.
(lib587, lib645)
- include shared symbols just once in re-used `lib*.c` sources.
(using `LIB*_C` guards.)
- drop re-used `lib*.c` sources where they were identical or
unused.
- make macros global.
- #undef macros before use.
What remain is the entry functions `test`, and `unit_setup`,
`unit_stop` in unit tests.
Also:
- fix formatting and other minor things along the way.
- add `const` where possible.
- sync some symbol names between tests.
- drop `mk-bundle-hints.sh` that's no longer necessary.
Closes#17468
- replace ß (scharfes S) with links.
- replace § (section sign) with links.
- replace 🙏 emoji with `🙏`.
Supported by GitHub, Forgejo/Gitea and most likely GitLab.
- docs/libcurl/curl_mprintf.md: replace Unicode ± with `{+|-}`.
- docs/CIPHERS.md: URL encode Unicode in URLs.
- lib1560: use hex encoding in `räksmörgås.se`.
- unit1307: use hex encoding in `Lindmätarv`.
- drop LATIN SMALL LETTER A WITH ACUTE exception.
No longer appears in tests.
This leaves the single character exception: `ö`
And file exceptions holding contributor names.
Follow-up to 9243ed59b3#17329
Follow-up to 838dc53bb7#17247Closes#17335
Move curlx_ functions into its own subdir.
The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.
The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.
The strcase defines are not curlx_ functions and should not be used by
tool or server code.
dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.
When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.
Assisted-by: Jay Satiro
Closes#17253
If the CURLU handle already holds a proper URL, otherwise it is an
error.
Verified by test 1560
Fixes#17188
Reported-by: zopsicle on github
Closes#17189
Treat %2e and %2E to be "dot equivalents" in the function and remove
such sequences as well, according to RFC 3986 section 5.2.4. That is
also what the browsers do.
This DOES NOT consider %2f sequences in the path to be actual slashes,
so there is no removal of dots for those.
This function does not decode nor encode any percent sequences.
Also switched the code to use dynbuf.
Extends test 1395 and 1560 to verify.
Assisted-by: Demi Marie Obenour
Fixes#16869Closes#16870
- cmake: disable test bundles for clang-tidy builds.
clang-tidy ignores #included .c sources, and incompatible with unity
and bundles. It caused clang-tidy ignoring all test sources. It also
means this is the first time tests sources are checked with
clang-tidy. (autotools doesn't run it on tests.)
- cmake: update description for `CURL_TEST_BUNDLES` option.
- fix tests using special `CURLE_*` enums that were missing from
`curl/curl.h`. Add them as reserved codes.
- fix about ~50 other issues detected by clang-tidy: unchecked results,
NULL derefs, memory leaks, casts to enums, unused assigments,
uninitialized `errno` uses, unchecked `open`, indent, and more.
- drop unnecessary casts (lib1533, lib3207).
- suppress a few impossible cases with detailed `NOLINT`s.
- lib/escape.c: drop `NOLINT` no longer necessary.
Follow-up to 72abf7c13a#13862 (possibly)
- extend two existing `NOLINT` comments with details.
Follow-up to fabfa8e402#15825Closes#16756
- fix redirect from file:// URL with query part
- find_host_sep() simplify
- urlencode_str() simplify
- redirect_url() simplify
- made more const char *
- add more redirect URL test cases to test 1560
Closes#16498
The redirect logic was broken when the redirect-to URL was a relative
URL only as a fragment or query (starting with '#' or '?').
Extended test 1560 to reproduce, then verify.
Reported-by: Jeroen Ooms
Fixes#15836Closes#15848
As the parsing and address "regeneration" are done anyway, we might as
well use the updated version in the result and thereby A) get a
normalized (and lower cased) version of the address and B) avoid a
strcpy().
Updated test 1560 to verify.
Closes#15143
It was previously wrongly verifying the input in its URL encoded format
when setting the hostname component with curl_url_set(), so it wrongly
rejected '%'.
Now it URL decodes the name appropriately before the check.
Added tests to lib1560 to verify that a fine %-code is okay and that a
bad %-code (that decodes to '%') is rejected.
Regression from 0a0c9b6dfa, shipped in 8.0.0
Fixes#14656
Reported-by: Venkat Krishna R
Closes#14657
Used for extracting:
- when used asking for a scheme, it will return CURLUE_NO_SCHEME if the
stored information was a guess
- when used asking for a URL, the URL is returned without a scheme, like
when previously given to the URL parser when it was asked to guess
- as soon as the scheme is set explicitly, it is no longer internally
marked as guessed
The idea being:
1. allow a user to figure out if a URL's scheme was set as a result of
guessing
2. extract the URL without a guessed scheme
3. this makes it work similar to how we already deal with port numbers
Extend test 1560 to verify.
Closes#13616
Before this patch, the result code was a mixture of `int` and
`CURLcode`.
Also adjust casts and fix a couple of minor issues found along the way.
Cherry-picked from #13489Closes#13600
By default the API inhibits empty queries and fragments extracted.
Unless this new flag is set.
This also makes the behavior more consistent: without it set, zero
length queries and fragments are considered not present in the URL. With
the flag set, they are returned as a zero length strings if they were in
fact present in the URL.
This applies when extracting the individual query and fragment
components and for the full URL.
Closes#13396
Using the URL API for a redirect URL when the redirected-to string
starts with a hash, ie is only a fragment, the API would produce the
wrong final URL.
Adjusted test 1560 to test for several new redirect cases.
Closes#13394
I implemented the IDN functions for macOS and iOS using Unicode
libraries coming with macOS and iOS.
Builds and runs here on macOS 14.2.1. Also verified to load and
run on older macOS version 10.13.
Build requires macOS SDK 13 or equivalent.
Set `-DUSE_APPLE_IDN=ON` CMake option to enable it.
With autotools and other build tools, set these manual options:
```
CPPFLAGS=-DUSE_APPLE_IDN
LIBS=-licucore
```
Completes TODO 1.6.
TODO: add autotools option and feature-detection.
Refs: #5330#5371
Co-authored-by: Viktor Szakats
Closes#13246
- delete redundant warning suppressions for `-Wformat-nonliteral`.
This now relies on `CURL_PRINTF()` and it's theoratically possible
that this macro isn't active but the warning is. We're ignoring this
as a corner-case here.
- replace two pragmas with code changes to avoid the warnings.
Follow-up to aee4ebe591#12803
Follow-up to 0923012758#12540
Follow-up to 3829759bd0#12489
Reviewed-by: Daniel Stenberg
Closes#12812
- enable `-Wsign-conversion` warnings, but also setting them to not
raise errors.
- fix `-Warith-conversion` warnings seen in CI.
These are triggered by `-Wsign-converion` and causing errors unless
explicitly silenced. It makes more sense to fix them, there just a few
of them.
- fix some `-Wsign-conversion` warnings.
- hide `-Wsign-conversion` warnings with a `#pragma`.
- add macro `CURL_WARN_SIGN_CONVERSION` to unhide them on a per-build
basis.
- update a CI job to unhide them with the above macro:
https://github.com/curl/curl/actions/workflows/linux.yml -> OpenSSL -O3
Closes#12492
