curl-curl/docs/tests/HTTP.md

<!--
Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.

SPDX-License-Identifier: curl
-->

# The curl HTTP Test Suite

This is an additional test suite using a combination of Apache httpd and
nghttpx servers to perform various tests beyond the capabilities of the
standard curl test suite.

# Usage

The test cases and necessary files are in `tests/http`. You can invoke
`pytest` from there or from the top level curl checkout and it finds all
tests.

```sh
curl> pytest tests/http
platform darwin -- Python 3.9.15, pytest-6.2.0, py-1.10.0, pluggy-0.13.1
rootdir: /Users/sei/projects/curl
collected 5 items

tests/http/test_01_basic.py .....
```

Pytest takes arguments. `-v` increases its verbosity and can be used several
times. `-k <expr>` can be used to run only matching test cases. The `expr` can
be something resembling a python test or a string that needs to match test
cases in their names.

```sh
curl/tests/http> pytest -vv -k test_01_02
```

runs all test cases that have `test_01_02` in their name. This does not have
to be the start of the name.

Depending on your setup, some test cases may be skipped and appear as `s` in
the output. If you run pytest verbose, it also gives you the reason for
skipping.

# Prerequisites

You need:

1. a recent Python, `pytest` and the other modules listed in
   `tests/http/requirements.txt`
2. Apache httpd and its development files. On Debian/Ubuntu, the packages
   `apache2-bin` and `apache2-dev` have these.
3. the Apache `mod_ssl`, `mod_http2` and `mod_proxy` modules. On Debian/Ubuntu, these
   modules are part of the `apache2-bin` package, but other distributions may
   package them separately.
4. a local `curl` project build
5. optionally, `nghttpx` with HTTP/3 enabled or h3 test cases are skipped

### Configuration

Via curl's `configure` script you may specify:

  * `--with-test-nghttpx=<path-of-nghttpx>` if you have nghttpx to use
   somewhere outside your `$PATH`.

  * `--with-test-httpd=<httpd-install-path>` if you have an Apache httpd
   installed somewhere else. On Debian/Ubuntu it otherwise looks into
   `/usr/bin` and `/usr/sbin` to find those.

  * `--with-test-caddy=<caddy-install-path>` if you have a Caddy web server
   installed somewhere else.

  * `--with-test-vsftpd=<vsftpd-install-path>` if you have a vsftpd ftp
   server installed somewhere else.

  * `--with-test-danted=<danted-path>` if you have `dante-server` installed

## Usage Tips

Several test cases are parameterized, for example with the HTTP version to
use. If you want to run a test with a particular protocol only, use a command
line like:

```sh
curl/tests/http> pytest -k "test_02_06 and h2"
```

Test cases can be repeated, with the `pytest-repeat` module (`pip install
pytest-repeat`). Like in:

```sh
curl/tests/http> pytest -k "test_02_06 and h2" --count=100
```

which then runs this test case a hundred times. In case of flaky tests, you
can make pytest stop on the first one with:

```sh
curl/tests/http> pytest -k "test_02_06 and h2" --count=100 --maxfail=1
```

which allow you to inspect output and log files for the failed run. Speaking
of log files, the verbosity of pytest is also used to collect curl trace
output. If you specify `-v` three times, the `curl` command is started with
`--trace`:

```sh
curl/tests/http> pytest -vvv -k "test_02_06 and h2" --count=100 --maxfail=1
```

all of curl's output and trace file are found in `tests/http/gen/curl`.

## Writing Tests

There is a lot of [`pytest` documentation](https://docs.pytest.org/) with
examples. No use in repeating that here. Assuming you are somewhat familiar
with it, it is useful how *this* general test suite is setup. Especially if
you want to add test cases.

### Servers

In `conftest.py` 3 "fixtures" are defined that are used by all test cases:

1. `env`: the test environment. It is an instance of class
   `testenv/env.py:Env`. It holds all information about paths, availability of
   features (HTTP/3), port numbers to use, domains and SSL certificates for
   those.
2. `httpd`: the Apache httpd instance, configured and started, then stopped at
   the end of the test suite. It has sites configured for the domains from
   `env`. It also loads a local module `mod_curltest?` and makes it available
   in certain locations. (more on mod_curltest below).
3. `nghttpx`: an instance of nghttpx that provides HTTP/3 support. `nghttpx`
   proxies those requests to the `httpd` server. In a direct mapping, so you
   may access all the resources under the same path as with HTTP/2. Only the
   port number used for HTTP/3 requests are different.

`pytest` manages these fixture so that they are created once and terminated
before exit. This means you can `Ctrl-C` a running pytest and the server then
shutdowns. Only when you brutally chop its head off, might there be servers
left behind.

### Test Cases

Tests making use of these fixtures have them in their parameter list. This
tells pytest that a particular test needs them, so it has to create them.
Since one can invoke pytest for a single test, it is important that a test
references the ones it needs.

All test cases start with `test_` in their name. We use a double number scheme
to group them. This makes it ease to run only specific tests and also give a
short mnemonic to communicate trouble with others in the project. Otherwise
you are free to name test cases as you think fitting.

Tests are grouped thematically in a file with a single Python test class. This
is convenient if you need a special "fixture" for several tests. "fixtures"
can have "class" scope.

There is a curl helper class that knows how to invoke curl and interpret its
output. Among other things, it does add the local CA to the command line, so
that SSL connections to the test servers are verified. Nothing prevents anyone
from running curl directly, for specific uses not covered by the `CurlClient`
class.

### mod_curltest

The module source code is found in `testenv/mod_curltest`. It is compiled
using the `apxs` command, commonly provided via the `apache2-dev` package.
Compilation is quick and done once at the start of a test run.

The module adds 2 "handlers" to the Apache server (right now). Handler are
pieces of code that receive HTTP requests and generate the response. Those
handlers are:

* `curltest-echo`: hooked up on the path `/curltest/echo`. This one echoes
  a request and copies all data from the request body to the response body.
  Useful for simulating upload and checking that the data arrived as intended.

* `curltest-tweak`: hooked up on the path `/curltest/tweak`. This handler is
  more of a Swiss army knife. It interprets parameters from the URL query
  string to drive its behavior.

  * `status=nnn`: generate a response with HTTP status code `nnn`.
  * `chunks=n`: generate `n` chunks of data in the response body, defaults to 3.
  * `chunk_size=nnn`: each chunk should contain `nnn` bytes of data. Maximum is 16KB right now.
  * `chunkd_delay=duration`: wait `duration` time between writing chunks
  * `delay=duration`: wait `duration` time to send the response headers
  * `body_error=(timeout|reset)`: produce an error after the first chunk in the response body
  * `id=str`: add `str` in the response header `request-id`

`duration` values are integers, optionally followed by a unit. Units are:

  * `d`: days (probably not useful here)
  * `h`: hours
  * `mi`: minutes
  * `s`: seconds (the default)
  * `ms`: milliseconds

As you can see, `mod_curltest`'s tweak handler allows Apache to simulate many
kinds of responses. An example of its use is `test_03_01` where responses are
delayed using `chunk_delay`. This gives the response a defined duration and the
test uses that to reload `httpd` in the middle of the first request. A graceful
reload in httpd lets ongoing requests finish, but closes the connection
afterwards and tears down the serving process. The following request then needs
to open a new connection. This is verified by the test case.