This document is targeted at cppTango developers only.

## General considerations

We have a large variety of distributions, and OS's, to check cppTango against.
In order to not create a mess in CI in the following we have listed a few
general ideas of how the tests are, and should be, organized:

- Each job MUST use one of the predefined templates:
  - `job-template-with-tests` if you want to run the tests or
  - `job-template-no-test-run` if you do not want to run the tests.

  If you have been longer with us, the variable `RUN_TESTS` has been replaced by this
  job template selection.
- Each job MUST inherit the global list of variables via `<<: *variables`
- Each job MUST use one of the predefined docker images, see [here]( https://gitlab.com/tango-controls/docker/ci/cpptango).
  If none of the available ones fit your purpose, check with the rest of the
  developers if one should either extend an existing one or create a new one.
  Best place to discuss this, is an issue.
- If you have one of the following commands in your `.gitlab-ci.yml`, then this
  indicates bad practice.  Try to avoid this under all circumstances!
  - `docker exec`: Should not be needed because you start already with the correct image
  - `cmake`: Configuration should always be done via `ci/config.sh`
    (`cmake-with-locations-and-min-versions-XXX` is an exception as it does test cmake configuration specialities.)
  - `make`: Building should always be done via `ci/build.sh`
  - `git clone`: The docker image should already include all required dependencies
  - `apt-get`/`dnf`/`apk`: The docker image should already have all required
    packages installed
- All docker images use a non-root user `tango` as default user, `sudo` is
  available in all images which also have docker installed and are able to run the tests.

### Useful scripts in CI and their supported environment variables

- `ci/build.sh`:

   - `TANGO_BUILD_TARGET`, target to build, defaults to `all`, a list of
     available targets can be outputted via `cmake --build . --target help`
     (only available when using the makefile generator)

- `ci/check-abi-api-compliance-check.sh`:

   - `CI_TARGET_BRANCH`, defaults to `main`

- `ci/config.sh`:

   This script accepts all cmake options listed in [Install.md](./Install.md)
   as environment variables and will pass them to cmake.

   Additionally supported:
     - `CMAKE_EXPORT_COMPILE_COMMANDS`, defaults to `off`
     - `CMAKE_UNITY_BUILD`, defaults to `off`
     - `CMAKE_UNITY_BUILD_BATCH_SIZE`, defaults to `0`
     - `OMNIORB_VERSION` for overriding the version check

- `ci/print_coredumps.sh`:

   - None

- `ci/test.sh`:

   - `GITLAB_CI`, set by gitlab CI only to enable specific tweaks, don't set yourself
   - `SKIP_TESTS`, regular expression passed to `ctest -E` for tests to skip

### Updating the CI docker images

- Check with the other developers if this is the right approach
- Change the docker images, see
  [here](https://gitlab.com/tango-controls/docker/ci/cpptango/super) for the
  super project with all of them
- Make a merge request (MR1) on the docker image project in
  https://gitlab.com/tango-controls/docker/ci/cpptango. The CI will build a
  docker image and push that to the registry. It is named like the branch.
- Make a merge request (MR2) against cppTango and use the branch of MR1
- Once review has passed on MR1 and MR2, get MR1 merged and tag the repo
- Update MR2 to point to the newly created tag
- Pass review again and get MR2 merged

### Running the tests

Running the tests requires docker. See
[here](https://docs.docker.com/engine/install) for install instructions.

Once you have installed docker, you can run the tests from `build/` via:

```bash
ctest
```

#### Excluding IDL5 tests

All of our test executables are run against two versions of the device servers.
The default "IDL6 servers", where device classes inherit from
`Tango::Device_6Impl` and the "IDL5 servers", where device classes inherit from
`Tango::Device_5Impl`.

There are two copies of each test registered with `ctest`:  One with the
`idl5::` prefix, which runs against the IDL5 servers; and, one without the
prefix, which runs against the IDL6 servers.

To avoid running the IDL5 versions of the tests, pass `-E "idl5::"` to ctest to
exclude these tests:

```bash
ctest -E "idl5::"
```

#### Run individual tests

For debugging a single test execute the following:

```
ctest -R event::per_event --output-on-failure -V
```

Test output and device server logs are collected in `build/tests/results`.

For more details on testing with CTest, see [here](https://cmake.org/Wiki/CMake/Testing_With_CTest).

#### Skipping catch2 tests on Windows

Catch2 Tests can be skipped on Windows using the "[skipwin32]" tag for the scenario.  For example,

```
SCEANRIO("My test that does not work on Windows", "[skipwin32]")
```

This should be used for tests that compile and should work on Windows, but for some reason the tests fails on this platform.  The commit message that introduces this tag should provide an explanation why the test needs to be skipped on Windows.

If the test does not compile on Windows, because it is testing scenarios not available on Windows, exclude the test from the build using a generator expression.  For example,

    $<$<BOOL:${HAS_FORK}>:catch2_fork.cpp>

Unfortunately, you should not use the "WIN32" flag in this generator expression as it is not defined on non-Windows platforms and this leads to CMake developer warnings.  To get around this, you need to introduce an additional variable like the above.

#### Setting environment up manually

The test runner automatically starts database and all required
device servers for each test. If you want to set up the environment
manually, from `build/` directory run:

```bash
source ./tests/environment/setup_database.sh  # source to get TANGO_HOST
./tests/environment/setup_devices.sh
# attach the debugger or perform some additional configuration
TANGO_TEST_CASE_SKIP_FIXTURE=1 ctest -V -R ds_cache -E "idl5::"
killall DevTest FwdTest
docker stop tango_cs mysql_db
```

The above starts the latest IDL version (i.e. IDL6) of the servers and, in
general, the "idl5::" versions of the tests will fail against these servers. So,
we pass `-E "idl5::"` to `ctest` to avoid running these tests.

To start IDL5 versions of the servers run `./tests/environment/setup_devices.sh`
with `IDL_SUFFIX=_IDL5` set:

```bash
source ./tests/environment/setup_database.sh  # source to get TANGO_HOST
IDL_SUFFIX=_IDL5 ./tests/environment/setup_devices.sh
# attach the debugger or perform some additional configuration
TANGO_TEST_CASE_SKIP_FIXTURE=1 ctest -V -R ds_cache -R "idl5::"
killall DevTest FwdTest
docker stop tango_cs mysql_db
```

Here the IDL6 versions of the tests will fail, so we must exclude them by
passing `-R "idl5::" to `ctest`.

#### Execucting gitlab CI jobs locally

For faster development it is possible to execute gitlab CI jobs locally on the development machine.

* Install gitlab runner, see https://docs.gitlab.com/runner/install
* `gitlab-runner exec docker --env "MAKEFLAGS=-j NUMBER_OF_CPUS" JOBNAME`
