diff --git a/.github/workflows/system-tests-pr.yml b/.github/workflows/system-tests-pr.yml index a2ab0fc48..a2b28f077 100644 --- a/.github/workflows/system-tests-pr.yml +++ b/.github/workflows/system-tests-pr.yml @@ -12,5 +12,3 @@ jobs: with: suites: release build_args: "TUTORIALS_REF:${{ github.event.pull_request.head.sha }}" - system_tests_branch: develop - log_level: "INFO" \ No newline at end of file diff --git a/tools/tests/README.md b/tools/tests/README.md index 164ed6962..f60eb5031 100644 --- a/tools/tests/README.md +++ b/tools/tests/README.md @@ -6,142 +6,155 @@ keywords: pages, development, tests summary: "Test complete simulations combining preCICE components of specific versions." --- -The tutorials repository hosts cases that need multiple components from the preCICE ecosystem to run. This directory provides tools that can automatically run complete simulations, using different versions of each component, and compare the results to references. While the main purpose is to run complete tests in the continuous integration workflows of preCICE, you can also run these tests on your laptop. +The tutorials repository hosts cases that need multiple components from the preCICE ecosystem to run. Te system tests automatically run tutorials combining the required components and compare the numerical results to references. While the main purpose is to run complete tests in the continuous integration workflows of preCICE, you can also run these tests on your laptop. -Read more about the system tests in the publication [System Regression Tests for the preCICE Coupling Ecosystem](https://doi.org/10.14279/eceasst.v83.2614). +## Running -[![ECEASST](https://img.shields.io/badge/DOI-10.14279%2Feceasst.v83.2614-green)](https://doi.org/10.14279/eceasst.v83.2614) +The main workflow for the user is executing the `systemtests.py` script, which is the same that the [GitHub Actions workflow](https://github.com/precice/tutorials/actions/workflows/system-tests-latest-components.yml) executes. Depending on the options given to the script, it reads in the respective metadata files and generates `docker-compose.yaml` files that can start a fully-defined coupled simulation. For arguments that are not provided, default values from `components.yaml` are used. + +### Manual and nightly runs on GitHub + +The [System tests (manual/nightly)](https://github.com/precice/tutorials/actions/workflows/system-tests-latest-components.yml) workflow executes the `release` test suite nightly and can also be triggered manually. + +On the workflow page, click `Run workflow`. The default values will execute the `release` test suite using the latest `develop` branches of every component. If you want to override the version of some component, specify it in the respective field. Commit hashes, branches, and tags are all accepted. Branches and tags will get automatically resolved to their current commit on GitHub before starting any test, and all tests will use the same version of any common component. -## Running the system tests +The available test suites are found in [`tests.yaml`](https://github.com/precice/tutorials/blob/develop/tools/tests/tests.yaml) and common values are: -The main workflow for the user is executing the `systemtests.py` script. Depending on the options given to the script, it reads in the respective metadata files and generates `docker-compose.yaml` files that can start a fully-defined coupled simulation. +- `quickstart`, `elastic-tube-1d`, or any other tutorial (see [exceptions](https://github.com/precice/tutorials/issues/448)) +- `openfoam-adapter`, `micro-manager`, `fmi-runner`, or similar test cases involving the respective component +- `precice` is a subset of cases that cover a range of preCICE features +- `release` is for all available but some very long or known to fail tests +- `extra` is for some longer tests +- `expected-to-fail`, `selected`, and `system-tests-dev` for some special cases -### Running the tests for a preCICE release +The `Use workflow from` is a default option of GitHub Actions that concerns the GHA workflow file itself. -Workflow for the preCICE v3 release testing: +### Running from a pull request -1. Collect the Git commits/tags of all components you want to test. The caching mechanism cannot detect changes based on branch names. The same effect might be encountered when rebasing and force-pushing the release branch. -2. In your terminal, navigate to the tutorials repository -3. Trigger the GitHub Actions Workflow. Until we merge the workflow to develop, this can only happen via the [GitHub CLI](https://cli.github.com/): +Several repositories include a workflow that allows triggering the system tests by adding the `trigger-system-tests` label to the pull request. The event is triggered only at the moment of adding the label, so you need to remove the label and add it again if needed. - ```bash - gh workflow run system-tests-latest-components.yml -f suites=release - ``` +See the [system tests workflow in the OpenFOAM adapter](https://github.com/precice/openfoam-adapter/blob/develop/.github/workflows/system-tests.yaml) for an example. - More arguments are available, for example: +### Running from the GitHub CLI - ```bash - gh workflow run system-tests-latest-components.yml -f suites=release -f build_args="PLATFORM:ubuntu2404,PRECICE_REF:develop" -f log_level="DEBUG" --ref=develop - ``` +The [GitHub CLI](https://cli.github.com/) allows triggering workflows of a GitHub project. For example: - The `build_args` override the defaults set in `tools/tests/components.yaml`. +```bash +gh workflow run system-tests-latest-components.yml -f suites=release +``` + +More arguments are available, for example: + +```bash +gh workflow run system-tests-latest-components.yml -f suites=release -f build_args="PLATFORM:ubuntu2404,PRECICE_REF:develop" -f log_level="DEBUG" --ref=develop +``` -4. Go to the tutorials [Actions](https://github.com/precice/tutorials/actions) page and find the running workflow -5. Check the status and the runtimes of each tutorial: +The `build_args` override the defaults set in `tools/tests/components.yaml`. - - Very small build times mean that the test is using cached container layers - - Most commonly, you will see tests failing with `Fieldcompare returned non zero exit code`. You will need to check the logs, but if the fieldcompare time is significant, this typically means that the numerical results differ above the tolerance (the test works!). +### Running locally -6. Download the build artifacts from Summary > runs. +To run locally, you will need Docker, Docker Compose, and Python 3. - - In there, inspect the per-stage logs (`system-tests-build.log`, `system-tests-run.log`, `system-tests-compare.log`). - - The produced results are in `precice-exports/`, the reference results in `reference-results-unpacked`. - - Compare using, e.g., ParaView or [fieldcompare](https://gitlab.com/dglaeser/fieldcompare): `fieldcompare dir precice-exports/ reference/`. The `--diff` option will give you `precice-exports/diff_*.vtu` files, while you can also try different tolerances with `-rtol` and `-atol`. +Navigate into the directory `tools/tests/` of the tutorials, make a Python virtual environment, and install the dependencies: -### Running specific test suites +```bash +python -m venv .venv && source .venv/bin/activate +python -m pip install -r requirements.txt +``` To test a certain test-suite defined in `tests.yaml`, use: ```bash -python3 systemtests.py --suites=fenics-adapter, +python systemtests.py --suites=quickstart ``` -To discover all tests, use `python print_test_suites.py`. +This will build and connect Docker containers and run tutorials in the `runs/` directory in the root of the tutorials. -To be able to fill in the right case tuple into the `tests.yaml`, you can use the `python3 print_case_combinations.py` script. +To clean up at the end, you might want to run a `docker system prune -a` and remove the `runs/` directory to save space. -## Running the system tests on GitHub Actions +There are also some auxiliary scripts (run without arguments): -Go to Actions > [System tests (latest components)](https://github.com/precice/tutorials/actions/workflows/system-tests-latest-components.yml) to see this workflow. +- `print_test_suites.py`: Print all test suites defined in `tests.yaml` +- `print_case_combinations.py`: Print all possible combinations of participants in a tutorial, using its `metadata.yaml`. -After bringing these changes to `master`, the manual triggering option should be visible on the top right. Until that happens, we can only trigger this workflow manually from the [GitHub CLI](https://github.blog/changelog/2021-04-15-github-cli-1-9-enables-you-to-work-with-github-actions-from-your-terminal/): +## Understanding the logs -```shell -gh workflow run system-tests-latest-components.yml -f suites=fenics-adapter --ref=develop -``` +For a local run, look into the `tutorials/runs/` directory, where you will find a time-stamped directory for every test executed. -Another example, to use the latest releases and enable debug information of the tests: +Each of these directories includes the usual tutorial case files and logs, as well as: -```shell -gh workflow run system-tests-latest-components.yml -f suites=fenics-adapter -f build_args="PRECICE_REF:v3.1.1,PRECICE_PRESET:production-audit,OPENFOAM_ADAPTER_REF:v1.3.0,PYTHON_BINDINGS_REF:v3.1.0,FENICS_ADAPTER_REF:v2.1.0,SU2_VERSION:7.5.1,SU2_ADAPTER_REF:64d4aff,DEALII_ADAPTER_REF:02c5d18,TUTORIALS_REF:340b447" -f log_level=DEBUG --ref=develop -``` +1. `system-tests-build.log`: The logs of building the respective components. +2. `system-tests-run.log`: The logs of running the simulation (intermixed, from all participants). +3. `system-tests-compare.log`: The logs for the comparison to the reference results. + +In addition, in the directories of the cases executed, you can find `system-tests-.log` files. + +### Numerical regressions + +When the tests fail at the results comparison step, this typically means that there are numerical regressions (unless something wrong went undetected in a previous step). We use [fieldcompare](https://gitlab.com/dglaeser/fieldcompare) to compare all [preCICE exports](https://precice.org/configuration-export.html) to reference results generated from a previous run. Relevant files: + +- `precice-exports/`: The coupling meshes of the test run. +- `reference-results/`: The coupling meshes of the reference run, as stored on Git LFS, expanded into `reference-results-unpacked`. +- `diff-results/`: Numerical difference of the results in the two directories (computed with `fieldcompare dir --diff precice-exports/ reference/`). These are only present on failed comparisons. + +To reproduce the comparison locally, use the [same fieldcompare command](https://github.com/precice/tutorials/blob/develop/tools/tests/docker-compose.field_compare.template.yaml): -where the `*_REF` should be a specific [commit-ish](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddefcommit-ishacommit-ishalsocommittish). - -Example output: - -```text -Run cd tools/tests - cd tools/tests - python systemtests.py --build_args=PRECICE_REF:v3.1.1,PRECICE_PRESET:production-audit,OPENFOAM_ADAPTER_REF:v1.3.0,PYTHON_BINDINGS_REF:v3.1.0,FENICS_ADAPTER_REF:v2.1.0 --suites=fenics-adapter --log_level=DEBUG - cd ../../ - shell: /usr/bin/bash -e {0} -INFO: About to run the following systemtest in the directory /home/precice/runners_root/actions-runner-tutorial/_work/tutorials/tutorials/runs: - [Flow over heated plate (fluid-openfoam, solid-fenics)] -INFO: Started running Flow over heated plate (fluid-openfoam, solid-fenics), 0/1 -DEBUG: Checking out tutorials master before copying -From https://github.com/precice/tutorials - * [new branch] master -> master -DEBUG: Building docker image for Flow over heated plate (fluid-openfoam, solid-fenics) -DEBUG: Running tutorial Flow over heated plate (fluid-openfoam, solid-fenics) -DEBUG: Running fieldcompare for Flow over heated plate (fluid-openfoam, solid-fenics) -DEBUG: extracting /home/precice/runners_root/actions-runner-tutorial/_work/tutorials/tutorials/flow-over-heated-plate/reference-results/fluid-openfoam_solid-fenics.tar.gz into /home/precice/runners_root/actions-runner-tutorial/_work/tutorials/tutorials/runs/flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723/reference_results -Using log_level: DEBUG -+---------------------------------------------------------+---------+-------------------+-----------------+-----------------------+ -| systemtest | success | building time [s] | solver time [s] | fieldcompare time [s] | -CRITICAL: Fieldcompare returned non zero exit code, therefore Flow over heated plate (fluid-openfoam, solid-fenics) failed -INFO: Running Flow over heated plate (fluid-openfoam, solid-fenics) took 280.5861554039875 seconds -ERROR: Failed to run Flow over heated plate (fluid-openfoam, solid-fenics) -+---------------------------------------------------------+---------+-------------------+-----------------+-----------------------+ -| Flow over heated plate (fluid-openfoam, solid-fenics) | 0 | 271.80 | 5.60 | 2.42 | -+---------------------------------------------------------+---------+-------------------+-----------------+-----------------------+ +```bash +fieldcompare dir precice-exports/ reference-results-unpacked// \ + --ignore-missing-reference-files \ + --ignore-unsupported-file-formats \ + -rtol 3e-7 ``` -In this case, building and running seems to work out, but the tests fail because the results differ from the reference results. This may be incorrect, as the previous step may have silently failed. +The differences are only shown per file, and there is no global metric or other summary (see [related discussion in fieldcompare](https://gitlab.com/dglaeser/fieldcompare/-/work_items/69)). + +Alternatively, [visualize the `precice-exports/diff_*.vtu` in ParaView](https://precice.org/configuration-export.html#visualization-with-paraview). + +## Extending -## Understanding what went wrong +### Adding new tests -The easiest way to debug a systemtest run is first to have a look at the output written into the action on GitHub. -If this does not provide enough hints, the next step is to download the generated `system_tests_run__` artifact. Note that by default this will only be generated if the systemtests fail. -Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains per-stage logs (`system-tests-build.log`, `system-tests-run.log`, `system-tests-compare.log`). These are a good starting point for further investigation. When fieldcompare runs with `--diff`, it writes VTK (.vtu) diff files under `precice-exports/`; if the comparison fails, those files are copied into a `diff-results/` subfolder in the same run directory (mirroring any subpaths under `precice-exports/`) so you can open them (e.g. in ParaView) to see where results differ from the reference. On successful comparisons, `diff-results/` is therefore absent. +Tests and test suites are defined in [`tests.yaml`](https://github.com/precice/tutorials/blob/develop/tools/tests/tests.yaml). By convention, every tutorial defines a test suite with the same name as its directory, and several test cases using combinations of the available participants. These test cases are later referenced by other test suites: these are typically the `release` and the test suites of different tested components. -## Adding new tests +The available cases are listed in the `metadata.yaml` of each tutorial. To add a new tutorial case as a test, add it to `metadata.yaml` and then define a test using it. Include that test in the relevant test suites. -### Adding tutorials +Use the `max_time` or `max_time_windows` parameters to restrict the runtime of the test to the first few coupling time windows, to save time. Aim for a runtime of less than a minute (assuming cached components), if possible. -In order for the systemtests to pick up the tutorial we need to define a `metadata.yaml` in the folder of the tutorial. There are a few `metadata.yaml` already present to get inspiration from. You can also have a look at the implementation details but normally the currently available ones should be easy to adopt. You can check your metadata parsing by `python print_metadata.py` and `python print_case_combinations.py` +You will need to define a reference results file. The reference results can and should be generated on GitHub using the [Generate reference results (manual)](https://github.com/precice/tutorials/actions/workflows/generate-reference-results-manual.yml) workflow for the respective test suite. You might want to temporarily set the `selected` test suite for requesting results only for a subset of test cases. -### Adding testsuites +Note that you will need to define the `TUTORIALS_REF` in the file [`reference_versions.yaml`](https://github.com/precice/tutorials/actions/workflows/generate-reference-results-manual.yml) to match the respective branch. Restore that to `develop` after that. See a [related issue](https://github.com/precice/tutorials/issues/844). -To add a testsuite just open the `tests.yaml` file and use the output of `python print_case_combinations.py` to add the right case combinations you want to test. Note that you can specify a `reference_result` which is not yet present. The `generate_reference_results.py` will pick that up and create it for you. -Note that its important to carefully check the paths of the `reference_result` in order to not have typos in there. Also note that same cases in different testsuites should use the same `reference_result`. +The results will be added to a Git LFS, but you will need special push access: just use the aforementioned GitHub Actions workflow, instead. -To cap the preCICE simulation time for a specific test without editing `precice-config.xml`, add an optional `max_time` (positive float, overrides ``) or `max_time_windows` (positive integer, overrides ``) field to the tutorial entry. Applies to both test runs and reference result generation. +### Adding new components -### Generate reference results +To add a new component, a few changes are needed: -Since we need data to compare against, you need to run `python generate_reference_results.py`. This process might take a while. -Please include the generated reference results in the pull request as they are strongly connected to the new testsuites. +1. In the `components.yaml`, add a new component, defining all the parameters that it might need. + - Add these parameters into the `reference_versions.yaml`. + - Add fields for these parameters into the `system-tests-latest-components.yml` workflow. +2. In the `dockerfiles//Dockerfile`, define a new stage for building your new component. Use the parameters you defined above. + - Defining a `_PR` variable will let you integrate the system tests into the respective component repository. +3. In the `component_templates.yaml`, define a component template. These are Jinja templates that are used by Docker Compose, and the main detail is how to run a simulation using that component. +4. Refer to the new component in the `metadata.yaml` of some tutorial and define some tests. -## Adding repositories +### Adding new repositories that can trigger the tests -If you want to trigger a testsuite from a new repository, you need to add a workflow file to that repository (under `.github/workflows/`). You can, for example, copy and adjust [the one from the OpenFOAM adapter](https://github.com/precice/openfoam-adapter/blob/develop/.github/workflows/system-tests.yaml). Then, you need a new label to trigger the workflow (e.g. `Issues`->`Labels`->`New label`). Last, in the [preCICE organization settings](https://github.com/organizations/precice/settings), you need to add the new repository to the action secret `WORKFLOW_DISPATCH_TOKEN` and to the default actions runner group. Adding the new label directly to the pull request in which you add the workflow should already trigger the system tests, compare the [pull request in the `precice` repository](https://github.com/precice/precice/pull/2052). +If you want to trigger the system tests from a new repository: + +1. Add a workflow file to that repository (under `.github/workflows/`). Example: [OpenFOAM adapter](https://github.com/precice/openfoam-adapter/blob/develop/.github/workflows/system-tests.yaml). +2. Create label as a trigger for workflow (under `Issues`->`Labels`->`New label`). +3. Give permissions: In the [preCICE organization settings](https://github.com/organizations/precice/settings), you need to add the new repository to the action secret `WORKFLOW_DISPATCH_TOKEN` and to the default actions runner group. Adding the new label directly to the pull request in which you add the workflow should already trigger the system tests. ## Implementation details -Each tutorial contains automation scripts (mainly `run.sh` and `clean.sh`), as well as metadata (`metadata.yaml`). The metadata file describes the available cases, how to run them, as well as their dependencies. A central `tests.yaml` file in this directory defines test suites, which execute different combinations of cases. The Python script `systemtests.py` executes the tests, allowing to filter for specific components or test suites. +Each tutorial contains automation scripts (mainly `run.sh` and `clean.sh`), as well as metadata (`metadata.yaml`). The metadata file describes the available cases, how to run them, as well as their dependencies. A central `tests.yaml` file defines test suites, which execute different combinations of cases. The Python script `systemtests.py` executes the tests, filter for specific test suites. + +Read more about the system tests in the publication [System Regression Tests for the preCICE Coupling Ecosystem](https://doi.org/10.14279/eceasst.v83.2614). + +[![ECEASST](https://img.shields.io/badge/DOI-10.14279%2Feceasst.v83.2614-green)](https://doi.org/10.14279/eceasst.v83.2614) -Let's dive deeper into some of these aspects. +
Expand the implementation details... ### General architecture @@ -153,7 +166,7 @@ Test steps include modifying the tutorial configuration files for the test syste Tests are executed by the `systemtests.py` script, which starts the Docker Compose. This can be executed locally, and it is the same script that GitHub Actions also execute. -The multi-stage Docker build allows building each component separately from the same Dockerfile, while Docker reuses cached layers. The Docker Compose services consider GitHub Actions Cache when building the services, although the cache is currently only updated, but not hit (see https://github.com/precice/tutorials/pull/372#issuecomment-1748335750). +The multi-stage Docker build allows building each component separately from the same Dockerfile, while Docker reuses cached layers. The Docker Compose services consider GitHub Actions Cache when building the services, although the cache is currently only updated, but not hit (see https://github.com/precice/tutorials/pull/372#issuecomment-1748335750). For this reason, we are running on a dedicated self-hosted runner. ### File structure @@ -252,7 +265,7 @@ Description: ### Components -The components mentioned in the Metadata are defined in the central `components.yaml` file. This file also specifies some arguments and their default values. These arguments can be anything, but most often they are version-related information. For example, the version of the OpenFOAM library used by the openfoam-adapter component, or the openfoam-adapter component version itself. For example: +The components mentioned in the Metadata are defined in the central `components.yaml` file. This file also specifies some arguments and their default values. These arguments can be anything, but most often they are version-related information. For example, the version of OpenFOAM used by the openfoam-adapter component, or the component version itself. For example: ```yaml openfoam-adapter: @@ -282,10 +295,11 @@ This `openfoam-adapter` component has the following attributes: #### Naming schema for build_arguments -Since the docker containers are still a bit mixed in terms of capabilities and support for different build_argument combinations the following rules apply: +The following rules apply for the `build_arguments`: - A build argument ending in `_REF` refers to a git commit-ish (like a tag or commit) being used to build the image. -- Some workflows set variables ending in `_PR`. These specify the GitHub pull request which provides the above `_REF` and can be on a fork. +- The `repository` parameter of any `_REF` argument points to the repository where the respective Git reference should be resolved. This is assumed to be consistent across components and the Dockerfile (and could be simplified). +- Some workflows set variables ending in `_PR`. These specify the GitHub pull request, which provides the above `_REF` and can be on a fork. - A build argument ending in `_VERSION` refers to the version of a third-party dependency to use (e.g., DUNE). - All other `build_arguments` are free of rules and up to the container maintainer. @@ -308,54 +322,15 @@ command: > This template defines: -- `image`: The base Docker image for this component, including a Git reference (tag), provided to the template as argument (e.g., by the `systemtests.py` script). +- `image`: The base Docker image for this component, including a Git reference (tag), provided to the template as an argument (e.g., by the `systemtests.py` script). - `depends_on`: Other services this service depends upon, typically a preparation service that fetches all components and tutorials. - `volumes`: Directories mapped between the host and the container. Apart from directories relating to the users and groups, this also defines where to run the cases. - `command`: How to run a case depending on this component, including how and where to redirect any screen output. -### Tests - -Concrete tests are specified centrally in the file `tests.yaml`. For example: - -```yaml -test_suites: - openfoam-adapter: - tutorials: - - path: flow-over-heated-plate - case_combination: - - fluid-openfoam - - solid-openfoam - reference_result: ./flow-over-heated-plate/reference-results/fluid-openfoam_solid-openfoam.tar.gz - - path: flow-over-heated-plate - case_combination: - - fluid-openfoam - - solid-fenics - reference_result: ./flow-over-heated-plate/reference-results/fluid-openfoam_solid-fenics.tar.gz - timeout: 1200 -``` - -The optional `timeout` field (in seconds) sets the maximum time for the solver run and fieldcompare phases of that specific case. If omitted, it defaults to `GLOBAL_TIMEOUT` (currently 300s (5min), overridable via the `PRECICE_SYSTEMTESTS_TIMEOUT` environment variable). - -This defines the test suite `openfoam-adapter`, with a case combination to run. - -### Generate Reference Results - -#### via GitHub workflow (recommended) - -The preferred way of adding reference results is via the manual `Generate reference results (manual)` workflow. This takes two inputs: - -- `from_ref`: branch where the new test configuration (e.g added tests, new reference_versions.yaml) is -- `commit_msg`: commit message for adding the reference results into the branch - -The workflow will checkout the `from_ref`, take the status of the systemtests of that branch and execute `python generate_reference_results.py`, upload the LFS objects into the self-hosted LFS server and add a commit with `commit_msg` onto the `from_ref` branch. - -#### manually +### Timeouts -In order to generate the reference results edit the `reference_versions.yaml` to match the required `build_arguments` otherwise passed via the cli. -Executing `generate_reference_results.py` will then generate the following files: +A `GLOBAL_TIMEOUT` is used for all operations. Its default value is 600s (5min), it is set in the beginning of [`Systemtests.py`](https://github.com/precice/tutorials/blob/develop/tools/tests/systemtests/Systemtest.py), and it can be overridden via the `PRECICE_SYSTEMTESTS_TIMEOUT` environment variable. -- all distinct `.tar.gz` defined in the `tests.yaml` -- a `reference_results.md` in the tutorial folder describing the arguments used and a sha-1 hash of the `tar.gz` archive. +Tests can define a different `timeout` in their `tests.yaml` entry, which applies to the running and results comparison steps. -The reference result archive will later be unpacked again during the systemtest and compared using `fieldcompare` -Please note that these files should always be kept in the git lfs. +
diff --git a/tools/tests/build_docker_images.py b/tools/tests/build_docker_images.py index 6a0d1cd07..da1df5a5f 100644 --- a/tools/tests/build_docker_images.py +++ b/tools/tests/build_docker_images.py @@ -19,12 +19,12 @@ def main(): parser.add_argument( '--build_args', type=str, - help='Comma-separated list of arguments provided to the components like openfoam:2102,pythonbindings:latest') + help='Comma-separated list of component build arguments (e.g., "PRECICE_REF:develop,OPENFOAM_ADAPTER_REF:develop")') parser.add_argument('--rundir', type=str, help='Directory to run the systemstests in.', nargs='?', const=PRECICE_TESTS_RUN_DIR, default=PRECICE_TESTS_RUN_DIR) parser.add_argument('--log_level', choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'], - default='INFO', help='Set the logging level') + default='INFO', help='Set the logging level of the system tests scripts.') # Parse the command-line arguments args = parser.parse_args() diff --git a/tools/tests/reference_versions.yaml b/tools/tests/reference_versions.yaml index c7c3e4c04..6c7e69563 100644 --- a/tools/tests/reference_versions.yaml +++ b/tools/tests/reference_versions.yaml @@ -22,7 +22,7 @@ OPENFOAM_ADAPTER_REF: "2c3062c" # develop, May 27, 2026 PRECICE_REF: "v3.4.1" PYTHON_BINDINGS_REF: "v3.4.0" SU2_ADAPTER_REF: "5abe79b" # develop, May 27, 2026 -TUTORIALS_REF: "mercurydpm-tests" +TUTORIALS_REF: "develop" # Additional settings PRECICE_PRESET: "production-audit" \ No newline at end of file diff --git a/tools/tests/systemtests.py b/tools/tests/systemtests.py index bc56a6a84..3183b0ca6 100644 --- a/tools/tests/systemtests.py +++ b/tools/tests/systemtests.py @@ -30,12 +30,12 @@ def main(): parser.add_argument( '--build_args', type=str, - help='Comma-separated list of arguments provided to the components like openfoam:2102,pythonbindings:latest') + help='Comma-separated list of component build arguments (e.g., "PRECICE_REF:develop,OPENFOAM_ADAPTER_REF:develop")') parser.add_argument('--rundir', type=str, help='Directory to run the systemstests in.', nargs='?', const=PRECICE_TESTS_RUN_DIR, default=PRECICE_TESTS_RUN_DIR) parser.add_argument('--log_level', choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'], - default='INFO', help='Set the logging level') + default='INFO', help='Set the logging level of the system tests scripts.') # Parse the command-line arguments args = parser.parse_args()