System Regression Tests for the preCICE Coupling Ecosystem

Abstract

Simulation codes are often implemented as one single application or library, which often clearly defines the system under testing and an oracle for regression tests. For some simulations, coupling multiple codes is necessary. In the case of the coupling library preCICE, a realistic system of at least two coupling participants (typically each a different simulation code) consists not only of the library, but of a complete software stack of components underneath each coupling participant. Each component is developed in a separate software repository, and the simulation and screen output often varies between executions or dependency versions. These make preCICE an interesting study case for system regression tests, requiring solutions to issues not typically faced by stand-alone research software projects. In this paper, we evaluate common testing approaches for solving the challenges faced when testing the preCICE ecosystem, and we present a new framework for system regression tests for preCICE, based on connected Docker containers with cached layers, version control of and numerical comparisons to reference results, and interaction of GitHub Actions workflows across repositories.