Easy, Collaborative Benchmarking with the Extended Benchmarking Automation Tool xbat

Abstract

Research software development for compute- and data-intensive problems increasingly requires balancing scientific objectives with performance optimisation, performance reproducibility, and resource efficiency. However, many research software engineers (RSEs) face barriers in accessing the hardware expertise and tools necessary for performance analysis, especially in high-performance computing (HPC) environments where understanding low-level metrics is essential for sustainable software development. Moreover, collecting and interpreting performance metrics can be challenging and error-prone, particularly for RSEs without in-depth hardware expertise. To address these challenges, we present the open-source software extended benchmarking automation tool (xbat) as an easy-to-use, software-agnostic tool to automate (continuous) benchmarking and to simplify performance analysis. xbat unifies the benchmarking lifecycle from job configuration to analysis in a single, accessible platform and enables collaborative benchmarking by sharing results and configurations to facilitate replication within projects. We focus on the architecture, capabilities, and features of xbat, demonstrating its ability to simplify (continuous) benchmarking and assist RSEs in improving the performance of their research software. In addition, we present the xbat command line interface (CLI) which allows easy integration into non-interactive workflows. We showcase four use cases, namely the closed-source Gurobi Optimizer software, a co-scheduling example of the memory-bound application Stream with the compute-bound application Linpack, the open-source molecular dynamics research software ls1 mardyn using multiple nodes, and lbmpy for fluid simulation on a graphics processing unit (GPU).