Dual backend design for sustainable and mockable data communication software

Abstract

Scientific software for data acquisition and processing is often developed over extended periods of time, during which hardware, communication protocols, and requirements evolve. Careful software design and abstractions that enable easy updates and component replacement are therefore essential, in particular for big science experiments whose software must be maintained for decades. This paper advocates for a single-interface, dual-backend design: the same interface is used to interact with a real communication backend used in production and a mock backend. The interface abstraction encapsulates the real backend, enabling easy maintenance and replacement, while the mock backend provides consistent, lightweight substitutes that are indispensable for fast unit and integration testing and simulating complex exchanges without any external infrastructure. Using a real and a mock backend simultaneously further allows real communication to be recorded and replayed later to reproduce complex scenarios for debugging or testing. This design has been implemented in a collection of libraries called phoenix [P. a], which abstracts and mocks a clock and a generic socket interface. A mock backend and a ZeroMQ backend are provided in C++, Rust, and Python, all relying on a compatible serialization library. This basis enables the implementation of data processing pipelines that are multi-language, consistent, mockable, and well encapsulated, therefore easily maintainable and extensible. The development of these tools is funded by the Horizon Europe project OSCARS [Osc], with the goal of supporting diverse scientific use cases.