Autonomic Orchestration of In-situ and In-transit Data Analytics for Simulation Studies
Xiaorui Du, Adriano Pimpini, Andrea Piccione, Zhuoxiao Meng, Anibal Siguenza-Torres, Stefano Bortoli, Alois Knoll, and Alessandro Pellegrini
Published in: Proceedings of the 2023 Winter Simulation Conference
Download PDF
Abstract:
Modern parallel/distributed simulations can produce large amounts of data. The historical approach of performing analyses at the end of the simulation is unlikely to cope with modern, extremely large-scale analytics jobs. Indeed, the I/O subsystem can quickly become the global bottleneck. Similarly, processing on-the-fly the data produced by simulations can significantly impair the performance in terms of computational capacity and network load.
We present a methodology and reference architecture for constructing an autonomic control system to determine at runtime the best placement for data processing (on simulation nodes or a set of external nodes). This allows for a good tradeoff between the load on the simulation’s critical path and the data communication system. Our preliminary experimentation shows that autonomic orchestration is crucial to improve the global performance of a data analysis system, especially when the simulation node’s rate of data production varies during simulation.
BibTeX Entry:
author = {Du, Xiaorui and Pimpini, Adriano and Piccione, Andrea and Meng, Zhuoxiao and Siguenza-Torres, Anibal and Bortoli, Stefano and Knoll, Alois and Pellegrini, Alessandro},
booktitle = {Proceedings of the 2023 Winter Simulation Conference},
title = {Autonomic Orchestration of In-situ and In-transit Data Analytics for Simulation Studies},
year = {2023},
month = dec,
publisher = {IEEE},
series = {WSC}
}