Optimised lambda architecture for monitoring wlcg using spark and spark streaming

Abstract

Introduction: The Worldwide LHC Computing Grid (WLCG) pro-vides storage and computational power for four LHC experiments (ALICE, ATLAS, CMS and LHCb). The WLCG infrastructure is a highly distributed and heterogeneous plat-form with various middleware characteristics, job submission and execution tools, and diverse methods of transferring and accessing the dataset. The high computation activity and distributed nature of the WLCG make the system very complex. Hence, it increases the probability of failures or inefficiencies. Efficient monitoring is necessary in order to present a comprehensive strategy to recognise and resolve any issues within such a dispersed infrastructure. It is also important for the effective utilisation of computational and data resources.