KDET-HPFL: A Personalized Federated Learning Framework for Multimodal Pedestrian Detection With Adaptive Feature Selection

Abstract

Pedestrian detection plays a critical role in intelligent perception systems in autonomous vehicles, which directly influences the reliability and safety of the overall system. Advanced in-vehicle sensor technology has enabled the continuous evolution of pedestrian detection systems by leveraging heterogeneous multimodal inputs such as RGB, infrared, depth, Light Detection And Ranging, and event data. Nevertheless, establishing a robust pedestrian detection system that is capable of integrating and processing such heterogeneous multimodal data effectively remains a significant challenge. At the same time, growing concerns about data privacy among automobile manufacturers have hindered further advances in detection model performance by restricting the sharing of private data within the industry. In this paper, a novel personalised federated learning framework, Kolmogorov-Arnold network-based Dual Expert Transformer Heterogeneous Personalized Federated Learning (KDET-HPFL), is proposed for multimodal pedestrian detection. To be specific, the KDET pedestrian detector is developed based on an expert feature selection module (which is designed to adaptively choose essential features from multimodal data) and a Group-Rational Kolmogorov-Arnold Network module, which enhances the feature extraction capabilities and improves the detection performance effectively. The HPFL framework is proposed for data privacy protection on heterogeneous multimodal data, where a cross-client aggregation (CCA) method is put forward by integrating different aggregation methods for certain layers in the KDET detector. With CCA, the HPFL framework achieves personalised feature retention of multimodal data pairs on multiple clients and improved model aggregation effect for each client. Experimental findings reveal that the proposed KDET-HPFL framework outperforms some existing personalised federated learning frameworks for pedestrian detection on four public datasets (i.e., LLVIP, STCrowd, InOutDoor, and EventPed) with mAP scores of 73.74%, 75.39%, 66.14%, and 79.57%, respectively.