Language-guided zero-shot segmentation with multi-angle reprojection for point cloud analysis

Abstract

Virtual Reality applications increasingly demand accurate 3D representations of real-world environments. While LiDAR point clouds capture physical spaces with high fidelity, they typically lack semantic labels, limiting their direct use for tasks such as object recognition, interaction modeling, and automation in immersive environments or digital twin systems. We present a LAnguage-guided zero-shot 3D SEgmentation and Reprojection tool (LASER), an engineered zero-shot segmentation tool that extends the state of the art by introducing language-guided 3D object detection for enhanced usability and accuracy. Unlike its predecessors, LASER uses an ensemble of GroundingDINO and Segment Anything Model as its backbone to process natural language queries and user-specified object categories, automated multi-view orthophoto generation with dynamic angles for optimal view selection, a confidence-weighted fusion algorithm for efficient 2D-3D reprojection, and a semantically labelled mesh output. The LASER pipeline begins by collecting point cloud data using LiDAR sensors, filtering the point cloud into ground and non-ground components, improving segmentation efficiency. It then generates multi-angle 2D orthophotos and perspective views, incorporating a user-guided angle selection module to optimise scene coverage. Then GroundingDINO detects objects based on textual descriptions, and Segment Anything Model subsequently refines these into segmentation masks. The core innovation of LASER lies in its confidence-weighted reprojection algorithm, which fuses multiple 2D segmentation results back into 3D space, ensuring higher segmentation accuracy and spatial consistency. The resulting semantically labelled assets can be exported in standard formats or iteratively refined through viewpoint adjustments or text prompt modifications. Our application of LASER to real-world 3D scans of construction sites demonstrates its effectiveness in delivering high segmentation precision, enhanced user interactivity, and seamless integration into virtual reality workflows. To comprehensively evaluate the proposed tool on diverse point cloud scans, we also presented the performance on four different test cases using two different scans (3DSES and Toronto3D) with both indoor and outdoor scenes. The results show consistent performance across scans. Finally, feature-based comparison with state-of-the-art approaches shows that LASER is an optimised tool for enriching static, open-world 3D scans with semantic labels, offering an alternative to existing state-of-the-art methods for niche applications.