VR-Deform: real-time visual and haptic interaction with deformable bodies using XPBD in virtual reality

Abstract

In this paper, we simulate a lifelike interaction with deformable bodies in virtual reality (VR) and provide force haptic feedback. Our method achieves physically compelling responses and stable feedback, which are essential for natural interaction with soft virtual components. Existing systems often suffer from instability or latency and rely on rigid body or primitive soft body models, which limit their application. We present a real-time VR framework that employs Extended Position Based Dynamics (XPBD) to enable deformable body simulation alongside stable haptic feedback. In our scenes, the proposed system achieves up to 96.3 FPS visual rendering on the VR system’s head-mounted display (HMD) while maintaining stability without frame loss or lag. Designed for lightweight deployment on consumer-grade hardware, our implementation integrates a haptic glove to capture finger motion and deliver multi-point tactile feedback. The simulation uses a unified solver that incorporates a set of position-based constraints and accounts for haptic force feedback. Our method supports scenes of up to 500k vertices and requires no external tracking, relying solely on the HMD and controllers for positional input. We conducted a user study in which participants interacted with the system and rated deformations and haptic feedback. Results showed the visuals and the haptic responses were convincing and plausible. We demonstrate that our framework enables immersive, stable, deformable body interaction with synchronised visual and haptic feedback, supporting applications in medical simulation, physical skill training, and entertainment.