Digital human model reconstructions using advanced deep learning methods

Abstract

Over the past few decades, 3D digital human modeling has emerged as a vibrant field of research, playing a foundational role in various applications such as film production, sports, medical sciences, and human-computer interaction. Early research efforts predominantly focused on artist-driven modeling techniques or relied on expensive scanning equipment. Our objective, however, is to leverage recent advances in deep learning technology to automatically generate personalized virtual avatars using only low-cost monocular cameras. In this dissertation, we present significant advancements in 3D digital human reconstruction from monocular images. By developing methods that effectively integrate temporal information and realistically reconstruct from sparse data, we address this challenging task. Given images and videos captured from monocular cameras, we have, for the first time, successfully reconstructed not only the 3D pose but also the complete 3D geometry of a person, including facial features, hair, and clothing. In our initial work, we trained a neural network with a partial attention mechanism to estimate 3D human poses from a single image. The network outputs a 3D mesh model that encapsulates body shape and posture but lacks surface details. This approach yielded promising results. In subsequent work, we advanced the optimization of the nude model obtained from the first study by incorporating multi-view human images to reduce errors caused by occlusions. By predicting the pose for each frame, we re-aligned the standard model and projected it onto each image for further optimization. We then employed shape-from-shading techniques to enhance surface details. In this dissertation, we explore methods for digital human reconstruction from monocular images and videos, enhanced by deep learning techniques. We present reconstruction approaches that focus on accuracy, simplicity, usability, and visual fidelity, utilizing multi-view image optimization. Through extensive evaluations, we provide a thorough analysis of key parameters, reconstruction quality, and the robustness of our methods. For the first time, our approach enables camera-based, user-friendly digitization for personal users, opening up exciting new applications such as telepresence and virtual try-on in online fashion shopping.