Disparity estimation and enhancement from holoscopic elemental images

Abstract

3D technology has acquired an enormous interest in the last few decades compared to standard 2D imaging. It has the advantage of projecting depth and motion parallax, pro viding a more realistic and natural image. Holoscopic imaging is a promising technique that captures full-colour spatial images using a single-aperture system. A micro-lens array shows the scene fromslightly varying angles between neighbouring lenses, mimicking the f ly’s eye approach and capturing four-dimensional information on a two-dimensional sur face. This technique has the potential to overcome key limitations of traditional 2D imag ing issues like depth, scalability, and multi-perspective problems due to its simple data collection and visualisation method, which provides robust and scalable spatial informa tion, and the additional features of motion parallax, binocular disparity, and convergence. Unlike stereoscopic technology, a holoscopic system is underexplored. Researchers con front a lack of holoscopic cameras to choose from: Lytro (no longer available) and Raytrix (industrial application, more expensive). The need for a low-cost holoscopic image simu lator and dataset to experiment with configurations, build prototypes, and use ground-truth data for exploration, benchmarking, and evaluation is evident. This research introduces a 4D Plenoptic function-based synthetic holoscopic image simulator. The simulator was developed to produce raw holoscopic images faster and simpler than prior versions. A comprehensive dataset with several micro-lens array configurations was developed and provided via the simulator. This dataset can be used to assess the resolution trade-off be tween angular and spatial data to help researchers choose micro-lens array configurations. Aninnovative disparity map generation technique is proposed, which produces a disparity mapof ascene from a single holoscopic image based on angular information preserved in its micro-images, also known as elemental images (EIs) The use of EIs for disparity es timation instead of traditional viewpoint images (VPIs) has not been extensively studied. This research aims to explore the feasibility of utilising angular perspective information instead of spatial orthographic information. Computing the disparity starts with enhanc ing the quality of EIs, which often suffer from low resolution and lack of texture, a pre processing phase is carried out using noise reduction and contrast enhancement. The dis parity between EIs pixels is calculated using the Semi-Global Block Matching (SGBM) technique, which is improved by employing a multi-resolution approach to overcome the resolution constraints of EIs as well as performing a content-aware analysis to dynami cally modify the SGBMwindowsize settings generating disparities across different levels of texture and complexity within the EIs. Finally, a weighted least squares (WLS) filter is used to improve the optimisation process. In addition, EIs with inaccurate backgrounds are detected and corrected with the use of a background mask and neighbouring EIs that contain accurate background information. The evaluation has revealed that the suggested technique has successfully generated disparity maps that surpass the accuracy of VPIs in real images and outperform two state-of-the-art deep learning algorithms. The method was also evaluated across many EI resolutions to test out the ideal resolution. Current disparity estimation approaches based on EIs suffer from significant performance decreases, particularly in texture-less regions. A novel method is proposed for performing EIs labelling and grouping directly from a holoscopic image by selecting the appropriate EIs that include the same object to use this information to reduce the disparity error in the texture-less regions. To begin, a subset of VPIs are extracted from the holoscopic image and subjected to conventional image segmentation. Next, labels are applied to the EIs that correspond to each object segmented from the VPIs using VPIs/EIs pixel mapping. To further improve the segmentation, we employ content-based image retrieval (CBIR), in which the query images are automatically chosen from the previously established seg mentation. The texture-less elemental images of the disparity map computed from the holoscopic image are updated using the labelling data. Positive findings from the evalua tion indicate that the proposed technique has helped reduce the disparity map error. The evaluation result outperformed state-of-the-art depth generation techniques, and the proposed technique enlarges the industrial application of 3D imaging applications such as AR/VR, inspection, robotics, security and entertainment.