Hole-filling Method Research Articles

Freehand 3D Ultrasound Imaging Based on Probe-mounted Vision and IMU System

ObjectivesFreehand three-dimensional (3D) ultrasound (US) is of great significance for clinical diagnosis and treatment, it is often achieved with the aid of external devices (optical and/or electromagnetic, etc.) that monitor the location and orientation of the US probe. However, this external monitoring is often impacted by imaging environment such as optical occlusions and/or electromagnetic (EM) interference. MethodsTo address the above issues, we integrated a binocular camera and an inertial measurement unit (IMU) on a US probe. Subsequently, we built a tight coupling model utilizing the unscented Kalman algorithm based on Lie groups (UKF-LG), combining vision and inertial information to infer the probe's movement, through which the position and orientation of the US image frame are calculated. Finally, the volume data was reconstructed with the voxel-based hole-filling method. ResultsThe experiments including calibration experiments, tracking performance evaluation, phantom scans, and real scenarios scans have been conducted. The results show that the proposed system achieved the accumulated frame position error of 3.78 mm and the orientation error of 0.36° and reconstructed 3D US images with high quality in both phantom and real scenarios. ConclusionsThe proposed method has been demonstrated to enhance the robustness and effectiveness of freehand 3D US. Follow-up research will focus on improving the accuracy and stability of multi-sensor fusion to make the system more practical in clinical environments.

Research on point cloud hole filling and 3D reconstruction in reflective area

3D reconstruction is the process of obtaining the three-dimensional shape or surface structure of an object, which is widely used in advanced manufacturing fields such as automotive, aerospace, industrial inspection, and reverse engineering. However, due to the structural characteristics of the component itself, the reflective properties of the coating material, and other factors, there may be specular reflection during image acquisition, making it difficult to achieve complete 3D reconstruction of the component. This paper proposes a method to address the problem of incomplete 3D reconstruction of strongly reflective objects by recognizing outlier points and filling point cloud holes. The proposed View-Transform-PointNet outlier point recognition network improves the alignment of the initial point cloud plane and implements secondary alignment of the point cloud based on the perpendicularity between the outlier plane in mixed reflection and the point cloud plane. The point cloud hole-filling method is based on the principle of outlier formation and approximates a local Gaussian distribution to linear variation. The distance between the end of each outlier plane and the real surface is calculated to repair the depth information of outlier points. The proposed method achieves a 39.4% increase in the number of point cloud filling, a 45.2% increase in the number of triangular mesh faces, a 46.9% increase in surface area, and a chamfer distance (CD) of 0.4471009, which is better than existing geometric repair methods in terms of standard deviation and smoothness. The method improves the alignment of initial point cloud planes and enhances the accuracy of outlier point recognition, which are the main innovative points of this study. The 3D reconstruction of the repaired point cloud model is achieved through Poisson equation and parameter adjustment. The proposed method reduces the error caused by large curvature in the boundary region and improves the smoothness and accuracy of the reconstructed model.

A hole-filling based approach to controlling structural complexity in topology optimization

Shape and topology optimization problems are usually associated with geometrical restrictions. Effective control of structural complexity during the optimization process is important for various considerations, e.g., functionality, manufacturability, and aesthetics. Most existing approaches characterize structural complexity as the number of cavities. However, for three-dimensional structures, the genus (i.e., the number of tunnels) should also be considered as an additional topological constraint. In this paper, a hole-filling method is integrated into the bi-directional evolutionary structural optimization (BESO) computational framework to control the number and size of existing cavities and tunnels during the form-finding process. In the hole-filling method, excess cavities are filled with solid material and excess tunnels are covered by building sheet-like patches. The minimum size of the truss-like components and the minimum thickness of the sheet-like components can be controlled separately. Several two- and three-dimensional compliance minimization problems are presented to demonstrate the effectiveness and potential applications of the proposed approach. The structural performances of the optimized structures with and without complexity control are compared and analyzed. The results show that the developed methodology can produce structurally efficient designs with controllable topologies. It is also demonstrated that the generation of sheet-like components can considerably increase the stiffness of the optimized structure. The proposed approach is capable of generating diverse and competitive designs for architects and engineers to achieve a fine balance between architectural novelty and structural efficiency.

Defect inspection for underwater structures based on line-structured light and binocular vision.

Surface defect inspection for underwater structures is important. However, the inspection technologies based on passive vision cannot meet accuracy requirements. In this paper, we propose a two-stage method based on structured light images for defect detection. In the first stage, light stripes are extracted based on the analysis of hue, saturation, value (HSV) space and gray space. Then a hole-filling method is applied to ensure stripe integrity. In the second stage, depth information for all light stripes is calculated to synthesize a depth map, which is segmented for defect localization and measurement. Experimental results have verified the feasibility and effectiveness of our method.

Virtual Viewpoint Film and Television Synthesis Based on the Intelligent Algorithm of Wireless Network Communication for Image Repair

With the development of the computer vision field, the acquisition of scene depth information is one of the important topics in the three‐dimensional reconstruction of the computer vision field, and its significance is particularly important. The purpose of this paper is to study the virtual viewpoint video synthesis for image restoration based on the intelligent algorithm of wireless network communication. Aiming at the hole problem caused by the change of occlusion relationship, this paper proposes a hole‐filling method based on background recognition. A threshold segmentation algorithm is used to reduce the filling priority of foreground pixels at the boundary of the hole and fully solve the hole problem. This paper also proposes a wireless sensor network node positioning model with swarm intelligence algorithm, which combines swarm intelligence algorithm with some key issues of wireless sensor network, speeds up the convergence, and improves the traditional intelligence algorithm. According to the experimental data in this paper, the algorithm in this paper is about 20% higher than the traditional algorithm in PSNR. On SSIM, the performance of the algorithm in this paper is 4.6% higher than the traditional algorithm at most, and the lowest is 2.2%.

Reliable fusion of ToF and stereo data based on joint depth filter

To obtain reliable depth images with high resolution, a novel method is proposed in this study that fuses data acquired from time-of-flight (ToF) and stereo cameras, through which the advantages of both active and passive sensing are utilised. Based on the classic error model of the ToF, gradient information is introduced to establish the likelihood distribution for all disparity candidates. The stereo likelihood is estimated in parallel based on a 3D adaptive support-weight approach. The two independent likelihoods are unified using a maximum likelihood estimation, a process also referred to as a joint depth filter herein. Conventional post-processing methods such as a mutual consistency check are also used after applying a joint depth filter. We also propose a novel hole-filling method based on the seed-growing algorithm to retrieve missing disparities. Experiment results show that the proposed fusion method can produce reliable high-resolution depth maps and outperforms other compared methods.

Topological constraints in 2D structural topology optimization

One of the straightforward definitions of structural topology optimization is to design the optimal distribution of the holes and the detailed shape of each hole implicitly in a fixed discretized design domain. However, typical numerical instability phenomena of topology optimization, such as the checkerboard pattern and mesh dependence, all take the form of an unexpected number of holes in the optimal result in standard density-type design methods, such as SIMP and ESO. Typically, the number of holes is indirectly controlled by tuning the value of the radius of the filter operator during the optimization procedure, in which the choice of the value of the filter radius is one of the most opaque and confusing issues for a beginner unfamiliar with the structural topology optimization algorithm. Based on the soft-kill bi-directional evolutionary structural optimization (BESO) method, an optimization model is proposed in this paper in which the allowed maximal number of holes in the designed structure is explicitly specified as an additional design constraint. The digital Gauss-Bonnet formula is used to count the number of holes in the whole structure in each optimization iteration. A hole-filling method (HFM) is also proposed in this paper to control the existence of holes in the optimal structure. Several 2D numerical examples illustrate that the proposed method cannot only limit the maximum number of holes in the optimal structure throughout the whole optimization procedure but also mitigate the phenomena of the checkerboard pattern and mesh dependence. The proposed method is expected to provide designers with a new way to tangibly manage the optimization procedure and achieve better control of the topological characteristics of the optimal results.

A Fast Hole-Filling Method for Triangular Mesh in Additive Repair

In the triangular meshes obtained in additive repair, it is a challenge to find one single hole-filling method to close all holes and make the filling patches assort with surrounding meshes well with low time complexity, which is mainly caused by the shape complexity and size difference of the various holes, especially in the fields of intelligent manufacturing, 3D measurement, and reverse engineering. Therefore, it is reasonable to adopt different algorithms to fill different types of holes. In this research, a fast hole-filling method for triangular mesh is proposed based on the hole size. First, a group of basic concepts is defined to make them uniform throughout the whole text, followed by the descriptions of hole detection and boundary cleaning. Second, three different algorithms are developed to fill the small-sized, middle-sized, and large-sized holes classified by hole size respectively, which can fill all the detected holes in a fast and proper manner. Finally, two experiments are carried out to verify the efficiency, robustness, and ability to recover the shape of our method. Compared to two state-of-the-art hole-filling methods in the first experiment, the quantitative evaluation results demonstrate that our proposed method is much faster than them with the ability to guarantee the regularity of most filling triangles. The second experiment proves that our method can produce satisfactory filling results by making the filling patches be compatible with surrounding meshes well.

Hole-Filling of RealSense Depth Images Using a Color Edge Map

The RealSense camera frequently includes hole regions—unfilled regions where depth information are missing in the depth image. The holes cause serious problems in applications that use depth sensors and often straddle the background and object simultaneously. Accordingly, a hole must be filled by considering the boundary of the object. Fortunately, the object boundary inside the hole can be estimated by the RealSense color image. This paper proposes a hole-filling method in which filling converges to the object boundary from both border pixels. However, the color edge elements of the object boundary are often missing. Because the missing edge highly influences the hole-filling procedure, the boundary of the object close to the camera should be filled as completely as possible. Consequently, we introduce a simple method of filling the gap between two edge endpoints. Furthermore, we can address the situation in which there are one or more edge pixels between two border pixels. We repeatedly change the filling direction by alternating between the horizontal and vertical directions until no hole remains. Our experiment reveals that the proposed method outperforms other filling methods when numerically comparing the hole-filled depth image with the ground truth image for four methods: root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity index map (SSIM), and correlation coefficient (CC). The proposed method also produces a superior visual when we observe the 3D stereo image pair constructed using an original color image and a left-view color image generated with depth-image-based rendering (DIBR).

View synthesis based on spatio-temporal continuity

Free viewpoint video is generated on the basis of a video plus depth (V+D) virtual point rendering framework. Because of the limited bandwidth of video transmission, depth-image-based rendering (DIBR) has become a common method. Most DIBR methods not only have holes and ghost artifacts but also have problems with time continuity, leading to frequent flickers. In this paper, we make full use of time domain information in video sequences and adjacent frames to extract the static background image of the whole scene. Furthermore, we propose a weighted-fusion hole-filling method based on static background to fill holes and maintain time continuity. Experimental results show that the proposed method can improve the quality of virtual view images and strengthen spatio-temporal continuity.

A Comparison of Hole-filling Methods for 3D Medical Data Reconstruction and Visualization

3D medical imaging can help the physicians to understand the patient anatomy, such as 3D ultrasound and 3D CT scan. In the case of 3D ultrasound reconstruction, the pixel nearest neighbour is one of the popular methods used. The hole-filling method in pixel nearest neighbour can fill in the empty voxel data which is not captured by the ultrasound. Thus, this paper studied the various hole-filling methods in the pixel nearest neighbour method to reconstruct the missing voxels. Besides that, an alternative method is also introduced based on the modified butterfly interpolation scheme. The experiment setup is designed to test the efficiency of the hole-filling method for 3D medical data visualisation by using mean absolute error as well as qualitatively compare the visualization of the reconstruction results. The proposed method can extract smooth skin from the reconstruction volume, although it has a high average MAE result.

Automatic lung nodule detection system using image processing techniques in computed tomography

Diagnosing and treating lung cancer at an early stage can improve the survival rate of patients. This study attempted to develop a computer-aided detection (CAD) system. In order to include all nodule types in the detection, this study proposes an image processing method for detecting ground glass opacity (GGO), part solid, and solid nodules in chest computed tomography. The process comprises image preprocessing, lung segmentation, nodule enhancement, candidate detection, and reduction of false positives. For lung segmentation, the edge searching method replaces the computing-intensive iterative hole-filling method. In order to extract nodules with extensively distributed gray levels, image accumulation is used in the nodule enhancement to rapidly enhance the gray level of individual nodules. In order to reduce false positives, the support vector machine (SVM) is applied twice. On the first run, the candidate nodules are obtained by using 4 two-dimensional features, and the classification result is obtained by using 11 three-dimensional features on the second run. This study used 667 lung nodules for experiment and evaluation. The proposed system can detect GGO, part solid, and solid nodules and takes only 0.1 s to process a single image. The total sensitivity of the system is more than 92.05%. The system excels at detecting small nodules in the range of 5 mm–9 mm with a sensitivity of 93.73% and GGO with a sensitivity of 93.02%. The results showed that the proposed rapid detection system has high sensitivity and low false positives, contributing to helping the clinicians’ diagnosis.

New Hole-Filling Method Using Extrapolated Spatio-Temporal Background Information for a Synthesized Free-View

This paper introduces a new hole-filling method using extrapolated spatio-temporal background information to obtain a synthesized free-view. New temporal background modeling is proposed, which incorporates stationary temporal information in the hole-filling process and preserves the temporal consistency of synthesized views. A background codebook is distinguished from a non-overlapped patch-based codebook, which contributes to extracting reliable temporal background information. Furthermore, a depth-map driven spatial local background estimation is also addressed to discriminate the background holes in each disocclusion and to define two spatial BG constraints that represent the lower and upper bounds of a background candidate. Holes are filled by comparing the similarities between the temporal background information and the spatial background constraints. In addition, a depth map-based ghost removal filter is described to solve the problem of the non-fit between a color image and the corresponding depth map of a virtual view. Finally, an exemplar-based inpainting is applied to fill in the remaining holes with a priority function that includes a new depth term. The experimental results demonstrated that the proposed method led to results that promised subjective and objective improvement over state-of-the-art methods.

Color Correction and Depth-Based Hierarchical Hole Filling in Free Viewpoint Generation

In this paper, a high-quality depth-image-based-rendering-based free viewpoint generation model is proposed, which consists of three parts. First, a boundary-aware 3-D warping is introduced for projecting the side views from original view plane to target virtual view plane. Specifically, a high-pass filter is used to remove the ghost contours along object boundaries. Second, a color correction-based image blending (CC-IB) procedure is put forward for synthesizing the two warped side views. During the proposed CC-IB, occluded regions are color-corrected and reconstructed in the first place, on the basis of an equivalent substitution thought and a polynomial fitting method, then color-continuous virtual view can be produced by blending the un-occluded regions and the reconstructed disocclusions. Finally, a depth guided hierarchical hole-filling method is presented for filling the remaining holes in the blended virtual view. With the assistance of depth, background pixels are preferentially selected for filling the holes. Experimental results show that the proposed model can produce high-quality virtual views with comfortable visual perceptions. Comparisons with other state-of-the-art models demonstrate that the proposed method can achieve good performance in both visual quality and quantitative evaluation.

시점의 공간 시간적 특징 기반 외삽 영상 홀 채움 기법의 개선

본 논문에서는 외삽으로 가상 시점 영상 생성 시에 발생하는 홀 영역을 효과적으로 채우는 VST 기반의 홀 채움 기법을 제안한다. 기존의 VST 기법은 스테레오 영상을 이용하여 내삽 영상을 생성한다. 가상 시점 생성 시 발생하는 홀 영역을 채우기 위해 시점, 공간적, 시간적 유사성을 가지는 8개의 주변 영상을 탐색한다. 하지만 탐색하는 영상의 개수가 많을 뿐만 아니라 각각의 영상을 풀 서치 방식으로 탐색하기 때문에 속도가 매우 느리다는 단점을 가지고 있다. 외삽 시점에 적용하는 경우에는 내삽 시점보다 홀 영역이 더 많이 생성되기 때문에 VST 기법을 적용하면 시간이 더 많이 소요된다. 따라서 본 논문에서는 외삽 영상 생성 시 검색 범위를 줄여 계산량을 줄이는 동시에 성능은 유지하는 VST 기반의 홀 채움 기법을 제안한다. 먼저 참조 영상의 사용 통계를 분석하여 참조 영상의 개수를 줄였다. 두 번째로 참조 영상에서 블록 탐색 시 n-step 블록 탐색 기법을 적용하였다. 실험을 통해 제안하는 기법이 VST 기법에 비해 속도는 약 8.5배, PSNR은 약 0.78dB 상승한 것을 확인하였다.

Multiview Stereoscopic Video Hole Filling Considering Spatiotemporal Consistency and Binocular Symmetry for Synthesized 3D Video

This paper proposes a new hole-filling method with spatiotemporal consistency and binocular symmetry for synthesized 3D videos in view extrapolation. Disocclusion regions in the synthesized views at virtual viewpoints result in regions with missing content. These regions will be referred to as hole regions. To provide the high-quality synthesized 3D videos via 3D display, the hole regions need to be filled considering the characteristics of human visual perception. From the perceptual point of view, binocular asymmetry between synthesized left- and right-eye videos (i.e., stereo pair video) is one of the most important factors that induce visual discomfort in stereoscopic viewing. In addition, binocular symmetry without temporal consistency between temporally neighboring frames could cause visual discomfort by annoying flickering artifacts. In this paper, to maintain the spatiotemporal consistency and binocular symmetry in synthesized 3D videos at multiple virtual viewpoints, we propose a global optimization-based hole-filling method using the information from the already filled adjacent view and previous frame. Furthermore, to reduce the computational cost of the global optimization, we propose a label propagation method, which propagates reliable labels used in the adjacent view and previous frame to the target image to be filled. The performance of the proposed method has been evaluated by objective assessments of 3D image quality, temporal consistency, and computational efficiency. In addition, subjective assessment is conducted for measuring the visual comfort and overall quality. The experimental results proved that the proposed method provides hole filling results with spatiotemporal consistency and binocular symmetry.

Real-time RGB-D image stitching using multiple Kinects for improved field of view

This article concerns the problems of a defective depth map and limited field of view of Kinect-style RGB-D sensors. An anisotropic diffusion based hole-filling method is proposed to recover invalid depth data in the depth map. The field of view of the Kinect-style RGB-D sensor is extended by stitching depth and color images from several RGB-D sensors. By aligning the depth map with the color image, the registration data calculated by registering color images can be used to stitch depth and color images into a depth and color panoramic image concurrently in real time. Experiments show that the proposed stitching method can generate a RGB-D panorama with no invalid depth data and little distortion in real time and can be extended to incorporate more RGB-D sensors to construct even a 360° field of view panoramic RGB-D image.

A fast and automatic hole-filling method based on feature line recovery

ABSTRACTIn this paper, an automatic method to fill holes in triangle mesh models is proposed. Different types of holes can be handled by recovering feature lines that pass through them, and then the recovered feature lines are used for splitting complex holes into small and simple ones, which simplifies the hole-filling problem. Two kinds of splits are considered, curve split and corner split. For curve split, Euler spirals are constructed as feature lines. The required end-points and their corresponding tangent vectors can be automatically identified to initialize the curve. For corner split, the corner point contained in the hole is recovered by optimizing a quadric equation, then multiple feature lines can be constructed automatically. An advantage of this method is that it works on the neighborhood of each hole, which makes it possible to handle huge models with high efficiency compared to other state-of-the-art methods.

개선된 패치 매칭을 이용한 깊이 영상 기반 렌더링의 홀 채움 방법

개선된 패치 매칭을 이용한 깊이 영상 기반 렌더링의 홀 채움 방법

The hole-filling method and multiscale algorithm for the heat transfer performance of periodic porous materials

In this paper, we consider the transient heat transfer problem with rapidly oscillating coefficients in periodic porous materials. The hole-filling method through filling all holes with a very compliant material is investigated, and as the thermal conductivity parameters of the weak phase go to zero, the proof of the limiting process is derived in detail. Then, the multiscale analysis and computations based on the hole-filling method for the associated multiphase problem in a domain without holes are proposed. Finally, some numerical results are given, which show a good agreement and verify the feasibility of the hole-filling method.