Three-dimensional Shape Of Object Research Articles

Three-Dimensional Dense Reconstruction: A Review of Algorithms and Datasets

Three-dimensional dense reconstruction involves extracting the full shape and texture details of three-dimensional objects from two-dimensional images. Although 3D reconstruction is a crucial and well-researched area, it remains an unsolved challenge in dynamic or complex environments. This work provides a comprehensive overview of classical 3D dense reconstruction techniques, including those based on geometric and optical models, as well as approaches leveraging deep learning. It also discusses the datasets used for deep learning and evaluates the performance and the strengths and limitations of deep learning methods on these datasets.

Deep Learning-Based Ensemble Approach for Autonomous Object Manipulation with an Anthropomorphic Soft Robot Hand

Autonomous object manipulation is a challenging task in robotics because it requires an essential understanding of the object’s parameters such as position, 3D shape, grasping (i.e., touching) areas, and orientation. This work presents an autonomous object manipulation system using an anthropomorphic soft robot hand with deep learning (DL) vision intelligence for object detection, 3D shape reconstruction, and object grasping area generation. Object detection is performed using Faster-RCNN and an RGB-D sensor to produce a partial depth view of the objects randomly located in the working space. Three-dimensional object shape reconstruction is performed using U-Net based on 3D convolutions with bottle-neck layers and skip connections generating a complete 3D shape of the object from the sensed single-depth view. Then, the grasping position and orientation are computed based on the reconstructed 3D object information (e.g., object shape and size) using U-Net based on 3D convolutions and Principal Component Analysis (PCA), respectively. The proposed autonomous object manipulation system is evaluated by grasping and relocating twelve objects not included in the training database, achieving an average of 95% successful object grasping and 93% object relocations.

Visual sensitivity at the service of action control in posterior parietal cortex.

The posterior parietal cortex (PPC) serves as a crucial hub for the integration of sensory with motor cues related to voluntary actions. Visual input is used in different ways along the dorsomedial and the dorsolateral visual pathways. Here we focus on the dorsomedial pathway and recognize a visual representation at the service of action control. Employing different experimental paradigms applied to behaving monkeys while single neural activity is recorded from the medial PPC (area V6A), we show how plastic visual representation can be, matching the different contexts in which the same object is proposed. We also present data on the exchange between vision and arm actions and highlight how this rich interplay can be used to weight different sensory inputs in order to monitor and correct arm actions online. Indeed, neural activity during reaching or reach-to-grasp actions can be excited or inhibited by visual information, suggesting that the visual perception of action, rather than object recognition, is the most effective factor for area V6A. Also, three-dimensional object shape is encoded dynamically by the neural population, according to the behavioral context of the monkey. Along this line, mirror neuron discharges in V6A indicate the plasticity of visual representation of the graspable objects, that changes according to the context and peaks when the object is the target of one's own action. In other words, object encoding in V6A is a visual encoding for action.

A simple and highly accurate method for fuzzy edge-based 3D morphometry

As the level of engineering technology increases, the mechanical problems posed in the area of mechanics become complex and diverse. The traditional measurement methods and content of past measurements no longer meet the needs, so more advanced experimental methods are needed to accurately measure the three-dimensional shape of objects and their deformations. Nitrate ester plasticized polyether (NEPE) propellants, are the highest energy solid propellant that have been applied in public reports in the world. Due to the material properties of the material itself, its mechanical properties cannot be accurately measured by conventional methods. This paper proposes a simple and highly accurate optical measurement method to study the volume change rate, Poisson’s ratio and true stress-strain curves of NEPE at different displacement loading rates of NEPE. In this paper, the strain of solid propellant NEPE in three directions during unidirectional stretching was measured by defocusing method and digital image correlation(DIC) method and the volume change rate, Poisson’s ratio and true stress-strain curves of NEPE under different stretching rates were obtained. The measured damage initiation engineering strain of NEPE at different tensile strain rates is 0.4687.

Implementation of dual-energy material decomposition technique in stationary CT baggage scanner withπ-angle sparsity for enhancing threat detection

Two-dimensional X-ray inspection systems are widely used in aviation security applications; however, they have inherent limitations in recognizing the three-dimensional (3D) shapes of hidden objects. Therefore, there is a growing demand for the implementation of advanced 3D X-ray inspection systems at airports for more accurate detection of threats in luggage and personal belongings. In this study, we designed a new stationary computed tomography (CT) baggage scanner with π-angle sparsity (i.e., 20 pairs of X-ray sources and line detectors were placed within a scan angle of 180°) and compressed sensing (CS)-based reconstruction, and implemented a dual-energy material decomposition (DEMD) technique in the proposed system to separate soft and dense materials of an examined object to enhance threat detection. To validate the efficacy of the proposed approach (CS/180°/P20), we conducted a feasibility study using numerical simulation before its practical implementation. Polychromatic projections were emulated at X-ray tube voltages of 60 and 140 kVp, and DEMD was applied to the projections prior to CT reconstruction. Conventional and dual-energy CT images were reconstructed using both standard filtered-backprojection (FBP) and state-of-the-art CS-based algorithms to compare the image quality. According to our simulation results, the CS-reconstructed images were almost unaffected by the clearly evident streak artifacts on the FBP-reconstructed images because of the use of 20 extreme sparse-view projections, and the image quality of the dual-energy CT images obtained using the proposed CT configuration was similar to that obtained using the conventional CT configuration with 720 dense projections, indicating the efficacy of the proposed approach. Consequently, high-quality dual-energy CT images of soft and dense materials were successfully obtained using the proposed stationary CT configuration.

Dynamic 3D phase-shifting profilometry based ona corner optical flow algorithm.

Real-time 3D reconstruction has been applied in many fields, calling for many ongoing efforts to improve the speed and accuracy of the used algorithms. Phase shifting profilometry based on the Lucas-Kanade optical flow method is a fast and highly precise method to construct and display the three-dimensional shape of objects. However, in this method, a dense optical flow calculation is required for the modulation image corresponding to the acquired deformed fringe pattern, which consumes a lot of time and affects the real-time performance of 3D reconstruction and display. Therefore, this paper proposes a dynamic 3D phase shifting profilometry based on a corner optical flow algorithm to mitigate this issue. Therein, the Harris corner algorithm is utilized to locate the feature points of the measured object, so that the optical flow needs to calculate for only the feature points which, greatly reduces the amount of calculation time. Both our experiments and simulations show that our method improves the efficiency of pixel matching by four times and 3D reconstruction by two times.

Shape-from-focus reconstruction using block processing followed by local heat-diffusion-based refinement

Shape from focus (SFF) is a method of recovering the three-dimensional shape of objects from the image sequence through various focusing measurement operators, and is widely used in the fields of medical diagnosis, PCB inspection, industrial manufacturing, and robotic operations. In SFF, the wrong focus measurement causes an inaccurate reconstruction of the object. Generally, to improve the accuracy of focusing measurements, many focus measure (FM) operators are proposed, which is a method that improves the focus volume (FV) performance. However, due to the influence of blur diffusion, which refers to the overlapping of blur circles between different regions, the calculation results of the FM operator in the local window often do not meet the stability requirements of complex scenarios. It is difficult to provide high-precision reconstruction results in practical applications. To overcome this limitation, we proposed an SFF framework based on block processing and heat-diffusion refinement to obtain a refined depth map (RDM). The acquisition of an RDM is mainly divided into two steps. First, the image is divided into blocks to reduce the effect of blur diffusion, and the FM operator is used to calculate the focus measurement value of each block to form the FV. The initial depth map (IDM) is obtained by searching for the best focus position in the FV. The second step is to refine the IDM based on the heat-diffusion principle to obtain the best focus position for all pixels in each block and derive the RDM. The results of experiments conducted on synthetic and real objects demonstrate the superiority of our method in terms of accuracy and its high stability in different application scenarios.

Visual perception: Contours that crack the ambiguity conundrum.

A new study shows how the brain exploits the parts of images where surfaces curve out of view to recover both the three-dimensional shape and material properties of objects. This sheds light on a long-standing 'chicken-and-egg' problem in perception research.

Scan-free time-of-flight-based three-dimensional imaging through a scattering layer.

Reconstructing an object's three-dimensional shape behind a scattering layer with a single exposure is of great significance in real-life applications. However, due to the little information captured by a single exposure while strongly perturbed by the scattering layer and encoded by free-space propagation, existing methods cannot achieve scan-free three-dimensional reconstruction through the scattering layer in macroscopic scenarios using a short acquisition time of seconds. In this paper, we proposed a scan-free time-of-flight-based three-dimensional reconstruction method based on explicitly modeling and inverting the time-of-flight-based scattering light propagation in a non-confocal imaging system. The non-confocal time-of-flight-based scattering imaging model is developed to map the three-dimensional object shape information to the time-resolved measurements, by encoding the three-dimensional object shape into the free-space propagation result and then convolving with the scattering blur kernel derived from the diffusion equation. To solve the inverse problem, a three-dimensional shape reconstruction algorithm consisting of the deconvolution and diffractive wave propagation is developed to invert the effects caused by the scattering diffusion and the free-space propagation, which reshapes the temporal and spatial distribution of scattered signal photons and recovers the object shape information. Experiments on a real scattering imaging system are conducted to demonstrate the effectiveness of the proposed method. The single exposure used in the experiment only takes 3.5s, which is more than 200 times faster than confocal scanning methods. Experimental results show that the proposed method outperforms existing methods in terms of three-dimensional reconstruction accuracy and imaging limit subjectively and objectively. Even though the signal photons captured by a single exposure are too highly scattered and attenuated to present any valid information in time gating, the proposed method can reconstruct three-dimensional objects located behind the scattering layer of 9.6 transport mean free paths (TMFPs), corresponding to the round-trip scattering length of 19.2 TMFPs.

Understanding microtopography changes in agricultural landscapes through precision assessments of digital surface models by the UAV-RTK-PPK method without ground control points

Soil erosion on agricultural land causes severe environmental problems and damages crop plants. Structure-from-motion with multiview stereo (SfM-MVS) together with data obtained via unmanned aerial vehicles (UAVs) helps understand spatiotemporal changes in the ground surface if there is enough precise data. However, installing ground control points in the field is labor intensive and disturbs field conditions. Here, images georeferenced using the Real-Time Kinematic (RTK) method were further improved using Post-Processing Kinematic (PPK) analysis. This study aims to verify whether the UAV-RTK-PPK method can create precise digital surface models to evaluate topographic changes owing to erosion without ground control points. The effects of the camera's interior orientation parameters settings and the addition of oblique images to nadiral images on the precision were also examined to reduce the positional error by the doming and bowling phenomena. Field observations were conducted in a 150 m × 25 m potato field with 156 poles to consider the three-dimensional object shapes in the verification. The doming or bowling phenomena occurred using only the nadiral images. The precision of the digital surface model was approximately 0.40 and 0.20 m (in the Z-direction) using the estimated and fixed camera's interior orientation parameters, respectively. In contrast, the digital surface models produced by adding oblique images to the nadiral images had a precision of approximately 0.04 m, regardless of the camera's interior orientation parameters. Thus, a high-precision digital surface model was obtained from images acquired using the UAV-RTK-PPK and SfM-MVS methods without ground control points. The digital surface model positional precision was improved by adding oblique images to the nadiral images without having to consider the camera's interior orientation parameters. Topographic changes >0.05 m after erosion by heavy rainfall were evaluated without ground control points using the UAV-RTK-PPK method with oblique and nadiral images. Therefore, our method can monitor the erosion of agricultural fields that may cause crop damage, such as the greening of potatoes.

Method of unwrapped phase error correction in phase-measuring profilometry based on adaptive local window

Phase measurement profilometry (PMP) is widely used to reconstruct three-dimensional (3D) shape of objects. However, errors are introduced during standard phase unwrapping process, which considerably brings down measurement accuracy and lowers method reliability. This paper proposes a novel method to identify the errors, and subsequently correct these errors through erecting adaptive local windows. Under proposal, two robust algorithms are developed to correct these errors in initial rudimentary correction as well as in subsequent refined error removal. The adaptive local window is designed as such that the fringe order corresponding to each period of the wrapping phase is independent of each other. Therefore, it avoids the error transmission in the correction process. Experiments are carried out on absolute phase error correction and 3D point cloud reconstruction. Experimental results verified the developed method in terms of reducing the stratification of 3D point cloud and improving the quality of 3D measurement.

Calibration method for fringe projection profilometry based on rational function lens distortion model

The calibration of the fringe projection profilometry (FPP) system is susceptible to the camera lens distortion. This paper presents a calibration method based on the rational function lens distortion model. In this method, we integrate the rational function into the mathematical expression of the out-of-plane height to mitigate the impact of lens distortion. With arbitrarily arranged measurement components, we can flexibly measure the three-dimensional (3D) shape of objects with no expensive auxiliaries, high-precision setups, or complicated projector calibration. First, we project modulated sinusoidal fringe patterns with high frequency to the concentric circle grids calibration board to accurately estimate the unwrapped phase value, the pixel coordinate, and the out-of-plane height at each feature point. Then, the 3D coordinates can be recovered by determining the coefficients of the high order phase, coordinate, and height (PCH) mapping. Experiments on computer simulation and real data demonstrate that the modified calibration contributes to the flexibility and accuracy of the 3D shape measurement.

Vibration Distribution Measurement of Car Door and Engine Head Using OPPA Vibration Distribution Analyzer

In order to address the issue of vibration, it is crucial to accurately measure the vibration distribution. The authors previously developed the one-pitch phase analysis (OPPA) method, which allows for rapidly capturing the three-dimensional shape of a flat object. By integrating this method into a system, an OPPA vibration distribution measurement system was created, utilizing a line light source consisting of LEDs or optical fibers and also a high-speed camera to measure the vibrations of three-dimensional objects without physical contact. To further extend the application of the OPPA method to larger objects, such as cars, in this paper, a new system is introduced using a commercially available projector using a liquid crystal display (LCD) instead of a liner light source and a glass grating. This new system, which employs an ultra-short throw projector, is highly sensitive in displacement measurements and provides a wide-area analysis. These kinds of projectors produce noises at the frequency of the cooling fan and the refresh rate of the LCD. However, in this study, these noise sources were also examined. The capabilities of the new system are demonstrated through its application to the measurement of vibrations in a car door and an engine head. The measurement system and examples of its application are presented.

Line-encoded structured light measurement method in measuring shiny and transparent objects

Structured light measurement (SLM) has been widely used in acquiring the three-dimensional (3D) shape of objects because of its high precision and high speed. However, the intensity based structured light coding strategies like sinusoidal patterns is difficult to measure transparent and shiny objects with high dynamic range surface reflectance due to inaccurate light intensity information. Divergent multi-line structured light projection and traditional gray-code method avoids this problem by extracting lines. However, traditional gray-code method requires additional many patterns to distinguish the order of lines, which reduces the measurement efficiency. To overcome defects, this paper proposes a line-encoded few-pattern SLM method in measuring shiny and transparent objects. In this method, we simultaneously extract the edge line, the center line and obtain their orders by projecting only five patterns. Many experiments were implemented to prove its accuracy and robustness. Because only five patterns are required, the proposed method greatly accelerate the process of reconstruction, which is indispensable for 3D real-time reconstruction. We conducted a dental model dynamic scanning experiment to prove it.

Computing Magnifier for Refining the Position and Shape of Three-Dimensional Objects in Acoustic Sensing

Computing Magnifier for Refining the Position and Shape of Three-Dimensional Objects in Acoustic Sensing

Does shading on great argus Argusianus argus feathers create a three-dimensional illusion?

Many animals use shading to infer the three-dimensional (3D) shape of objects, and mimicking natural shading patterns can produce the illusion of 3D form on a flat surface. Over 150 years ago, Charles Darwin noted that the ocelli (eyespots) on the feathers of the great argus Argusianus argus, when held vertically during courtship displays to females, were perfectly shaded to resemble 3D hemispheres to human viewers. We tested whether these ocelli appear 3D to birds by training chickens Gallus gallus domesticus to select images of either convex or concave shapes using shading cues, and then presenting them with images of great argus ocelli. Chickens successfully learned how to discriminate between convex and concave shapes, and treated the great argus pheasant ocelli in the same way as convex training stimuli. Our findings are consistent with previous studies that birds can perceive 3D shape from shading cues in a similar manner to humans. The perception of great argus ocelli as consistent with 3D shape by avian viewers suggests that shape illusions can play a role in male courtship.

Spatial phase-shifting profilometry by use of polarization for measuring 3D shapes of metal objects.

In this paper, we present a polarization spatial phase-shifting method for fringe projection profilometry. It enables us to measure the three-dimensional shape of a metal object in a fast way requiring only a single-shot implementation. With this method, a couple of projectors are equipped, in front of their lens, with linear polarization filters having orthogonal polarization directions, so that they can simultaneously cast two sinusoidal fringe patterns having different phase shifts onto the measured metal surfaces without mixture. To register the two projected patterns, we suggest a fringe alignment method based on the epipolar geometry between the projectors. By taking advantage of the property of metal surfaces in maintaining polarization state of incident light, the deformed fringe patterns on the measured surfaces are captured by using two coaxially-arranged polarization cameras. As a result, the fringe phases are calculated by using a two-step phase-shifting algorithm and further the 3D shapes of the measured surfaces are reconstructed. Experimental results demonstrate the proposed method to be valid and efficient in measuring metal objects.

IMPROVEMENT OF THE THREE-DIMENSIONAL VISUALIZATION METHODS FOR DETERMINATION OF MORPHOLOGICAL FEATURES OF PHASE MICROOBJECTS

The use of methods of physical and mathematical three-dimensional modelling of phase microobjects, in particular blood erythrocytes, was proposed in order to determine morphological changes. The method of digital signal processing based on double Fourier transform, with the imposition of a band pass filter, the method of restoring the three-dimensional shape of objects from two-dimensional images and the method of spectral analysis were considered in detail improved performance. On the basis of dimensional and aberration calculations, the optimal elements in terms of optical parameters were selected and an experimental stand was assembled. Interference digital images of normal human erythrocytes and altered under the action of osmotic pressure, images of peripheral blood cells for illumination by three types of coherent radiation were obtained. To implement three-dimensional modelling developed a program in Python using Numpy and Open CV libraries and a program that allows to build and visualize three-dimensional models of two-dimensional images, running Windows, written in C ++. A DirectX library and a graphics accelerator are used to display 3D models. The method of visualization of three-dimensional models on two-dimensional images makes it possible to change the scale of the surface and the type of projection used, to work in real time in interactive mode. The method of spectral analysis with processing of images of erythrocytes for enlightenment obtained in three frequency bands of coherent light, after further correlation of the obtained results will significantly improve the accuracy of determining its morphological parameters. The proposed improved methods allow using them to determine the state of the plasma membranes of erythrocytes in order to assess possible pathologies, which will increase the effectiveness of laboratory diagnostic methods.

Characteristic Analysis of Data Preprocessing for 3D Point Cloud Classification Based on a Deep Neural Network: PointNet

Laser scanning is a noncontact and nondestructive technique that captures the three-dimensional (3D) shape of objects as point clouds. Deep neural networks have been widely used to classify the 3D shapes of point clouds. In applying deep learning on point clouds, point cloud preprocessing is the first step. This study was conducted to analyze 3D shape classification characteristics using a deep neural network, PointNet, with a point cloud dataset, ModelNet40, for four preprocessing cases: random, scaling, zero-mean, and normalization. For each preprocessing case, the minimum and maximum coordinates of the point clouds and 3D shape classification performance are investigated. The results show that normalization preprocessing exhibits the most significant improvement in classification performance, and the zero-mean method is particularly effective. The findings indicate that proper preprocessing, such as normalization, should be performed before deep learning when the mean coordinates and scale of the point clouds differ significantly.

High dynamic range real-time 3D measurement based on Fourier transform profilometry

The high dynamic range (HDR) fringe projection technique is widely used to measure the three-dimensional (3D) shapes of objects with shiny surfaces; however, it tends to compromise when measuring the dynamic objects. To reduce the number of images and improve the real-time performance, a Fourier transform profilometry (FTP)-based method that can increase the dynamic range for real-time 3D measurement is proposed. Firstly, single-color patterns are projected onto the measured object, and one captured color image can be separated into three monochrome images with different intensities by taking advantage of the different responses of R, G, and B channels. Second, the HDR fringe pattern is generated by compositing three monochrome images, and then processed by the background-normalized algorithm to obtain the final fringe pattern. Subsequently, FTP is employed to retrieve the phase map and obtain the 3D shape. The experimental results demonstrated that the proposed method can measure the dynamic objects with shiny surfaces in real-time 3D measurement system.