Three-dimensional Computer Vision Research Articles

Hierarchical channel-spatial interaction network for partial-to-partial point cloud registration

ABSTRACT Partial-to-partial point cloud registration is a fundamental and challenging task in the three-dimensional computer vision. In this letter, we propose a novel Hierarchical Channel-Spatial Interaction Network (HCSINet), which can hierarchically explore information interaction in terms of the channel and spatial dimensions, enabling well partial point cloud perception for partial-to-partial registration. Specifically, the Channel-wise Feature Interaction (CFI) module is regarded as the basic unit of the encoder sub-network, which achieves feature hierarchical interaction to learn the overlapping scores and intuitively improve the importance of overlapping areas. Furthermore, to improve the discrimination of the overlapping points, the Co-Attention-based Spatial Association (CA-SA) module progressively excavates multi-scale spatial correlation of overlapping points by conjointly computing collaborative attention. Experimental results verify the validity of our HCSINet, which achieves competitive partial-to-partial registration performances compared to existing traditional and deep learning-based methods. The source code is publicly available https://github.com/lxnudt/HCSINet.

Integrated convolutional neural networks for joint super-resolution and classification of radar images

Deep learning techniques have been widely used for two-dimensional (2D) and three-dimensional (3D) computer vision problems, such as object detection, super-resolution (SR) and classification to name a few. Radar images suffer from poor resolution as compared to optical images, hence developing a high-accuracy model to solve computer vision problems, such as a classifier, is a challenge. This is because of the lack of high-frequency details in the input images which makes it difficult for the classifier model to generate accurate predictions. Ways of addressing this challenge include training the learning model with a large dataset or using a more complicated model, such as deeper layer architecture. However, employing such approaches might result in the overfitting of the model, where the model might not generalize well on previously unseen data. Also, generating a large dataset for training the model is a challenging task, especially in the case of radar images. An alternate solution for achieving high accuracy in radar classification problems is provided in this paper wherein a CNN-enabled super-resolution (SR) model is integrated with the classifier model. The SR model is trained to generate high-resolution (HR) millimeter-wave (mmW) images from any input low-resolution (LR) mmW images. These resolved images from the SR model will be used by the classifier model to classify the input images into appropriate classes, consisting of threat and non-threat objects. The training data for the dual CNN layers are generated using a numerical model of a near-field coded-aperture computational imaging (CI) system. This trained dual CNN model is tested with simulated data generated from the CI numerical model wherein a high classification accuracy of 95% and a fast inference time of 0.193 s are obtained, making it suitable for real-time automated threat classification applications. For fair comparison, the developed CNN model is also validated with experimentally generated reconstruction data, in which case, a classification accuracy of 94% is obtained.

Point Cloud Registration Based on Local Variation of Surface Keypoints

Keypoint detection plays a pivotal role in three-dimensional computer vision, with widespread applications in improving registration precision and efficiency. However, current keypoint detection methods often suffer from poor robustness and low discriminability. In this study, a novel keypoint detection approach based on the local variation of surface (LVS) is proposed. The LVS keypoint detection method comprises three main steps. Firstly, the surface variation index for each point is calculated using the local coordinate system. Subsequently, points with a surface variation index lower than the local average are identified as initial keypoints. Lastly, the final keypoints are determined by selecting the minimum value within the neighborhood from the initial keypoints. Additionally, a sampling consensus correspondence estimation algorithm based on geometric constraints (SAC-GC) for efficient and robust estimation of optimal transformations in correspondences is proposed. By combining LVS and SAC-GC, we propose a coarse-to-fine point cloud registration algorithm. Experimental results on four public datasets demonstrate that the LVS keypoint detection algorithm offers improved repeatability and robustness, particularly when dealing with noisy, occluded, or cluttered point clouds. The proposed coarse-to-fine point cloud registration algorithm also exhibits enhanced robustness and computational efficiency.

Information geometry based extreme low-bit neural network for point cloud

Deep learning has significantly advanced three-dimensional computer vision applied to point clouds. Nevertheless, the substantial consumption of time, storage, and energy substantially limits its deployment on edge devices with constrained resources. Extremely low bit quantization has received wide attention due to its extremely high compression ratio, but the problem of a significant drop in accuracy cannot be ignored. To alleviate the obvious accuracy degradation for extreme low-bit quantization, this paper proposes a novel compression framework for binary and ternary neural networks applied to point clouds, which introduces information geometry to model quantization to compensate the severe feature manifold distortion. It applies differential geometry on manifolds to study the implicit information of the point cloud feature data. Based on the theoretical analysis from the novel perspective of information geometry, two optimization modules are proposed to alleviate severe geometry distortions on differential manifolds. The first module, scaling recovery, provides layer-wise scaling parameters to reduce geometric distortion caused by quantization. The second module, Pooling Recovery, is specially designed to alleviate more severe pooling geometry distortions in point clouds. These two modules benefit both binary and ternary neural networks with ignored overheads. For ternary quantization, optimizations on convolution weights and gradients are additionally introduced. The proposed self-adaptive gradient estimation provides a more accurate approximation to the non-differential ternary staircase function. Convolution weight optimization is implemented on an information-geometry optimized model to achieve even higher accuracy and less memory consumption. Experimental results validate that the proposed models significantly outperform state-of-the-art methods and demonstrate better scalability. Overall, this compression framework has the potential to facilitate the deployment of deep learning models on edge devices with limited resources, opening up new opportunities for applications of three-dimensional computer vision.

Computer Vision Driven Precision Dairy Farming for Efficient Cattle Management

Precision Dairy Farming (PDF)” or “The Per Animal Approach” can be enhanced through the implementation of three-dimensional computer vision, which offers improved cattle identification, disease monitoring, and growth assessment. The integration of 3D vision systems is particularly vital for advancing dairy farming practices in the next generation. These systems facilitate the automation of various animal husbandry tasks, including monitoring, herding, feeding, milking, and bedding of animals. The applications of 3D computer vision in PLF encompass diverse platforms, such as 3D camera installations for monitoring cow walking postures, and intelligent systems that interact safely with animals, capable of identifying dairy cattle and detecting health indicators like animal identification, recognition, body condition score, and lameness. To be effective, systems must be adaptable to unconstrained environments, varying herd characteristics, weather conditions, farmyard layouts, and animal-machine interaction scenarios. Considering these requirements, this paper proposes the application of emerging computer vision and artificial intelligence techniques in dairy farming. This review encourages future research in three-dimensional computer vision for cattle growth management and its potential extension to other livestock and wild animals

Software for brain tumor diagnosis on magnetic resonance imaging

BACKGROUND: The main reason for the development and implementation of artificial intelligence (AI) technologies in neuro-oncology is the high prevalence of brain tumors reaching up to 200 cases per 100,000 population. The incidence of a primary focus in the brain is 5%10%; however, 60%70% of those who die from malignant neoplasms have metastases in the brain. Magnetic resonance imaging (MRI) is the most common method for primary non-invasive diagnosis of brain tumors and monitoring disease progression. One of the challenges is the classification of tumor types and determination of clinical parameters (size and volume) for the conduct, diagnosis, and treatment procedures, including surgery.
 AIM: To develope a software module for the differential diagnosis of brain neoplasms on MRI images.
 METHODS: The software module is based on the developed Siberian Brain Tumor Dataset (SBT), which contains information on over 1000 neurosurgical patients with fully verified (histologically and immunohistochemically) postoperative diagnoses. The data for research and development was presented by the Federal Neurosurgical Center (Novosibirsk). The module uses two- and three-dimensional computer vision models with pre-processed MRI sequence data included in the following packages: pre-contrast T1-weighted image (WI), post-contrast T1-WI, T2-WI, and T2-WI with fluid-attenuated inversion-recovery technique. The models allow to detect and recognize with high accuracy 4 types of neoplasms, such as meningioma, neurinoma, glioblastoma, and astrocytoma, and segment and distinguish components and sizes: ET (tumor core absorbing Gd-containing contrast), TC (tumor core) = ET + Necr (necrosis) + NenTu, and WT (whole tumor) = TC + Ed (peritumoral edema).
 RESULTS: The developed software module shows high segmentation results on SBT by Dice metric for ET 0.846, TC 0.867, WT 0.9174, Sens 0.881, and Spec 1.000 areas. The testing and validation were done at the international BraTS Challenge 2021 competition. The test dataset yielded DiceET 0.86588, DiceTC 0.86932, and DiceWT 0.921 values, placing the developed software module in the top ten. According to the classification, the results demonstrate high accuracy rates of up to 92% in patient analysis (up to 89% in slice analysis), a very high potential, and a perspective for future research in this area.
 CONCLUSIONS: The developed software module may be used for training specialists and in clinical diagnostics.

A Fast and Reliable Solution to PnP, Using Polynomial Homogeneity and a Theorem of Hilbert

One of the most-extensively studied problems in three-dimensional Computer Vision is “Perspective-n-Point” (PnP), which concerns estimating the pose of a calibrated camera, given a set of 3D points in the world and their corresponding 2D projections in an image captured by the camera. One solution method that ranks as very accurate and robust proceeds by reducing PnP to the minimization of a fourth-degree polynomial over the three-dimensional sphere S3. Despite a great deal of effort, there is no known fast method to obtain this goal. A very common approach is solving a convex relaxation of the problem, using “Sum Of Squares” (SOS) techniques. We offer two contributions in this paper: a faster (by a factor of roughly 10) solution with respect to the state-of-the-art, which relies on the polynomial’s homogeneity; and a fast, guaranteed, easily parallelizable approximation, which makes use of a famous result of Hilbert.

Machine learning applied to HR-pQCT images improves fracture discrimination provided by DXA and clinical risk factors.

Traditional analysis of High Resolution peripheral Quantitative Computed Tomography (HR-pQCT) images results in a multitude of cortical and trabecular parameters which would be potentially cumbersome to interpret for clinicians compared to user-friendly tools utilising clinical parameters. A computer vision approach (by which the entire scan is 'read' by a computer algorithm) to ascertain fracture risk, would be far simpler. We therefore investigated whether a computer vision and machine learning technique could improve upon selected clinical parameters in assessing fracture risk. Participants of the Hertfordshire Cohort Study (HCS) attended research visits at which height and weight were measured; fracture history was determined via self-report and vertebral fracture assessment. Bone microarchitecture was assessed via HR-pQCT scans of the non-dominant distal tibia (Scanco XtremeCT), and bone mineral density measurement and lateral vertebral assessment were performed using dual-energy X-ray absorptiometry (DXA) (Lunar Prodigy Advanced). Images were cropped, pre-processed and texture analysis was performed using a three-dimensional local binary pattern method. These image data, together with age, sex, height, weight, BMI, dietary calcium and femoral neck BMD, were used in a random-forest classification algorithm. Receiver operating characteristic (ROC) analysis was used to compare fracture risk identification methods. Overall, 180 males and 165 females were included in this study with a mean age of approximately 76years and 97 (28%) participants had sustained a previous fracture. Using clinical risk factors alone resulted in an area under the curve (AUC) of 0.70 (95% CI: 0.56-0.84), which improved to 0.71 (0.57-0.85) with the addition of DXA-measured BMD. The addition of HR-pQCT image data to the machine learning classifier with clinical risk factors and DXA-measured BMD as inputs led to an improved AUC of 0.90 (0.83-0.96) with a sensitivity of 0.83 and specificity of 0.74. These results suggest that using a three-dimensional computer vision method to HR-pQCT scanning may enhance the identification of those at risk of fracture beyond that afforded by clinical risk factors and DXA-measured BMD. This approach has the potential to make the information offered by HR-pQCT more accessible (and therefore) applicable to healthcare professionals in the clinic if the technology becomes more widely available.

Two- and Three-Dimensional Computer Vision Techniques for More Reliable Body Condition Scoring

This article identifies the essential technologies and considerations for the development of an Automated Cow Monitoring System (ACMS) which uses 3D camera technology for the assessment of Body Condition Score (BCS). We present a comparison of a range of common techniques at the different developmental stages of Computer Vision including data pre-processing and the implementation of Deep Learning for both 2D and 3D data formats commonly captured by 3D cameras. This research focuses on attaining better reliability from one deployment of an ACMS to the next and proposes a Geometric Deep Learning (GDL) approach and evaluating model performance for robustness from one farm to another in the presence of background, farm, herd, camera pose and cow pose variabilities.

Spectral-Polarization Systems of Three-Dimensional Computer Vision Based on Acousto-Optic Filtering

Development of two-channel acousto-optic spectral devices with enhanced functionality is described. Approaches to development of multifunctional acousto-optic devices capable to register different types of optical information (spatial distribution, spectra, polarisation characteristics) are discussed. Opportunities to register multi-dimensional complementary data based on Bragg light diffraction on ultrasonic waves in a uniaxial crystal are analysed. It is demonstrated that such approach may be developed on the basis of implementation of simultaneous diffraction of two light beams in planes rotated relative to each other by a small angle. Such enhancement of the acousto-optic filter operational concept allows to go beyond the classic constraints imposed on their characteristics.

Estimating economic benefit of sugar beet based on three-dimensional computer vision: a case study in Inner Mongolia, China

Selecting and breeding crop varieties with high economic benefits is of great significance for social stability and development. The economic benefit of crops is usually reflected by the purchase price. Traditional estimation of economic benefits using purchase price formula based on manual measured traits is time-consuming. Structure-from-Motion in conjunction with multi-view stereo (SFM-MVS) method could extract plant phenotypic traits and has the potential for the efficient and timely estimation of economic benefits for sugar beet. In this study, a framework was developed to obtain phenotypic traits in order to estimate the economic benefits of sugar beet with 207 genotypes based on the calculation of a non-linear formula and the partial least square regression (PLSR) model. The first part of the framework was the designing of a low-cost portable equipment that can be used to obtain multi-view images of taproot in order to facilitate its three-dimensional (3D) reconstruction based on SFM-MVS method. The following part was the development of an automated pipeline for estimating ten traits from the reconstructed 3D taproot. Good agreement was found between measured and estimated traits with R2 >0.97. The PLSR model constructed using the data in 2018 was used to predict the data in 2019 with moderate performance (R2 = 0.5). A new PLSR model built using 70 % of the data collected in 2018 and 2019 could predict the remaining 30 % of the data with a higher R2 of 0.61. The model built with multi-years data had a higher accuracy in estimating phenotypic traits, which suggests that PLSR model can estimate beet economic benefit by using the SFM-MVS method with multi-year data. The current method is more efficient than the manual measurement and may provide a basis for selecting and cultivating sugar beet with high economic benefit.

A laboratory prototype of automatic pavement crack sealing based on a modified 3D printer

ABSTRACT Cracking is a common fault in asphalt and concrete pavements, which causes water damage and further defects if not repaired in timely fashion. Conventional pavement crack sealing methods based on machines and manual operations are generally subjective, labour-intensive, risky, and inefficient. A laboratory prototype of an automatic pavement crack sealing platform is proposed in this paper, which uses a modified three-dimensional (3D) printer and computer vision. A modified 3D printer based on fused deposition modelling (FDM) was combined with an image capturing platform, an image processing algorithm and a path planning method to form the automated pavement crack sealing platform, which can automatically detect pavement cracks and seal them with bitumen emulsion sealant. Specimens of concrete pavement slabs with cracks were produced in the laboratory to test the proposed method, and the cracks were then detected and sealed by the proposed platform. The results show that 3D printing is an effective method for automated pavement crack sealing, which is recommended in the field of automatic road maintenance and repair.

The generalized orthogonal Procrustes problem in the high noise regime

AbstractWe consider the problem of estimating a cloud of points from numerous noisy observations of that cloud after unknown rotations and possibly reflections. This is an instance of the general problem of estimation under group action, originally inspired by applications in three-dimensional imaging and computer vision. We focus on a regime where the noise level is larger than the magnitude of the signal, so much so that the rotations cannot be estimated reliably. We propose a simple and efficient procedure based on invariant polynomials (effectively: the Gram matrices) to recover the signal, and we assess it against fundamental limits of the problem that we derive. We show our approach adapts to the noise level and is statistically optimal (up to constants) for both the low and high noise regimes. In studying the variance of our estimator, we encounter the question of the sensivity of a type of thin Cholesky factorization, for which we provide an improved bound which may be of independent interest.

Multi-Head Attentional Point Cloud Classification and Segmentation Using Strictly Rotation-Invariant Representations

Point cloud processing plays an increasingly essential role in three-dimensional (3D) computer vision target, scene parsing, environmental perception, etc. Compared with using aligned point cloud data for classification and segmentation, the strictly rotation-invariant representations show enough robustness. Inspired by the great success of deep learning, we propose a novel neural network called Multi-head Attentional Point Cloud Classification and Segmentation Using Strictly Rotation-invariant Representations. Our research focuses on processing the point cloud rotated in any direction effectively and precisely. First of all, the strictly rotation-invariant point cloud representations are obtained through point projection. Then we apply a multi-head attentional convolution layer (MACL) using attention coding to develop the performance of point cloud feature extraction. Finally, our network assigns different responses and recognizes the overall geometry well through a key point descriptor, adding to the global feature. Our method can explore more in-depth information for accuracy enhancement with attention pooling and multi-layer perceptron (MLP) based on an advanced DenseNet. Our network enjoys 90.63% and 87.50% classification accuracy testing on ModelNet10 and ModelNet40, and 75.15% intersection over union metric (mIoU) evaluating on ShapeNet Part dataset, remaining under any rotation. Rotating experimental results indicate that our framework realizes better point cloud classification and segmentation performance than most state-of-the-art methods.

Highly accurate 3D reconstruction based on a precise and robust binocular camera calibration method

The precision of the camera calibration is one of the key factors that affect attitude measurement accuracy in many computer vision tasks. This study proposes a new calibration approach for binocular cameras. Firstly, based on singular value decomposition, the best transformation matrix to the essential matrix is approximated as the initial guess, which is solved in using the Frobenius norm. Secondly, the initial guess is refined through maximum likelihood estimation. A new calculating expression is derived for computing the relative position matrix of the binocular cameras. The Levenberg–Marquardt algorithm is then implemented to refine the initial guess. Large sets of synthesised and real point correspondences were tested to demonstrate the validity of the proposed method. Extensive experiments demonstrated that the proposed method outperforms the state-of-the-art methods. The error rate of the proposed method was 0.5% for the length test and about 1% for the angle test at a range of 1 m. This method can advance three-dimensional (3D) computer vision one additional step from laboratory environments to real-world use.

Quantifying Physical Activity in Young Children Using a Three-Dimensional Camera.

The purpose of this study was to determine the feasibility and validity of using three-dimensional (3D) video data and computer vision to estimate physical activity intensities in young children. Families with children (2–5-years-old) were invited to participate in semi-structured 20-minute play sessions that included a range of indoor play activities. During the play session, children’s physical activity (PA) was recorded using a 3D camera. PA video data were analyzed via direct observation, and 3D PA video data were processed and converted into triaxial PA accelerations using computer vision. PA video data from children (n = 10) were analyzed using direct observation as the ground truth, and the Receiver Operating Characteristic Area Under the Curve (AUC) was calculated in order to determine the classification accuracy of a Classification and Regression Tree (CART) algorithm for estimating PA intensity from video data. A CART algorithm accurately estimated the proportion of time that children spent sedentary (AUC = 0.89) in light PA (AUC = 0.87) and moderate-vigorous PA (AUC = 0.92) during the play session, and there were no significant differences (p > 0.05) between the directly observed and CART-determined proportions of time spent in each activity intensity. A computer vision algorithm and 3D camera can be used to estimate the proportion of time that children spend in all activity intensities indoors.

The Control of Holographic 3D Towercopter Model

Abstract Many teachers and universities try to find various ways how to improve the education process. Usage of interactive technologies and the three-dimensional computer vision brings new perspective of understanding problems related to the issue, makes scientific facts more attractive and improves student results. The paper describes a new system for interactive teaching using holographic 3D simulations. It can help with better illustration of physical meaning of abstract mathematical model describing a dynamical system. The presented system is developed for educational purposes, mainly for teaching of control theory. The created platform consists of two parts. The first one is dealing with visualization of the 3D model in the holographic device. The second part is devoted to the control of the model on the base of predefined control structures. Background of the application is driven by Scilab simulation environment. The advantage of the system is that it allows to upload new controllers for the simulation. Students can design their own controllers and to use them for the predefined 3D model. Comparison of results from simulations respecting various setting of controllers or even different control structures helps to understand the subject matter and to improve the quality of learning. The functionality of the whole system is demonstrated on the experiment with towercopter. The created model is based on the real towercopter plant situated in our laboratory.

Representation of freshwater aquaculture fish behavior in low dissolved oxygen condition based on 3D computer vision

Dissolved oxygen (DO) plays an important role in industrialized freshwater aquaculture. Such deficiencies such as the high cost of water-quality monitoring system and the failure to accurately monitor or describe aquaculture water-quality existed in freshwater aquaculture water-quality monitoring system. Here, a kind of representation method applied to characterize industrialized aquaculture fish behavior in different degrees of DO deficiency is based on three-dimensional (3D) Computer Vision. 3D coordinate values of aquaculture fishes in water acquired from 3D Computer Vision Device by processing aquaculture fish image are applied to represent such parameters as the average activity and height of aquaculture fish in water. This method for representing different behaviors of industrialized freshwater aquaculture fish under the condition of anoxia is realized by using these parameters and combing with the experience of aquaculture. The results show that the representation of industrialized freshwater aquaculture fish based on 3D Computer Vision System can be applied to describe industrialized aquaculture fish behavior and effectively compensate for the shortfall spatial location of aquaculture fish unable to acquire from 2D monitoring system, which is helpful for the accurate and reasonable control of DO in aquaculture.

Polarimetric target depth sensing in ambient illumination based on polarization-coded structured light.

Depth sensing is a basic issue in three-dimensional computer vision, and structured light is one of the most prevailing methods for it. However, complex surroundings and strong ambient illumination are fairly unfavorable to depth sensing based on structured light. Complex surroundings increase computation overhead and require extra effort to be separated from the target object. Strong ambient illumination is unfavorable to the signal-noise ratio of structured light and, thus, increases the difficulty of decoding. In this paper, we propose that the polarization-coded structured light is capable of target enhanced depth sensing in ambient illumination. We present the polarimetric principle, an improved algorithm of polarization-coded structured light, and signal-noise-ratio analysis in ambient illumination. Experimental results show that polarization-coded structured light is efficient and robust for target depth sensing of a complicated environment. The polarization-coded structured light is promising to the target depth sensing in an outdoor scenario and industrial inspection.

3D CV Descriptor on Parallel Heterogeneous Platforms

Embedded three-dimensional (3D) Computer Vision (CV) is considered a technology enabler for future consumer applications, attracting a wide interest in academia and industry. However, 3D CV processing is a computation-intensive task. Its high computational cost is directly related to the processing of 3D point clouds, with the 3D descriptor computation representing one of the main bottlenecks. Understanding the main computational challenges of 3D CV applications, as well as the key characteristics, enabling features, and limitations of current computing platforms, is clearly strategic to identify the directions of evolution for future embedded processing systems targeting 3D CV. In this work, an innovative and complex 3D descriptor (called SHOT) has been ported on a high-end and an embedded computing platform. The high-end system is composed by a high-performance Intel CPU coupled with a Nvidia GPU. The embedded platform is, instead, composed by an ARM-based processor, coupled with the STHORM accelerator. STHORM is a many-core low-power accelerator developed by ST Microelectronics, featuring up to 64 computational units. The SHOT descriptor has been parallelized using the OpenCL programming model for both platforms. Finally, we have performed an in-depth performance comparison and analysis between general-purpose processors and accelerators in both high-end and embedded domains, discussing and highlighting the main differences in the Hardware/Software (HW/SW) design methodologies and approaches between high-end and embedded systems targeting 3D CV applications.