3D Pixel Research Articles

Utilizing Spectral, Structural and Textural Features for Estimating Oat Above-Ground Biomass Using UAV-Based Multispectral Data and Machine Learning.

Accurate and timely monitoring of biomass in breeding nurseries is essential for evaluating plant performance and selecting superior genotypes. Traditional methods for phenotyping above-ground biomass in field conditions requires significant time, cost, and labor. Unmanned Aerial Vehicles (UAVs) offer a rapid and non-destructive approach for phenotyping multiple field plots at a low cost. While Vegetation Indices (VIs) extracted from remote sensing imagery have been widely employed for biomass estimation, they mainly capture spectral information and disregard the 3D canopy structure and spatial pixel relationships. Addressing these limitations, this study, conducted in 2020 and 2021, aimed to explore the potential of integrating UAV multispectral imagery-derived canopy spectral, structural, and textural features with machine learning algorithms for accurate oat biomass estimation. Six oat genotypes planted at two seeding rates were evaluated in two South Dakota locations at multiple growth stages. Plot-level canopy spectral, structural, and textural features were extracted from the multispectral imagery and used as input variables for three machine learning models: Partial Least Squares Regression (PLSR), Support Vector Regression (SVR), and Random Forest Regression (RFR). The results showed that (1) in addition to canopy spectral features, canopy structural and textural features are also important indicators for oat biomass estimation; (2) combining spectral, structural, and textural features significantly improved biomass estimation accuracy over using a single feature type; (3) machine learning algorithms showed good predictive ability with slightly better estimation accuracy shown by RFR (R2 = 0.926 and relative root mean square error (RMSE%) = 15.97%). This study demonstrated the benefits of UAV imagery-based multi-feature fusion using machine learning for above-ground biomass estimation in oat breeding nurseries, holding promise for enhancing the efficiency of oat breeding through UAV-based phenotyping and crop management practices.

A Study of 3D model voxelization method for artificial intelligence learning

<p class="abstract">The 3D reconstruction technology which is one of various restoration technologies, implements and shapes 2D pixels of an object that actually exists in a 3D form. As an accurate 3D model for artificial intelligence learning information pre-processing is required. The pre-processing needs to accurately size information and coordinate information of a 3D object for artificial intelligence learning. Also, the 3D model data generated during the preprocessing can be represented as a point cloud, which is point-based coordinate data. In this study, the pixel data was analyzed by voxelizing the 3D model for artificial intelligence learning, and the 3D model data was digitized to form a mesh file. In this study, 3D modeling was done with a 3D modeling tool and the object was exported to STL. In addition, it was converted into mesh file data including 3D coordinate data in Python language on Google’s colab platform. The mesh data created in this way is used in DataSet for artificial intelligence learning (CNN, RNN, GAN). Currently, there are many datasets for 2D artificial intelligence learning, but this study provided an opportunity to collect 3D artificial intelligence learning datasets. The research results can be used in the fields of robots, autonomous driving, games, and product design.</p>

Efficient and Accurate Computed Tomography-Based Stone Volume Determination: Development of an Automated Artificial Intelligence Algorithm.

Given the shortcomings of current stone burden characterization (maximum diameter or ellipsoid formulas), we sought to investigate the diagnostic accuracy and precision of a University of California, Irvine-developed artificial intelligence (AI) algorithm for determining stone volume determination. A total of 322 noncontrast CT scans were retrospectively obtained from patients with a diagnosis of urolithiasis. The largest stone in each noncontrast CT scan was designated the "index stone." The 3D volume of the index stone using 3D Slicer technology was determined by a validated reviewer; this was considered the "ground truth" volume. The AI-calculated index stone volume was subsequently compared with ground truth volume as well with the scalene, prolate, and oblate ellipsoid formulas estimated volumes. There was a nearly perfect correlation between the AI-determined volume and the ground truth (R=0.98). While the AI algorithm was efficient for determining the stone volume for all sizes, its accuracy improved with larger stone size. Moreover, the AI stone volume produced an excellent 3D pixel overlap with the ground truth (Dice score=0.90). In comparison, the ellipsoid formula-based volumes performed less well (R range: 0.79-0.82) than the AI algorithm; for the ellipsoid formulas, the accuracy decreased as the stone size increased (mean overestimation: 27%-89%). Lastly, for all stone sizes, the maximum linear stone measurement had the poorest correlation with the ground truth (R range: 0.41-0.82). The University of California, Irvine AI algorithm is an accurate, precise, and time-efficient tool for determining stone volume. Expanding the clinical availability of this program could enable urologists to establish better guidelines for both the metabolic and surgical management of their urolithiasis patients.

Microfluidics on lensless, semiconductor optical image sensors: challenges and opportunities for democratization of biosensing at the micro-and nano-scale

Abstract Optical image sensors are 2D arrays of pixels that integrate semiconductor photodiodes and field effect transistors for efficient photon conversion and processing of generated electrons. With technological advancements and subsequent democratization of these sensors, opportunities for integration with microfluidics devices are currently explored. 2D pixel arrays of such optical image sensors can reach dimensions larger than one centimeter with a sub-micrometer pixel size, for high spatial resolution lensless imaging with large field of view, a feat that cannot be achieved with lens-based optical microscopy. Moreover, with advancements in fabrication processes, the field of microfluidics has evolved to develop microfluidic devices with an overall size below one centimeter and individual components of sub-micrometer size, such that they can now be implemented onto optical image sensors. The convergence of these fields is discussed in this article, where we review fundamental principles, opportunities, challenges, and outlook for integration, with focus on contact-mode imaging configuration. Most recent developments and applications of microfluidic lensless contact-based imaging to the field of biosensors, in particular those related to the potential for point of need applications, are also discussed.

Multiple face images encryption based on a new non-adjacent dynamic coupled mapping lattice

There have been limited research studies on encryption algorithms specifically designed for certain encryption scenarios. Aiming at different types of face images, this paper presents a multiple face images encryption algorithm based on a new non-adjacent coupled mapping lattice with dynamic parameter (NCMLP). NCMLP has a more comprehensive parameter range, better chaos, and more complex dynamic behavior Compared to classical non-adjacent coupled lattice mapping, which is suitable for encryption. The encryption algorithm firstly uses face detection technology to extract multiple face information and to generate a double-layer key, then scrambles the face information by three-dimensional bit-level cyclic shift, and finally encrypts the global information using 3D pixel level scrambling and 3D double index diffusion. It is worth mentioning that the algorithm has high security and can be applied to ordinary images. The experimental results and safety analysis show that the information entropy can exceed 7.9992, the histogram variance can be less than 1072, the pixel number change rate can reach 99.6323%, and the unified average change intensity can reach 33.5495%, all of which are very close to the ideal values.

Prediction of optimal 3D printing orientation using vertically sparse voxelization and modified support structure tomography

PurposeIncreasingly 3D printing is used for parts of garments or for making whole garments due to their flexibility and comfort and for functionalizing or enhancing the aesthetics of the final garment and hence adding value. Many of these applications rely on complex programming of the 3D printer and are usually provided by the vendor company. This paper introduces a simpler, easier platform for designing 3D-printed textiles, garments and other artifacts, by predicting the optimal orientation of the target objects to minimize the use of plastic filaments.Design/methodology/approachThe main idea is based on the shadow-casting analogy, which assumes that the volume of the support structure is similar to that of the shadow from virtual sunlight. The triangular elements of the target object are converted into 3D pixels with integer-based normal vectors and real-numbered coordinates via vertically sparse voxelization. The pixels are classified into several groups and their noise is suppressed using a specially designed noise-filtering algorithm called slot pairing. The final support structure volume information was rendered as a two-dimensional (2D) figure, similar to a medical X-ray image. Thus, the authors named their method modified support structure tomography.FindingsThe study algorithm showed an error range of no more than 1.6% with exact volumes and 6.8% with slicing software. Moreover, the calculation time is only several minutes for tens of thousands of mesh triangles. The algorithm was verified for several meshes, including the cone, sphere, Stanford bunny and human manikin.Originality/valueSimple hardware, such as a CPU, embedded system, Arduino or Raspberry Pi, can be used. This requires much less computational resources compared with the conventional g-code generation. Also, the global and local support structure is represented both quantitatively and graphically via tomographs.

Painting with Data: Visually based open-source tool for geo-computing

In this article we present, the conceptual and technical background of a tool for mapping and geo-computing, Painting with Data (PWD). PWD was inspired by early computer mapping algorithms that focused on drawing as an intuitive interface between spatial data and architectural practice. PWD embraces the potential of such techniques, coupled with alternative computational representations, modes of interaction, and computational interfaces, to encourage public participation in planning and design. Essential to this task, is PWD’s integration of open-source, collaborative, interactive, and web-based technologies to create an online software with a visually based approach to spatial analysis and mapping that dramatically reduces the steep learning curve required for GIS software. As a high-level graphical interface, PWD allows users to iterate by intuitively creating spatial models on-the-fly based on their subjective understanding of information. In PWD, we deploy voxels, a data structure that organizes information as 3D pixels, which allows users to compute with spatial information visually, to potentially inform the ways in which users build quantitative models. In PWD, multiple layers of information can be visualized concurrently, and visual correlations can be made instantly. Users build spatial models by directly manipulating the map itself instead of manipulating a database that then produces a map. PWD’s high-level interaction is made possible by custom data structures that leverage GPU processing, which makes them significantly faster than traditional topological data structures. Computationally, user interactions generate visualization specifications and declarative queries that are compiled and executed by the platform. Lastly, PWD introduces a visual programming language, which enables intuitive geospatial modeling and visualization. Practical work has shown the value of PWD’s approach to design students, planning agencies and community non-profits. Throughout the process, PWD’s open-source spatial models generated a user community with more than 3000 users, including designers, students, and educators.

3D pose estimation and localization of construction equipment from single camera images by virtual model integration

Computer vision technologies are receiving much attention for automatic site monitoring. However, previous approaches mainly focused on two-dimensional (2D) pixel information, and there was a limit to providing detailed three-dimensional (3D) pose information. Some studies have built a training database only using virtual models, but there were still limitations in representing the actual equipment. To address these limitations, the authors propose a method for estimating 3D poses of equipment from single camera images by integrating virtual models. The proposed method consists of three main processes: (1) 2D – 3D annotation using a virtual model; (2) preprocessing for pose estimation in two ways (i.e., image matching or key points detection); and (3) 3D pose localization. As a result, the root mean square error for the 3D pose of construction equipment was 1.49 m (using image matching) and 1.56 m (using key points detection). The results imply that the proposed method successfully estimated the 3D pose of the equipment with a single camera. The findings of this study can contribute to detailed productivity or safety analysis based on more diverse 3D information than 2D pixel-based analysis.

Influence of the number of measurements on detecting internal defects using photothermal super resolution reconstruction with random pixel patterns

ABSTRACT In this work, the limits of using spatially structured heating combined with subsequent photothermal super resolution reconstruction for the thermographic detection and resolution of internal defects are investigated. The approach is based on the application of modern high-power laser projector technology, which is used to repeatedly project varying spatially structured 2D pixel patterns to photothermally heat the object under test. After processing the generated thermographic data using nonlinear convex optimisation in conjunction with exploiting the joint-sparse nature of the defect signals within the individual measurements, a high-resolution 2D-sparse defect/inhomogeneity map is obtained. The main focus of the investigation is set on the influence of the number of individual measurements on the achievable reconstruction quality. Using numerical simulations based on an analytical representation of the forward solution to the underlying inverse problem, the convergence rate over performed measurements of the achievable reconstruction quality is determined. Finally, all findings are experimentally validated by reconstructing a set of internal defects in an additively manufactured sample. In this work, it is shown that for a variety of different defect separation distances, the projection of 50 different pixel patterns allows for a good trade-off between experimental complexity and reconstruction quality.

Cross-Modal Contrastive Learning for Domain Adaptation in 3D Semantic Segmentation

Domain adaptation for 3D point cloud has attracted a lot of interest since it can avoid the time-consuming labeling process of 3D data to some extent. A recent work named xMUDA leveraged multi-modal data to domain adaptation task of 3D semantic segmentation by mimicking the predictions between 2D and 3D modalities, and outperformed the previous single modality methods only using point clouds. Based on it, in this paper, we propose a novel cross-modal contrastive learning scheme to further improve the adaptation effects. By employing constraints from the correspondences between 2D pixel features and 3D point features, our method not only facilitates interaction between the two different modalities, but also boosts feature representations in both labeled source domain and unlabeled target domain. Meanwhile, to sufficiently utilize 2D context information for domain adaptation through cross-modal learning, we introduce a neighborhood feature aggregation module to enhance pixel features. The module employs neighborhood attention to aggregate nearby pixels in the 2D image, which relieves the mismatching between the two different modalities, arising from projecting relative sparse point cloud to dense image pixels. We evaluate our method on three unsupervised domain adaptation scenarios, including country-to-country, day-to-night, and dataset-to-dataset. Experimental results show that our approach outperforms existing methods, which demonstrates the effectiveness of the proposed method.

Subsampling of 3D Pixel Blocks as a Video Compression Method for Analog Transmission

Hybrid digital–analog transmission of video signals enables flexibility in dividing video information into two parts to utilize the available bandwidth better. This study proposes a compression scheme to reduce the utilized bandwidth. The scheme uses different subsampling in three-dimensional (3D) blocks, where subsampling factors are selected to minimize reconstruction distortion. The study evaluates various methods for subsampling and reconstruction to find the best combination in terms of reconstruction quality and complexity. Results show that medium-quality reconstructions can be obtained for compression ratios of about 0.125–0.3 samples per pixel.

3-D Scene Flow Estimation on Pseudo-LiDAR: Bridging the Gap on Estimating Point Motion

3D scene flow characterizes how the points at the current time flow to the next time in the 3D Euclidean space, which possesses the capacity to infer autonomously the non-rigid motion of all objects in the scene. The previous methods for estimating scene flow from images have limitations, which split the holistic nature of 3D scene flow by estimating optical flow and disparity separately. Learning 3D scene flow from point clouds also faces the difficulties of the gap between synthesized and real data and the sparsity of LiDAR point clouds. In this paper, the generated dense depth map is utilized to obtain explicit 3D coordinates, which achieves direct learning of 3D scene flow from 2D images. The stability of the predicted scene flow is improved by introducing the dense nature of 2D pixels into the 3D space. Outliers in the generated 3D point cloud are removed by statistical methods to weaken the impact of noisy points on the 3D scene flow estimation task. Disparity consistency loss is proposed to achieve more effective unsupervised learning of 3D scene flow. The proposed method of self-supervised learning of 3D scene flow on real-world images is compared with a variety of methods for learning on the synthesized dataset and learning on LiDAR point clouds. The comparisons of multiple scene flow metrics are shown to demonstrate the effectiveness and superiority of introducing pseudo-LiDAR point cloud to scene flow estimation.

TCAD Analysis of Leakage Current and Breakdown Voltage in Small Pitch 3D Pixel Sensors.

Small-pitch 3D pixel sensors have been developed to equip the innermost layers of the ATLAS and CMS tracker upgrades at the High Luminosity LHC. They feature 50 × 50 and 25 × 100 μm2 geometries and are fabricated on p-type Si-Si Direct Wafer Bonded substrates of 150 μm active thickness with a single-sided process. Due to the short inter-electrode distance, charge trapping effects are strongly mitigated, making these sensors extremely radiation hard. Results from beam test measurements of 3D pixel modules irradiated at large fluences (1016neq/cm2) indeed demonstrated high efficiency at maximum bias voltages of the order of 150 V. However, the downscaled sensor structure also lends itself to high electric fields as the bias voltage is increased, meaning that premature electrical breakdown due to impact ionization is a concern. In this study, TCAD simulations incorporating advanced surface and bulk damage models are used to investigate the leakage current and breakdown behavior of these sensors. Simulations are compared with measured characteristics of 3D diodes irradiated with neutrons at fluences up to 1.5 × 1016neq/cm2. The dependence of the breakdown voltage on geometrical parameters (e.g., the n+ column radius and the gap between the n+ column tip and the highly doped p++ handle wafer) is also discussed for optimization purposes.

A binocular vision-based underwater object size measurement paradigm: Calibration-Detection-Measurement (C-D-M)

In this study, we propose a binocular vision-based underwater object size measurement paradigm: Calibration-Detection-Measurement (C-D-M). In the C-procedure, we construct an underwater refraction model to calculate basic parameters and refractive parameters of an underwater binocular vision system in waterproof housing. These parameters are key to realising the conversion of 2D pixel coordinates to 3D world coordinates. In the D-procedure, we attempt to address the difficult problem of underwater object detection via multi-feature joint perception, and accurately segment the regions of underwater objects in binocular images. Subsequently, the object regions and 3D reconstruction parameters are input into the M-procedure to measure the sizes of underwater objects. Specifically, a novel contour curve matching strategy is presented instead of dense matching to acquire reliable matching points. Subsequently, a 3D contour curve is reconstructed to calculate the key sizes of objects. Finally, comprehensive evaluations are conducted to demonstrate the effectiveness of our C-D-M paradigm.

An attitude estimation algorithm for the telescopic arm of the boarding bridge based on YOLOv5 and EPnP

Aiming at the attitude estimation of the telescopic arm of the boarding bridge in the process of docking with the offshore wind turbine, an attitude estimation algorithm for the telescopic arm of the boarding bridge based on YOLOv5 and EPnP is proposed in this paper. YOLOv5 algorithm is used to detect four marks in the offshore wind turbine logo image, and then the EPnP algorithm is used to solve attitude angles of the telescopic arm relative to the landing port of the offshore wind turbine according to the 2D pixel coordinates of the center point of each mark. The experimental results show that the attitude estimation error of the proposed algorithm is less than 0.4° and the data update frequency of the algorithm implemented in NVIDIA Jetson AGX Xavier is 25Hz, which meets the real-time and accuracy requirements for the attitude detection of the boarding bridge telescopic arm.

Deep learning model-based brand design 3D image construction

Abstract In order to have a better product display and thus attract consumers’ purchases and increase the economic benefits of the enterprise, in this paper, we propose a deep learning model for brand 3D image design. A feedforward neural network that estimates the error of previous layers based on the error of the output layer assigns the convolutional kernel weight parameters of the network in the interval and stops when the error reaches a preset accuracy or reaches a preset maximum learning count. The locally-aware convolutional neural network acquires local features that are finer than the global features and outputs the feature maps of the convolutional layers after passing the activation function to calculate the sensitivity of the sampled layer units. Given the sensitivity information of the feature map, the gradient of the kernel function weights is obtained, and the updated parameters are trained to achieve feature map recursion and solve the image boundary problem. A 3D recurrent neural network is constructed using data-driven multiple or single images, transformed into a low-dimensional feature matrix, processed with 3D pixel data, extracted perceptual features, and generated high-resolution images. The analysis of the results shows that the CD value of the used model is 0.477 and the EMD value is 0.579, which makes the constructed 3D images with more obvious detail levels and more accurate structural design, while the model of Pixel2Mesh focuses more on surface information, so the generated model is more realistic and closer to the real image.

Volumetric flame reconstructions in an optical engine cylinder involving refraction and blockage

Practical applications of computed tomography (CT) in optical engines require an advanced algorithm that can correct the light refraction via optical windows and reconstruct the 3D signal field partially blocked by structural obstacles. In this work, an advanced CT algorithm is designed for optical engines to simultaneously eliminate the imaging distortion by refraction and diminish the reconstruction errors using partial signal blocking. By combining the pinhole model and Snell’s law, the ray tracings from discretized 3D voxels in the measurement domain to 2D pixels in the imaging planes are accurately calculated, thus restoring the distortion in recorded projections. Besides, by deciding the locations and numbers of voxels that actually participate in iterative CT calculation, the iterative update process of voxel intensity becomes independent of the blocked rays, reducing the reconstruction errors. The algorithm is then numerically validated by reconstructing a simulated signal phantom inside an optical cylinder with a lightproof obstacle between the phantom and a recording camera, which imitates the refraction and blocking conditions in practical optical engines. Moreover, experimental demonstration is performed by reconstructing practical premixed flames inside optical engines. Both the simulation and the experiment present significantly enhanced flame chemiluminescence reconstruction by applying the optimized CT algorithm compared to the original algorithm utilized in open space applications.

AWafer-Scale Nanoporous 2D Active Pixel Image Sensor Matrix with High Uniformity, High Sensitivity, and Rapid Switching.

2D transition-metal dichalcogenides (TMDs) havebeen successfully developed as novel ubiquitous optoelectronics owing to their excellent electrical and optical characteristics. However, active-matrix image sensors based on TMDs have limitations owing to the difficulty of fabricating large-area integrated circuitry and achieving high optical sensitivity. Herein, a large-area uniform, highly sensitive, and robust image sensor matrix with active pixels consisting of nanoporous molybdenum disulfide (MoS2 ) phototransistors and indium-gallium-zinc oxide (IGZO) switching transistors is reported. Large-area uniform 4-inch wafer-scale bilayer MoS2 films are synthesized by radio-frequency (RF) magnetron sputtering and sulfurization processes and patterned to be a nanoporous structure consisting of an array of periodic nanopores on the MoS2 surface via block copolymer lithography. Edge exposure on the nanoporous bilayer MoS2 induces the formation of subgap states, which promotes a photogating effect to obtain an exceptionally high photoresponsivity of 5.2 × 104 A W-1 . A 4-inch-wafer-scale image mapping is successively achieved using this active-matrix image sensor by controlling the device sensing and switching states. The high-performance active-matrix image sensor is state-of-the-art in 2D material-based integrated circuitry and pixel image sensor applications.

Long-Term Visual Simultaneous Localization and Mapping: Using a Bayesian Persistence Filter-Based Global Map Prediction

With the rapidly growing demand for accurate localization in real-world environments, visual simultaneous localization and mapping (SLAM) has received significant attention in recent years. However, those existing methods still suffer from the degradation of localization accuracy in long-term changing environments. To address these problems, we propose a novel long-term SLAM system with map prediction and dynamics removal. First, a visual point-cloud matching algorithm is designed to efficiently fuse 2D pixel information and 3D voxel information. Second, each map point is classified into three types: static, semistatic, and dynamic based on the Bayesian persistence filter (BPF). Then we remove the dynamic map points to eliminate the influence of those map points. We can obtain a global predicted map by modeling the time series of semistatic map points. Finally, we incorporate the predicted global map into a state-of-the-art SLAM method, achieving an efficient visual SLAM system for long-term, dynamic environments. Extensive experiments are carried out on a wheelchair robot in an indoor environment over several months. The results demonstrate that our method has better map prediction accuracy and achieves more robust localization performance.

System tests of the ATLAS ITk planar and 3D pixel modules

In order to validate the design of the new all-silicon Inner Tracker (ITk) for ATLAS for the HL-LHC, a series of system tests has been performed, to assess the performance of prototype planar and 3D pixel modules arranged into serial power chains mounted onto realistic mechanical structures. In this paper, the prototype loaded local supports and test infrastructure is described and the key results presented.