Single Camera System Research Articles

A direct comparison of a next generation hyperspectral camera to state-of-the-art

Hyperspectral camera technology is advancing rapidly, and this paper seeks to compare a state-of-the-art industrial dual-camera setup to a single-camera system employing the latest chip technology (IMX990 from Sony). The hyperspectral cameras are compared over both the Visual and Short-Wave Infrared range (400–1700 nm) of the electromagnet spectrum. The spectral range and resolution, as well as spatial parameters and spectroscopic information are quantified with comparable optics, electronics, and test targets. Generally, enhanced spectral detail and reduced noise were observed for the single-camera compared to its peers. Thus, the IMX990 shows promising performance for the new generation of hyperspectral cameras directly relevant to industrial applications, such as detection, documentation, and sorting.

Stereo matching algorithm using deep learning and edge-preserving filter for machine vision

Machine vision research began with a single-camera system, but these systems had various limitations from having just one point-of-view of the environment and no depth information, therefore stereo cameras were invented. This paper proposes a hybrid method of a stereo matching algorithm with the goal of generating an accurate disparity map critical for applications such as 3D surface reconstruction and robot navigation to name a few. Convolutional neural network (CNN) is utilised to generate the matching cost, which is then input into cost aggregation to increase accuracy with the help of a bilateral filter (BF). Winner-take-all (WTA) is used to generate the preliminary disparity map. An edge-preserving filter (EPF) is applied to that output based on a transform that defines an isometry between curves on the 2D image manifold in 5D and the real line to eliminate these artefacts. The transform warps the input signal adaptively to allow linear 1D filtering. Due to the filter's resistance to high contrast and brightness, it is effective in refining and removing noise from the output image. Based on experimental research employing a Middlebury standard validation benchmark, this approach gives high accuracy with an average non-occluded error of 6.71% comparable to other published methods.

A Collaborative Visual Sensing System for Precise Quality Inspection at Manufacturing Lines

Visual sensing has been widely adopted for quality inspection in production processes. This paper presents the design and implementation of a smart collaborative camera system, called BubCam , for automated quality inspection of manufactured ink bags in Hewlett-Packard (HP) Inc.’s factories. Specifically, BubCam estimates the volume of air bubbles in an ink bag, which may affect the printing quality. The design of BubCam faces challenges due to the dynamic ambient light reflection, motion blur effect, and data labeling difficulty. As a starting point, we design a single-camera system which leverages various deep learning (DL)-based image segmentation and depth fusion techniques. New data labeling and training approaches are proposed to utilize prior knowledge of the production system for training the segmentation model with a small dataset. Then, we design a multi-camera system which additionally deploys multiple wireless cameras to achieve better accuracy due to multi-view sensing. To save power of the wireless cameras, we formulate a configuration adaptation problem and develop the single-agent and multi-agent deep reinforcement learning (DRL)-based solutions to adjust each wireless camera’s operation mode and frame rate in response to the changes of presence of air bubbles and light reflection. The multi-agent DRL approach aims to reduce the retraining costs during the production line reconfiguration process by only retraining the DRL agents for the newly added cameras and the existing cameras with changed positions. Extensive evaluation on a lab testbed and real factory trial shows that BubCam outperforms six baseline solutions including the current manual inspection and existing bubble detection and camera configuration adaptation approaches. In particular, BubCam achieves 1.3x accuracy improvement and 300x latency reduction, compared with the manual inspection approach.

An Image Processing Approach for Real-Time Safety Assessment of Autonomous Drone Delivery

The aim of producing self-driving drones has driven many researchers to automate various drone driving functions, such as take-off, navigation, and landing. However, despite the emergence of delivery as one of the most important uses of autonomous drones, there is still no automatic way to verify the safety of the delivery stage. One of the primary steps in the delivery operation is to ensure that the dropping zone is a safe area on arrival and during the dropping process. This paper proposes an image-processing-based classification approach for the delivery drone dropping process at a predefined destination. It employs live streaming via a single onboard camera and Global Positioning System (GPS) information. A two-stage processing procedure is proposed based on image segmentation and classification. Relevant parameters such as camera parameters, light parameters, dropping zone dimensions, and drone height from the ground are taken into account in the classification. The experimental results indicate that the proposed approach provides a fast method with reliable accuracy based on low-order calculations.

Reliability and Concurrent Validity of a Markerless, Single Camera, Portable 3D Motion Capture System for Assessment of Glenohumeral Mobility.

Recent technological advancements have enabled medical, sport, and fitness professionals to utilize digital tools that assist with conducting movement examinations and screenings. One such advancement has been the implementation of a single camera, markerless, and portable 3D motion capture system designed to obtain ROM measurements for multiple body parts simultaneously. However, the reliability and validity of a markerless 3D motion capture system that uses a single camera has not been established. The purpose of this study was to investigate the reliability and concurrent validity of this 3D motion capture system compared to a goniometer in assessing ROM of the glenohumeral joint. Quasi-experimental reliability, convenience sampling. Forty healthy volunteers (mean ± SD, age 27.4 ± 12.4 years, height 173.4 ± 11.1 cm, weight 72.1 ± 16.2 kg) participated in this study. Intrarater reliability was analyzed by ICC(2,k) with a 95% CI using two repeated trials for each shoulder movement (flexion, abduction, external rotation, internal rotation) that were recorded simultaneously via two methods: a standard goniometer and a 3D motion capture system. Concurrent validity was analyzed using Pearson Correlation Coefficient (r). The intrarater reliability between the two instruments for glenohumeral motions yielded an overall ICC of 0.82 (95% CI: 0.74-0.88) indicating good reliability for both instruments.The 3D motion capture system demonstrated strong correlations with goniometry for shoulder flexion (r = 0.67), abduction (r = 0.63), and external rotation (r = 0.76), and very strong correlation for shoulder internal rotation (r = 0.84). Results from this study indicated that a markerless, single camera, portable 3D motion capture system can be a reliable and valid tool to assess glenohumeral joint ROM in comparison to a standard goniometer. 3.

Fast and accurate flow measurement through dual-camera light field particle image velocimetry and ordered-subset algorithm

Light field particle image velocimetry (LF-PIV) can measure the three-dimensional (3D) flow field via a single perspective and hence is very attractive for applications with limited optical access. However, the flow velocity measurement via single-camera LF-PIV shows poor accuracy in the depth direction due to the particle reconstruction elongation effect. This study proposes a solution based on a dual-camera LF-PIV system along with an ordered-subset simultaneous algebraic reconstruction technique (OS-SART). The proposed system improves the spatial resolution in the depth direction and reduces the reconstruction elongation. The OS-SART also reduces the computational time brought by the dual-camera LF-PIV. Numerical reconstructions of the particle fields and Gaussian ring vortex field are first performed to evaluate the reconstruction accuracy and efficiency of the proposed system. Experiments on a circular jet flow are conducted to further validate the velocity measurement accuracy. Results indicate that the particle reconstruction elongation is reduced more than 10 times compared to the single-camera LF-PIV and the reconstruction efficiency is improved at least twice compared to the conventional SART. The accuracy is improved significantly for the ring vortex and 3D jet flow fields compared to the single-camera system. It is therefore demonstrated that the proposed system is capable of measuring the 3D flow field fast and accurately.

Simultaneous fluorescence imaging of bowel perfusion and ureter delineation using methylene blue: a demonstration in a porcine model

BackgroundIntraoperative near-infrared fluorescence imaging (NIRF) with preoperative optical dye administration is a promising technique for quick and easy intraoperative visualization of the ureter and for an improved, real-time assessment of intestinal perfusion. During colorectal surgery, there is a need for simultaneous non-invasive ureteral imaging and bowel perfusion assessment, using one single camera system. The purpose of this study is to investigate the feasibility of simultaneous intestinal perfusion and ureteral imaging using a single commercially available NIRF imaging system.MethodsSix Landrace pigs underwent laparotomy under general anesthesia in this experiment. An intravenous (IV) dose of 0.2 mg/kg indocyanine green (ICG) was given to assess bowel perfusion. Two pairs received a methylene blue (MB) iv injection of 0.75, 0.50 or 0.25 mg/kg respectively to investigate ureteral visualization. Quest Spectrum Fluorescence Camera (Quest Medical Imaging, Middenmeer, The Netherlands) was used for NIRF imaging.ResultsUreter visualization and bowel perfusion under NIRF imaging was achieved in all animals. All ureters were visible after five to ten minutes and remained clearly visible until the end of every experiment (120–420 min). A mixed model analysis did not show any significant differences neither between the three groups nor over time. Importantly, we demonstrated that bowel perfusion could be visualized with methylene blue (MB) as well. We observed no interference between ICG and MB and a faster washout of MB.ConclusionWe successfully demonstrated simultaneous fluorescence angiography with ICG and ureteral imaging with MB in the same surgical procedure, with the same commercially available NIRF imaging equipment. More importantly, we showed that the use MB is adequate for bowel perfusion assessment and ureter visualization with this NIRF imaging system. Besides, MB showed an earlier washout time, which can be clinical beneficial as a repeated dye injection may be necessary during a surgical procedure.

Detection of Empty/Occupied States of Parking Slots in Multicamera system using Mask R-CNN Classifier

A fast growth of vehicles in big cities has an impact of arising road loads and difficulty of finding empty parking spaces. One solution to cope with the problem is to develop a parking management system which can provide useful information of available parking spaces to the potential users. This paper discusses about a new multicamera arrangement and the function to evaluate the empty/occupied states of the parking slots, as an alternative solution to the existing single camera system, The system adopted Mask R-CNN for its classifier, because of its capability to provide the polygon outputs for its detected objects, compared with the existing bounding box outputs provided by other classifiers. The proposed function has optimized the available information from all cameras, by considering the relative position of each camera to the parking spaces, and also capable of overcoming occlusion problem occurs in some cameras, The experiment shows that the capability of overcoming the occlusion problem has been validated, and its performance to evaluate the empty/occupied states of the parking slots was better than the single camera system to a certain threshold.

Depth perception in single camera system using focus blur and aperture number

Depth perception in single camera system using focus blur and aperture number

VICO-DR: A Collaborative Virtual Dressing Room for Image Consulting

In recent years, extended reality has increasingly been used to enhance the shopping experience for customers. In particular, some virtual dressing room applications have begun to develop, as they allow customers to try on digital clothes and see how they fit. However, recent studies found that the presence of an AI or a real shopping assistant could improve the virtual dressing room experience. In response to this, we have developed a collaborative synchronous virtual dressing room for image consulting that allows customers to try on realistic digital garments chosen by a remotely connected human image consultant. The application has different features for the image consultant and the customer. The image consultant can connect to the application, define a database of garments, select different outfits with different sizes for the customer to try, and communicate with the customer through a single RGB camera system. The customer-side application can visualize the description of the outfit that the avatar is wearing, as well as the virtual shopping cart. The main purpose of the application is to offer an immersive experience, ensured by the presence of a realistic environment, an avatar that resembles the customer, a real-time physically-based cloth simulation algorithm, and a video-chat system.

Deep learning-enabled invalid-point removal for spatial phase unwrapping of 3D measurement

The single-camera system projecting single-frequency patterns is the ideal option among all proposed Fringe Projection Profilometry (FPP) systems in terms of measurement speed and system cost. This system necessitates a robust spatial phase unwrapping algorithm. However, robust spatial phase unwrapping remains a challenge in complex scenes. Quality-guided spatial phase unwrapping needs more efficient ways to identify unreliable points in phase maps before unwrapping. End-to-end deep learning spatial phase unwrapping face generality and interpretability problems. This paper proposes a hybrid spatial phase unwrapping method combining deep learning-enabled invalid-point removal and traditional path-following. This hybrid method demonstrates better robustness than traditional quality-guided methods, better interpretability than end-to-end deep learning schemes, and generality on unseen data. Experiments on the real dataset of multiple illumination conditions and multiple FPP systems differing in image resolution, fringe number, fringe direction, and optics wavelength verify the effectiveness of the proposed method.

Online position correction approach of an industrial robot by using a new photogrammetric measurement system

Six-axis industrial robots are widely used, mainly because of their versatility. However, they have one drawback: they are not as accurate as conventional machine tools. There are currently a number of approaches to increase the accuracy of these robots. This paper reports on the online correction of an ABB IRB 120 industrial robot using a novel photogrammetric measurement device, the Multi Aperture Positioning System (MAPS). As described by Luhmann, multi-camera solutions were previously required for the online measurement of the six degrees of freedom (6DOF = position and orientation in 3D space) of the robot TCP. MAPS offers the advantages of a single-camera system and at the same time the features and accuracy of a multi-camera system. To increase the accuracy of the robot, a correction algorithm is developed. It uses the measured values from MAPS to calculate the deviation of the robot Tool Center Point (TCP) position from the planned position and corrects it online if necessary. In this paper, three experiments are presented and their results discussed. Two to determine the position and trajectory accuracy of the robot and one to determine its z-stability on a trajectory in the xy plane. In each of these experiments, the robot end effector is tracked with the MAPS measurement device. After the actual state of the robot was recorded, all experiments were performed again, this time with the online correction. In all experiments, a significant increase in the accuracy of the robot can be observed. For example, the mean absolute error (MAE) of the position accuracy improved by a factor of 7 from 0.0207 to 0.0029 mm and the path accuracy (MAE) from 0.1140 to 0.0151 mm.

Three-dimensional shape and deformation measurement on complex structure parts

Stereo digital image correlation technique (stereo-DIC or 3D-DIC) has been widely used in three-dimensional (3D) shape and deformation measurement due to its high accuracy and flexibility. But it is a tough task for it to deal with complex structure components because of the severe perspective distortion in two views. This paper seeks to resolve this issue using a single-camera system based on DIC-assisted fringe projection profilometry (FPP). A pixel-wise and complete 3D geometry of complex structures can be reconstructed using the robust and efficient Gray-coded method based on a FPP system. And then, DIC is just used to perform the temporal matching and complete full-field pixel-to-pixel tracking. The in- and out-of-plane deformation are obtained at the same time by directly comparing the accurate and complete 3D data of each corresponding pixel. Speckle pattern design and fringe denoising methods are carefully compared and chosen to simultaneously guarantee the measuring accuracy of 3D shape and deformation. Experimental results demonstrate the proposed method is an effective means to achieve full-field 3D shape and deformation measurement on complex parts, such as honeycomb structure and braided composite tube, which are challenging and even impossible for the traditional stereo-DIC method.

Automated variety trial plot growth and flowering detection for maize and soybean using machine vision

Hundreds of crop variety trial sites are operated across Australia with up to 100 small plots (20 m2) that require manual, labour-intensive monitoring for emergence, height, canopy cover and flowering status. Machine vision systems can reduce labour in monitoring, and infield fixed cameras are most suitable to provide low-cost continuous sensing without requiring labour of travelling to sites. Height is commonly detected using stereo cameras; however, a low-cost single camera system is preferable for broad scale use at variety trial sites. Existing low-cost fixed camera systems assess multiple plots in the image’s field of view from a fixed mask that needs to be manually updated as the crop grows. Perspective transformation could be applied automatically to identify plot locations in the image. Existing systems also analyse multiple images independently and filter results to remove impacts from lighting variations. An alternative approach is to only analyse images in the same lighting conditions each day to reduce the need for filtering. In addition, a series of daily images can be used to track multiple leaf positions to monitor plant growth. A machine vision system has been developed that combines these technologies to track plant height from a series of daily images selected in the same lighting conditions, in combination with perspective transformation, in multiple plots in the camera’s field of view. Emergence, canopy cover and flowering status were also identified at the canopy surface in each plot using colour segmentation and/or shape analysis. The algorithms were evaluated on 5 and 10 m towers monitoring randomised trials of 16 maize and soybean plots and detected maize flowering date within one day, soybean height (RMSE = 18.38 cm; R2 = 0.880), maize height (RMSE = 47.73 cm; R2 = 0.838), soybean canopy cover (RMSE = 22.14%; R2 = 0.818) and maize canopy cover (RMSE = 14.01%; R2 = 0.750). The larger error in maize height detection was due to the flowers being tracked by the algorithm instead of vegetation. Further work is required to transfer the algorithms to other crops and varieties.

Deep Learning-Enabled Spatial Phase Unwrapping for 3d Measurement

In terms of 3D imaging speed and system cost, the single-camera system projecting single-frequency patterns is the ideal option among all proposed Fringe Projection Profilometry (FPP) systems. This system necessitates a robust spatial phase unwrapping (SPU) algorithm. However, robust SPU remains a challenge in complex scenes. Quality-guided SPU algorithms need more efficient ways to identify the unreliable points in phase maps before unwrapping. End-to-end deep learning SPU methods face generality and interpretability problems. This paper proposes a hybrid method combining deep learning and traditional path-following for robust SPU in FPP. This hybrid SPU scheme demonstrates better robustness than traditional quality-guided SPU methods, better interpretability than end-to-end deep learning scheme, and generality on unseen data. Experiments on the real dataset of multiple illumination conditions and multiple FPP systems differing in image resolution, the number of fringes, fringe direction, and optics wavelength verify the effectiveness of the proposed method.

Compressive imaging for thwarting adversarial attacks on 3D point cloud classifiers

Three dimensional (3D) point cloud classifiers are used extensively in safety crucial applications such as autonomous cars, face recognition, military applications, and many more. Despite the critical importance of their reliability, 3D classifiers are prone to adversarial attacks that can be crafted in the real world. While it is possible to use known methods to prevent adversarial attacks, they can be easily counter-attacked, leading to an arms race between the attacker and the defender. Here, we propose to use 3D compressive sensing to recover an original label of the 3D object. Since compressive sensing inherently encodes the 3D signal, it also prevents the arms race between the attacker and the defender. The 3D compressive sensing we consider is a single pixel camera (SPC) system that can be implemented in Light Detection and Ranging (LiDAR) systems.

Rotation Correction Method Using Depth-Value Symmetry of Human Skeletal Joints for Single RGB-D Camera System

Rotation Correction Method Using Depth-Value Symmetry of Human Skeletal Joints for Single RGB-D Camera System

In situ Estimation of Coral Recruitment Patterns From Shallow to Mesophotic Reefs Using an Optimized Fluorescence Imaging System

Coral recruitment represents a key element for coral reef persistence and resilience in the face of environmental disturbances. Studying coral recruitment patterns is fundamental for assessing reef health and implementing appropriate management strategies in an era of climate change. The FluorIS system has been developed to acquire high resolution, wide field-of-view (FOV) in situ images of coral recruits fluorescence and has proven successful in shallow reef environments. However, up to now, its applicability to mesophotic coral ecosystems remains unknown due to the complexity of the system and the limited time available when working at mesophotic depth. In this study we optimized the FluorIS system by utilizing a single infrared-converted camera instead of the bulkier regular dual-camera system, substantially reducing the system complexity and significantly decreasing the acquisition time to an average of 10 s for a set of 3 images. Moreover, the speed-FluorIS system is much more economical, decreasing the cost of the full set-up by roughly 40% compared to the original dual-camera system. We tested the utility of the speed-FluorIS by surveying coral recruits across shallow and mesophotic reefs of the Red Sea (Gulf of Eilat) and Bermuda, two of the most northerly reefs in the world with markedly different substrate and topography, and demonstrate that the modified system enables fast imaging of fluorescence to study coral recruitment patterns over a broader range of depths and reef topographies than previous fluorescence methods. Our single-camera system represents a valuable, non-invasive and rapid underwater tool which will help standardize surveys and long-term monitoring of coral recruits, contributing to our understanding of these vital and delicate early life stages of corals.

Defocus particle tracking: a comparison of methods based on model functions, cross-correlation, and neural networks

Defocus particle tracking (DPT) has gained increasing importance for its use to determine particle trajectories in all three dimensions with a single-camera system, as typical for a standard microscope, the workhorse of today’s ongoing biomedical revolution. DPT methods derive the depth coordinates of particle images from the different defocusing patterns that they show when observed in a volume much larger than the respective depth of field. Therefore it has become common for state-of-the-art methods to apply image recognition techniques. Two of the most commonly and widely used DPT approaches are the application of (astigmatism) particle image model functions (MF methods) and the normalized cross-correlations between measured particle images and reference templates (CC methods). Though still young in the field, the use of neural networks (NN methods) is expected to play a significant role in future and more complex defocus tracking applications. To assess the different strengths of such defocus tracking approaches, we present in this work a general and objective assessment of their performances when applied to synthetic and experimental images of different degrees of astigmatism, noise levels, and particle image overlapping. We show that MF methods work very well in low-concentration cases, while CC methods are more robust and provide better performance in cases of larger particle concentration and thus stronger particle image overlap. The tested NN methods generally showed the lowest performance, however, in comparison to the MF and CC methods, they are yet in an early stage and have still great potential to develop within the field of DPT.

Depth perception in single rgb camera system using lens aperture and object size: a geometrical approach for depth estimation

In recent years, with increase in concern about public safety and security, human movements or action sequences are highly valued when dealing with suspicious and criminal activities. In order to estimate the position and orientation related to human movements, depth information is needed. This is obtained by fusing data obtained from multiple cameras at different viewpoints. In practice, whenever occlusion occurs in a surveillance environment, there may be a pixel-to-pixel correspondence between two images captured from two cameras and, as a result, depth information may not be accurate. Moreover use of more than one camera exclusively adds burden to the surveillance infrastructure. In this study, we present a mathematical model for acquiring object depth information using single camera by capturing the in focused portion of an object from a single image. When camera is in-focus, with the reference to camera lens center, for a fixed focal length for each aperture setting, the object distance is varied. For each aperture reading, for the corresponding distance, the object distance (or depth) is estimated by relating the three parameters namely lens aperture radius, object distance and object size in image plane. The results show that the distance computed from the relationship approximates actual with a standard error estimate of 2.39 to 2.54, when tested on Nikon and Cannon versions with an accuracy of 98.1% at 95% confidence level.