Camera Platform Research Articles

Mozart: A Mobile ToF System for Sensing in the Dark Through Phase Manipulation

Sensing in low-light and dark environments has a wide range of applications. However, existing sensing technologies suffer several major challenges, such as excessive noise and low resolution. In this work, we propose Mozart - a new mobile sensing system that leverages off-the-shelf Time-of-Flight (ToF) depth cameras to generate high-resolution and rich-in-texture maps for applications in dark scenarios. The design of Mozart is based on our key observation that the phase components of ToF measurements can be manipulated to expose texture information. Through in-depth analysis of the physical reflection model, we show that the textures can be exposed and enhanced using highly compute-efficient phase manipulation functions. Moreover, by exploiting the physics texture models, we propose an autoencoderbased unsupervised learning approach that can automatically learn efficient representations from phase components to generate high-resolution maps. We implemented Mozart on several Android smartphone models and an edge testbed with standalone ToF camera platforms for various applications in the dark. The results show that Mozart can work in real time and delivers significant improvement over existing sensing technologies. The demo video at https:// www.youtube.com/watch?v=L_sxyTZxIdU shows that Mozart offers a low-cost, highperformance sensing technology for nextgeneration applications in the dark.

EHNet: Efficient Hybrid Network with Dual Attention for Image Deblurring

The motion of an object or camera platform makes the acquired image blurred. This degradation is a major reason to obtain a poor-quality image from an imaging sensor. Therefore, developing an efficient deep-learning-based image processing method to remove the blur artifact is desirable. Deep learning has recently demonstrated significant efficacy in image deblurring, primarily through convolutional neural networks (CNNs) and Transformers. However, the limited receptive fields of CNNs restrict their ability to capture long-range structural dependencies. In contrast, Transformers excel at modeling these dependencies, but they are computationally expensive for high-resolution inputs and lack the appropriate inductive bias. To overcome these challenges, we propose an Efficient Hybrid Network (EHNet) that employs CNN encoders for local feature extraction and Transformer decoders with a dual-attention module to capture spatial and channel-wise dependencies. This synergy facilitates the acquisition of rich contextual information for high-quality image deblurring. Additionally, we introduce the Simple Feature-Embedding Module (SFEM) to replace the pointwise and depthwise convolutions to generate simplified embedding features in the self-attention mechanism. This innovation substantially reduces computational complexity and memory usage while maintaining overall performance. Finally, through comprehensive experiments, our compact model yields promising quantitative and qualitative results for image deblurring on various benchmark datasets.

Estimating Leaf Area Index in Apple Orchard by UAV Multispectral Images with Spectral and Texture Information

The Leaf Area Index (LAI) strongly influences vegetation evapotranspiration and photosynthesis rates. Timely and accurately estimating the LAI is crucial for monitoring vegetation growth. The unmanned aerial vehicle (UAV) multispectral digital camera platform has been proven to be an effective tool for this purpose. Currently, most remote sensing estimations of LAIs focus on cereal crops, with limited research on economic crops such as apples. In this study, a method for estimating the LAI of an apple orchard by extracting spectral and texture information from UAV multispectral images was proposed. Specifically, field measurements were conducted to collect LAI data for 108 sample points during the final flowering (FF), fruit setting (FS), and fruit expansion (FE) stages of apple growth in 2023. Concurrently, UAV multispectral images were obtained to extract spectral and texture information (Gabor transform). The Support Vector Regression Recursive Feature Elimination (SVR-REF) was employed to select optimal features as inputs for constructing models to estimate the LAI. Finally, the optimal model was used for LAI mapping. The results indicate that integrating spectral and texture information effectively enhances the accuracy of LAI estimation, with the relative prediction deviation (RPD) for all models being greater than 2. The Categorical Boosting (CatBoost) model established for FF exhibits the highest accuracy, with a validation set R2, root mean square error (RMSE), and RPD of 0.867, 0.203, and 2.482, respectively. UAV multispectral imagery proves to be valuable in estimating apple orchard LAIs, offering real-time monitoring of apple growth and providing a scientific basis for orchard management.

Assessment of three-dimensional large-scale deformations during drying of hydrogels via a structure from motion photogrammetry

A low-cost, efficient and accurate structure from motion (SfM) photogrammetry is used to reconstruct the complex morphologies of unevenly deforming hydrogels during the drying process. The primary setups comprise a camera and rotary platform, which are low-cost and works stably for most drying conditions. The dynamic three-dimensional (3D) reconstruction of cylindrical hydrogel samples is achieved by SfM through the generation of point clouds, and the 3D morphologies are calculated by a Poisson reconstruction algorithm based on these point clouds. The accuracy of SfM in quantitatively measuring large-scale deformation is validated, and its applications in drying studies are explored. The results show that as drying proceeds, the maximum measurement deviation of the hydrogel volume between SfM and the traditional buoyancy force method is 1.93% at a volume shrinkage of 69%, confirming the accuracy of the current method for measuring large-scale drying shrinkage. Based on SfM, the temporal evolution of the cross section of the hydrogel is obtained. Comparing to conventional drying experiments that assess the dryness of deformed materials based on overall moisture content changes, SfM enables the evaluation of localized complete drying by quantitative analysis of deformation parameter variations.

Robotic‑assisted minimally invasive pancreas surgery. Review

The use robotics in surgery is gaining momentum. This approach holds substantial promise in pancreas surgery. Minimally invasive techniques have been increasingly used in oncologic surgery and pancreatic surgery in particular. Minimally invasive pancreatectomy (MIP) has been performed with increasing frequency in all pancreatic pathologies, including pancreatic neuroendocrine tumors (PNETs), including pancreaticoduodenectomy (PD), distal pancreatectomy (DP), and pancreatic enucleation. Robotic‑assisted surgery using the DaVinci platform has accelerated the adoption of MIP. The proportion of robotic DP between 2015 and 2016 was nearly 4 times greater than between 2010 and 2012 (16% vs 4%), with a similar increase in proportion of robotic PD (7% vs 2%) over the same time frame. Advantages of robotic surgery over laparoscopy include three‑dimensional visualization, increased degrees of motion with endo‑articulation, stable camera platform, surgeon ergonomics, and single surgeon’s ability to control four instruments. Systematic adoption of MIP has been facilitated by the creation of multiple multicenter training programs in robotic DP and PD, aiming to standardize the oncologic safety and technical performance of these operations. Robotic surgery for pancreatic lesions and malignancies has become well accepted and is expanding to more and more center annually. The number of centers using robotics in pancreatic surgery is rapidly increasing. The most studied robotic pancreas surgeries are PD and DP. Most studies are in their early phases, but they report that robotic pancreas surgery is safe feasible. Robotic pancreas surgery offers several advantages over open and laparoscopic techniques. Data regarding costs of robotics versus conventional techniques is still lacking. Robotic pancreas surgery is still in its early stages. It holds promise to become the new surgical standard for pancreatic resections in the future, however, more research is still needed to establish its safety, cost effectiveness and efficacy in providing the best outcomes.

Vulnerability analysis of super high-rise building security system based on Bayesian network and digital twin technology

Recent years have seen the increasing need to prevent illegal intrusions in super high-rise buildings. To improve the security system of super high-rise buildings by adding dynamic vulnerability assessments for illegal intrusion risks and rectification plan optimization capabilities, while also to increase its visualization level, a method utilizing digital twin technology and integrating Bayesian networks is proposed. To validate this method, a case study within a prominent super high-rise building in the Central Business District (CBD) of Beijing is conducted, in which the vulnerability analysis results have shown that among all the tertiary parent nodes, the coverage extent of the video surveillance system most significantly influences the vulnerability. By using the comprehensive blind spot analysis function of the established digital twin platform, thoughtful additions and angle adjustments of surveillance cameras have successfully increased the score of the security level from 84 to 90. This has proved that the method proposed in this study can provide decision support for the vulnerability assessment and improvement for the security system of super high-rise buildings.

Dynamic Multiple Object Segmentation with Spatio-Temporal Filtering.

This article primarily focuses on the localization and extraction of multiple moving objects in images taken from a moving camera platform, such as image sequences captured by drones. The positions of moving objects in the images are influenced by both the camera's motion and the movement of the objects themselves, while the background position in the images is related to the camera's motion. The main objective of this article was to extract all moving objects from the background in an image. We first constructed a motion feature space containing motion distance and direction, to map the trajectories of feature points. Subsequently, we employed a clustering algorithm based on trajectory distinctiveness to differentiate between moving objects and the background, as well as feature points corresponding to different moving objects. The pixels between the feature points were then designated as source points. Within local regions, complete moving objects were segmented by identifying these pixels. We validated the algorithm on some sequences in the Video Verification of Identity (VIVID) program database and compared it with relevant algorithms. The experimental results demonstrated that, in the test sequences when the feature point trajectories exceed 10 frames, there was a significant difference in the feature space between the feature points on the moving objects and those on the background. Correctly classified frames with feature points accounted for 67% of the total frames.The positions of the moving objects in the images were accurately localized, with an average IOU value of 0.76 and an average contour accuracy of 0.57. This indicated that our algorithm effectively localized and segmented the moving objects in images captured by moving cameras.

Based on an Overview of Crack Detection

This Crack detection, as an important part of concrete structural health monitoring, aims to reflect the stress state and damage degree of the structure. Traditional concrete crack detection mainly relies on manual visual identification, which has low detection efficiency and accuracy. In addition, manual visual detection has problems such as being greatly affected by lighting conditions, being unable to cover overhead locations such as bridge towers and high piers, and being highly subjective. In recent years, in order to solve the above problems, researchers at home and abroad have developed concrete crack detection equipment based on digital image technology, such as inspection vehicles equipped with high-resolution cameras, unmanned aerial vehicles, crawling robots, and so on. Meanwhile, efficient and accurate crack detection algorithms are the basis for realizing accurate crack identification. How to balance the detection speed and accuracy has been the focus of academic attention. This paper reviews the research progress of concrete crack detection equipment based on digital image technology at home and abroad in recent years, including camera platform, calibration method, preprocessing, traditional and deep learning algorithms, crack feature extraction, image stitching, and 3D representation and monitoring of cracks. In addition, the deficiencies in the research are summarized.

Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks.

Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.

Development of a multi-object tracking algorithm with untrained features of object matching

The problem of multiple object tracking is one of the most difficult tasks in computer vision. The article is devoted to a task of multiple object tracking on video footage received from an unmanned aerial vehicle. Unlike a static camera platform, the mobile platform causes an accidental camera movement, which leads to sudden changes in the position, angle and scale of objects. Such aspects considerably hinder efficient object tracking. In this paper, we explore the possibilities of improving the tracking quality in the case of camera movements. We significantly outperform ByteTrack algorithm, one of the best tracking algorithms for the MOT Challenge dataset, on the Visdrone 2019 dataset.

Camera-Based Short Physical Performance Battery and Timed Up and Go Assessment for Older Adults with Cancer.

This paper presents an automatic camera-based device to monitor and evaluate the gait speed, standing balance, and 5 Times Sit-Stand (5TSS) tests of the Short Physical Performance Battery (SPPB) and the Timed Up and Go (TUG) test. The proposed design measures and calculates the parameters of the SPPB tests automatically. The SPPB data can be used for physical performance assessment of older patients under cancer treatment. This stand-alone device has a Raspberry Pi (RPi) computer, three cameras, and two DC motors. The left and right cameras are used for gait speed tests. The center camera is used for standing balance, 5TSS, and TUG tests and for angle positioning of the camera platform toward the subject using DC motors by turning the camera left/right and tilting it up/down. The key algorithm for operating the proposed system is developed using Channel and Spatial Reliability Tracking in the cv2 module in Python. Graphical User Interfaces (GUIs) in the RPi are developed to run tests and adjust cameras, controlled remotely via smartphone and its Wi-Fi hotspot. We have tested the implemented camera setup prototype and extracted all SPPB and TUG parameters by conducting several experiments on a human subject population of 8 volunteers (male and female, light and dark complexions) in 69 test runs. The measured data and calculated outputs of the system consist of tests of gait speed (0.041 to 1.92 m/s with average accuracy of >95%), and standing balance, 5TSS, TUG, all with average time accuracy of >97%.

PlasPi TDM: Augmentation of a low-cost camera platform for advanced underwater physical-ecological observations

Marine ecosystem dynamics in the context of climate change is a growing scientific, political and social concern requiring regular monitoring through appropriate observational technologies and studies. Thus, a wide range of tools comprising chemical, biogeochemical, physical, and biological sensors, as well as other platforms exists for marine monitoring. However, their high acquisition and maintenance costs are often a major obstacle, especially in low-income developing countries. We designed an advanced low-cost synoptic marine ecosystem observation system that operates at relatively high temporal frequencies, named PlasPi TDM. This instrument is an improved version of the camera system (PlasPI marine cameras) developed in 2020 by Autun Purser from the Alfred Wegener Institute Helmholtz Center for Polar and Marine Research (Germany), and collaborators. It incorporates several innovative developments such as multispectral (records the spectrum of any object photographed), temperature and pressure sensors. The PlasPi TDM operates to a depth of 200 m. The various field deployments demonstrate the operational capability of the PlasPi TDM for different applications and illustrate its considerable potential for in-situ observations and marine surveillance in Africa. This device is intended as an open-source project and its continued development is encouraged for a more integrated, sustainable and low-cost ocean observing system.

Fish behavior in response to an approaching underwater camera

Responses of fish to camera survey gear can influence abundance estimates, and fish behaviors such as attraction or avoidance of an approaching camera platform can bias fish counts. In addition, artificial lighting is generally required for camera-based surveys in boreal systems at > 30 m depth. This study recorded the responses of fish and their apparent orientation as seen in camera images taken in two different areas in Alaska. Stereo-image methods were used to estimate fish movement, range from camera, and the movement of the camera platform itself by estimating the spatial relationship between fish and the seafloor. Change in the apparent position of feature points along the seafloor and the fish was used to derive fish and camera speed. A total of 670 individual fish across 19 near-bottom species were analyzed from a dataset of 3463 image frames from 112 separate camera deployments. Individual fish species were combined into groups and analyzed for reactions, orientation toward the camera, nearest approach to the camera, and swimming speed. Most fish did not react strongly to the camera. A number of distinct patterns in reactions were observed, with roundfish exhibiting the highest level of attraction to the camera and flatfish showing the highest number of strong avoidance reactions. Significant differences in approach distance and fish swimming speed across analysis groups, reactions and regions were found, with highest fish speeds observed with roundfish and closest approach of the camera observed with demersal rockfish and flatfish. The findings demonstrate distinct differences in reactions among groups and suggest that the potential attraction response of roundfish might lead to positive biases for density. Flatfish were reactive mostly at close ranges, and rockfishes exhibited neutral and moderate reactions, which may signal reasonable estimates of abundance for these groups using camera-based surveys.

An autonomous system design for mold loading on press brake machines using a camera platform, deep learning, and image processing

An autonomous system design for mold loading on press brake machines using a camera platform, deep learning, and image processing

An embarrassingly simple approach for visual navigation of forest environments.

Navigation in forest environments is a challenging and open problem in the area of field robotics. Rovers in forest environments are required to infer the traversability of a priori unknown terrains, comprising a number of different types of compliant and rigid obstacles, under varying lighting and weather conditions. The challenges are further compounded for inexpensive small-sized (portable) rovers. While such rovers may be useful for collaboratively monitoring large tracts of forests as a swarm, with low environmental impact, their small-size affords them only a low viewpoint of their proximal terrain. Moreover, their limited view may frequently be partially occluded by compliant obstacles in close proximity such as shrubs and tall grass. Perhaps, consequently, most studies on off-road navigation typically use large-sized rovers equipped with expensive exteroceptive navigation sensors. We design a low-cost navigation system tailored for small-sized forest rovers. For navigation, a light-weight convolution neural network is used to predict depth images from RGB input images from a low-viewpoint monocular camera. Subsequently, a simple coarse-grained navigation algorithm aggregates the predicted depth information to steer our mobile platform towards open traversable areas in the forest while avoiding obstacles. In this study, the steering commands output from our navigation algorithm direct an operator pushing the mobile platform. Our navigation algorithm has been extensively tested in high-fidelity forest simulations and in field trials. Using no more than a 16 × 16 pixel depth prediction image from a 32 × 32 pixel RGB image, our algorithm running on a Raspberry Pi was able to successfully navigate a total of over 750 m of real-world forest terrain comprising shrubs, dense bushes, tall grass, fallen branches, fallen tree trunks, small ditches and mounds, and standing trees, under five different weather conditions and four different times of day. Furthermore, our algorithm exhibits robustness to changes in the mobile platform's camera pitch angle, motion blur, low lighting at dusk, and high-contrast lighting conditions.

First Demonstration of Compton Camera Used for X-Ray Fluorescence Imaging.

X-ray fluorescence computed tomography (XFCT) is a promising approach used for obtaining the distribution of high-Z elements in the target object. The characteristic energy of X-ray fluorescence (XRF) photons makes XFCT have higher sensitivity and contrast ratio. Conventional XFCT systems usually require mechanical collimators, which leads to a huge loss of incident photons and reduce photon collection efficiency. The Compton camera is an imaging modality that uses electronic collimation to obtain the incident direction of the photons, which brings advantages in the detection area and X-ray photon collection efficiency. Compton cameras can also achieve three-dimensional (3D) imaging with one or several views of scanning without rotation. Therefore, it is a good idea to realize XRF imaging by using Compton cameras, which may provide more potential applications and imaging possibilities, such as handheld XRF imaging or image-guided interventional operation. In this work, we demonstrate the first Compton camera platform which is used for XRF imaging. The proposed X-ray fluorescence Compton camera (XFCC) mainly consists of a conventional X-ray tube (150kVp) and a Timepix3 photon-counting detector (PCD). A PMMA phantom with insertions containing different concentrations of 4%, 6%, 8%, and 10% (weight/volume) Gd solution is scanned by a fan-beam X-ray. Besides, a Compton camera-based X-ray fluorescence imaging reconstruction method (CCFIRM) is developed to solve specific problems in XFCC reconstruction. The reconstruction images and the quantitative analyses of the contrast-to-noise ratio (CNR) are presented. The results indicate that the detectability limit of the proposed XFCC platform is 3.5139 wt.% (CNR =4 ).

Design of Fruit-Carrying Monitoring System for Monorail Transporter in Mountain Orchard

The real-time monitoring and detection of the fruit carrying for monorail transporter in the mountain orchard are significant for the transporter scheduling and safety. In this paper, we present a fruit carrying monitoring system, including the pan-tilt camera platform, AI edge computing platform, improved detection algorithm and the web client. The system used a pan-tilt camera to capture images of the truck body of the monorail transporter, realizing monitoring of fruit carrying. Besides, we present an improved fruit carrying detection algorithm based on YOLOv5s, taking the “basket”, “orange” and “fullbasket” as the object. We introduced the improved attention mechanism E-CBAM (Efficient-Convolutional Block Attention Module) based on CBAM, into the C3 module in the neck network of YOLOv5s. Focal loss was introduced to improve the classification and confidence loss to improve detection accuracy; to deploy the model on the embedded platform better, we compressed the model through the EagleEye pruning algorithm to reduce the parameters and improve the detection speed. The experiment was performed on the custom fruit-carrying datasets, the mAP was 91.5%, which was 9.6%, 9.9% and 12.0% higher than that of Faster-RCNN, RetinaNet-Res50 and YOLOv3-tiny, respectively, and detection speed at Jetson Nano was 72[Formula: see text]ms/img. The monitoring system and detection algorithm proposed in the paper can provide technical support for the safe transportation of monorail transporter and scheduling transportation equipment more efficiently.

Time-resolved determination of velocity fields on the example of a simple water channel flow, using a real-time capable FPGA-based camera platform

Field Programmable Gate Arrays (FPGAs) have become a widely available computational resource for real-time image processing and can be used in time-critical optical process measurements. In this study, we investigate the applicability of the spatial filtering technique for time-resolved velocity field determination, on the example of a simple water tunnel flow. For this, the spatial filtering technique was implemented for a custom FPGA platform and analyzed regarding the effect of spatial and temporal resolution on the accuracy of the results. It was possible to prove the algorithm as suitable for velocity field estimation and identify algorithm parameters, which allow for increasing the accuracy of the results.Graphical abstract

Continuous organ perfusion monitoring using indocyanine green in a piglet model.

Unrecognized organ hypoperfusion may cause major postoperative complications with detrimental effects for the patient. The use of Indocyanine Green (ICG) to detect organ hypoperfusion is emerging but the optimal methodology is still uncertain. The purpose of this study was to determine the feasibility of real-time continuous quantitative perfusion assessment with Indocyanine Green (ICG) to monitor organ perfusion during minimally invasive surgery using a novel ICG dosing regimen and quantification software. In this experimental porcine study, twelve subjects were administered a priming dose of ICG, followed by a regimen of high-frequency (1 dose per minute), low-dose bolus injections with weight-adjusted (0.008mg/kg) ICG allowing for continuous perfusion monitoring. In each pig, one randomly assigned organ of interest [stomach (n = 3), ascending colon (n = 3), rectum (n = 3) and spleen (n = 3)] was investigated with varying camera conditions. Video recording was performed with the 1588 AIM Stryker camera platform and subsequent quantitative analysis of the ICG signal were performed using a research version of a commercially available surgical real-time analysis software. Using a high-frequency, low-dose bolus ICG regimen, fluorescence visualization and quantification in abdominal organs were successful in the stomach (3/3), ascending colon (1/3), rectum (2/3), and the spleen (3/3). ICG accumulation in the tissue over time did not affect the quantification process. Considerable variation in fluorescence signal was observed between organs and between the same organ in different subjects. Of the different camera conditions investigated, the highest signal was achieved when the camera was placed 7.5cm from the target organ. This proof-of-concept study finds that real-time continuous perfusion monitoring in different abdominal organs using ICG is feasible. However, the study also finds a large variation in fluorescence intensity between organs and between the same organ in different subjects while using a fixed weight-adjusted dosing regimen using the same camera setting and placement.

Minimum Cable Tensions and Tension Sensitivity for Long-Span Cable-Driven Camera Robots with Applications to Stability Analysis

Employing cables with strong flexibility and unidirectional restraints to operate a camera platform leads to stability issues for a camera robot with long-span cables considering the cable mass. Cable tensions, which are the constraints for the camera platform, have a critical influence on the stability of the robot. Consequently, this paper focuses on two special problems of minimum cable tension distributions (MCTDs) within the workspace and the cable tension sensitivity analysis (CTSA) for a camera robot by taking the cable mass into account, which can be used to investigate the stability of the robot. Firstly, three minimum cable tension distribution indices (MCTDIs) were proposed for the camera robot. An important matter is that the three proposed MCTDIs, which represent the weakest constraints for the camera platform, can be employed for investigating the stability of the robot. In addition, a specified minimum cable tension workspace (SMCTW) is introduced, where the minimum cable tension when the camera platform is located at arbitrary position meets the given requirement. Secondly, the CTSA model and cable tension sensitivity analysis index (CTSAI) for the camera robot were proposed with grey relational analysis method, in which the influence mechanism and influence degree of the positions of the camera platform relative to cable tensions was investigated in detail. Lastly, the reasonableness of the presented MCTDIs and the method for the CTSA with applications in the stability analysis of the camera robot were supported by performing some simulation studies.