Standard Camera Research Articles

LiDAR-360 RGB Camera-360 Thermal Camera Targetless Calibration for Dynamic Situations

Integrating multiple types of sensors into autonomous systems, such as cars and robots, has become a widely adopted approach in modern technology. Among these sensors, RGB cameras, thermal cameras, and LiDAR are particularly valued for their ability to provide comprehensive environmental data. However, despite their advantages, current research primarily focuses on the one or combination of two sensors at a time. The full potential of utilizing all three sensors is often neglected. One key challenge is the ego-motion compensation of data in dynamic situations, which results from the rotational nature of the LiDAR sensor, and the blind spots of standard cameras due to their limited field of view. To resolve this problem, this paper proposes a novel method for the simultaneous registration of LiDAR, panoramic RGB cameras, and panoramic thermal cameras in dynamic environments without the need for calibration targets. Initially, essential features from RGB images, thermal data, and LiDAR point clouds are extracted through a novel method, designed to capture significant raw data characteristics. These extracted features then serve as a foundation for ego-motion compensation, optimizing the initial dataset. Subsequently, the raw features can be further refined to enhance calibration accuracy, achieving more precise alignment results. The results of the paper demonstrate the effectiveness of this approach in enhancing multiple sensor calibration compared to other ways. In the case of a high speed of around 9 m/s, some situations can improve the accuracy about 30 percent higher for LiDAR and Camera calibration. The proposed method has the potential to significantly improve the reliability and accuracy of autonomous systems in real-world scenarios, particularly under challenging environmental conditions.

AI-driven canine cataract detection: a machine learning approach using support vector machine

ABSTRACT The pivotal role of the Support Vector Machine becomes evident in addressing the impact of aging and genetics on canine cataract development. Timely detection, crucial for preventing illness in dogs, is facilitated by Artificial Intelligence. This proves especially beneficial for veterinarians grappling with detection challenges. Within the realm of Artificial Intelligence, the subset of Machine Learning emerges as a promising avenue for tackling intricate tasks. The current study introduces a model for grading canine cataracts through images captured on standard mobile phone cameras. Notably, the proposed system centers around harnessing the power of the Support Vector Machine, a potent Machine Learning algorithm tailored for classifying (grading) cataracts based on images. Rigorous evaluation, employing K-fold Cross-validation, validates the images’ reliability. Both binary-class and multi-class experimentation contribute to the dataset. Impressively, the accuracy rates for the trained classification model stand at 83% (without Cross-validation) and 81% (with Cross-validation for K = 10), underscoring the robustness of the approach.

A geometric model with stochastic error for abnormal motion detection of portal crane bucket grab

Abnormal swing angle detection of bucket grabs is crucial for efficient harbor operations. In this study, we develop a practically convenient swing angle detection method for crane operation, requiring only a single standard surveillance camera at the fly-jib head, without the need for sophisticated sensors or markers on the payload. Specifically, our algorithm takes the video images from the camera as input. Next, a fine-tuned ‘the fifth version of the You Only Look Once algorithm’ (YOLOv5) model is used to automatically detect the position of the bucket grab on the image plane. Subsequently, a novel geometric model is constructed, which takes the pixel position of the bucket grab, the steel rope length provided by the Programmable Logic Controller (PLC) system, and the optical lens information of the camera into consideration. The key parameters of this geometric model are statistically estimated by a novel iterative algorithm. Once the key parameters are estimated, the algorithm can automatically detect swing angles from video streams. Being analytically simple, the computation of our algorithm is fast, as it takes about 0.01 s to process one single image generated by the surveillance camera. Therefore, we are able to obtain an accurate and fast estimation of the swing angle of an operating crane in real-time applications. Simulation studies are conducted to validate the model and algorithm. Real video examples from Qingdao Seaport under various weather conditions are analyzed to demonstrate its practical performance.

Development of a method to use standard hospital gamma cameras as triage whole body monitors in UK emergencies

This paper outlines the process by which a medical gamma camera can be utilised to support assessment of internal radionuclides for the public. While hospital based gamma cameras are able to detect photopeaks, they are often limited to an energy range of 40-540 keV. However, radionuclides with photopeak energies above 540 keV can still be detected as the partial collection of photon energy increases the count rate at lower energies. By combining extensive mathematical modelling with empirical calibration of multiple gamma cameras it is possible to develop a linear correlation between the efficiency of counting point sources and the overall counting efficiency for the camera. Once established, a simple protocol can be used to characterise any gamma camera, using optimal system settings, and hence generate a system efficiency with sufficient accuracy to allow the camera to be used in a triage process to committed effective doses of 2 mSv.&#xD.

Near-field terahertz electro-optical imaging based on a polarization image sensor

Abstract This paper presents a hyperspectral microscopy system that offers two-dimensional (2D) measurement of the spectral phase and amplitude information of terahertz (THz) radiation without the need for raster scanning. To achieve this, a new THz imaging method is introduced, wherein the distribution of the THz electric field is spatially measured using the electro-optic effect with a commercial polarization image sensor. This method enables the direct measurement of polarization components, eliminating the need for the polarization optics usually required in conventional electro-optical imaging. The performance of this imaging method is compared with a conventional 2D imaging system based on a standard visible camera. Finally, the sub-wavelength resolution capabilities of this new sensor are demonstrated by imaging a sample in the near field.

FBG Interrogator Using a Dispersive Waveguide Chip and a CMOS Camera.

Optical sensors using fiber Bragg gratings (FBGs) have become an alternative to traditional electronic sensors thanks to their immunity against electromagnetic interference, their applicability in harsh environments, and other advantages. However, the complexity and high cost of the FBG interrogation systems pose a challenge for the wide deployment of such sensors. Herein, we present a clean and cost-effective method for interrogating an FBG temperature sensor using a micro-chip called the waveguide spectral lens (WSL) and a standard CMOS camera. This interrogation system can project the FBG transmission spectrum onto the camera without any free-space optical components. Based on this system, an FBG temperature sensor is developed, and the results show good agreement with a commercial optical spectrum analyzer (OSA), with the respective wavelength-temperature sensitivity measured as 6.33 pm/°C for the WSL camera system and 6.32 pm/°C for the commercial OSA. Direct data processing on the WSL camera system translates this sensitivity to 0.44 μm/°C in relation to the absolute spatial shift of the FBG spectra on the camera. Furthermore, a deep neural network is developed to train the spectral dataset, achieving a temperature resolution of 0.1 °C from 60 °C to 120 °C, while direct processing on the valley/dark line detection yields a resolution of 7.84 °C. The proposed hardware and the data processing method may lead to the development of a compact, practical, and low-cost FBG interrogator.

Validity of Valor Inertial Measurement Unit for Upper and Lower Extremity Joint Angles.

Inertial measurement units (IMU) are increasingly utilized to capture biomechanical measures such as joint kinematics outside traditional biomechanics laboratories. These wearable sensors have been proven to help clinicians and engineers monitor rehabilitation progress, improve prosthesis development, and record human performance in a variety of settings. The Valor IMU aims to offer a portable motion capture alternative to provide reliable and accurate joint kinematics when compared to industry gold standard optical motion capture cameras. However, IMUs can have disturbances in their measurements caused by magnetic fields, drift, and inappropriate calibration routines. Therefore, the purpose of this investigation is to validate the joint angles captured by the Valor IMU in comparison to an optical motion capture system across a variety of movements. Our findings showed mean absolute differences between Valor IMU and Vicon motion capture across all subjects' joint angles. The tasks ranged from 1.81 degrees to 17.46 degrees, the root mean squared errors ranged from 1.89 degrees to 16.62 degrees, and interclass correlation coefficient agreements ranged from 0.57 to 0.99. The results in the current paper further promote the usage of the IMU system outside traditional biomechanical laboratories. Future examinations of this IMU should include smaller, modular IMUs with non-slip Velcro bands and further validation regarding transverse plane joint kinematics such as joint internal/external rotations.

ChromaFlash: Snapshot Hyperspectral Imaging Using Rolling Shutter Cameras

Hyperspectral imaging captures scene information across narrow, contiguous bands of the electromagnetic spectrum. Despite its proven utility in industrial and biomedical applications, its ubiquity has been limited by bulky form factors, slow capture times, and prohibitive costs. In this work, we propose a generalized approach to snapshot hyperspectral imaging that only requires a standard rolling shutter camera and wavelength-adjustable lighting. The crux of this approach entails using the rolling shutter as a spatiotemporal mask, varying incoming light quicker than the camera's frame rate in order for the captured image to contain rows of pixels illuminated at different wavelengths. An image reconstruction pipeline then converts this coded image into a complete hyperspectral image using sparse optimization. We demonstrate the feasibility of this approach by deploying a low-cost system called ChromaFlash, which uses a smartphone's camera for image acquisition and a series of LEDs to change the scene's illumination. We evaluated ChromaFlash through simulations on two public hyperspectral datasets and assessed its spatial and spectral accuracy across various system parameters. We also tested the real-world performance of our prototype by capturing diverse scenes under varied ambient lighting conditions. In both experiments, ChromaFlash outperformed state-of-the-art techniques that use deep learning to convert RGB images into hyperspectral ones, achieving snapshot performance not demonstrated by prior attempts at accessible hyperspectral imaging.

An automated lightweight approach for detecting dead fish in a recirculating aquaculture system

Accurate methods for detecting dead fish are of great significance for fisheries because of their potential to improve production and reduce water pollution. Small-scale fisheries typically apply manual observation, but this approach is labor-intensive and time-consuming, which limits its application on large-scale farms. Advances in computer vision tools have provided the foundation for revolutionary methods of automated identification of individual-animal behavior. However, computer vision systems often suffer from unstable accuracy, low efficiency, and limited detection capability. Thus, the aim of this study was to design a deep learning system, designated “Deadfish-YOLO”, to detect dead fish based solely on standard underwater camera images with no additional hardware. First, eight selected rearing tanks were monitored by a self-developed computer version system. From these tank images, a dataset containing 18,114 frames was built by manual labeling. Next, a lightweight backbone network was generated with YOLOv4 to ensure fast computations, and an attention mechanism was introduced into the model to suppress unimportant features. Finally, the ReLU-memristor-like activation function was adopted to improve neural-network performance. The accuracy and processing speed of Deadfish-YOLO were superior to those of other state-of-the-art single- and two-stage detection models; Deadfish-YOLO ran at 85 frames per second with a mean average precision of 0.946 and an average ratio of 0.924 between intersection and union. These results demonstrate that Deadfish-YOLO may be used to automatically monitor dead fish in real circulating aquaculture systems. In addition, the results of this study should facilitate the wider application of artificial-intelligence-based animal monitoring in aquaculture.

A Hybrid Neuromorphic Object Tracking and Classification Framework for Real-Time Systems.

Deep learning inference that needs to largely take place on the "edge" is a highly computational and memory intensive workload, making it intractable for low-power, embedded platforms such as mobile nodes and remote security applications. To address this challenge, this article proposes a real-time, hybrid neuromorphic framework for object tracking and classification using event-based cameras that possess desirable properties such as low-power consumption (5-14 mW) and high dynamic range (120 dB). Nonetheless, unlike traditional approaches of using event-by-event processing, this work uses a mixed frame and event approach to get energy savings with high performance. Using a frame-based region proposal method based on the density of foreground events, a hardware-friendly object tracking scheme is implemented using the apparent object velocity while tackling occlusion scenarios. The frame-based object track input is converted back to spikes for TrueNorth (TN) classification via the energy-efficient deep network (EEDN) pipeline. Using originally collected datasets, we train the TN model on the hardware track outputs, instead of using ground truth object locations as commonly done, and demonstrate the ability of our system to handle practical surveillance scenarios. As an alternative tracker paradigm, we also propose a continuous-time tracker with C++ implementation where each event is processed individually, which better exploits the low latency and asynchronous nature of neuromorphic vision sensors. Subsequently, we extensively compare the proposed methodologies to state-of-the-art event-based and frame-based methods for object tracking and classification, and demonstrate the use case of our neuromorphic approach for real-time and embedded applications without sacrificing performance. Finally, we also showcase the efficacy of the proposed neuromorphic system to a standard RGB camera setup when simultaneously evaluated over several hours of traffic recordings.

Comparison of the perceived image quality of intraoral orthodontic photographs taken with DSLR camera and mobile phone camera: A double-blinded prospective study

Clinical orthodontic photography is a vital skill that every orthodontist should master to record the patients’ details and to permit the orthodontist to carefully plan, monitor and execute the treatment. With the advancement of technology, some clinicians opt to take intraoral photographs with their mobile phone rather than DSLR camera. Hence, this study aimed to answer one main question: whether there was any significant difference in the perceived quality image between intraoral photos taken with a mobile phone and a standard DSLR camera. The cameras used were a DSLR (Nikon D300s with AF-S Micro NIKKOR 105mm lens and NIKON R1C1 Twin Flash) and Mobile Phone (Apple I-Phone 11 with Selfie Ring Light). Assessment of 20 sets of intraoral photographs (100 individual images) by five IIUM orthodontists using a perceived quality Likert scale of Zero (0) to Ten (10). The assessors and the lead investigator were blinded to the source of the photographs. Reliability was evaluated using a test-retest method on 4 sets of intraoral photographs (20 individual images), a few weeks after their initial assessment. There was no significant difference (p=0.35) in perceived quality of intraoral photographs taken between DSLR and mobile phone, with the mean value of 7.34 and 7.12 respectively. Reliability was good (ICC=0.549). This prospective study showed that there was no statistical difference between the perceived quality of intraoral orthodontic photographs taken with a DSLR camera and a Mobile Phone camera.

Dental cavity analysis, prediction, localization, and quantification using computer vision

Dental health assessment is a critical component of overall well-being, and advancements in computer vision and deep learning have opened new avenues for automating and enhancing this process. In this study, we present a comprehensive approach to dental cavity analysis, spanning localization, quantification, and visualization. Our methodology leveraged a diverse dataset of colored dental images that had been meticulously augmented and annotated. The You Only Look Once model was employed for precise dental cavity localization, providing bounding box predictions. Remarkably, these results were obtained based on images from standard device cameras. Subsequently, we introduced the use of the segment anything model segmentation model, known for its zero-shot generalization capabilities, to focus on the exact areas of dental cavities. This approach enhanced the granularity of our analysis, providing dental professionals with detailed visualizations for precise diagnosis. During the quantification phase, we extracted cavity areas from bounding box coordinates, enabling accurate measurement of cavity sizes. The model achieved a notable mean average precision of 0.732, an accuracy of 0.789, and a recall of 0.701. Moreover, the model converged quickly, with most metrics achieving near-optimal results after 100 iterations. This quantitative data augments traditional diagnosis methods, facilitating more informed treatment decisions.

Analysis of the ice front propagation in heart tissues: First results

Experimental tests were conducted simultaneously using an infrared (IR) camera and a standard camera in the visible range (VR) to detect the ice front propagation as a function of time and space in different sections of pig heart laid on the surface of a cold spherical aluminum source (30 mm in diameter). The goal of the experiments was to simulate the conditions during the medical practice of internal cryotherapy, which aims to eliminate the tissues responsible for atrial fibrillation in the human heart. Freezing temperatures and related times were also measured, although they were affected by several uncertainties associated with the tests, such as air trapping between the metal sphere and flesh, inhomogeneity of the flesh samples, and the low spatial resolution and accuracy of the IR camera. A comparison with the results of previously conducted numerical simulations (Potenza et al., 2023) [1] is also discussed.

Interference-enhanced micro-vision-based single-shot imaging of five degrees-of-freedom error motions for ultra-precision rotary axes

The measurement of five degrees-of-freedom (5-DOF) error motions, including radial, axial, and tilt motions, is crucial for ultra-precision rotary axes, which are key components of ultra-precision machine tools and instrumentation. In this study, we propose an interference-enhanced micro-vision technique to concurrently derive the 5-DOF error motions from a single-shot two-dimensional image, which was captured by a standard industrial camera equipped with an interference objective lens. By consolidating the essential features into a single optical path, the interference-enhanced micro-vision technique ingeniously merges machine micro-vision and modified white-light interference to detect in-plane and out-of-plane motions. Numerical simulations demonstrated, the basic principle for deriving the 5-DOF error motions, and the magnification of objective lens had inconsistent effects on the measurement accuracy for the radial and tilt motions, i.e. higher magnification led to higher radial accuracy but lower tilt accuracy. As practical application, the error motion detection capability was demonstrated by simultaneously measuring the 5-DOF synchronous and asynchronous error motions for a typical air bearing spindle at rotation speeds of 8.33, 108.33, and 308.33 rpm. The synchronous errors were nearly identical at various spindle speeds. However, because of system dynamics, increased vibrations were observed to be superimposed on the basic tilt error motions as the spindle speeds increased, which were verified by the vibration marks imprinted on the turned surfaces. For the 5-DOF motion measurements, the least-square fitting using large-volume edge and greyscale data of the captured image enabled super-high resolutions, despite using a camera with a relatively large pixel size and low bit depth. These results demonstrate that the proposed interference-enhanced micro-vision technique is a simple and effective tool for measuring spatial error motions in ultra-precision rotary axes.

Novel plenoptic Imager for 3D Particle Tracking Velocimetry for Fluid Structure Interactions

We show a novel plenoptic camera architecture and demonstrate its ability to perform three-dimensional three-component velocimetry using standard multi camera processing techniques. The novel architecture needs only a custom lens assembly with no modification to the camera body. This allows the use of any camera with an appropriate sensor size. The architecture directly creates multiple views of a scene side by side on the camera sensor which can be separated and treated as if they originated from independent cameras. Standard calibration techniques for creating 3D to 2D correspondence on the images can then be implemented to reconstruct 3D scene information. For this case we focus on performing 3D PTV using the commercial software DaVis and the "shake-the-box" algorithm. We look at the flow induced by an oscillating rod and compare the results with the analytical solution to validate the results.

Camera traps equipped with macro lenses as a tool for monitoring arboreal small mammals: a case study in an agroecosystem (NE Italy)

Despite their increasing use, camera traps as a monitoring tool for arboreal small mammals leave room for further improvements to increase their effectiveness. In the summer of 2023, we conducted a small mammal survey in a wooded area of a lowland agroecosystem in the Veneto region, using standard camera traps equipped with macro lenses for close-up shooting. This camera trap technique made it possible to contact three species of small mammals in the tree-shrub layer: Eurasian red squirrel Sciurus vulgaris, wood mouse Apodemus sylvaticus, and black rat Rattus rattus. The use of macro lenses combined with the standard camera trapping technique made it possible to obtain better quality images and more information even on smaller species compared to more traditional camera traps.

Georectifying drone image data over water surfaces without fixed ground control: Methodology, uncertainty assessment and application over an estuarine environment

Light-weight consumer-grade drones have the potential to provide geospatial image data to study a broad range of oceanic processes. However, rigorously tested methodologies to effectively and accurately geolocate and rectify these image data over mobile and dynamic water surfaces, where temporally fixed points of reference are unlikely to exist, are limited. We present a simple to use automated workflow for georectifying individual aerial images using position and orientation data from the drone's on-board sensor (i.e. direct-georectification). The presented methodology includes correcting for camera lens distortion and viewing angle and exploits standard mathematics and camera data processing techniques. The method is used to georectify image datasets from test flights with different combinations of altitude and camera angle. Using a test site over land, directly-georectified images, as well as the same images georectified using standard photogrammetry software, are evaluated using a network of known ground control points. The novel methodology performs well with the camera at nadir (both 10 m and 25 m above ground level) and exhibits a mean spatial accuracy of ±1 m. The same accuracy is achieved when the camera angle is 30° at 10 m above ground level but decreases to ±2.9 m at 30° and 25 m. The accuracy changes because the uncertainties are a function of the altitude and angle of the camera versus the ground. Drone in-flight positioning errors can reduce the accuracy further to ±5 m with the camera at 30° and 25 m. An ensemble approach is used to map the uncertainties within the camera field-of-view to show how they change with viewing distance and drone position and orientation. The complete approach is demonstrated over an estuarine environment that includes the shoreline and open water, producing results consistent with the land-based field-tests of accuracy. Overall, the workflow presented here provides a low cost and agile solution for direct-georectification of drone-captured image data over water surfaces. This approach could be used for collecting and processing image data from drones or ship-mounted cameras to provide observations of ocean colour, sea-ice, ocean glitter, sea surface roughness, white-cap coverage, coastal water quality, and river plumes. The Python scripts for the complete image georectification workflow, including uncertainty map generation, are available from https://github.com/JamieLab/SArONG.

Compact multispectral light field camera based on an inkjet-printed microlens array and color filter array.

With emerging advanced optical sensing technologies and their wide-ranging applications, gathering comprehensive optical data from real-world scenes is becoming increasingly crucial for their accurate reconstruction and analysis. In order to capture both three-dimensional (3D) spatial and spectral information from a scene, multiple devices or time-intensive scanning processes are often involved. Here, we demonstrate a multispectral light field camera that allows for the simultaneous acquisition of 3D information and spectral data in a single snapshot. By utilizing inkjet printing as the fabrication technology, the miniaturized optical components in the camera were manufactured with high precision and can be integrated into a standard camera housing. Furthermore, the microlens arrays and the color filter arrays were fabricated on the same substrate, and a precise alignment between the two arrays was achieved. The compact multispectral camera opens the door to a multitude of possibilities for mobile applications, ranging from autonomous driving and consumer electronics such as smartphones to medical technology such as endoscopes.

Self-illuminated third-harmonic image upconversion.

We demonstrate third-harmonic upconversion imaging to visualize real time infrared images in the visible with standard CCD or CMOS silicon-based cameras operating at room temperature. Different from the usual sum-frequency mixing image upconversion, our third-harmonic image upconversion approach does not require an auxiliary IR source to illuminate a target. The upconversion system uses a passively Q-switched Nd3+:YVO4 laser operating at 1342 nm with two intracavity KTP crystals, obtaining a 447 nm upconverted image by a cascaded process where an upconverted image at 671 nm is also obtained as an intermediate step.

Microsaccade-inspired event camera for robotics.

Neuromorphic vision sensors or event cameras have made the visual perception of extremely low reaction time possible, opening new avenues for high-dynamic robotics applications. These event cameras' output is dependent on both motion and texture. However, the event camera fails to capture object edges that are parallel to the camera motion. This is a problem intrinsic to the sensor and therefore challenging to solve algorithmically. Human vision deals with perceptual fading using the active mechanism of small involuntary eye movements, the most prominent ones called microsaccades. By moving the eyes constantly and slightly during fixation, microsaccades can substantially maintain texture stability and persistence. Inspired by microsaccades, we designed an event-based perception system capable of simultaneously maintaining low reaction time and stable texture. In this design, a rotating wedge prism was mounted in front of the aperture of an event camera to redirect light and trigger events. The geometrical optics of the rotating wedge prism allows for algorithmic compensation of the additional rotational motion, resulting in a stable texture appearance and high informational output independent of external motion. The hardware device and software solution are integrated into a system, which we call artificial microsaccade-enhanced event camera (AMI-EV). Benchmark comparisons validated the superior data quality of AMI-EV recordings in scenarios where both standard cameras and event cameras fail to deliver. Various real-world experiments demonstrated the potential of the system to facilitate robotics perception both for low-level and high-level vision tasks.