Camera Hardware Research Articles

Image super-resolution based on improved ESRGAN and its application in camera calibration

Camera calibration is a key technique in the field of computer vision. The Zhang’s method is currently the most commonly used camera calibration method, but its accuracy is mainly dependent on the image quality, thusly constrained by camera hardware conditions. Considering that high-performance cameras are often expensive, a cost-effective method to improve the quality of calibration images is of great practical value. Without changing the existing hardware conditions, image super-resolution can be considered to enhance the quality of the calibration image. Image super-resolution (SR) is the process of reconstructing an image from low resolution (LR) to high resolution (HR). It can enhance the clarity and detail of the calibration images, which will be beneficial in improving the accuracy of camera calibration. However, there is very little research on the application of image super-resolution to camera calibration. Therefore, this study innovatively proposed a method and process for applying image super-resolution in camera calibration. Firstly, the ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) was optimized and improved, followed by the proposal of the MMRSRGAN (Multi-scale Multi-adaptive-weights Residual Super-Resolution Generative Adversarial Networks). Two methods for local image super-resolution were also proposed for efficient training and testing of datasets, and two evaluation metrics based on reprojection error were proposed evaluate the effectiveness of the proposed model. Training and testing experiments of image super-resolution for camera calibration were conducted on both virtual and real image datasets. The proposed MMRSRGAN was compared with several advanced image super-resolution networks developed in recent years across multiple dimensions and indicators. The positive results have demonstrated the feasibility of applying image super-resolution to camera calibration and showcased the good performance of the MMRSRGAN.

Camera-in-the-loop based test scenario generation method for pedestrian collision avoidance system

The Pedestrian Collision Avoidance System (PCAS) of Intelligent Vehicle (IV) can be effective in preventing the occurrence of traffic accidents. However, the complicated operation environments introduce great challenges to the camera used by the PCAS. Therefore, the camera based PCAS should be fully tested and evaluated before deployment. The traditional simulation test for the camera based PCAS attempted to use geometric or physical simulation models, which have low reality and are suitable for the primary stage of the PCAS development. Camera-in-the-Loop (CIL) test is one of Hardware-in-the-Loop methods that embeds the real camera hardware into the virtual simulation system to test the camera. CIL can utilize the real hardware response while overcoming the common simulation weakness of fidelity. In this paper, we construct a CIL test platform, and propose the CIL based test scenarios generation and scenario parameter impact evaluation method for PCAS. First, we construct the CIL test platform whose image quality and functional confidence are both validated to prove CIL credibility. Second, the PCAS under the test is analyzed and the corresponding test scenario parameters are designed. In order to accelerate the test scenario generation, a Greedy Based Combination test method (GBC) based on the CIL is proposed. The Chi-square analysis and two-factor of variance analysis verification methods are used to analyze the influence of individual and multiple scenario parameters on the PCAS performance. The experiment results show that the GBC improves the test speed by 12 times compared to the traversal test, and the frequency ratio of each scenario parameter is no more than 3% different from that of the traversal test. Also, GBC has an equivalent ability to find the PCAS collision scenarios parameter combination to the traversal test.

Generating real-scene hologram through light field imaging and deep learning

Addressing the challenge of acquiring holograms from real-world scenes, this study introduces a novel approach leveraging light field cameras to capture light field data, which is subsequently transformed into authentic scene holograms. This methodology integrates light field imaging technology with a pre-trained deep neural network. To compensate for the limitations inherent in camera hardware, a super-resolution algorithm is employed. The conversion of light field information into RGB-D data facilitates its input into the deep neural network, enabling the inference of corresponding real-world scene holograms. Empirical evidence demonstrates that the system is capable of inferring high-resolution (1920 × 1080) real-world scene holograms within a timeframe of 5 s, utilizing hardware comprising an NVIDIA RTX 3060.

73‐1: Implementation of Eye‐Tracking Technology Based on Vehicle Glasses‐Free 3D Prism Display

With the continuous development of in‐car display applications, the demand for in‐car glasses‐free 3D display screens is gradually increasing. This article adopts a high‐resolution 4K display screen based on prism technology， and is equipped with a human eye tracking system implemented using a regular monocular camera. This configuration enables the in‐car instrument display to provide a high‐quality glasses‐free 3D display effect within a large visual range, addressing the limited viewing area issue associated with traditional glasses‐free 3D displays. Due to constraints in camera hardware conditions, we introduced a motion compensation and motion prediction algorithm. Through experimental validation, this algorithm significantly enhances the performance of human eye tracking using a regular camera. A comparison with traditional eye‐tracking methods demonstrates the substantial advantages of this algorithm in improving the performance of in‐car 3D display technology.

Object Tracking Using Computer Vision: A Review

Object tracking is one of the most important problems in computer vision applications such as robotics, autonomous driving, and pedestrian movement. There has been a significant development in camera hardware where researchers are experimenting with the fusion of different sensors and developing image processing algorithms to track objects. Image processing and deep learning methods have significantly progressed in the last few decades. Different data association methods accompanied by image processing and deep learning are becoming crucial in object tracking tasks. The data requirement for deep learning methods has led to different public datasets that allow researchers to benchmark their methods. While there has been an improvement in object tracking methods, technology, and the availability of annotated object tracking datasets, there is still scope for improvement. This review contributes by systemically identifying different sensor equipment, datasets, methods, and applications, providing a taxonomy about the literature and the strengths and limitations of different approaches, thereby providing guidelines for selecting equipment, methods, and applications. Research questions and future scope to address the unresolved issues in the object tracking field are also presented with research direction guidelines.

Towards robotic disassembly: A comparison of coarse-to-fine and multimodal fusion screw detection methods

To accelerate the transition towards a more circular economy it is believed essential to increase the economic viability and hence the adoption of de- and remanufacturing processes. For this, industry 4.0 technologies, including robotization, computer vision and artificial intelligence, are commonly considered as the main enablers to reduce costs while boosting performances of de- and remanufacturing processes. However, the essential step in de- and remanufacturing of screw detection for robotic disassembly is still confronted with various challenges, such as false negatives due to deep screw holes and false positives caused by similar patterns and colors, e.g. letters in labels. Previous studies improved the detection accuracy by capturing images of only a fine part of the product and neglected the negative impact of this imaging approach on the overall time efficiency of the disassembly process. Therefore, two of the most distinct approaches are evaluated in the presented research: 1. A Coarse-to-Fine detection method, where potential screws across the entire product are detected by an area camera in the coarse stage to generate an optimal unscrewing path and screw types and locations are further refined in a second instance, the fine stage. 2. A multimodal fusion detection method, where a combination of a high-resolution color line and 3D triangulation line camera system captures full-product images allowing a single run screw detection in the RGB, Intensity and Depth (RGBID) images. Furthermore, a pooling-based aligned fusion module (PAlign) is introduced to improve the feature representation and mitigate the encountered spatial misalignment of different modality branches. Two industrially representative datasets comprising images of laptops and hard disk drives are collected for the training and evaluation of the screw detection and unscrewing task. Experimental results show that the RGBID fusion method with PAlign module achieves an unscrewing success rate of 80.4 % with an average detection and unscrewing speed of 2.27 s per screw, which is almost double the speed when compared to the Coarse-to-Fine approach. Through a detailed analysis comparing these most distinct screw detection methods in terms of imaging approach, lighting and camera hardware (1/2, area/line, low/high resolution, RGB/RGBID, 600/25.000€), this study provides concrete insights and recommendations aimed at enhancing the efficiency of disassembly systems for sustainable demanufacturing.

Edge Boosted Global Awared Low-light Image Enhancement Network

Low-light images are captured in situations where the lighting is poor or the camera hardware is not capable of producing good quality images. These types of images tend to have low contrast, blurry details, noise, and color distortion. In computer vision applications, image brightness plays a crucial role, and therefore, low-light image enhancement is used as a preprocessing step. In this study, we have improved the Low-Light Enhancement Network with Global Awareness (GLADNet) method by adding a UNet-based edge information extraction unit. The channel attention mechanism was also incorporated into the edge information extraction unit to achieve color preservation. Our experiments show that our proposed method has achieved higher PSNR, SSIM, and FSIM metrics compared to reference images. Additionally, it has produced lower NIQE and BRISQUE values for non-reference performance evaluation. Moreover, our proposed method removes noise better and produces visual results that are closer to the target images.

Learning a physics-based filter attachment for hyperspectral imaging with RGB cameras

Countless RGB cameras are ubiquitously distributed in our daily lives, serving to perceive and depict the diverse colors of the world. Reconstructing hyperspectral images (HSI) from these trichromatic cameras emerges as a promising solution to address the limitations of existing, costly hyperspectral imaging systems. The performance of HSI reconstruction relies heavily on the camera spectral response (CSR). Thus, designing a better CSR and putting it into practice is the critical issue for RGB-based HSI reconstruction. However, the CSR curves designed in the existing works are overly random, making them challenging to manufacture directly. Additionally, the designed CSR curves require modifications to the camera hardware, resulting in the loss of RGB imaging functionality. In this paper, we propose a hyperspectral imaging system, which involves enhancing the CSR curve of existing RGB cameras and preserving RGB imaging functionality by adding a learnable physics-based spectral filter. Specifically, we first parameterize the spectral filter transmittance as a function of the filter thicknesses, based on the physical constraints of the multilayer interference principle. Then, we propose a joint optimization framework in which the thicknesses of the filter and the hyperspectral reconstruction network are optimized. In this manner, the thicknesses of the filter are obtained and used to manufacture the filter directly. Finally, we construct a prototype and verify the benefits of our spectral filter design method through experiments including both synthetic data and real images.

Multi-Scale Image- and Feature-Level Alignment for Cross-Resolution Person Re-Identification

Cross-Resolution Person Re-Identification (re-ID) aims to match images with disparate resolutions arising from variations in camera hardware and shooting distances. Most conventional works utilize Super-Resolution (SR) models to recover Low Resolution (LR) images to High Resolution (HR) images. However, because the SR models cannot completely compensate for the missing information in the LR images, there is still a large gap between the HR image recovered from the LR images and the real HR images. To tackle this challenge, we propose a novel Multi-Scale Image- and Feature-Level Alignment (MSIFLA) framework to align the images on multiple resolution scales at both the image and feature level. Specifically, (i) we design a Cascaded Multi-Scale Resolution Reconstruction (CMSR2) module, which is composed of three cascaded Image Reconstruction (IR) networks, and can continuously reconstruct multiple variables of different resolution scales from low to high for each image, regardless of image resolution. The reconstructed images with specific resolution scales are of similar distribution; therefore, the images are aligned on multiple resolution scales at the image level. (ii) We propose a Multi-Resolution Representation Learning (MR2L) module which consists of three-person re-ID networks to encourage the IR models to preserve the ID-discriminative information during training separately. Each re-ID network focuses on mining discriminative information from a specific scale without the disturbance from various resolutions. By matching the extracted features on three resolution scales, the images with different resolutions are also aligned at the feature-level. We conduct extensive experiments on multiple public cross-resolution person re-ID datasets to demonstrate the superiority of the proposed method. In addition, the generalization of MSIFLA in handling cross-resolution retrieval tasks is verified on the UAV vehicle dataset.

Curvelet coefficient prediction-based image super-resolution method for precision measurement

The precision of image-based measurements is mainly limited by the resolution of the camera hardware. Currently, increasing the resolution of measurement images by super-resolution method is an effective way to improve the precision of vision measurement. However, most image super-resolution methods are only based on the prediction of spatial information, therefore the reconstruction quality and localization accuracy of the edge regions are often insufficient to meet the requirements of precision measurements. For this reason, a Curvelet coefficient prediction method for image super-resolution (CPSR) is proposed to achieve accurate super-resolution results of edge regions in this paper. Firstly, the image is decomposed into sub-bands of Curvelet coefficients at different scale to extract frequency features. Then, deep residual networks are built and trained to fit the mapping function between low- and high-resolution coefficient sub-bands. In addition, Curvelet loss and edge localization loss functions are designed to obtain Curvelet coefficient errors of each scale and edge localization errors of sub-pixel level. The proposed method is evaluated using public super-resolution datasets and actual precision measurement images. Experimental results show that CPSR generates images with better visual effects, and has strong generalization ability for the reconstruction of different edge patterns. Furthermore, compared with the common super-resolution method based on deep learning, vision measurements on the reconstructed image of CPSR achieve smaller measurement errors, indicating that CPSR achieves more accurate edge localization. Experimental results verified the effectiveness of the proposed method.

Denoising Method Based on Salient Region Recognition for the Spatiotemporal Event Stream.

Event cameras are the emerging bio-mimetic sensors with microsecond-level responsiveness in recent years, also known as dynamic vision sensors. Due to the inherent sensitivity of event camera hardware to light sources and interference from various external factors, various types of noises are inevitably present in the camera's output results. This noise can degrade the camera's perception of events and the performance of algorithms for processing event streams. Moreover, since the output of event cameras is in the form of address-event representation, efficient denoising methods for traditional frame images are no longer applicable in this case. Most existing denoising methods for event cameras target background activity noise and sometimes remove real events as noise. Furthermore, these methods are ineffective in handling noise generated by high-frequency flickering light sources and changes in diffused light reflection. To address these issues, we propose an event stream denoising method based on salient region recognition in this paper. This method can effectively remove conventional background activity noise as well as irregular noise caused by diffuse reflection and flickering light source changes without significantly losing real events. Additionally, we introduce an evaluation metric that can be used to assess the noise removal efficacy and the preservation of real events for various denoising methods.

Lessons Learned in Change Management in Deploying Novel Informatics Solutions: Experience Implementing a Point-of-Care Patient Photography System with Radiography

We describe implementation of a point-of-care system for simultaneous acquisition of patient photographs along with portable radiographs at a large academic hospital. During the implementation process, we observed several technical challenges in the areas of (1) hardware—automatic triggering for photograph acquisition, camera hardware enclosure, networking, and system server hardware and (2) software—post-processing of photographs. Additionally, we also faced cultural challenges involving workflow issues, communication with technologists and users, and system maintenance. We describe our solutions to address these challenges. We anticipate that these experiences will provide useful insights into deploying and iterating new technologies in imaging informatics.

0944 Telemedicine (TM) to provide outpatient care for sleep disorders in complex neurological patients in an academic Medical Center

Abstract Introduction Since the widespread adoption of telemedicine (TM) as a reimbursable option for the delivery of healthcare, (particularly clinic visits), the field of sleep medicine has been a beneficiary. In complex neurological patients, it was hypothesized that physical and cognitive constraints would limit the use of TM. This analysis was undertaken to evaluate patient care strategies in an academic neurology department based in a large metropolitan city Methods 2964 adult patients who opted for neurology clinic visits via video TM in the last 6 months were included from electronic medical record review at the University of Texas Health Sciences Center in Houston. Of these, 557 were seen in sleep medicine specialty. A successful visit was defined as a simultaneous connection of at least 2 parties for a minimum duration of time defined by institutional guidelines. Patients were analyzed in age subgroups (< 30, 31-50, 51-70 and >70 years of age) as well as race (Asian, Black, Hispanic, White and Others). Neurological diagnoses in these patients included stroke, epilepsy, movement disorders, cognitive disorders, neuromuscular conditions and less commonly multiple sclerosis Results Sleep medicine providers comprise of 2/53 (3.7%) of the department’s clinical providers, but accounted for 18.7% of TM visits. 97% (182/186) of new patients and 95% (354/371) of established patients had successful TM visits. Among general neurology patients, these numbers were 89% and 88% respectively. There was a statistically significant difference between successful visits in the sleep medicine and overall neurology patient groups (p value 0.0003 for new, 0.0001 for established patients). Among sleep medicine patients, percentage failures were 2.9, 3.3, 4.1 and 3.9 among the respective age groups. Percentage failures across the respective racial groups were 0, 6.1, 4.8, 4.2 and 1.9. Reasons cited for unsuccessful visits included camera, microphone and general hardware failures. Surprisingly network failure was not reported Conclusion The success of telemedicine clinic visits for the management of sleep disorders is not impacted by physical and cognitive impairment in patients with complex neurological conditions. Further analysis to include success of sleep diagnostics and treatment of sleep disorders (start-to-finish pathway), via telemedicine in this complex population is underway Support (if any) None

Udo Krüger Charts TechnoTeam's Journey

Udo Krüger Charts TechnoTeam's Journey

Design of intelligent visitor system based on cloud and edge collaborative computing

With the maturity of face recognition and speech recognition technologies, artificial intelligence (AI) cloud and edge computing collaboration have become a new research direction. In many enterprises and government departments, there are certain management requirements for visitors, usually using traditional manual records or computer-aided manual management. These methods require certain personnel management costs, and they face underlying problems concerned with personal identification as well as security. In this paper, we analyze the functions and features of cloud-edge collaboration and discuss the edge intelligence technology in the cloud-edge collaboration environment. Then by combining the architecture of the intelligent visitor system, we apply AI cloud and edge computing to collaboratively solve critical issues faced by the visitor system, such as real-time and data authenticity. The intelligent visitor system employs a Rockchip RK3399 motherboard, ID card reader, microphone array, camera, and other hardware to build an edge computing environment. Combined with Baidu AI cloud, the system has an intelligent visitor system with face recognition and voice interaction capabilities, which can realize verification of visitor information, voice self-registration, and automatic measurement of visitors’ body temperature and other functions.

E2SRI: Learning to Super-Resolve Intensity Images From Events

An event camera reports per-pixel intensity differences as an asynchronous stream of events with low latency, high dynamic range (HDR), and low power consumption. This stream of sparse/dense events limits the direct use of well-known computer vision applications for event cameras. Further applications of event streams to vision tasks that are sensitive to image quality issues, such as spatial resolution and blur, e.g., object detection, would benefit from a higher resolution of image reconstruction. Moreover, despite the recent advances in spatial resolution in event camera hardware, the majority of commercially available event cameras still have relatively low spatial resolutions when compared to conventional cameras. We propose an end-to-end recurrent network to reconstruct high-resolution, HDR, and temporally consistent grayscale or color frames directly from the event stream, and extend it to generate temporally consistent videos. We evaluate our algorithm on real-world and simulated sequences and verify that it reconstructs fine details of the scene, outperforming previous methods in quantitative quality measures. We further investigate how to (1) incorporate active pixel sensor frames (produced by an event camera) and events together in a complementary setting and (2) reconstruct images iteratively to create an even higher quality and resolution in the images.

A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging

Hyperspectral imaging enables many versatile applications for its competence in capturing abundant spatial and spectral information, which is crucial for identifying substances. However, the devices for acquiring hyperspectral images are typically expensive and very complicated, hindering the promotion of their application in consumer electronics, such as daily food inspection and point-of-care medical screening, etc. Recently, many computational spectral imaging methods have been proposed by directly reconstructing the hyperspectral information from widely available RGB images. These reconstruction methods can exclude the usage of burdensome spectral camera hardware while keeping a high spectral resolution and imaging performance. We present a thorough investigation of more than 25 state-of-the-art spectral reconstruction methods which are categorized as prior-based and data-driven methods. Simulations on open-source datasets show that prior-based methods are more suitable for rare data situations, while data-driven methods can unleash the full potential of deep learning in big data cases. We have identified current challenges faced by those methods (e.g., loss function, spectral accuracy, data generalization) and summarized a few trends for future work. With the rapid expansion in datasets and the advent of more advanced neural networks, learnable methods with fine feature representation abilities are very promising. This comprehensive review can serve as a fruitful reference source for peer researchers, thus paving the way for the development of computational hyperspectral imaging.

SR-DSFF and FENet-ReID: A Two-Stage Approach for Cross Resolution Person Re-Identification.

In real-life scenarios, the accuracy of person re-identification (Re-ID) is subject to the limitation of camera hardware conditions and the change of image resolution caused by factors such as camera focusing errors. People call this problem cross-resolution person Re-ID. In this paper, we improve the recognition accuracy of cross-resolution person Re-ID by enhancing the image enhancement network and feature extraction network. Specifically, we treat cross-resolution person Re-ID as a two-stage task: the first stage is the image enhancement stage, and we propose a Super-Resolution Dual-Stream Feature Fusion sub-network, named SR-DSFF, which contains SR module and DSFF module. The SR-DSFF utilizes the SR module recovers the resolution of the low-resolution (LR) images and then obtains the feature maps of the LR images and super-resolution (SR) images, respectively, through the dual-stream feature fusion with learned weights extracts and fuses feature maps from LR and SR images in the DSFF module. At the end of SR-DSFF, we set a transposed convolution to visualize the feature maps into images. The second stage is the feature acquisition stage. We design a global-local feature extraction network guided by human pose estimation, named FENet-ReID. The FENet-ReID obtains the final features through multistage feature extraction and multiscale feature fusion for the Re-ID task. The two stages complement each other, making the final pedestrian feature representation has the advantage of accurate identification compared with other methods. Experimental results show that our method improves significantly compared with some state-of-the-art methods.

Application of Image Processing Sensor and Pattern Recognition in Detection of Bearing Surface Defects

In order to solve the problem that the traditional detection technology can not meet the requirements of online detection, a visual detection device for bearing inner ring defects based on image processing and pattern recognition is proposed in this paper. The device systematically designs an image acquisition device of bearing inner ring based on CCD. In the hardware scheme, the appropriate lens, camera, light source, and other related hardware are selected according to the actual needs, a complete image acquisition platform of bearing inner ring is built, and the software function design is completed. The simulation results show that the qualified rate of machine detection is 98.6%, the missed detection rate is 0, and the false detection rate is 1.4%, which are better than manual detection. Conclusion. The test results show that the bearing inner ring defect detection system can detect the surface defects of bearing inner ring quickly, stably, and reliably, and the detection efficiency and accuracy are higher than the original manual detection method, so it has a good application prospect.

A comparative study on the effect of grid printing methods and camera hardware in automatic strain measurement through CGA

Strain analysis is an important step in sheet metal forming operations to identify the potential locations of failure. Circle grid analysis is a widely used method to evaluate the strains in sheet metal forming, where deformed circles on a formed component are measured to evaluate the strains at different locations. In this paper, two different USB cameras are used to capture the images of the deformed grids. The images are processed using the in-house software developed in MATLAB for strain measurement. The effect of image acquisition hardware and grid printing methods on strain measurement accuracy has been analyzed. For this, the sheets are printed with circular grids by screen printing, laser engraving, and electrochemical etching. The sheets are then deformed in incremental forming to produce different levels of strain in the formed components. The deformed grids on the formed components are captured using proposed USB cameras with appropriate attachments and processed by in-house strain measurement software. The strain measured using proposed hardware and software is also compared with the commercial strain measurement system. The mean absolute percentage error in strain measurement with the proposed USB cameras was found to be 0.671 and 0.761. The statistical results show that there is no evidence to support the hypothesis that the two hardware perform differently on average. Hence, the proposed hardware and software can be used in industry and research laboratories for strain analysis in sheet metal forming operations.