Camera Image Processing Research Articles

Camera Image Processing on ESP32 Microcontroller with Help of Convolutional Neural Network

This paper analyzes a common ESP32 microcontroller with a built-in camera for image classification tasks using a convolutional neural network. ESP32 is commonly used in IoT devices to read data and control sensors, so its computing power is not significant, which has a positive effect on the cost of the device. The prevalence of ultra-low power embedded devices such as ESP32 will allow the widespread use of artificial intelligence built-in IoT devices. The duration of photographing and photo processing is obtained in the paper, as this can be a bottleneck of the microcontroller, especially together with machine learning algorithms. Deployed convolutional neural network, pre-trained on another device, MobileNet architecture on microcontroller and proved that ESP32 capacity is sufficient for simultaneous operation of both the camera and convolutional neural network.

Fall Detection for Shipboard Seafarers Based on Optimized BlazePose and LSTM.

Aiming to avoid personal injury caused by the failure of timely medical assistance following a fall by seafarer members working on ships, research on the detection of seafarer’s falls and timely warnings to safety officers can reduce the loss and severe consequences of falls to seafarers. To improve the detection accuracy and real-time performance of the seafarer fall detection algorithm, a seafarer fall detection algorithm based on BlazePose–LSTM is proposed. This algorithm can automatically extract the human body key point information from the video image obtained by the vision sensor, analyze its internal data correlation characteristics, and realize the process from RGB camera image processing to seafarer fall detection. This fall detection algorithm extracts the human body key point information through the optimized BlazePose human body key point information extraction network. In this section, a new method for human bounding-box acquisition is proposed. In this study, a head detector based on the Vitruvian theory was used to replace the pre-trained SSD body detector in the BlazePose preheating module. Simultaneously, an offset vector is proposed to update the bounding box obtained. This method can reduce the frequency of repeated use of the head detection module. The algorithm then uses the long short-term memory neural network to detect seafarer falls. After extracting fall and related behavior data from the URFall public data set and FDD public data set to enrich the self-made data set, the experimental results show that the algorithm can achieve 100% accuracy and 98.5% specificity for the seafarer’s falling behavior, indicating that the algorithm has reasonable practicability and strong generalization ability. The detection frame rate can reach 29 fps on a CPU, which can meet the effect of real-time detection. The proposed method can be deployed on common vision sensors.

Imaging simulation of the AMCW ToF camera based on path tracking.

The time-of-flight (ToF) camera suffers from many error factors, such as multiple reflection or multipath interference and electronic and optical shot noise, making it difficult to simulate its imaging process. Aiming to test the ToF camera algorithm, it is important to obtain a depth image affected by these error factors. In order to model the light propagation behavior and the sensor effect in the imaging process of the ToF camera, an amplitude modulated continuous-wave (AMCW) ToF camera imaging simulation method based on path tracking is presented by deducing the path tracking algorithm model in the AMCW ToF camera theoretically and by realizing the physically based simulation by introducing the infrared bidirectional reflectance distribution function (BRDF) data of the actual materials. According to the constructed error evaluation indexes, the correctness of the imaging simulation method is verified based on the ground experiment. The mean absolute error (MAE) and root mean square error (RMSE) are 10.32 mm and 15.12 mm, respectively, which are less than the error results of the other two comparative simulation methods. The results show that the proposed method is reasonable and can provide reliable data support for AMCW ToF hardware development and algorithm testing.

Detection of AI-Manipulated Fake Faces via Mining Generalized Features

Recently, AI-manipulated face techniques have developed rapidly and constantly, which has raised new security issues in society. Although existing detection methods consider different categories of fake faces, the performance on detecting the fake faces with “unseen” manipulation techniques is still poor due to the distribution bias among cross-manipulation techniques. To solve this problem, we propose a novel framework that focuses on mining intrinsic features and further eliminating the distribution bias to improve the generalization ability. First, we focus on mining the intrinsic clues in the channel difference image (CDI) and spectrum image (SI) view of two different aspects, including the camera imaging process and the indispensable step in AI manipulation process. Then, we introduce the Octave Convolution and an attention-based fusion module to effectively and adaptively mine intrinsic features from CDI and SI view of these two different but intrinsic aspects. Finally, we design an alignment module to eliminate the bias of manipulation techniques to obtain a more generalized detection framework. We evaluate the proposed framework on four categories of fake faces datasets with the most popular and state-of-the-art manipulation techniques and achieve very competitive performances. We further conduct experiments on cross-manipulation techniques, and the results of our method show the superior advantages on improving generalization ability.

Underwater large field of view 3D imaging based on fisheye lens

As for the 3D imaging of seabed topography, seabed sunken ships and large underwater buildings, the field of view is a significant technical index of the 3D imaging system. In this paper, a binocular vision imaging system based on fisheye lens is applied to satisfy the requirements for 3D imaging of large underwater scenes. Considering the severe image distortion caused by the large wide-angle of fisheye lens and the failure of pinhole imaging model underwater, the imaging process of underwater fisheye camera is re-described through the parametric representation of four-dimensional light field, and the multi-layer refraction imaging model of underwater binocular stereo vision is established based on fisheye camera. A conversion imaging model is constructed to convert the corrected underwater fisheye image into the equivalent image taken in the air. Combined with the existing stereo matching methods, the converted images realize the 3D reconstruction of a large-scale underwater scene. Experimental results show that the corrected underwater fisheye images have a small average reprojection error, which evidences the effectiveness of the proposed calibration method. In the experiment of 3D reconstruction for the large scene composed of multiple underwater ceramic discs, the reconstructed targets have good 3D imaging shapes, which indicates the developed underwater binocular fisheye camera imaging system has the ability to perform 3D imaging and make observations of large underwater scenes.

Image Quality and System Performance XIX Conference Overview and Papers Program

We live in a visual world. The perceived quality of images is of crucial importance in industrial, medical, and entertainment application environments. Developments in camera sensors, image processing, 3D imaging, display technology, and digital printing are enabling new or enhanced possibilities for creating and conveying visual content that informs or entertains. Wireless networks and mobile devices expand the ways to share imagery and autonomous vehicles bring image processing into new aspects of society. The power of imaging rests directly on the visual quality of the images and the performance of the systems that produce them. As the images are generally intended to be viewed by humans, a deep understanding of human visual perception is key to the effective assessment of image quality.

Assessing the Spectral Characteristics of Dye- and Pigment-Based Inkjet Prints by VNIR Hyperspectral Imaging.

Fine art photography, paper documents, and other parts of printing that aim to keep value are searching for credible techniques and mediums suitable for long-term archiving purposes. In general, long-lasting pigment-based inks are used for archival print creation. However, they are very often replaced or forged by dye-based inks, with lower fade resistance and, therefore, lower archiving potential. Frequently, the difference between the dye- and pigment-based prints is hard to uncover. Finding a simple tool for countrified identification is, therefore, necessary. This paper assesses the spectral characteristics of dye- and pigment-based ink prints using visible near-infrared (VNIR) hyperspectral imaging. The main aim is to show the spectral differences between these ink prints using a hyperspectral camera and subsequent hyperspectral image processing. Two diverse printers were exploited for comparison, a hobby dye-based EPSON L1800 and a professional pigment-based EPSON SC-P9500. The identical prints created via these printers on three different types of photo paper were recaptured by the hyperspectral camera. The acquired pixel values were studied in terms of spectral characteristics and principal component analysis (PCA). In addition, the obtained spectral differences were quantified by the selected spectral metrics. The possible usage for print forgery detection via VNIR hyperspectral imaging is discussed in the results.

Assessment of Do Son Beach Changes using Monitoring Camera Images

In recent years, camera technology has been used in the field of beach variation and beach protection. This study presents the results of processing images of Do Son beach based on images. Camera image processing is calibrated based on real measurements of control points. Camera images are processed to capture shoreline and beach developments for the period from August 2018 to October 2019. The results show that Do Son beach has short term and seasonal fluctuations. Beach geomorphology tends to reach equilibrium form.

Discrete and Continuum Approaches for Modeling Solids Motion Inside a Rotating Drum at Different Regimes

In this work, the performance of discrete and continuum computational models for addressing granular flow dynamics in a rotating drum at different regimes is studied. The results are compared to the experimental observations obtained by image processing of a high-speed camera on a pilot plant rotating drum. For the discrete modeling, Discrete Elements Method (DEM) through the open-source software LIGGGHTS(R) is used, while for the continuum model, the μ(I)-rheology is implemented in the general structure of a Volume-Of-Fluid (VOF) solver of the OpenFOAM(R) platform. Four test cases consisting of different sets of particles filling and rotational speed are considered and the results are analyzed in terms of solids distribution, the velocity of the particles, and mixing patterns. The solids distribution and velocities for each one of the tests considered are fairly similar between both computational techniques and the experimental observations. In general, DEM results show a higher level of agreement with the experiments, with minor differences that might be irrelevant in some cases (e.g., more splashing of particles for the fastest regimes). Among the drawbacks of the continuum model, it was unable to predict the slumping regime observed experimentally which can be attributed to the lack of a yield criterion and a slower dragging of the granular material when the drum is being accelerated, which can be attributed to the need of adding non-local effects to the rheology. On the other hand, the dynamic of the bed in the rolling and cascading regimes are accurately predicted by the continuum model in less time than DEM, even in a pilot plant scale system. These results suggest that the use of a continuum model with granular fluid rheology is more suited for simulating industrial-scale rotating drums at different regimes than DEM, but only if all the phenomenological features (i.e., yield criteria and non-local effects) are taken into account in the model.

Dynamic camera configuration learning for high-confidence active object detection

The performance of object detection is closely related to the quality of input images. However, the current image acquisition is purely guided by human visual perception, and such camera imaging process ignores the subsequent application. In this context, detection performance is impacted by imaging configuration and dynamic camera motion. To address the above problems, an active object detection framework is proposed in this paper, which aims to build the bridge between imaging configuration and object detection task. Within the proposed framework, a dynamic camera configuration learning approach is presented based on deep reinforcement learning, where the camera is actively controlled to maximize the detection performance. Through iterated interactions between imaging, control and object detection, the deep gap between perception and cognition in the object detection system is eliminated, and the transformation from physical imaging to purposeful imaging is realized. The effectiveness and advantages of the proposed framework are demonstrated in three dynamic environments.

Guest Editorial Software Defined Internet of Vehicles

Internet of Vehicles (IoV) is a large interactive network composed of information such as vehicle location, speed and route. Vehicles can collect their own environment and state information through GPS, RFID, sensors, camera image processing, and other devices. They can transmit their various information to the central processing unit through Internet technology. These large amounts of vehicle information can be analyzed and processed through computer technology to calculate the optimal route for different vehicles, and report road conditions in time and schedule signal light cycles. Internet of Vehicles (IoV) is a large-scale system network for wireless communication and information exchange between vehicles and people, vehicles and roads, vehicles and the Internet, which is based on intra-vehicle network, inter-vehicle network and vehicle mobile Internet, in accordance with the agreed communication protocols and data interaction standards. The system realizes the integrated network of intelligent traffic management, intelligent dynamic information service, and intelligent vehicle control by “filtering and cleaning” massive data and processing data on the platform. Internet of Vehicles (IoV) system utilizes advanced IoT technology, cloud computing, and big data to make the system fully aware of roads and traffic. It enables all vehicles to collect information through their own environment and state, and upload all kinds of information to the Internet big data platform. The central processing unit collects, analyzes, and processes a large amount of uploaded information. The system will control every vehicle involved in the traffic and control every road in real time to provide users with traffic efficiency and safety.

Effect of gas-liquid two-phase macro-mixing on the uniformity of temperature distribution in a top-blown stirred tank using image analysis

This paper investigates the effect of two-phase mixing of gas and liquid in a top-blowing mixing tank on the uniformity of temperature distribution, using hot heat transfer oil and air as the working medium. Results show that temperature difference is the main factor influencing the temperature field uniformity in the mixing tank, and the uniformity of the temperature distribution increases with the temperature difference increased, while the increase of the air flow and the insertion depth of the gun has less influence on the uniformity of the temperature distribution. In addition, a new threshold selection method is proposed for grayscale maps with temperature fields corresponding to different color gradations, in which the overall mean is replaced by the mean of the grayscale values in the target region, and a highly correlated model between the temperature field and the grayscale map is determined with a correlation coefficient of 0.98. Using a thermal imaging camera and digital image processing techniques, combining with a deviation approach, a relative ratio η between the uniformity of the two-dimensional temperature field distribution and the uniformity of the pixel distribution of its corresponding grayscale map is established. In particular, the indicator shows an interesting trend with an increasing temperature difference, i.e., the ratio is close to 1. The effect of bubble-flow topology on the uniformity of temperature distribution was explored, and it was found that betti numbers averages were negatively correlated with temperature distribution uniformity factor. Finally, a correlation model (correlation coefficient of 0.95) between the uniformity of the temperature field distribution and the average volumetric thermal coefficient was obtained experimentally.

Closed-loop deposition of martensitic stainless steel during laser additive manufacturing to control microstructure and mechanical properties

A closed-loop laser additive manufacturing (CLAM) system was developed to monitor and control layer-by-layer fabrication via directed energy deposition (DED). The well-modified laser additive manufacturing (LAM) package utilizes a thermal infrared camera and thermal image processing algorithms to extract and control the thermal dynamics of the process, including (i) cooling rate, (ii) melt pool temperature, and (iii) heating rate. Martensitic stainless steel of grade S410-L was used to cumulatively deposit 5 layers on top of each other at different processing conditions without closed-loop control. Significant deviations of cooling rate and melt pool temperature are observed within different layers of a thin-wall structure without feedback control in the open-loop state. To assess the effects of cooling rate control on final LAM properties, three samples with different setpoint cooling rates of 550, 1100, and 1750 °C/s are tested under closed-loop conditions. Microstructural and microhardness maps and profiles of open-loop and closed-loop samples reveal highly uniform and controlled hardness profiles when closed-loop control is applied, in contrast to highly varying properties in samples produce in open-loop operation. The current research results illustrate how one can achieve controlled microstructural characteristics and mechanical properties through real-time control of the cooling rate during the LAM process.

A Raspberry Pi‐based camera system and image processing procedure for low cost and long‐term monitoring of forest canopy dynamics

Abstract In‐situ, high‐frequency and long‐term monitoring of forest canopy development is vital for improved understanding in many fields of ecology. Although there are well‐established commercial instruments measuring light interception and digital hemispherical photography methods, the equipment cost (thousands of pounds) and lack of field robustness make these instruments not suitable for long‐term unattended measurements and can only give the high‐resolution spatial sampling usually required in forests when used manually, with substantial effort. Here we present a low‐cost Raspberry Pi camera system that is robust and suitable for the acquisition of long‐term, high‐frequency forest canopy images. The low cost of each camera means multiple systems could be used to provide the required spatial sampling. We also present an open‐source R script for post‐processing these images to provide a semi‐automated processing chain, easily adaptable for use at other forest sites. Four individual camera systems were tested at a deciduous oak forest in SE England over three growing seasons. These ground‐based camera systems were used to take upward‐facing images of a forest canopy every 30 min. The timing of canopy seasonal development, derived from the novel Raspberry Pi method, compared well with an established technique based on canopy light interception measurements. With a cost of approximately £80, these systems have the potential to enable unattended widespread forest canopy monitoring at a very low cost, providing high spatial and temporal resolution that has previously been unattainable.

Microfluidic protein detection and quantification using droplet morphology

Sensitive, inline detection of proteins is required for post-chromatographic analyses in proteomics, cell-based assays, and drug discovery workflows. Among the common inline methods, post-column derivatization requires chemical labels, while label-free methods are either expensive (mass spectrometry) or have limited sensitivity at small length scales (UV–Vis). This paper presents a label-free detection technique based on the concept that dissolved proteins can function as surfactants and decrease the dynamic interfacial tension (IFT) of an immiscible (water–oil) interface. Existing methods for measuring IFT, such as axisymmetric drop shape analysis (ADSA), operate in batch mode and are not suitable for continuous detection. Here we show that a microfluidic flow-focusing droplet generator operating at a frequency of > 100 Hz can track IFT changes continuously, with high temporal resolution and small detection volumes. Variations in protein concentration alter the size and shape of the drops/plugs formed, and these changes can be quantified in time using a high-speed camera and in-house image processing software. Moreover, the continuously refreshing interface alleviates issues related to surface aging. Two applications are demonstrated: (1) direct injection of a single protein into a microfluidic chip. (2) post-column detection of protein mixtures separated by high performance size exclusion chromatography (SEC HPLC). Of interest, the dynamic range of protein (bovine serum albumin, BSA) was 50 -104 μg/ml without using HPLC unit. The lowest limit of detection without HPLC unit was ~ 1 μg/ml of thyroglobulin protein in a 1 nl droplet, which equates to 1 fg of total protein. When used as a detector, the aforementioned detection method offered a sensitivity of six orders of magnitude higher than conventional UV–VIS detectors.

The 20k Samples-Per-Second Real Time Detection of Acoustic Vibration Based on Displacement Estimation of One-Dimensional Laser Speckle Images.

Audio signal acquisition using a laser speckle image is an appealing topic since it provides an accurate and non-contact solution for vibration measurement. However, due to the limitation of camera frame rate and image processing speed, previous research could not achieve real time reconstruction of an audio signal. In this manuscript, we use a one-dimensional laser speckle image to measure the acoustic vibration of sound source and propose a fast and sub-pixel accuracy algorithm to estimate the displacement of captured one-dimensional laser speckle images. Compared with previous research, the proposed method is faster and more accurate in displacement estimation. Owing to this, the frequency bandwidth and the robustness are significantly increased. Experiment results show that the proposed system can achieve 20k samples-per-second sampling rate, and the audio signal can be reconstructed with high quality in real time.

A Contactless Approach to Monitor Rail Vibrations

Continuous welded rails (CWR) are subjected to thermal effects that may lead to buckling or fracture during warm or cold seasons, respectively. The modal characteristics (frequency and mode shapes) of CWR may reveal important information about the thermal stress that can be used to prevent rail failures. The primary objective of this study is to prove a contactless method to monitor the vibration and to extract the modal characteristics of rails using a high-speed camera and advanced image processing. This study is the first step towards a general noninvasive monitoring paradigm aimed at measuring axial stress in CWR. To prove the principles of the proposed paradigm, a finite element model of an unrestrained rail segment under varying length, boundary conditions, and axial stresses was formulated. The results of the model were then used to interpret the experimental results relative to a 2.4 m-long rail subjected to compressive loading–unloading cycles. During the experiment, the rail was subjected to the impact of an instrumented hammer and the triggered vibration was recorded with a high-speed camera. The videos were then processed using the phase-based displacement extraction, motion magnification, as well as dynamic mode decomposition techniques to extract the modal characteristics of the specimen. The results show that the frequencies extracted from the images matched well those obtained with two conventional accelerometers bonded to the rail while the mode shapes extracted from the videos matched those predicted numerically. Additionally, the numerical analysis enabled the interpretation of some unexpected experimental results. The results presented here proved that the proposed method to infer axial stress in CWR requires proper modeling in order to link the modal characteristics of the rails to the axial stress. In the future, the finite element formulation presented here will be expanded to model CWR under given cross-ties and fasteners conditions in order to link the modal characteristics of the rail of interest to its axial stress.

A high performance real-time vision system for curved surface inspection

Surface quality plays an important role in inspection lines. In this paper, a novel imaging device combined with FPGA (Field Programmable Gate Array) based processing platform had been designed to detect and analyse curved surface defects for vision inspection. The optical imaging part was made by an optical device which can be used to collect curved surface features without anamorphose and a camera with 70k Hz linear CMOS was used to capture surface information. The FPGA based inspection platform had been developed for camera control and image processing. Inspecting experiments had been tested with an inspection accuracy of 0.2 mm x 0.2 mm which satisfied a 12 m/s real-time vision inspection line. This research result can be subsequently applied to various surface inspection scenarios.

A Fault-Tolerant Control Approach Based on Image Processing Applied to Three-Tank System

This paper presents an innovative and practical solution for fault-tolerant control of industrial systems based on hybrid redundant measurements. The paper proposes a fault-tolerant control scheme against single and multiple sensor faults that can occur in typical tank-based industrial applications. A robust fault detection filter based on $H_{-} /H_{\infty }$ index is designed to generate a residual signal that is robust against disturbance and sensitive to the fault, simultaneously. It also integrates camera information and image processing techniques in the control scheme to get level-measurement redundancy. The proposed scheme was tested using the well-known three-tank benchmark, with both offline and online computation. Results showed that the proposed scheme tolerated the sensor faults and preserved the correct operation of the tank levels in both single and multiple-sensor faults, without changing the original structure of the controller.

Pendeteksian Penyakit pada Daun Cabai dengan Menggunakan Metode Deep Learning

Chili is one of the most essential horticultural plants in Indonesia. In addition to the lack of supply of plants, the price of chili on the market has increased dramatically. The shortage is affected by unpredictable climate changes, which have to result in many chili plants suffering from crop failure. It was because the disease infects chili plants so that harvests are decreased. This work would incorporate Deep Learning for image processing in Disease Detection Systems. This disease detection method will be used to help users, in particular chili farmers, identify whether or not the leaves of their chili plants are contaminated with the disease. This system would take a picture of chili leaf using a Raspberry Pi camera and implement image processing on the chili leaf image to collect valuable information on the image to find out whether or not the chili leaf is contaminated with the disease. The purpose of this research is to make a desktop application for a disease detection system that has the ability to detect whether or not a chili leaf is infected by several diseases, display the condition of the chili leaves, display the type of disease that infects the chili leaves (if any), and provide a percentage probability of the system in detecting the image of the chili leaves correctly (whether it is healthy chili leaves or sick chili leaves). The system reaches 100 percent accuracy with good brightness and distance less than 1 meter, while the
 system reaches 68.8 percent accuracy with poor brightness and distance greater than or equal to 100 percent.
 
 Keywords— chili leaf; deep learning; disease detection; raspberry pi