Traditional Computer Vision Techniques Research Articles

Machine learning for quality assessment of ground-based optical images of satellites

Astronomical images collected by ground-based telescopes suffer from degradation and perturbations attributed to atmospheric turbulence. We investigate the application of convolutional neural networks (CNNs) to ground-based satellite imaging to address two existing problems. First, the multiframe blind deconvolution (MFBD) algorithms that can extract well-resolved images from these degraded data frames are computationally expensive, requiring supercomputing infrastructure for relatively fast performance and currently cannot be done in real time. Because of this, it is difficult to optimize collection parameters to maximize the likelihood of producing a resolved image with MFBD. Second, the space-object National Imagery Interpretability Rating Scale (SNIIRS) allows human analysts to provide a quantitative score of image quality based on identification of target features. It is naturally difficult to automate this scoring process, not only because the scale is based on identifiable features but also because the images may be in an almost-resolved image quality regime that is difficult to handle for traditional computer vision techniques. For both applications, we present our results using CNNs on data collected at the Maui Space Surveillance Site as well as a new synthetic dataset we introduce containing over a million SNIIRS rated pairings of perturbed and pristine ground-based satellite images.

A Deep Learning Pipeline for Identification of Motor Units in Musculoskeletal Ultrasound

Ultrasound imaging provides information from a large part of the muscle. It has recently been shown that ultrafast ultrasound imaging can be used to record and analyze the mechanical response of individual MUs using blind source separation. In this work, we present an alternative method - a deep learning pipeline - to identify active MUs in ultrasound image sequences, including segmentation of their territories and signal estimation of their mechanical responses (twitch train). We train and evaluate the model using simulated data mimicking the complex activation pattern of tens of activated MUs with overlapping territories and partially synchronized activation patterns. Using a slow fusion approach (based on 3D CNNs), we transform the spatiotemporal image sequence data to 2D representations and apply a deep neural network architecture for segmentation. Next, we employ a second deep neural network architecture for signal estimation. The results show that the proposed pipeline can effectively identify individual MUs, estimate their territories, and estimate their twitch train signal at low contraction forces. The framework can retain spatio-temporal consistencies and information of the mechanical response of MU activity even when the ultrasound image sequences are transformed into a 2D representation for compatibility with more traditional computer vision and image processing techniques. The proposed pipeline is potentially useful to identify simultaneously active MUs in whole muscles in ultrasound image sequences of voluntary skeletal muscle contractions at low force levels.

Automated pig counting using deep learning

Pig counting is one of the most critical topics in farming management and asset estimation. Due to its complexity, traditional agriculture method relies on manual counting, which is obviously inefficient and a waste of manpower. The challenging aspects like partial occlusion, overlapping and different perspectives even limit the usage of traditional computer vision techniques. In recent years, deep learning has become more and more popular for computer vision applications, because of its superior performance comparing to traditional methods. In this paper, we propose a deep learning solution to address the pig counting problem. We present a modified Counting Convolutional Neural Network (Counting CNN) model according to the structure of ResNeXt, and tune a series of experimental parameters. Our CNN model learns the mapping from the image feature to the density map, and obtains the total number of pigs in the entire image by integrating the density map. In order to validate the efficacy of our proposed method, we conduct experiments on a real-world dataset collected from actual piggery farming with 15 pigs in an image averagely. We achieve 1.67 Mean Absolute Error (MAE) per image and outperforms the competing algorithms, which strongly demonstrates that our proposed method can accurately estimate the number of pigs even if they are partially occluded in different perspectives. The detection speed, 42 ms per image, meets the requirements of agricultural application. We share our code and the first pig dataset we collected for pig counting at https://github.com/xixiareone/counting-pigs for livestock husbandry and science research community.

A 3D CNN-LSTM-Based Image-to-Image Foreground Segmentation

The video-based separation of foreground (FG) and background (BG) has been widely studied due to its vital role in many applications, including intelligent transportation and video surveillance. Most of the existing algorithms are based on traditional computer vision techniques that perform pixel-level processing assuming that FG and BG possess distinct visual characteristics. Recently, state-of-the-art solutions exploit deep learning models targeted originally for image classification. Major drawbacks of such a strategy are the lacking delineation of FG regions due to missing temporal information as they segment the FG based on a single frame object detection strategy. To grapple with this issue, we excogitate a 3D convolutional neural network (3D CNN) with long short-term memory (LSTM) pipelines that harness seminal ideas, viz., fully convolutional networking, 3D transpose convolution, and residual feature flows. Thence, an FG-BG segmenter is implemented in an encoder-decoder fashion and trained on representative FG-BG segments. The model devises a strategy called double encoding and slow decoding, which fuses the learned spatio-temporal cues with appropriate feature maps both in the down-sampling and up-sampling paths for achieving well generalized FG object representation. Finally, from the Sigmoid confidence map generated by the 3D CNN-LSTM model, the FG is identified automatically by using Nobuyuki Otsu's method and an empirical global threshold. The analysis of experimental results via standard quantitative metrics on 16 benchmark datasets including both indoor and outdoor scenes validates that the proposed 3D CNN-LSTM achieves competitive performance in terms of figure of merit evaluated against prior and state-of-the-art methods. Besides, a failure analysis is conducted on 20 video sequences from the DAVIS 2016 dataset.

Emotion Recognition from Speech Using the Bag-of-Visual Words on Audio Segment Spectrograms

It is noteworthy nowadays that monitoring and understanding a human’s emotional state plays a key role in the current and forthcoming computational technologies. On the other hand, this monitoring and analysis should be as unobtrusive as possible, since in our era the digital world has been smoothly adopted in everyday life activities. In this framework and within the domain of assessing humans’ affective state during their educational training, the most popular way to go is to use sensory equipment that would allow their observing without involving any kind of direct contact. Thus, in this work, we focus on human emotion recognition from audio stimuli (i.e., human speech) using a novel approach based on a computer vision inspired methodology, namely the bag-of-visual words method, applied on several audio segment spectrograms. The latter are considered to be the visual representation of the considered audio segment and may be analyzed by exploiting well-known traditional computer vision techniques, such as construction of a visual vocabulary, extraction of speeded-up robust features (SURF) features, quantization into a set of visual words, and image histogram construction. As a last step, support vector machines (SVM) classifiers are trained based on the aforementioned information. Finally, to further generalize the herein proposed approach, we utilize publicly available datasets from several human languages to perform cross-language experiments, both in terms of actor-created and real-life ones.

Urban Traffic Density Estimation Based on Ultrahigh-Resolution UAV Video and Deep Neural Network

This paper presents an advanced urban traffic density estimation solution using the latest deep learning techniques to intelligently process ultrahigh-resolution traffic videos taken from an unmanned aerial vehicle (UAV). We first capture nearly an hour-long ultrahigh-resolution traffic video at five busy road intersections of a modern megacity by flying a UAV during the rush hours. We then randomly sampled over 17 K 512×512 pixel image patches from the video frames and manually annotated over 64 K vehicles to form a dataset for this paper, which will also be made available to the research community for research purposes. Our innovative urban traffics analysis solution consists of an advanced deep neural network (DNN) based vehicle detection and localization, type (car, bus, and truck) recognition, tracking, and vehicle counting over time. We will present extensive experimental results to demonstrate the effectiveness of our solution. We will show that our enhanced single shot multibox detector (Enhanced-SSD) outperforms other DNN-based techniques and that deep learning techniques are more effective than traditional computer vision techniques in traffic video analysis. We will also show that ultrahigh-resolution video provides more information that enables more accurate vehicle detection and recognition than lower resolution contents. This paper not only demonstrates the advantages of using the latest technological advancements (ultrahigh-resolution video and UAV), but also provides an advanced DNN-based solution for exploiting these technological advancements for urban traffic density estimation.

Deep photovoltaic nowcasting

Predicting the short-term power output of a photovoltaic panel is an important task for the efficient management of smart grids. Short-term forecasting at the minute scale, also known as nowcasting, can benefit from sky images captured by regular cameras and installed close to the solar panel. However, estimating the weather conditions from these images—sun intensity, cloud appearance and movement, etc.—is a very challenging task that the community has yet to solve with traditional computer vision techniques. In this work, we propose to learn the relationship between sky appearance and the future photovoltaic power output using deep learning. We train several variants of convolutional neural networks which take historical photovoltaic power values and sky images as input and estimate photovoltaic power in a very short term future. In particular, we compare three different architectures based on: a multi-layer perceptron (MLP), a convolutional neural network (CNN), and a long short term memory (LSTM) module. We evaluate our approach quantitatively on a dataset of photovoltaic power values and corresponding images gathered in Kyoto, Japan. Our experiments reveal that the MLP network, already used similarly in previous work, achieves an RMSE skill score of 7% over the commonly-used persistence baseline on the 1-min future photovoltaic power prediction task. Our CNN-based network improves upon this with a 12% skill score. In contrast, our LSTM-based model, which can learn the temporal dependencies in the data, achieves a 21% RMSE skill score, thus outperforming all other approaches.

Artificial Intelligence for Water Quality Monitoring

Emerging trends in Artificial Intelligence (AI) have provided a path to make better predictions in a variety of fields, including the detection of medical diseases, weather patterns, water, food quality patterns, remote sensing etc. Deep learning, a branch of AI that uses deep convolutional neural networks (CNN) modeled after the human brain, have been prominently used for accurate identification of facial features, text, and voice. This technique is used predominantly to classify a set of objects, that may not necessarily have a fixed set of features, making it very difficult to detect programmatically. As an example, researchers have started implementing deep learning techniques to identify non-uniform cancer cells from the high-resolution microscopic images of tissue samples. Similarly, most of the current screening/diagnostic devices are color-based indicators. By looking at the amount of color appearance on sensor zones of such devices, one can identify the level of certain contaminants, diseases or infections. Typically, look-up tables will be provided to classify the level of sensing parameters based on color intensity. This is a task that would be extremely time consuming and challenging to do physically, given that a complete database mapping color to concentration would have to be created and a minor difference may or may not indicate a significant concentration change. Deep learning will be very helpful for accurate identification of color and its intensity on such diagnostic devices. In the present work, we have developed an AI-based mobile application platform, that can capture the sensor image using an inbuilt smartphone camera, identify the presence of sensing parameter and classify the level of sensing parameter based on color intensity identified in the training sets on the captured image using deep CNN algorithm. As a test case, we have implemented the developed AI-based mobile application platform for water quality monitoring for bacterial contamination. We used a low-cost rapid test kit i.e., Mobile Water Kit (MWK), developed by Gunda et al. [Anal. Methods, 2014, 6, 6236-6246 and Analyst, 2016, 141, 2920-2929] for monitoring the quality of water for bacterial contamination. MWK detects indicator bacteria (E. coli) in water samples within an hour, based on the appearance of pinkish red color on the surface of the sensing area. The color intensity represents the level of bacteria in water samples. Using the AI-based mobile app, we capture the image of the MWK sensing area (after testing water samples) and classify them into E. coli present images (i.e. E. coli images) and E. coli absent images (non-E. coli images). Deep learning works very well when there is an abundance of training data and there are certain factors that will make it difficult to programmatically distinguish between types. Using traditional computer vision techniques, one would scan the colors of each concentration. However, determining the color intensity for each concentration level is very difficult (especially because these are different shades of pinkish red for MWK). Using deep learning, this is made easy as the system determines these color intensities through training sets that have been provided statistically. In this present work, we have collected training data from MWK by testing the water samples with known concentrations of E. coli bacteria and then manually segregated the captured images based on whether the sample contains E. coli or not. Then we wrote a labeling script to label these images based on their type. We then used Google Tensorflow (a deep learning Artificial Intelligence open source tool) to distinguish between E. coli and non-E. coli images. Subsequently, we used the labeling script to classify whether an MWK tested image contains E. coli or not. As of now, we can classify the images with approximately 99% accuracy. We will also able to predict concentration levels using this method.

Using deep learning to segment breast and fibroglandular tissue in MRI volumes.

Automated segmentation of breast and fibroglandular tissue (FGT) is required for various computer-aided applications of breast MRI. Traditional image analysis and computer vision techniques, such atlas, template matching, or, edge and surface detection, have been applied to solve this task. However, applicability of these methods is usually limited by the characteristics of the images used in the study datasets, while breast MRI varies with respect to the different MRI protocols used, in addition to the variability in breast shapes. All this variability, in addition to various MRI artifacts, makes it a challenging task to develop a robust breast and FGT segmentation method using traditional approaches. Therefore, in this study, we investigated the use of a deep-learning approach known as "U-net." We used a dataset of 66 breast MRI's randomly selected from our scientific archive, which includes five different MRI acquisition protocols and breasts from four breast density categories in a balanced distribution. To prepare reference segmentations, we manually segmented breast and FGT for all images using an in-house developed workstation. We experimented with the application of U-net in two different ways for breast and FGT segmentation. In the first method, following the same pipeline used in traditional approaches, we trained two consecutive (2C) U-nets: first for segmenting the breast in the whole MRI volume and the second for segmenting FGT inside the segmented breast. In the second method, we used a single 3-class (3C) U-net, which performs both tasks simultaneously by segmenting the volume into three regions: nonbreast, fat inside the breast, and FGT inside the breast. For comparison, we applied two existing and published methods to our dataset: an atlas-based method and a sheetness-based method. We used Dice Similarity Coefficient (DSC) to measure the performances of the automated methods, with respect to the manual segmentations. Additionally, we computed Pearson's correlation between the breast density values computed based on manual and automated segmentations. The average DSC values for breast segmentation were 0.933, 0.944, 0.863, and 0.848 obtained from 3C U-net, 2C U-nets, atlas-based method, and sheetness-based method, respectively. The average DSC values for FGT segmentation obtained from 3C U-net, 2C U-nets, and atlas-based methods were 0.850, 0.811, and 0.671, respectively. The correlation between breast density values based on 3C U-net and manual segmentations was 0.974. This value was significantly higher than 0.957 as obtained from 2C U-nets (P<0.0001, Steiger's Z-test with Bonferoni correction) and 0.938 as obtained from atlas-based method (P=0.0016). In conclusion, we applied a deep-learning method, U-net, for segmenting breast and FGT in MRI in a dataset that includes a variety of MRI protocols and breast densities. Our results showed that U-net-based methods significantly outperformed the existing algorithms and resulted in significantly more accurate breast density computation.

Bio-inspired classifier for road extraction from remote sensing imagery

An adaptive approach for road extraction inspired by the mechanism of primary visual cortex (V1) is proposed. The motivation is originated by the characteristics in the receptive field from V1. It has been proved that human or primate visual systems can distinguish useful cues from real scenes effortlessly while traditional computer vision techniques cannot accomplish this task easily. This idea motivates us to design a bio-inspired model for road extraction from remote sensing imagery. The proposed approach is an improved support vector machine (SVM) based on the pooling of feature vectors, using an improved Gaussian radial basis function (RBF) kernel with tuning on synaptic gains. The synaptic gains comprise the feature vectors through an iterative optimization process representing the strength and width of Gaussian RBF kernel. The synaptic gains integrate the excitation and inhibition stimuli based on internal connections from V1. The summation of synaptic gains contributes to pooling of feature vectors. The experimental results verify the correlation between the synaptic gain and classification rules, and then show better performance in comparison with hidden Markov model, SVM, and fuzzy classification approaches. Our contribution is an automatic approach to road extraction without prelabeling and postprocessing work. Another apparent advantage is that our method is robust for images taken even under complex weather conditions such as snowy and foggy weather. © 2014 SPIE.

Optimal 3D reconstruction of coronary arteries for 3D clinical assessment

A novel method is presented for 3D reconstruction of the coronary arteries. The algorithm refines the point correspondences between the arteries visible in a pair of monoplane X-ray fluoroscopy images acquired at different incidences. Traditional computer vision techniques use the RANSAC methods to discard outliers in a list of corresponding points. However, as these methods work for rigid motion primarily, we introduce a curvature constraint that relates the 2D and 3D Frenet information of the coronary artery centerlines. This constraint takes into account non-rigid movement of the arteries and also serves as a refinement tool to discard potential outlier points. Results show that for synthetic experiments with left–right anterior oblique (LAO/RAO) and posterior–left lateral (PA/LAT) viewing angles, the average 3D RMS errors are respectively 3.1 mm and 1.9 mm for the RANSAC method, and decrease to 2.8 mm and 1.1 mm by using our curvature constraint methodology. Similarly, clinical validation is performed on two datasets and the average 2D retroprojection errors are 3.34 pixels and 2.24 pixels, respectively.