Computer Vision Pipeline Research Articles

Incremental Learning of Hand Symbols Using Event-Based Cameras

Conventional cameras create redundant output especially when the frame rate is high. Dynamic vision sensors (DVSs), on the other hand, generate asynchronous and sparse brightness change events only when an object in the field of view is in motion. Such event-based output can be processed as a 1D time sequence, or it can be converted to 2D frames that resemble conventional camera frames. Frames created, e.g., by accumulating a fixed number of events, can be used as input for conventional deep learning algorithms, thus upgrading existing computer vision pipelines through low-power, low-redundancy sensors. This paper describes a hand symbol recognition system that can quickly be trained to incrementally learn new symbols recorded with an event-based camera, without forgetting previously learned classes. By using the iCaRL incremental learning algorithm, we show that we can learn up to 16 new symbols using only 4000 samples for each symbol and achieving a final symbol accuracy of over 80%. The system achieves latency of under 0.5s and training requires 3 minutes for 5 epochs on an NVIDIA 1080TI GPU.

Quality Inspection of Remote Radio Units Using Depth-Free Image-Based Visual Servo With Acceleration Command

In this paper, the problem of quality inspection of remote radio units (RRUs) is approached using the image-based visual servo control. A novel computer vision pipeline is designed that recognizes the power port of RRU and tracks it from the stream of images. For the control part, a new depth-independent interaction matrix is designed that relates the depth information with the area of the region of interest surrounding the power port. Based on this, an acceleration command is generated to drive the robot's trajectories. Furthermore, a propotional-derivative (PD)-type controller is designed based on the idea of sliding surface in variable structure control. This reduces the number of design parameters to a single parameter. The designed controllers are proven to be stable using the Lyapunov stability analysis. Simulation results and experimental validations are provided to support the research arguments.

Removing segmentation inconsistencies with semi-supervised non-adjacency constraint.

The advent of deep learning has pushed medical image analysis to new levels, rapidly replacing more traditional machine learning and computer vision pipelines. However segmenting and labelling anatomical regions remains challenging owing to appearance variations, imaging artifacts, the paucity and variability of annotated data, and the difficulty of fully exploiting domain constraints such as anatomical knowledge about inter-region relationships. We address the last point, improving the network's region-labeling consistency by introducing NonAdjLoss, an adjacency-graph based auxiliary training loss that penalizes outputs containing regions with anatomically-incorrect adjacency relationships. NonAdjLoss supports both fully-supervised training and a semi-supervised extension in which it is applied to unlabeled supplementary training data. The approach substantially reduces segmentation anomalies on the MICCAI-2012, IBSRv2 brain MRI datasets and the Anatomy3 whole body CT dataset, especially when semi-supervised training is included.

Photometric Long-Range Positioning of LED Targets for Cooperative Navigation in UAVs

Autonomous flight with unmanned aerial vehicles (UAVs) nowadays depends on the availability and reliability of Global Navigation Satellites Systems (GNSS). In cluttered outdoor scenarios, such as narrow gorges, or near tall artificial structures, such as bridges or dams, reduced sky visibility and multipath effects compromise the quality and the trustworthiness of the GNSS position fixes, making autonomous, or even manual, flight difficult and dangerous. To overcome this problem, cooperative navigation has been proposed: a second UAV flies away from any occluding objects and in line of sight from the first and provides the latter with positioning information, removing the need for full and reliable GNSS coverage in the area of interest. In this work we use high-power light-emitting diodes (LEDs) to signalize the second drone and we present a computer vision pipeline that allows to track the second drone in real-time from a distance up to 100 m and to compute its relative position with decimeter accuracy. This is based on an extension to the classical iterative algorithm for the Perspective-n-Points problem in which the photometric error is minimized according to a image formation model. This extension allow to substantially increase the accuracy of point-feature measurements in image space (up to 0.05 pixels), which directly translates into higher positioning accuracy with respect to conventional methods.

Automated Measurement of Lumbar Lordosis on Radiographs Using Machine Learning and Computer Vision

Study Design:Cross sectional database study.Objective:To develop a fully automated artificial intelligence and computer vision pipeline for assisted evaluation of lumbar lordosis.Methods:Lateral lumbar radiographs were used to develop a segmentation neural network (n = 629). After synthetic augmentation, 70% of these radiographs were used for network training, while the remaining 30% were used for hyperparameter optimization. A computer vision algorithm was deployed on the segmented radiographs to calculate lumbar lordosis angles. A test set of radiographs was used to evaluate the validity of the entire pipeline (n = 151).Results:The U-Net segmentation achieved a test dataset dice score of 0.821, an area under the receiver operating curve of 0.914, and an accuracy of 0.862. The computer vision algorithm identified the L1 and S1 vertebrae on 84.1% of the test set with an average speed of 0.14 seconds/radiograph. From the 151 test set radiographs, 50 were randomly chosen for surgeon measurement. When compared with those measurements, our algorithm achieved a mean absolute error of 8.055° and a median absolute error of 6.965° (not statistically significant, P > .05).Conclusion:This study is the first to use artificial intelligence and computer vision in a combined pipeline to rapidly measure a sagittal spinopelvic parameter without prior manual surgeon input. The pipeline measures angles with no statistically significant differences from manual measurements by surgeons. This pipeline offers clinical utility in an assistive capacity, and future work should focus on improving segmentation network performance.

Edge-Based Live Video Analytics for Drones

Real-time video analytics on small autonomous drones poses several difficult challenges at the intersection of wireless bandwidth, processing capacity, energy consumption, result accuracy, and timeliness of results. In response to these challenges, this paper describes four strategies to build adaptive computer vision pipelines for domains such as search-and-rescue, surveillance, and wildlife conservation. Our experimental results show that a judicious combination of drone-based processing and edge-based processing can save substantial wireless bandwidth and thus improve scalability, without compromising result accuracy or latency.

Computer vision for LCA foreground modelling\u2014an initial pipeline and proof of concept software, lcopt-cv

PurposeThe majority of LCA studies begin with the drawing of a process flow diagram, which then needs to be translated manually into an LCA model. This study presents an initial image processing pipeline, implemented in an open-source software package, called lcopt-cv, which can be used to identify the boxes and links in a photograph of a hand-drawn process flow diagram and automatically create an LCA foreground model.MethodsThe computer vision pipeline consists of a total of 15 steps, beginning with loading the image file and conversion to greyscale. The background is equalised, then the foreground of the image is extracted from the background using thresholding. The lines are then dilated and closed to account for drawing errors. Contours in the image are detected and simplified, and rectangles (contours with four corners) are identified from the simplified contours as ‘boxes’. Links between these boxes are identified using a flood-filling technique. Heuristic processing, based on knowledge of common practice in drawing of process flow diagrams, is then performed to more accurately identify the typology of the identified boxes and the direction of the links between them.Results and discussionThe performance of the image processing pipeline was tested on four flow diagrams of increasing difficulty: one simple computer drawn diagram and three photographs of hand-drawn diagrams (a simple diagram, a complex diagram and a diagram with merged lines). A set of default values for the variables which define the pipeline was developed through trial and error. For the two simple flow charts, all boxes and links were identified using the default settings. The complex diagram required minor tweaks to the default values to detect all boxes and links. An ‘unstacking’ heuristic allowed the diagram with merged lines to be correctly processed. After some manual reclassification of link directions and process types, the diagrams were turned into LCA models and exported to open-source LCA software packages (lcopt and Brightway) to be verified and analysed.ConclusionsThis study demonstrates that it is possible to generate a fully functional LCA model from a picture of a flow chart. This has potentially important implications not only for LCA practitioners as a whole, but in particular for the teaching of LCA. Skipping the steep learning curve required by most LCA software packages allows teachers to focus on important LCA concepts, while participants maintain the benefits of experiential learning by doing a ‘real’ LCA.

Abstract P4-02-11: Computer vision detects morphological correlates of HER2 positive breast cancer in H&E stained histological images

Abstract Background/Objectives: The determination of HER2-positivity by IHC or FISH is critical for identifying patients most likely to benefit from anti-HER2 therapy. However, these methods do not always provide an accurate indication of HER2 overactivity, which can occur without gene amplification or overexpression of the HER2 protein. Our study objective was to determine if a deep learning convolutional neural network (CNN) could be trained, using IHC HER2 staining, to learn a morphological signature for HER2 positivity in H&E stained slides. Methods: For training, we used H&E images (whole slides scanned at 40x) from 10 HER+ patients (IHC 3+) and 15 HER2− patients (IHC 0 or 1+) along with their adjacent HER2 IHC images. We first annotated non-cancer regions in H&E images. We then identified tumor regions in HER2 IHC that were positive (intense/complete circumferential stain) or negative (no or weak stain). We ignored any regions with equivocal IHC response as well as whole slides of IHC 2+ patients. For rigorous testing, slides from a separate set of 7 HER2+ and 19 HER2− patients were used. Digitized slides and expert consensus IHC HER2 status for each patient were provided as part of an international HER2 IHC scoring competition organized by the University of Warwick. The computer vision pipeline comprised four-stages. First, we color-normalized the H&E images to reduce unwanted color variation between slides. Second, a pre-trained neural network (NN1) marked all nuclear centroids to make it easier for subsequent stages to focus on nuclear morphology and inter-nuclear spatial arrangements. Third, a neural network (NN2) trained on sub-images of size 100x100 centered at nuclear centroids classified nuclei as non-cancer vs. cancer. Fourth, a final neural network (NN3) sub-classified cancer nuclei into HER2+ or HER−. In H&E images of held-out test patients, the percent of cancer nuclei scored as HER2+ was analyzed. Results: NN2 had a classification accuracy exceeding 97% on a validation set of 25,000 nuclei set aside from training patients, while NN3 had a validation accuracy of 88% on 7,500 test nuclei. On the set aside test patients, among the seven HER2+ patients an average of 49.8% cancer nuclei were scored HER2+, while among the 19 HER2− test patients the corresponding proportion was only 24.7% (p < 0.01). The AUC of binary classification of test patients into HER2+/− based on percent of HER2+ nuclei was 0.815. Upon closer inspection, the H&E morphology of some of the misclassified HER− patients showed visual similarity to the correctly classified HER2− patients and vice versa. Conclusions: Even when morphological patterns associated with cancer subtypes are too subtle for humans to reliably detect, H&E stained slides analyzed by CNNs may be able to geographically map a HER2 signature. It is unclear if misclassification with respect to IHC status is a reflection of morphological confusion or discordance between genomic subtype and IHC response. By training multiple neural networks to detect morphological signatures corresponding to different molecular subtypes of breast cancer, we may be able to detect and study intra-tumor heterogeneity in a cost effective and/or complementary way compared to multi-region sequencing. Citation Format: Dhage S, Anand D, Kumar N, Gann PH, Sethi A. Computer vision detects morphological correlates of HER2 positive breast cancer in H&E stained histological images [abstract]. In: Proceedings of the 2018 San Antonio Breast Cancer Symposium; 2018 Dec 4-8; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2019;79(4 Suppl):Abstract nr P4-02-11.

MODELLING BUILDING COSTS FROM 3D BUILDING MODELS – ESTIMATING THE CONSTRUCTION EFFORT FROM IMAGE-BASED SURFACE MODELS OF DRY-STONE SHEPHERD SHELTERS (KRAS, SLOVENIA)

Abstract. In the second half of the 19th and early 20th century, sheep shepherds have built dry-stone shelters all over the Slovene Kras (or Karst) region. Despite being made out of stones that are interlocked without the use of any binding material, many of these vernacular constructions survived – even though sometimes only partially – the ravages of time. The fact that over one hundred fifty shepherd shelters are currently known is mainly due to the craftsmanship of their builders and thanks to (and even despite) their present location. A majority of these stone constructions can be found in areas that are nowadays forested, thus shielding them from weather-related or anthropogenic damage (because they are difficult to spot). This paper reports on the geometric documentation of those shelters using a photogrammetric computer vision pipeline, thereby mainly focussing on the difficulties that were encountered during this process. However, such image-based modelling approaches merely yield digital three-dimensional (3D) approximations of the shelters’ surface geometry (along with some sub-optimal colour data). Although these 3D surface models might be suitable to digitally preserve vulnerable vernacular buildings to some extent, they do not magically advance our understanding of them. The second part of this article focuses, therefore, on the extraction of archaeological information from these digital 3D constructions. More specifically, the total amount of stones, the total building time and the building cost regarding caloric energy expenditure are estimated for each of the digitised shelters. Although this assessment of architectural energetics provided useful insight into the building efforts and nutrient uptake of the shepherds, it also revealed many assumptions and shortcomings that often characterise archaeological information extraction from digital 3D models of buildings.

Detecting Hand Posture in Piano Playing Using Depth Data

AbstractWe present research for automatic assessment of pianist hand posture that is intended to help beginning piano students improve their piano-playing technique during practice sessions. To automatically assess a student's hand posture, we propose a system that is able to recognize three categories of postures from a single depth map containing a pianist's hands during performance. This is achieved through a computer vision pipeline that uses machine learning on the depth maps for both hand segmentation and detection of hand posture. First, we segment the left and right hands from the scene captured in the depth map using per-pixel classification. To train the hand-segmentation models, we experiment with two feature descriptors, depth image features and depth context features, that describe the context of individual pixels' neighborhoods. After the hands have been segmented from the depth map, a posture-detection model classifies each hand as one of three possible posture categories: correct posture, low wrists, or flat hands. Two methods are tested for extracting descriptors from the segmented hands, histograms of oriented gradients and histograms of normal vectors. To account for variation in hand size and practice space, detection models are individually built for each student using support vector machines with the extracted descriptors. We validate this approach using a data set that was collected by recording four beginning piano students while performing standard practice exercises. The results presented in this article show the effectiveness of this approach, with depth context features and histograms of normal vectors performing the best.

Rain Removal in Traffic Surveillance: Does it Matter?

Varying weather conditions, including rainfall and snowfall, are generally regarded as a challenge for computer vision algorithms. One proposed solution to the challenges induced by rain and snowfall is to artificially remove the rain from images or video using rain removal algorithms. It is the promise of these algorithms that the rain-removed image frames will improve the performance of subsequent segmentation and tracking algorithms. However, rain removal algorithms are typically evaluated on their ability to remove synthetic rain on a small subset of images. Currently, their behavior is unknown on real-world videos when integrated with a typical computer vision pipeline. In this paper, we review the existing rain removal algorithms and propose a new dataset that consists of 22 traffic surveillance sequences under a broad variety of weather conditions that all include either rain or snowfall. We propose a new evaluation protocol that evaluates the rain removal algorithms on their ability to improve the performance of subsequent segmentation, instance segmentation, and feature tracking algorithms under rain and snow. If successful, the de-rained frames of a rain removal algorithm should improve segmentation performance and increase the number of accurately tracked features. The results show that a recent single-frame-based rain removal algorithm increases the segmentation performance by 19.7% on our proposed dataset, but it eventually decreases the feature tracking performance and showed mixed results with recent instance segmentation methods. However, the best video-based rain removal algorithm improves the feature tracking accuracy by 7.72%.

A simple computer vision pipeline reveals the effects of isolation on social interaction dynamics in Drosophila.

Isolation profoundly influences social behavior in all animals. In humans, isolation has serious effects on health. Drosophila melanogaster is a powerful model to study small-scale, temporally-transient social behavior. However, longer-term analysis of large groups of flies is hampered by the lack of effective and reliable tools. We built a new imaging arena and improved the existing tracking algorithm to reliably follow a large number of flies simultaneously. Next, based on the automatic classification of touch and graph-based social network analysis, we designed an algorithm to quantify changes in the social network in response to prior social isolation. We observed that isolation significantly and swiftly enhanced individual and local social network parameters depicting near-neighbor relationships. We explored the genome-wide molecular correlates of these behavioral changes and found that whereas behavior changed throughout the six days of isolation, gene expression alterations occurred largely on day one. These changes occurred mostly in metabolic genes, and we verified the metabolic changes by showing an increase of lipid content in isolated flies. In summary, we describe a highly reliable tracking and analysis pipeline for large groups of flies that we use to unravel the behavioral, molecular and physiological impact of isolation on social network dynamics in Drosophila.

LSTM-based multi-label video event detection

Since large-scale surveillance videos always contain complex visual events, how to generate video descriptions effectively and efficiently without human supervision has become mandatory. To address this problem, we propose a novel architecture for jointly recognizing multiple events in a given surveillance video, motivated by the sequence to sequence network. The proposed architecture can predict what happens in a video directly without the preprocessing of object detection and tracking. We evaluate several variants of the proposed architecture with different visual features on a novel dataset perpared by our group. Moreover, we compute a wide range of quantitative metrics to evaluate this architecture. We further compare it to the popular Support Vector Machine-based visual event detection method. The comparison results suggest that the proposal method can outperform the traditional computer vision pipelines for visual event detection.

Deep bilinear features for Her2 scoring in digital pathology

Abstract We present an automated approach for rating HER2 over-expressions in given whole-slide images of breast cancer histology slides. The slides have a very high resolution and only a small part of it is relevant for the rating. Our approach is based on Convolutional Neural Networks (CNN), which are directly modelling the whole computer vision pipeline, from feature extraction to classification, with a single parameterized model. CNN models have led to a significant breakthrough in a lot of vision applications and showed promising results for medical tasks. However, the required size of training data is still an issue. Our CNN models are pre-trained on a large set of datasets of non-medical images, which prevents over-fitting to the small annotated dataset available in our case. We assume the selection of the probe in the data with just a single mouse click defining a point of interest. This is reasonable especially for slices acquired together with another sample. We sample image patches around the point of interest and obtain bilinear features by passing them through a CNN and encoding the output of the last convolutional layer with its second-order statistics. Our approach ranked second in the Her2 contest held by the University of Warwick achieving 345 points compared to 348 points of the winning team. In addition to pure classification, our approach would also allow for localization of parts of the slice relevant for visual detection of Her2 over-expression.

Automatically tracking neurons in a moving and deforming brain.

Advances in optical neuroimaging techniques now allow neural activity to be recorded with cellular resolution in awake and behaving animals. Brain motion in these recordings pose a unique challenge. The location of individual neurons must be tracked in 3D over time to accurately extract single neuron activity traces. Recordings from small invertebrates like C. elegans are especially challenging because they undergo very large brain motion and deformation during animal movement. Here we present an automated computer vision pipeline to reliably track populations of neurons with single neuron resolution in the brain of a freely moving C. elegans undergoing large motion and deformation. 3D volumetric fluorescent images of the animal’s brain are straightened, aligned and registered, and the locations of neurons in the images are found via segmentation. Each neuron is then assigned an identity using a new time-independent machine-learning approach we call Neuron Registration Vector Encoding. In this approach, non-rigid point-set registration is used to match each segmented neuron in each volume with a set of reference volumes taken from throughout the recording. The way each neuron matches with the references defines a feature vector which is clustered to assign an identity to each neuron in each volume. Finally, thin-plate spline interpolation is used to correct errors in segmentation and check consistency of assigned identities. The Neuron Registration Vector Encoding approach proposed here is uniquely well suited for tracking neurons in brains undergoing large deformations. When applied to whole-brain calcium imaging recordings in freely moving C. elegans, this analysis pipeline located 156 neurons for the duration of an 8 minute recording and consistently found more neurons more quickly than manual or semi-automated approaches.

Fine-Grained Car Detection for Visual Census Estimation

Targeted socio-economic policies require an accurate understanding of a country’s demographic makeup. To that end, the United States spends more than 1 billion dollars a year gathering census data such as race, gender, education, occupation and unemployment rates. Compared to the traditional method of collecting surveys across many years which is costly and labor intensive, data-driven, machine learning-driven approaches are cheaper and faster—with the potential ability to detect trends in close to real time. In this work, we leverage the ubiquity of Google Street View images and develop a computer vision pipeline to predict income, per capita carbon emission, crime rates and other city attributes from a single source of publicly available visual data. We first detect cars in 50 million images across 200 of the largest US cities and train a model to predict demographic attributes using the detected cars. To facilitate our work, we have collected the largest and most challenging fine-grained dataset reported to date consisting of over 2600 classes of cars comprised of images from Google Street View and other web sources, classified by car experts to account for even the most subtle of visual differences. We use this data to construct the largest scale fine-grained detection system reported to date. Our prediction results correlate well with ground truth income data (r=0.82), Massachusetts department of vehicle registration, and sources investigating crime rates, income segregation, per capita carbon emission, and other market research. Finally, we learn interesting relationships between cars and neighborhoods allowing us to perform the first large scale sociological analysis of cities using computer vision techniques.

People occupancy detection and profiling with 3D depth sensors for building energy management

In this paper, a computational framework for occupancy detection and profiling based exclusively on depth data is presented. 3D depth sensors offer many advantages against traditional video cameras. Occupants’ privacy can be assured more effectively because depth information is unsuitable to reveal the person's identity. Notable low-level computer vision tasks can be simplified, thus lightening the computational load. The presented framework is suitable for wall-mounting setups as well as for ceiling-mounting setups, and scales well with the number of people. To take full advantage of depth data and to accommodate specificities of crowded environments, several improvements to the standard computer vision pipeline are suggested. Firstly, the running Gaussian average background model is adapted to work with depth distances in crowded scenes. Secondly, the classical complete linkage agglomerative clustering is boosted by adding edge-based constraints specifically designed for people segmentation in depth data. Thirdly, to reliable discriminate people, specific depth-based features are defined to be used with a Real AdaBoost classifier. The preliminary results achieved by using two different depth sensors and synthetic data are very promising, outperforming existing approaches. Relevant applications for building energy management, such as occupancy profiling and construction of trajectories and density maps, have been also demonstrated.

A low-cost vehicle counter for next-generation ITS

This paper describes a vehicle counter application to be used in low-cost and low-complexity devices to be deployed in next-generation pervasive Intelligent Transport Systems. The first part of the paper introduces the Linesensor theory, which exploits the temporal redundancy of the movement to enable the processing of a 1D images (i.e. lines), thus reducing the complexity for extracting features and understanding the environment. Because of the high speed of the objects to be detected, the proposed application requires a very high frame rate and consequently an optimised design for the whole computer vision pipeline. For these reasons, in the second part of the paper, we propose a low-complexity background modelling algorithm permitting to extract information related to the whole image from a single metric. Our arguments demonstrate that the proposed algorithm has comparable performance in the segmentation operation as other state-of-the-art techniques, but reducing significantly the computational cost.

Common-sense reasoning for human action recognition

This paper presents a novel method that leverages reasoning capabilities in a computer vision system dedicated to human action recognition. The proposed methodology is decomposed into two stages. First, a machine learning based algorithm – known as bag of words – gives a first estimate of action classification from video sequences, by performing an image feature analysis. Those results are afterward passed to a common-sense reasoning system, which analyses, selects and corrects the initial estimation yielded by the machine learning algorithm. This second stage resorts to the knowledge implicit in the rationality that motivates human behaviour. Experiments are performed in realistic conditions, where poor recognition rates by the machine learning techniques are significantly improved by the second stage in which common-sense knowledge and reasoning capabilities have been leveraged. This demonstrates the value of integrating common-sense capabilities into a computer vision pipeline.

PIXHAWK: A micro aerial vehicle design for autonomous flight using onboard computer vision

We describe a novel quadrotor Micro Air Vehicle (MAV) system that is designed to use computer vision algorithms within the flight control loop. The main contribution is a MAV system that is able to run both the vision-based flight control and stereo-vision-based obstacle detection parallelly on an embedded computer onboard the MAV. The system design features the integration of a powerful onboard computer and the synchronization of IMU-Vision measurements by hardware timestamping which allows tight integration of IMU measurements into the computer vision pipeline. We evaluate the accuracy of marker-based visual pose estimation for flight control and demonstrate marker-based autonomous flight including obstacle detection using stereo vision. We also show the benefits of our IMU-Vision synchronization for egomotion estimation in additional experiments where we use the synchronized measurements for pose estimation using the 2pt+gravity formulation of the PnP problem.