Visual Recognition Tasks Research Articles

CondNet: Conditional Classifier for Scene Segmentation

The fully convolutional network (FCN) has achieved tremendous success in dense visual recognition tasks, such as scene segmentation. The last layer of FCN is typically a global classifier (1x1 convolution) to recognize each pixel to a semantic label. We empirically show that this global classifier, ignoring the intra-class distinction, may lead to sub-optimal results. In this work, we present a conditional classifier to replace the traditional global classifier, where the kernels of the classifier are generated dynamically conditioned on the input. The main advantages of the new classifier consist of: (i) it attends on the intra-class distinction, leading to stronger dense recognition capability; (ii) the conditional classifier is simple and flexible to be integrated into almost arbitrary FCN architectures to improve the prediction. Extensive experiments demonstrate that the proposed classifier performs favourably against the traditional classifier on the FCN architecture. The framework equipped with the conditional classifier (called CondNet) achieves new state-of-the-art performances on two datasets. The code and models are available at https://git.io/CondNet.

Competing Perceptual Salience in a Visual Word Recognition Task Differentially Affects Children With and Without Autism Spectrum Disorder.

Differences in visual attention have long been recognized as a central characteristic of autism spectrum disorder (ASD). Regardless of social content, children with ASD show a strong preference for perceptual salience-how interesting (i.e., striking) certain stimuli are, based on their visual properties (e.g., color, geometric patterning). However, we do not know the extent to which attentional allocation preferences for perceptual salience persist when they compete with top-down, linguistic information. This study examined the impact of competing perceptual salience on visual word recognition in 17 children with ASD (mean age 31 months) and 17 children with typical development (mean age 20 months) matched on receptive language skills. A word recognition task presented two images on a screen, one of which was named (e.g., Find the bowl!). On Neutral trials, both images had high salience (i.e., were colorful and had geometric patterning). On Competing trials, the distracter image had high salience but the target image had low salience, creating competition between bottom-up (i.e., salience-driven) and top-down (i.e., language-driven) processes. Though both groups of children showed word recognition in an absolute sense, competing perceptual salience significantly decreased attention to the target only in the children with ASD. These findings indicate that perceptual properties of objects can disrupt attention to relevant information in children with ASD, which has implications for supporting their language development. Findings also demonstrate that perceptual salience affects attentional allocation preferences in children with ASD, even in the absence of social stimuli. LAY SUMMARY: This study found that visually striking objects distract young children with autism spectrum disorder (ASD) from looking at relevant (but less striking) objects named by an adult. Language-matched, younger children with typical development were not significantly affected by this visual distraction. Though visual distraction could have cascading negative effects on language development in children with ASD, learning opportunities that build on children's focus of attention are likely to support positive outcomes.

On Generalization of Deep Neural Networks for Visual Recognition Tasks

On Generalization of Deep Neural Networks for Visual Recognition Tasks

Visual Hand Recognition in Hand Laterality and Self-Other Discrimination Tasks: Relationships to Autistic Traits and Positive Body Image.

In a study concerning visual body part recognition, a “self-advantage” effect, whereby self-related body stimuli are processed faster and more accurately than other-related body stimuli, was revealed, and the emergence of this effect is assumed to be tightly linked to implicit motor simulation, which is activated when performing a hand laterality judgment task in which hand ownership is not explicitly required. Here, we ran two visual hand recognition tasks, namely, a hand laterality judgment task and a self-other discrimination task, to investigate (i) whether the self-advantage emerged even if implicit motor imagery was assumed to be working less efficiently and (ii) how individual traits [such as autistic traits and the extent of positive self-body image, as assessed via the Autism Spectrum Quotient (AQ) and the Body Appreciation Scale-2 (BAS-2), respectively] modulate performance in these hand recognition tasks. Participants were presented with hand images in two orientations [i.e., upright (egocentric) and upside-down (allocentric)] and asked to judge whether it was a left or right hand (an implicit hand laterality judgment task). They were also asked to determine whether it was their own, or another person’s hand (an explicit self-other discrimination task). Data collected from men and women were analyzed separately. The self-advantage effect in the hand laterality judgment task was not revealed, suggesting that only two orientation conditions are not enough to trigger this motor simulation. Furthermore, the men’s group showed a significant positive correlation between AQ scores and reaction times (RTs) in the laterality judgment task, while the women’s group showed a significant negative correlation between AQ scores and differences in RTs and a significant positive correlation between BAS-2 scores and dprime in the self-other discrimination task. These results suggest that men and women differentially adopt specific strategies and/or execution processes for implicit and explicit hand recognition tasks.

A Double-Deep Spatio-Angular Learning Framework for Light Field-Based Face Recognition

Face recognition has attracted increasing attention due to its wide range of applications, but it is still challenging when facing large variations in the biometric data characteristics. Lenslet light field cameras have recently come into prominence to capture rich spatio-angular information, thus offering new possibilities for advanced biometric recognition systems. This paper proposes a double-deep spatio-angular learning framework for light field-based face recognition, which is able to model both the intra-view/spatial and inter-view/angular information using two deep networks in sequence. This is a novel recognition framework that has never been proposed in the literature for face recognition or any other visual recognition task. The proposed double-deep learning framework includes a long short-term memory (LSTM) recurrent network, whose inputs are VGG-Face descriptions, computed using a VGG-16 convolutional neural network (CNN). The VGG-Face spatial descriptions are extracted from a selected set of 2D sub-aperture (SA) images rendered from the light field image, corresponding to different observation angles. A sequence of the VGG-Face spatial descriptions is then analyzed by the LSTM network. A comprehensive set of experiments has been conducted using the IST-EURECOM light field face database, addressing varied and challenging recognition tasks. The results show that the proposed framework achieves superior face recognition performance when compared to the state of the art.

Repeated memory‐based assessments: Implications for clinical trials and practice

AbstractBackgroundDetailed longitudinal neuropsychological assessment in the Harvard Aging Brain Study (HABS) and related studies has revealed that subtle decrements in associative and semantic memory, as well as reduced practice effects for repeated items may be the earliest cognitive changes in preclinical AD. Utilizing digital, repeated, and remote assessment of these cognitive functions has the potential to increase the rapidity and ease with which these subtle changes can be detected. We previously showed that diminished practice effects on a challenging iPad‐based monthly face‐name memory paradigm was associated with neuroimaging markers of amyloid and tau in normal older adults participating in HABS. These findings inspired the development of the Boston Remote Assessment of Neurocognitive Health (BRANCH), a smartphone‐based associative and semantic memory assessment using stimuli and tasks relevant to everyday life designed for serial assessment.MethodWe describe the development, feasibility, and pilot data from a smartphone‐based memory assessment administered daily (over consecutive days) in normal older adults using their own devices. BRANCH includes 2 measures of paired associative learning (faces and names, groceries and prices), an associative memory test with facilitated encoding (categories), and a continuous visual recognition task (street signs). A total of 41 pilot participants (mean age=76.1; 64% female) completed 1 version of BRANCH in clinic and 4 versions remotely on their own mobile device for 4 consecutive days. Participants also completed standard paper and pencil testing and a questionnaire about their experience completing the task.ResultBRANCH was feasible with a total of 80% of individuals completing all assessments in the correct order. It was acceptable to participants with 64% finding the tasks at‐least somewhat to highly engaging. Older age and general cognition (e.g., MMSE) was associated with worse performance across memory tasks. Improved performance (practice effect) was observed on daily administration of paired associative memory tasks (Figure 1.).ConclusionA theoretically and evidence‐based digital memory assessment with ecologically‐valid tasks and stimuli is feasible for older adults to complete independently on their own devices. Digital assessment over multiple timepoints has the potential to drastically improve the efficiency with which cognitive performance and subtle decline can be captured.

Controversial stimuli: Pitting neural networks against each other as models of human cognition

Distinct scientific theories can make similar predictions. To adjudicate between theories, we must design experiments for which the theories make distinct predictions. Here we consider the problem of comparing deep neural networks as models of human visual recognition. To efficiently compare models’ ability to predict human responses, we synthesize controversial stimuli: images for which different models produce distinct responses. We applied this approach to two visual recognition tasks, handwritten digits (MNIST) and objects in small natural images (CIFAR-10). For each task, we synthesized controversial stimuli to maximize the disagreement among models which employed different architectures and recognition algorithms. Human subjects viewed hundreds of these stimuli, as well as natural examples, and judged the probability of presence of each digit/object category in each image. We quantified how accurately each model predicted the human judgments. The best-performing models were a generative analysis-by-synthesis model (based on variational autoencoders) for MNIST and a hybrid discriminative–generative joint energy model for CIFAR-10. These deep neural networks (DNNs), which model the distribution of images, performed better than purely discriminative DNNs, which learn only to map images to labels. None of the candidate models fully explained the human responses. Controversial stimuli generalize the concept of adversarial examples, obviating the need to assume a ground-truth model. Unlike natural images, controversial stimuli are not constrained to the stimulus distribution models are trained on, thus providing severe out-of-distribution tests that reveal the models’ inductive biases. Controversial stimuli therefore provide powerful probes of discrepancies between models and human perception.

Effects of age, word frequency, and noise on the time course of spoken word recognition.

This study assessed the effects of age, word frequency, and background noise on the time course of lexical activation during spoken word recognition. Participants (41 young adults and 39 older adults) performed a visual world word recognition task while we monitored their gaze position. On each trial, four phonologically unrelated pictures appeared on the screen. A target word was presented auditorily following a carrier phrase ("Click on the ________"), at which point participants were instructed to use the mouse to click on the picture that corresponded to the target word. High- and low-frequency words were presented in quiet to half of the participants. The other half heard the words in a low level of noise in which the words were still readily identifiable. Results showed that, even in the absence of phonological competitors in the visual array, high-frequency words were fixated more quickly than low-frequency words by both listener groups. Young adults were generally faster to fixate on targets compared to older adults, but the pattern of interactions among noise, word frequency, and listener age showed that older adults' lexical activation largely matches that of young adults in a modest amount of noise.

Individual differences among deep neural network models

Deep neural networks (DNNs) excel at visual recognition tasks and are increasingly used as a modeling framework for neural computations in the primate brain. Just like individual brains, each DNN has a unique connectivity and representational profile. Here, we investigate individual differences among DNN instances that arise from varying only the random initialization of the network weights. Using tools typically employed in systems neuroscience, we show that this minimal change in initial conditions prior to training leads to substantial differences in intermediate and higher-level network representations despite similar network-level classification performance. We locate the origins of the effects in an under-constrained alignment of category exemplars, rather than misaligned category centroids. These results call into question the common practice of using single networks to derive insights into neural information processing and rather suggest that computational neuroscientists working with DNNs may need to base their inferences on groups of multiple network instances.

Learning Domain‐Invariant and Discriminative Features for Homogeneous Unsupervised Domain Adaptation

A classifier trained on the label-rich source dataset tends to perform poorly on the unlabeled target dataset because of the distribution discrepancy across different datasets. Unsupervised domain adaptation aims to transfer knowledge from the labeled source dataset to the unlabeled target dataset to solve this problem. Most of the existing unsupervised domain adaptation methods only concentrate on learning domain-invariant features across different domains, but they neglect the discriminability of the learned features to satisfy the cluster assumption. In this paper, we propose Semantic pairwise centroid alignment (SPCA), which is a point-wise method to learn both domain-invariant and discriminative features for homogeneous unsupervised domain adaptation. SPCA utilizes a novel semantic centroid loss to reduce the intraclass distance in feature space by using source data and target High-confidence centroid points (HCCPs). Then a classifier trained on source features is expected to generalize well on target features. Extensive experiments on visual recognition tasks verify the effectiveness of the proposed SPCA and also demonstrate that both domaininvariant and discriminative features learned by SPCA can significantly boost the performance of homogeneous unsupervised domain adaptation.

MLRSNet: A multi-label high spatial resolution remote sensing dataset for semantic scene understanding

To better understand scene images in the field of remote sensing, multi-label annotation of scene images is necessary. Moreover, to enhance the performance of deep learning models for dealing with semantic scene understanding tasks, it is vital to train them on large-scale annotated data. However, most existing datasets are annotated by a single label, which cannot describe the complex remote sensing images well because scene images might have multiple land cover classes. Few multi-label high spatial resolution remote sensing datasets have been developed to train deep learning models for multi-label based tasks, such as scene classification and image retrieval. To address this issue, in this paper, we construct a multi-label high spatial resolution remote sensing dataset named MLRSNet for semantic scene understanding with deep learning from the overhead perspective. It is composed of high-resolution optical satellite or aerial images. MLRSNet contains a total of 109,161 samples within 46 scene categories, and each image has at least one of 60 predefined labels. We have designed visual recognition tasks, including multi-label based image classification and image retrieval, in which a wide variety of deep learning approaches are evaluated with MLRSNet. The experimental results demonstrate that MLRSNet is a significant benchmark for future research, and it complements the current widely used datasets such as ImageNet, which fills gaps in multi-label image research. Furthermore, we will continue to expand the MLRSNet. MLRSNet and all related materials have been made publicly available at https://data.mendeley.com/datasets/7j9bv9vwsx/1 and https://github.com/cugbrs/MLRSNet.git.

Long Short-Term Memory With Gate and State Level Fusion for Light Field-Based Face Recognition

Long Short-Term Memory (LSTM) is a prominent recurrent neural network for extracting dependencies from sequential data such as time-series and multi-view data, having achieved impressive results for different visual recognition tasks. A conventional LSTM network, hereafter referred only as LSTM network, can learn a model to posteriorly extract information from one input sequence. However, if two or more dependent sequences of data are simultaneously acquired, the LSTM networks may only process those sequences consecutively, not taking benefit of the information carried out by their mutual dependencies. In this context, this paper proposes two novel LSTM cell architectures that are able to jointly learn from multiple sequences simultaneously acquired, targeting to create richer and more effective models for recognition tasks. The efficacy of the novel LSTM cell architectures is assessed by integrating them into deep learning-based methods for face recognition with multi-view, light field images. The new cell architectures jointly learn the scene horizontal and vertical parallaxes available in a light field image, to capture richer spatio-angular information from both directions. A comprehensive evaluation, with the IST-EURECOM LFFD dataset using three challenging evaluation protocols, shows the advantage of using the novel LSTM cell architectures for face recognition over the state-of-the-art light field-based methods. These results highlight the added value of the novel cell architectures when learning from correlated input sequences.

Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision.

Deep feedforward neural network models of vision dominate in both computational neuroscience and engineering. The primate visual system, by contrast, contains abundant recurrent connections. Recurrent signal flow enables recycling of limited computational resources over time, and so might boost the performance of a physically finite brain or model. Here we show: (1) Recurrent convolutional neural network models outperform feedforward convolutional models matched in their number of parameters in large-scale visual recognition tasks on natural images. (2) Setting a confidence threshold, at which recurrent computations terminate and a decision is made, enables flexible trading of speed for accuracy. At a given confidence threshold, the model expends more time and energy on images that are harder to recognise, without requiring additional parameters for deeper computations. (3) The recurrent model’s reaction time for an image predicts the human reaction time for the same image better than several parameter-matched and state-of-the-art feedforward models. (4) Across confidence thresholds, the recurrent model emulates the behaviour of feedforward control models in that it achieves the same accuracy at approximately the same computational cost (mean number of floating-point operations). However, the recurrent model can be run longer (higher confidence threshold) and then outperforms parameter-matched feedforward comparison models. These results suggest that recurrent connectivity, a hallmark of biological visual systems, may be essential for understanding the accuracy, flexibility, and dynamics of human visual recognition.

Event-Based Visual Place Recognition With Ensembles of Temporal Windows

Event cameras are bio-inspired sensors capable of providing a continuous stream of events with low latency and high dynamic range. As a single event only carries limited information about the brightness change at a particular pixel, events are commonly accumulated into spatio-temporal windows for further processing. However, the optimal window length varies depending on the scene, camera motion, the task being performed, and other factors. In this research, we develop a novel ensemble-based scheme for combining temporal windows of varying lengths that are processed in parallel. For applications where the increased computational requirements of this approach are not practical, we also introduce a new "approximate" ensemble scheme that achieves significant computational efficiencies without unduly compromising the original performance gains provided by the ensemble approach. We demonstrate our ensemble scheme on the visual place recognition (VPR) task, introducing a new Brisbane-Event-VPR dataset with annotated recordings captured using a DAVIS346 color event camera. We show that our proposed ensemble scheme significantly outperforms all the single-window baselines and conventional model-based ensembles, irrespective of the image reconstruction and feature extraction methods used in the VPR pipeline, and evaluate which ensemble combination technique performs best. These results demonstrate the significant benefits of ensemble schemes for event camera processing in the VPR domain and may have relevance to other related processes, including feature tracking, visual-inertial odometry, and steering prediction in driving.

Simultaneous 3D hand detection and pose estimation using single depth images

Abstract In this paper, we investigate 3D hand pose estimation using single depth images. On the one hand, accurate hand localization is a crucial factor for pose estimation. On the other hand, multi-task learning methods have achieved great success in visual recognition tasks. Therefore, in this paper we proposed to simultaneously detect the hand location and estimate its 3D pose in a multi-task learning framework. We used 3D region proposal for 3D pose estimation, which searches possible hand locations in the 3D space. In the experimental part, the proposed method is evaluated on several benchmark datasets and shown it is comparable to most existing 3D hand pose estimation methods.

Single-Shot Object Detection with Split and Combine Blocks

Feature fusion is widely used in various neural network-based visual recognition tasks, such as object detection, to enhance the quality of feature representation. It is common practice for both the one-stage object detectors and the two-stage object detectors to implement feature fusion in feature pyramid networks (FPN) to enhance the capacity to detect objects of different scales. In this work, we propose a novel and efficient feature fusion unit, which is referred to as the Split and Combine (SC) Block, that splits the input feature maps into several parts, then processes these sub-feature maps with different emphasis, and finally gradually concatenates the outputs one-by-one. The SC block implicitly encourages the network to focus on features that are more important to the task, thus improving network efficiency and reducing inference computations. In order to prove our analysis and conclusions, a backbone network and an FPN employing this technique are assembled into a one-stage detector and evaluated on the MS COCO dataset. With the newly introduced SC block and other novel training tricks, our detector achieves a good speed-accuracy trade-off on COCO test-dev set, with 37.1% AP (average precision) at 51 FPS and 38.9% AP at 40 FPS.

Visual privacy-preserving level evaluation for multilayer compressed sensing model using contrast and salient structural features

Recognition and classification tasks in images or videos are ubiquitous, but they can lead to privacy issues. People increasingly hope that camera systems can record and recognize important events and objects, such as real-time recording of traffic conditions and accident scenes, elderly fall detection, and in-home monitoring. However, people also want to ensure these activities do not violate the privacy of users or others. The sparse representation classification and recognition algorithms based on compressed sensing (CS) are robust at recognizing human faces from frontal views with varying expressions and illuminations, as well as occlusions and disguises. This is a potential way to perform recognition tasks while preserving visual privacy. In this paper, an improved Gaussian random measurement matrix is adopted in the proposed multilayer CS (MCS) model to realize multiple image CS and achieve a balance between visual privacy-preserving and recognition tasks. The visual privacy-preserving level evaluation for MCS images has important guiding significance for image processing and recognition. Therefore, we propose an image visual privacy-preserving level evaluation method for the MCS model (MCS-VPLE) based on contrast and salient structural features. The basic concept is to use the contrast measurement model based on the statistical mean of the asymmetric alpha-trimmed filter and the salient generalized center-symmetric local binary pattern operator to extract contrast and salient structural features, respectively. The features are fed into a support vector regression to obtain the image quality score, and the fuzzy c-means algorithm is used for clustering to obtain the final evaluated image visual privacy-preserving score. Experiments on three constructed databases show that the proposed method has better prediction effectiveness and performance than conventional methods.

GMFAD: Towards Generalized Visual Recognition via Multilayer Feature Alignment and Disentanglement.

The deep learning based approaches which have been repeatedly proven to bring benefits to visual recognition tasks usually make a strong assumption that the training and test data are drawn from similar feature spaces and distributions. However, such an assumption may not always hold in various practical application scenarios on visual recognition tasks. Inspired by the hierarchical organization of deep feature representation that progressively leads to more abstract features at higher layers of representations, we propose to tackle this problem with a novel feature learning framework, which is called GMFAD, with better generalization capability in a multilayer perceptron manner. We first learn feature representations at the shallow layer where shareable underlying factors among domains (e.g., a subset of which could be relevant for each particular domain) can be explored. In particular, we propose to align the domain divergence between domain pair(s) by considering both inter-dimension and inter-sample correlations, which have been largely ignored by many cross-domain visual recognition methods. Subsequently, to learn more abstract information which could further benefit transferability, we propose to conduct feature disentanglement at the deep feature layer. Extensive experiments based on different visual recognition tasks demonstrate that our proposed framework can learn better transferable feature representation compared with state-of-the-art baselines.

A comprehensive survey on convolutional neural network in medical image analysis

CNN is inspired from Primary Visual (V1) neurons. It is a typical deep learning technique and can help teach machine how to see and identify objects. In the most recent decade, deep learning develops rapidly and has been well used in various fields of expertise such as computer vision and natural language processing. As the representative algorithm of deep learning, Convolution Neural Network (CNN) has been regarded as a breakthrough of historic significance in image processing and visual recognition tasks since the astonishing results achieved on ImageNet Large Scale Visual Recognition Competition (ILSVRC) Unlike methods based on handcrafted features, CNN models can build high-level features from low-level ones in a data-driven fashion and have displayed great potential in medical image analysis among the aspects of segmentation of histological images identification, lesion detection, tissue classification, etc. This paper provides a review on CNN from the perspectives of its basic mechanism introduction, structure, typical architecture and main application in medical image analysis through analyzing over 100 references from Google Scholar, PubMed, Web of Science and various sources published from 1958 to 2020.

Identification of White Matter Networks Engaged in Object (Face) Recognition Showing Differential Responses to Modulated Stimulus Strength.

Blood-oxygenation-level-dependent (BOLD) signals in magnetic resonance imaging indirectly reflect neural activity in cortex, but they are also detectable in white matter (WM). BOLD signals in WM exhibit strong correlations with those in gray matter (GM) in a resting state, but their interpretation and relationship to GM activity in a task are unclear. We performed a parametric visual object recognition task designed to modulate the BOLD signal response in GM regions engaged in higher order visual processing, and measured corresponding changes in specific WM tracts. Human faces embedded in different levels of random noise have previously been shown to produce graded changes in BOLD activation in for example, the fusiform gyrus, as well as in electrophysiological (N170) evoked potentials. The magnitudes of BOLD responses in both GM regions and selected WM tracts varied monotonically with the stimulus strength (noise level). In addition, the magnitudes and temporal profiles of signals in GM and WM regions involved in the task coupled strongly across different task parameters. These findings reveal the network of WM tracts engaged in object (face) recognition and confirm that WM BOLD signals may be directly affected by neural activity in GM regions to which they connect.