High-level Inference Research Articles

An active inference model of hierarchical action understanding, learning and imitation.

We advance a novel active inference model of the cognitive processing that underlies the acquisition of a hierarchical action repertoire and its use for observation, understanding and imitation. We illustrate the model in four simulations of a tennis learner who observes a teacher performing tennis shots, forms hierarchical representations of the observed actions, and imitates them. Our simulations show that the agent's oculomotor activity implements an active information sampling strategy that permits inferring the kinematic aspects of the observed movement, which lie at the lowest level of the action hierarchy. In turn, this low-level kinematic inference supports higher-level inferences about deeper aspects of the observed actions: proximal goals and intentions. Finally, the inferred action representations can steer imitative responses, but interfere with the execution of different actions. Our simulations show that hierarchical active inference provides a unified account of action observation, understanding, learning and imitation and helps explain the neurobiological underpinnings of visuomotor cognition, including the multiple routes for action understanding in the dorsal and ventral streams and mirror mechanisms.

Mu rhythm suppression over sensorimotor regions is associated with greater empathic accuracy.

When people encounter others’ emotions, they engage multiple brain systems, including parts of the sensorimotor cortex associated with motor simulation. Simulation-related brain activity is commonly described as a ‘low-level’ component of empathy and social cognition. It remains unclear whether and how sensorimotor simulation contributes to complex empathic judgments. Here, we combine a naturalistic social paradigm with a reliable index of sensorimotor cortex-based simulation: electroencephalography suppression of oscillatory activity in the mu frequency band. We recruited participants to watch naturalistic video clips of people (‘targets’) describing emotional life events. In two experiments, participants viewed these clips (i) with video and sound, (ii) with only video or (iii) with only sound and provided continuous ratings of how they believed the target felt. We operationalized ‘empathic accuracy’ as the correlation between participants’ inferences and targets’ self-report. In Experiment 1 (US sample), across all conditions, right-lateralized mu suppression tracked empathic accuracy. In Experiment 2 (Israeli sample), this replicated only when using individualized frequency-bands and only for the visual stimuli. Our results provide novel evidence that sensorimotor representations—as measured through mu suppression—play a role not only in low-level motor simulation, but also in higher-level inferences about others’ emotions, especially when visual cues are crucial for accuracy.

Higher-level Inferences in the Strong-Kleene Setting: A Proof-theoretic Approach

Building on early work by Girard (1987) and using closely related techniques from the proof theory of many-valued logics, we propose a sequent calculus capturing a hierarchy of notions of satisfaction based on the Strong Kleene matrices introduced by Barrio et al. (Journal of Philosophical Logic 49:93–120, 2020) and others. The calculus allows one to establish and generalize in a very natural manner several recent results, such as the coincidence of some of these notions with their classical counterparts, and the possibility of expressing some notions of satisfaction for higher-level inferences using notions of satisfaction for inferences of lower level. We also show that at each level all notions of satisfaction considered are pairwise distinct and we address some remarks on the possible significance of this (huge) number of notions of consequence.

Binocular contrast perception of gratings, noise, and natural images

The visual system is tasked with combining the two eye’s views into a single percept of the world. To understand the principles that underlie this combination, dichoptic stimuli (stimuli that have different levels of luminance and/or contrast in the two eyes) can be used to probe how each eye’s input contributes to the ultimate binocular percept. For example, contrast perception of simple dichoptic stimuli, such as sine wave gratings, is well-predicted as a weighted combination of the two eyes inputs typically biased towards the eye seeing higher contrast. We aimed to examine contrast perception of more complex dichoptic stimuli to determine how higher-order visual properties may influence the binocular percept. Using a perceptual matching task in which participants adjusted one stimulus (which was non-dichoptic) to match the perceived contrast of another stimulus (which was dichoptic), we examined the perceived contrast of three classes of stimuli: gratings, 1/f noise, and natural image patches. In addition to finding the closest perceptual match, we also asked participants to report on whether and how the two stimuli differed in appearance. We found that: 1) Spatial variation in the stimuli modulated binocular contrast perception; however, the effect depended on the stimulus type. Specifically, spatial manipulations that affected the perceptual matches for gratings (i.e., increasing or decreasing the contrast of the surrounding area in either eye) did not have a systematic effect on the matches for the noise and natural images. 2) Across all stimulus types, however, spatial manipulations did affect the proportion of trials in which participants reported finding an “exact match”. These results suggest that spatial frequency interactions and higher-level inferences can modulate both the interocular weighting and the perceptual interpretation during binocular combination.

Reinforcement Learning Based Querying in Camera Networks for Efficient Target Tracking

Surveillance camera networks are a useful monitoring infrastructure that can be used for various visual analytics applications, where high-level inferences and predictions could be made based on target tracking across the network. Most multi-camera tracking works focus on re-identification problems and trajectory association problems. However, as camera networks grow in size, the volume of data generated is humongous, and scalable processing of this data is imperative for deploying practical solutions. In this paper, we address the largely overlooked problem of scheduling cameras for processing by selecting one where the target is most likely to appear next. The inter-camera handover can then be performed on the selected cameras via re-identification or another target association technique. We model this scheduling problem using reinforcement learning and learn the camera selection policy using Q-learning. We do not assume the knowledge of the camera network topology but we observe that the resulting policy implicitly learns it. We evaluate our approach using NLPR MCT dataset, which is a real multi-camera multi-target tracking benchmark and show that the proposed policy substantially reduces the number of frames required to be processed at the cost of a small reduction in recall.

Stretch Sensor-Based Facial Expression Recognition and Classification Using Machine Learning

Sensor-based Facial expression recognition (FER) is an attractive research topic. Nowadays, FER is used for different application such as smart environments and healthcare solutions. The machine can learn human emotion by using FER technology. It is the primary and essential for quantitative analysis of human sentiments. FER is an image recognition problem within the broader field of computer vision. Face detection and tracking, reliable face recognition still present a considerable challenge for researchers in computer vision and pattern recognition. First, data processing and analytics are intensive and require a large number of computation resources and memory. Second, the fundamental technical limitation is its robustness in changes in the environment. Finally, illumination variation further complicates the design of robust algorithms because of changes in shadow casts. However, sensor-based FER overcomes all these limitations. Sensor technologies, especially low-power, wireless communication, high-capacity, and data processing have made substantial progress, making it possible for sensors to evolve from low-level data collection and transmission to high-level inference. This study aims to develop a stretchable sensor-based FER system. We use random forest machine learning algorithms used for training the FER model. Commercial stretchable facial expression dataset is simulated into the anaconda software. In this research, our stretch sensor FER dataset obtained around 95% accuracy for four different emotions (Neutral, Happy, Sad, and Disgust).

HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor.

Cameras are widely adopted for high image quality with the rapid advancement of complementary metal-oxide-semiconductor (CMOS) image sensors while offloading vision applications’ computation to the cloud. It raises concern for time-critical applications such as autonomous driving, surveillance, and defense systems since moving pixels from the sensor’s focal plane are expensive. This paper presents a hardware architecture for smart cameras that understands the salient regions from an image frame and then performs high-level inference computation for sensor-level information creation instead of transporting raw pixels. A visual attention-oriented computational strategy helps to filter a significant amount of redundant spatiotemporal data collected at the focal plane. A computationally expensive learning model is then applied to the interesting regions of the image. The hierarchical processing in the pixels’ data path demonstrates a bottom-up architecture with massive parallelism and gives high throughput by exploiting the large bandwidth available at the image source. We prototype the model in field-programmable gate array (FPGA) and application-specific integrated circuit (ASIC) for integrating with a pixel-parallel image sensor. The experiment results show that our approach achieves significant speedup while in certain conditions exhibits up to 45% more energy efficiency with the attention-oriented processing. Although there is an area overhead for inheriting attention-oriented processing, the achieved performance based on energy consumption, latency, and memory utilization overcomes that limitation.

Imprecise neural computations as a source of adaptive behaviour in volatile environments.

In everyday life, humans face environments that feature uncertain and volatile or changing situations. Efficient adaptive behaviour must take into account uncertainty and volatility. Previous models of adaptive behaviour involve inferences about volatility that rely on complex and often intractable computations. Because such computations are presumably implausible biologically, it is unclear how humans develop efficient adaptive behaviours in such environments. Here, we demonstrate a counterintuitive result: simple, low-level inferences confined to uncertainty can produce near-optimal adaptive behaviour, regardless of the environmental volatility, assuming imprecisions in computation that conform to the psychophysical Weber law. We further show empirically that this Weber-imprecision model explains human behaviour in volatile environments better than optimal adaptive models that rely on high-level inferences about volatility, even when considering biologically plausible approximations of such models, as well as non-inferential models like adaptive reinforcement learning.

A Survey on Bayesian Deep Learning

A comprehensive artificial intelligence system needs to not only perceive the environment with different “senses” (e.g., seeing and hearing) but also infer the world’s conditional (or even causal) relations and corresponding uncertainty. The past decade has seen major advances in many perception tasks, such as visual object recognition and speech recognition, using deep learning models. For higher-level inference, however, probabilistic graphical models with their Bayesian nature are still more powerful and flexible. In recent years, Bayesian deep learning has emerged as a unified probabilistic framework to tightly integrate deep learning and Bayesian models. 1 In this general framework, the perception of text or images using deep learning can boost the performance of higher-level inference and, in turn, the feedback from the inference process is able to enhance the perception of text or images. This survey provides a comprehensive introduction to Bayesian deep learning and reviews its recent applications on recommender systems, topic models, control, and so on. We also discuss the relationship and differences between Bayesian deep learning and other related topics, such as Bayesian treatment of neural networks.

Intelligent querying for target tracking in camera networks using deep Q-learning with n-step bootstrapping

Surveillance camera networks are a useful infrastructure for various visual analytics applications, where high-level inferences and predictions could be made based on target tracking across the network. Most multi-camera tracking works focus on target re-identification and trajectory association problems to track the target. However, since camera networks can generate enormous amount of video data, inefficient schemes for making re-identification or trajectory association queries can incur prohibitively large computational requirements. In this paper, we address the problem of intelligent scheduling of re-identification queries in a multi-camera tracking setting. To this end, we formulate the target tracking problem in a camera network as an MDP and learn a reinforcement learning based policy that selects a camera for making a re-identification query. The proposed approach to camera selection does not assume the knowledge of the camera network topology but the resulting policy implicitly learns it. We have also shown that such a policy can be learnt directly from data. Using the NLPR MCT and the Duke MTMC multi-camera multi-target tracking benchmarks, we empirically show that the proposed approach substantially reduces the number of frames queried.

Enhancing image retrieval for complex queries using external knowledge sources

Annotation-based image retrieval associates textual descriptions to images based on human perception. A user query, composed of keywords of choice and for retrieval, are usually matched lexically with the textual descriptions associated for stored images to extract the best matches. This paradigm will not produce appropriate desired results for complex queries if a semantic approach is not considered. This paper proposes an image retrieval framework which integrates external knowledge sources for obtaining a higher-level inference that can both handle complex queries and increase the number of relevant retrievals. The framework includes a parser where a semantic representation graph is initially generated from both image captions and query. The semantic representation of image captions is stored in the form of Resource Description Framework (RDF) triples, while the user query is translated into a SPARQL language query. For better query understanding, the external knowledge sources (ConceptNet, WordNet), are next fused together with the parser’s output in a significant process named query expansion to infer combined and expanded knowledge about the terms used in the query. Also, the expansion process generates a set of expansion rules to semantically expand the user query to adapt the inferred knowledge. The expanded query is matched against the stored RDF triplets to indicate the best matched image retrievals. Retrievals are eventually ranked using a relation similarity metric to obtain a ranked list of relevant images. Experimental studies carried on two Flickr datasets show that the proposed framework outperforms related work with 40% increase in the number of relevant retrievals at almost full accuracy. The framework achieves additionally an average increase for the accuracy at given k in the range of 50–72% for up to the tenth retrieval.

A Geolocation Analytics-Driven Ontology for Short-Term Leases: Inferring Current Sharing Economy Trends

Short-term property rentals are perhaps one of the most common traits of present day shared economy. Moreover, they are acknowledged as a major driving force behind changes in urban landscapes, ranging from established metropolises to developing townships, as well as a facilitator of geographical mobility. A geolocation ontology is a high level inference tool, typically represented as a labeled graph, for discovering latent patterns from a plethora of unstructured and multimodal data. In this work, a two-step methodological framework is proposed, where the results of various geolocation analyses, important in their own respect, such as ghost hotel discovery, form intermediate building blocks towards an enriched knowledge graph. The outlined methodology is validated upon data crawled from the Airbnb website and more specifically, on keywords extracted from comments made by users of the said platform. A rather solid case-study, based on the aforementioned type of data regarding Athens, Greece, is addressed in detail, studying the different degrees of expansion & prevalence of the phenomenon among the city’s various neighborhoods.

The Verriest Lecture: Adventures in blue and yellow.

Conventional models of color vision assume that blue and yellow (along with red and green) are the fundamental building blocks of color appearance, yet how these hues are represented in the brain and whether and why they might be special are questions that remain shrouded in mystery. Many studies have explored the visual encoding of color categories, from the statistics of the environment to neural processing to perceptual experience. Blue and yellow are tied to salient features of the natural color world, and these features have likely shaped several important aspects of color vision. However, it remains less certain that these dimensions are encoded as primary or "unique" in the visual representation of color. There are also striking differences between blue and yellow percepts that may reflect high-level inferences about the world, specifically about the colors of light and surfaces. Moreover, while the stimuli labeled as blue or yellow or other basic categories show a remarkable degree of constancy within the observer, they all vary independently of one another across observers. This pattern of variation again suggests that blue and yellow and red and green are not a primary or unitary dimension of color appearance, and instead suggests a representation in which different hues reflect qualitatively different categories rather than quantitative differences within an underlying low-dimensional "color space."

Integration of contextual-pragmatic and phonetic information in speech perception: An eye-tracking study

Pragmatic information, such as inferences regarding upcoming coreference, has been shown to influence phonetic perception (Rohde & Ettlinger, 2012). Pragmatic information, however, comes in many forms. Using a Visual World Paradigm, tracking listeners’ categorical responses and the time-course of information integration via eye movements, we investigated whether and how a different kind of pragmatic information, the contrastive function of prenominal adjectives (Sedivy, Tanenhaus, Chambers, & Carlson, 1999), can affect listeners’ perception of voicing in initial plosives. Our results suggest that the pragmatic contrast inference did not affect the behavioral judgments on phonetic categorization, but it did have an (albeit limited) influence during the online processing of voice onset time (VOT). Our findings suggest that different kinds of higher-level pragmatic inferences are not uniform in how (successfully) they are integrated with low-level phonetic properties in real-time comprehension.

From high-level inference algorithms to efficient code

Probabilistic programming languages are valuable because they allow domain experts to express probabilistic models and inference algorithms without worrying about irrelevant details. However, for decades there remained an important and popular class of probabilistic inference algorithms whose efficient implementation required manual low-level coding that is tedious and error-prone. They are algorithms whose idiomatic expression requires random array variables that are latent or whose likelihood is conjugate . Although that is how practitioners communicate and compose these algorithms on paper, executing such expressions requires eliminating the latent variables and recognizing the conjugacy by symbolic mathematics. Moreover, matching the performance of handwritten code requires speeding up loops by more than a constant factor. We show how probabilistic programs that directly and concisely express these desired inference algorithms can be compiled while maintaining efficiency. We introduce new transformations that turn high-level probabilistic programs with arrays into pure loop code. We then make great use of domain-specific invariants and norms to optimize the code, and to specialize and JIT-compile the code per execution. The resulting performance is competitive with manual implementations.

Stimulus complexity shapes response correlations in primary visual cortex

Spike count correlations (SCCs) are ubiquitous in sensory cortices, are characterized by rich structure, and arise from structured internal dynamics. However, most theories of visual perception treat contributions of neurons to the representation of stimuli independently and focus on mean responses. Here, we argue that, in a functional model of visual perception, featuring probabilistic inference over a hierarchy of features, inferences about high-level features modulate inferences about low-level features ultimately introducing structured internal dynamics and patterns in SCCs. Specifically, high-level inferences for complex stimuli establish the local context in which neurons in the primary visual cortex (V1) interpret stimuli. Since the local context differentially affects multiple neurons, this conjecture predicts specific modulations in the fine structure of SCCs as stimulus identity and, more importantly, stimulus complexity varies. We designed experiments with natural and synthetic stimuli to measure the fine structure of SCCs in V1 of awake behaving macaques and assessed their dependence on stimulus identity and stimulus statistics. We show that the fine structure of SCCs is specific to the identity of natural stimuli and changes in SCCs are independent of changes in response mean. Critically, we demonstrate that stimulus specificity of SCCs in V1 can be directly manipulated by altering the amount of high-order structure in synthetic stimuli. Finally, we show that simple phenomenological models of V1 activity cannot account for the observed SCC patterns and conclude that the stimulus dependence of SCCs is a natural consequence of structured internal dynamics in a hierarchical probabilistic model of natural images.

Model-based fMRI reveals dissimilarity processes underlying base rate neglect.

Extensive evidence suggests that people use base rate information inconsistently in decision making. A classic example is the inverse base rate effect (IBRE), whereby participants classify ambiguous stimuli sharing features of both common and rare categories as members of the rare category. Computational models of the IBRE have posited that it arises either from associative similarity-based mechanisms or from dissimilarity-based processes that may depend on higher-level inference. Here we develop a hybrid model, which posits that similarity- and dissimilarity-based evidence both contribute to the IBRE, and test it using functional magnetic resonance imaging data collected from human subjects completing an IBRE task. Consistent with our model, multivoxel pattern analysis reveals that activation patterns on ambiguous test trials contain information consistent with dissimilarity-based processing. Further, trial-by-trial activation in left rostrolateral prefrontal cortex tracks model-based predictions for dissimilarity-based processing, consistent with theories positing a role for higher-level symbolic processing in the IBRE.

The Effect of Edge Bundling and Seriation on Sensemaking of Biclusters in Bipartite Graphs.

Exploring coordinated relationships (e.g., shared relationships between two sets of entities) is an important analytics task in a variety of real-world applications, such as discovering similarly behaved genes in bioinformatics, detecting malware collusions in cyber security, and identifying products bundles in marketing analysis. Coordinated relationships can be formalized as biclusters. In order to support visual exploration of biclusters, bipartite graphs based visualizations have been proposed, and edge bundling is used to show biclusters. However, it suffers from edge crossings due to possible overlaps of biclusters, and lacks in-depth understanding of its impact on user exploring biclusters in bipartite graphs. To address these, we propose a novel bicluster-based seriation technique that can reduce edge crossings in bipartite graphs drawing and conducted a user experiment to study the effect of edge bundling and this proposed technique on visualizing biclusters in bipartite graphs. We found that they both had impact on reducing entity visits for users exploring biclusters, and edge bundles helped them find more justified answers. Moreover, we identified four key trade-offs that inform the design of future bicluster visualizations. The study results suggest that edge bundling is critical for exploring biclusters in bipartite graphs, which helps to reduce low-level perceptual problems and support high-level inferences.

Action Choice and Outcome Congruency Independently Affect Intentional Binding and Feeling of Control Judgments

Sense of agency (SoA) refers to the subjective experience that one is in control of their actions and the consequences of these actions. The SoA is a complex phenomenon, influenced by a weighted combination of various prospective (pre-movement) and retrospective (post-movement) processes and factors related to action choice, action selection fluency, action-outcome associations and higher-level inferences. In the current study, we examined the effect of the congruency between actions and outcomes in a context where the choice-level of actions was varied from 1 to 4. The actions consisted of right, left, up and down key presses while the outcomes were visual representations of the actions (i.e., right, left, up and down-pointing arrowheads). Participants performed either an instructed action or freely selected an action among two, three, or four alternatives. Each action randomly produced either a congruent or an incongruent outcome, depending on the matching between the direction of the key press and the direction of the outcome arrowhead. Participants estimated the delay between their actions and the observed outcomes and reported their feeling of control (FoC) over the outcomes. Interval estimations were used as an indirect measure of the SoA to quantify the intentional binding effect, which refers to the perceived temporal attraction between voluntary actions and their outcomes. The results showed that both intentional binding and FoC were enhanced as the choice-level was increased from 1 to 4. Additionally, intentional binding and FoC over the outcomes were stronger when actions produced congruent compared to incongruent outcomes. These results provide additional evidence that both intentional binding and FoC are sensitive to the number of action alternatives and the congruency between actions and their outcomes. Importantly, the current study suggests that these prospective and retrospective cues might independently influence intentional binding and FoC judgments.

GoDP: Globally Optimized Dual Pathway deep network architecture for facial landmark localization in-the-wild

Facial landmark localization is a fundamental module for pose-invariant face recognition. The most common approach for facial landmark detection is cascaded regression, which is composed of two steps: feature extraction and facial shape regression. Recent methods employ deep convolutional networks to extract robust features for each step, while the whole system could be regarded as a deep cascaded regression architecture. In this work, instead of employing a deep regression network, a Globally Optimized Dual-Pathway (GoDP) deep architecture is proposed to identify the target pixels through solving a cascaded pixel labeling problem without resorting to high-level inference models or complex stacked architecture. The proposed end-to-end system relies on distance-aware soft-max functions and dual-pathway proposal-refinement architecture. Results show that it outperforms the state-of-the-art cascaded regression-based methods on multiple in-the-wild face alignment databases. The model achieves 1.84 normalized mean error (NME) on the AFLW database [1], which outperforms 3DDFA [2] by 61.8%. Experiments on face identification demonstrate that GoDP, coupled with DPM-headhunter [3], is able to improve rank-1 identification rate by 44.2% compare to Dlib [4] toolbox on a challenging database.