Ensemble Coding Research Articles

Ensemble averaging: What can we learn from skewed feature distributions?

Many studies have shown that observers can accurately estimate the average feature of a group of objects. However, the way the visual system relies on the information from each individual item is still under debate. Some models suggest some or all items sampled and averaged arithmetically. Another strategy implies "robust averaging," when middle elements gain greater weight than outliers. One version of a robust averaging model was recently suggested by Teng et al. (2021), who studied motion direction averaging in skewed feature distributions and found systematic biases toward their modes. They interpreted these biases as evidence for robust averaging and suggested a probabilistic weighting model based on minimization of the virtual loss function. In four experiments, we replicated systematic skew-related biases in another feature domain, namely, orientation averaging. Importantly, we show that the magnitude of the bias is not determined by the locations of the mean or mode alone, but is substantially defined by the shape of the whole feature distribution. We test a model that accounts for such distribution-dependent biases and robust averaging in a biologically plausible way. The model is based on well-established mechanisms of spatial pooling and population encoding of local features by neurons with large receptive fields. Both the loss functions model and the population coding model with a winner-take-all decoding rule accurately predicted the observed patterns, suggesting that the pooled population response model can be considered a neural implementation of the computational algorithms of information sampling and robust averaging in ensemble perception.

InfoMax Classification-Enhanced Learnable Network for Few-Shot Node Classification

Graph neural networks have a wide range of applications, such as citation networks, social networks, and knowledge graphs. Among various graph analyses, node classification has garnered much attention. While many of the recent network embedding models achieve promising performance, they usually require sufficient labeled nodes for training, which does not meet the reality that only a few labeled nodes are available in novel classes. While few-shot learning is commonly employed in the vision and language domains to address the problem of insufficient training samples, there are still two characteristics of the few-shot node classification problem in the non-Euclidean domain that require investigation: (1) how to extract the most informative knowledge for a class and use it on testing data and (2) how to thoroughly explore the limited number of support sets and maximize the amount of information transferred to the query set. We propose an InfoMax Classification-Enhanced Learnable Network (ICELN) to address these issues, motivated by Deep Graph InfoMax (DGI), which adapts the InfoMax principle to the summary representation of a graph and the patch representation of a node. By increasing the amount of information that is shared between the query nodes and the class representation, an ICELN can transfer the maximum amount of information to unlabeled data and enhance the graph representation potential. The whole model is trained using an episodic method, which simulates the actual testing environment to ensure the meta-knowledge learned from previous experience may be used for entirely new classes that have not been studied before. Extensive experiments are conducted on five real-world datasets to demonstrate the advantages of an ICELN over the existing few-shot node classification methods.

Effects of temporal properties of facial expressions on the perceived intensity of emotion.

A series of multiple facial expressions can be temporally perceived as an averaged facial expression in a process known as ensemble perception. This study examined the effect of temporal parameters on the perceived intensity of facial expression in each emotion, and how the effect varies with autistic traits in typically developing people. In the experiment, we presented facial expressions that switched from emotional to neutral expressions, and vice versa, for 3 s. Participants rated the overall perceived intensity of the facial emotions as a whole rather than rating individual items within the set. For the two tasks, a ratio of duration of emotional faces to duration of neutral faces (emotional ratio) and the timing for transitions were manipulated individually. The results showed that the intensity of facial emotion was perceived more strongly when the presentation ratio increased and when the emotional expression was presented last. The effects were different among the emotions (e.g. relatively weak in the anger expression). Moreover, the perceived intensity of angry expressions decreased with autistic traits. These results suggest that the properties and individual differences in the facial ensemble of each emotion affect emotional perceptions.

Abstractive Summarization for Video: A Revisit in Multistage Fusion Network With Forget Gate

Multimodal abstractive summarization for videos is an emerging task, aiming to generate a summary from multi-source information (i.e. video, audio transcript). The challenge is how to merge multimodal long sequences to capture rich semantic information without allowing possible noise from either lengthy modal sequence to degrade the other modality and thus hurt the entire model. To address the issues, we propose a multistage fusion network with forget gate (MFFG), which selectively integrates multi-source information through the cross-fusion in encoding and hierarchical fusion in decoding between modalities, and design a fusion forget gate module to suppress the flow of potential multimodal noise of multi-source long sequence. Meanwhile, considering that the source text in this task is lengthy and has the same distribution as the output summary text, we inherit the partial structure of the MFFG model and again propose its variant, single-stage fusion network with forget gate (SFFG), which simplifies the fusion schema, and leverages the long source text to enhance the representation of the target summary. Experimental results on How2 dataset and How2-300 dataset demonstrate the superiority of the two multimodal fusion methods. Further, we provide a version of ASR transcription data of How2 dataset to evaluate model performance under noisy scenarios, and experimental results show obvious advantages of our proposed models over prior systems.

Pulkovo Airport Terminal in Leningrad is a Masterpiece of Domestic Modernism

This article is devoted to the Pulkovo airport terminal complex in Leningrad – Saint-Petersburg. The article offers comprehensive research on the architectural project (the project leader A.V. Zhuk) and its implementation as an architectural phenomenon in a sociocultural context. The research methodology is based on historical, chronological, structural, functional, and semiotic approaches. Based on historical and cultural analysis of the transformation and improvement of the project during the period of its realisation, the article shows a few innovative functions of the airport, besides its primary technological function, such as: representative function; memorial function, preserving results of the creative team's work and builders work; artistic function within the ensemble perception of the project. The article also shows the project's dynamic of development and enhancement throughout fulfilled work on it. The article illustrates the design solutions embodied in full in the construction, reflects the scale of construction and improvement of professional methods and techniques. The author also reviewed and analysed texts and documents contemporary to the architectural conception and its implementation. Based on the research, the author makes a conclusion about the Pulkovo airport being an object, perfect by its function and artistic image, representing a synthesis of innovative methods, taking landscape features into account, and translating the author's cultural experience. The relevance of the preservation of the airport is related to its characteristics, which are unique for world architecture and Saint-Petersburg's environment. The scientific novelty of the research lies in the statement about the building of Pulkovo airport being a perfect example of an embodiment of rationalism ideas in conjunction with the creation of vivid artistic imagery. Also, the thesis is presented, about the creative team successfully combining bold technological innovations with a humanistic focus of the functional object.

Color diversity judgments in peripheral vision: Evidence against "cost-free" representations.

Is visual perception "rich" or "sparse?" One finding supporting the "rich" hypothesis shows that a specific visual summary representation, color diversity, is represented "cost-free" outside focally-attended regions in dual-task paradigms [1]. Here, we investigated whether this "cost-free" phenomenon for color diversity perception extends to peripheral vision. After replicating previous findings and verifying that color diversity is represented "cost-free" in central vision, we performed two experiments: in our first experiment, we extended the paradigm to peripheral vision and found that in minimally-attended regions of space, color diversity perception was impaired. In a second and final experiment, we added confidence judgments to our task, and found that participants maintained high levels of metacognitive awareness of impaired performance in minimally-attended visual areas in the periphery. These findings provide evidence that color perception may be partially attention-dependent in peripheral vision, and challenge previous views on both sides of the rich vs. sparse debate.

The Relation between Ensemble Coding of Length and Orientation Does Not Depend on Spatial Attention

Most people are good at estimating summary statistics for different features of groups of objects. For instance, people can selectively attend to different features of a group of lines and report ensemble properties such as the mean length or mean orientation and there are reliable individual differences in such ensemble judgment abilities. Our recent study found decisive evidence in support of a correlation between the errors on mean length and mean orientation judgments (r = 0.62). The present study investigates one possible mechanism for this correlation. The ability to allocate spatial attention to single items varies across individuals, and in the recent study, this variability could have contributed to both judgments because the location of lines was unpredictable. Here, we replicate this prior work with arrays of lines with fully predictable spatial locations, to lower the contribution of the ability to distribute attention effectively over all items in a display. We observed a strong positive correlation between errors on the length and orientation averaging tasks (r = 0.65). This provides evidence against individual differences in spatial attention as a common mechanism supporting mean length and orientation judgments. The present result aligns with the growing evidence for at least one ensemble-specific ability that applies across different kinds of features and stimuli.

Historical Floods on the Spanish Mediterranean Basin: A Methodological Proposal for the Classification of Information at High Spatio–Temporal Resolution—AMICME Database (CE 1035–2022)

The uncertainties raised by climate change in the Mediterranean region justify a detailed study on rainfall regime over a long time scale. This study is not just about reconstructing and characterizing climate variability, but also identifying the hydrometeorological episodes of greater severity and low frequency. These phenomena, such as floods and droughts, affect the availability of water resources, generate casualties and have significant impacts on human activity. In this context of global warming, knowledge of past extreme hydrometeorological events has gained importance over time. Moreover, the ability to classify these past extreme events in order to compare them with current ones strengthens research capabilities. For this reason, this paper proposes a multivariable classification system for flood episodes, both historical and recent. The classification considers the physical conditions of a flood, its damage to structures and its social impacts. The proposed methodology has been applied to the AMICME flood database for the Spanish Mediterranean catchment areas from CE 1035 to 2022. This paper shows the application of this methodology for six flood episodes from different centuries and locations. It has been possible to display all the indexed information for the different flood cases of every episode through cartographic summary representations. Results prove the adaptability of the classification system to different hydrographical and historical contexts.

Auto-regressive extractive summarization with replacement

Auto-regressive extractive summarization approaches determine sentence extraction probability conditioning on previous decisions by maintaining a partial summary representation. Despite its popularity, the framework has two main drawbacks: 1) the partial summary representation is irresolutely denoted by a weighted summation of all the processed sentences without any filtering, resulting in a noisy representation and degrading the effectiveness of extracting subsequent sentences; 2) earlier sentences are biased towards a higher extraction probability due to the sequential nature of sequence tagging. To address these two problems, we propose the Auto-regressive Extractive Summarization with Replacement (AES-Rep), a novel auto-regressive extractive summarization model. In particular, the AES-Rep model consists of two main modules: the extraction decision module that determines whether a sentence should be extracted, and the replacement locater module that enables extracted deficient sentences to be replaced with latter sentences by comparing their expressiveness with respect to the main idea of the document. These modules update the partial summary with explicit actions using elaborated multidimensional guidance. We conduct extensive experiments on the benchmark CNN and DailyMail datasets. Experimental results show that AES-Rep can achieve better performance compared with various strong baselines in terms of multiple ROUGE metrics.

Ensemble perception, categorization, and high-level visual search

Introduction. 1- Categorization is a fundamental sensory-motor ability facilitating rapid behavioral responses to complex situations. It is recognition of shared features or similarities between objects, events, or ideas, and their organization into associated abstract groups. Categorization allows responding similarly to category members: chairs are for sitting; food for eating; friends for approaching; foes for avoiding. 2- High-level visual search tests categorization, since it is based on requesting observers to find exemplars of named categories among heterogeneous objects. 3- Ensemble perception is the ability to efficiently represent groups of similar objects by their summary statistics. Studies found that observers are better at perceiving the mean of a set of images than remembering the presence of a particular member of the set. Observers also automatically and implicitly extract averages of sets of images, on-the-fly, on a trial-by-trial basis. Note that there is an inherent paradox in both categorization and ensemble perception: category exemplars and ensemble images share sufficient features to allow recognizing them as category or set members, but each one is different, or we would have a useless tautology. Methods. We now use the “individual differences” paradigm, asking if categorization, high-level visual search, and ensemble perception depend on the same or different underlying mechanisms. If separate mechanisms exist, then Nature’s manipulations (including in neurotypically-developed participants and those on the autism spectrum) may lead some individuals to have particularly sensitive mechanisms for one, and other individuals for the other. Crucially, performance subserved by different mechanisms should predict each other less than performance subserved by the same mechanism. Results. We find correlated performance for face detection and basic-level ensemble perception, but uncorrelated performance for complex categorization and ensemble perception or face detection. Conclusions. These results have important implications for placing these perceptual mechanisms at low-levels versus high-levels of the visual hierarchy.

Does attention influence who you follow in a crowd? Tracking neighbors vs. following your friends

Does attention increase the influence of neighbors in a crowd? Rio, Dachner, & Warren’s (2018) model assumes ensemble perception of crowd motion, and yet it seems that we can follow our friends and ignore others. We adapted the Multiple Object Tracking paradigm to test whether selective attention to a subset of the crowd spontaneously influences following behavior. In Experiment 1, the participant walked with a virtual crowd 9 neighbors, viewed in an HMD. At the beginning of each trial, a target set of 2-6 neighbors, or all 9 neighbors (control), blinked on and off; participants were instructed to attend to and track those neighbors as they walked with the crowd. The heading (walking direction) of targets or distractors was perturbed (±20°) midway through the trial. As a tracking check, a randomized memory probe appeared at the end of the trial. Participant heading, speed, facing direction (yaw angle) and probe responses were recorded. A linear mixed effects regression revealed that participant heading was influenced by the number of turning neighbors (χ2(1) = 17.19, p < 0.001), but not by attention to targets. Rather, participant heading was statistically equivalent to the average heading direction of the crowd (BF01 = 11.90), consistent with ensemble perception. Probe responses were highly accurate, indicating that participants successfully tracked targets. In Experiment 2, we tested whether participants can follow a group of “friends” when explicitly instructed to do so. Participants were first trained to recognize a set of avatar friends, which replaced blinking targets; otherwise the design was the same. In this case, participants fully segmented friends from a crowd of strangers and exclusively followed them (χ2(1) = 59.27, p<0.001), breaking ensemble perception. Subsequent experiments are investigating whether this “friend” effect is due to task-relevant attention (explicit instructions to follow targets) or grouping by familiarity.

Face Masks: Implications for Identity Face Processing in Ensemble Perception.

Face Masks: Implications for Identity Face Processing in Ensemble Perception.

Ensemble perception explained: A population response model of ensemble perception

Ensemble perception explained: A population response model of ensemble perception

Hidden by letters: How grouping lines into letters interferes with ensemble perception

Hidden by letters: How grouping lines into letters interferes with ensemble perception

Multisensory integration of orally-sourced gustatory and olfactory inputs to the posterior piriform cortex in awake rats.

Flavour refers to the sensory experience of food, which is a combination of sensory inputs sourced from multiple modalities during consumption, including taste and odour. Previous work has demonstrated that orally-sourced taste and odour cues interact to determine perceptual judgements of flavour stimuli, although the underlying cellular- and circuit-level neural mechanisms remain unknown. We recently identified a region of the piriform olfactory cortex in rats that responds to both taste and odour stimuli. Here, we investigated how converging taste and odour inputs to this area interact to affect single neuron responsiveness ensemble coding of flavour identity. To accomplish this, we recorded spiking activity from ensembles of single neurons in the posterior piriform cortex (pPC) in awake, tasting rats while delivering taste solutions, odour solutions and taste+odour mixtures directly into the oral cavity. Our results show that taste and odour inputs evoke highly selective, temporally-overlapping responses in multisensory pPC neurons. Comparing responses to mixtures and their unisensory components revealed that taste and odour inputs interact in a non-linear manner to produce unique response patterns. Taste input enhances trial-by-trial decoding of odour identity from small ensembles of simultaneously recorded neurons. Together, these results demonstrate that taste and odour inputs to pPC interact in complex, non-linear ways to form amodal flavour representations that enhance identity coding. KEY POINTS: Experience of food involves taste and smell, although how information from these different senses is combined by the brain to create our sense of flavour remains unknown. We recorded from small groups of neurons in the olfactory cortex of awake rats while they consumed taste solutions, odour solutions and taste+odour mixtures. Taste and smell solutions evoke highly selective responses. When presented in a mixture, taste and smell inputs interacted to alter responses, resulting in activation of unique sets of neurons that could not be predicted by the component responses. Synergistic interactions increase discriminability of odour representations. The olfactory cortex uses taste and smell to create new information representing multisensory flavour identity.

Temporal dynamics of neural ensemble coding of remembered target location in the primate prefrontal cortex

Temporal dynamics of neural ensemble coding of remembered target location in the primate prefrontal cortex

Limitations in the flexibility of multisensory ensemble coding

Limitations in the flexibility of multisensory ensemble coding

A General Ability for Ensemble Perception

A General Ability for Ensemble Perception

Memory matching features bias the ensemble perception of facial identity.

Humans have the ability to efficiently extract summary statistics (i.e., mean) from a group of similar objects, referred to as ensemble coding. Recent studies have demonstrated that ensemble perception of simple objects is modulated by the visual working memory (VWM) task through matching features in VWM. However, few studies have examined the extending scope of such a matching feature effect and the influence of the organization mode (i.e., the way of combining memory matching features with ensemble properties) on this effect. Two experiments were done to explore these questions. We used a dual-task paradigm for both experiments, which included a VWM task and a mean estimation task. Participants were required to adjust a test face to the mean identity face and report whether the irregular objects in a memory probe were identical or different to the studied objects. In Experiment 1, using identity faces as ensemble stimuli, we compared participants' performances in trials where a subset color matched that of the studied objects to those of trials without color-matching subsets. In Experiment 2, we combined memory matching colors with ensemble properties in common region cues and compared the effect with that of Experiment 1. Results of Experiments 1 and 2 showed an effect of the VWM task on high-level ensemble perception that was similar to previous studies using a low-level averaging task. However, the combined analysis of Experiments 1 and 2 revealed that memory matching features had less influence on mean estimations when matching features and ensemble properties combined in the common region than when combined as parts of a complete unit. These findings suggest that the impact of memory matching features is not limited by the level of stimulus feature, but can be impacted by the organization between matching features and ensemble target properties.

The relationship between ensemble representations of facial information

People accurately evaluate various types of facial information (gaze direction, facial expression, facial identity, and gender) of multiple faces. Considering such varieties, summarizing abilities of facial information might vary depending on its type because it is either changeable (e.g., gaze direction and expression) or invariant (e.g., identity and gender). The current study investigated the relationship between the averaging abilities of facial information using an individual difference approach and a dual-task paradigm to understand the effect of facial information type on the ensemble coding of facial information. We conducted two online experiments on the relationship between the averaging abilities of facial expressions and gaze direction (Experiment 1) and those of facial expressions and gender (Experiment 2). Participants were asked to estimate the average of each piece of facial information in the first and second blocks (single task), respectively, and both sequentially in the third and fourth blocks (dual task). We found that most of the error autocorrelations of facial information were high, indicating high measurement reliability. Participants’ abilities to average facial expressions were correlated with those to average gaze directions, but not with those to average gender information. That is, the ensemble processing of facial expressions is related to gaze directions, but not genders. These results suggest that ensemble representations of facial information regarding changeable properties differ from those of invariant ones.