Joint Attention Mechanism Research Articles

Image Captioning with a Joint Attention Mechanism by Visual Concept Samples

The attention mechanism has been established as an effective method for generating caption words in image captioning; it explores one noticed subregion in an image to predict a related caption word. However, even though the attention mechanism could offer accurate subregions to train a model, the learned captioner may predict wrong, especially for visual concept words, which are the most important parts to understand an image. To tackle the preceding problem, in this article we propose Visual Concept Enhanced Captioner, which employs a joint attention mechanism with visual concept samples to strengthen prediction abilities for visual concepts in image captioning. Different from traditional attention approaches that adopt one LSTM to explore one noticed subregion each time, Visual Concept Enhanced Captioner introduces multiple virtual LSTMs in parallel to simultaneously receive multiple subregions from visual concept samples. Then, the model could update parameters by jointly exploring these subregions according to a composite loss function. Technically, this joint learning is helpful in finding the common characters of a visual concept, and thus it enhances the prediction accuracy for visual concepts. Moreover, by integrating diverse visual concept samples from different domains, our model can be extended to bridge visual bias in cross-domain learning for image captioning, which saves the cost for labeling captions. Extensive experiments have been conducted on two image datasets (MSCOCO and Flickr30K), and superior results are reported when comparing to state-of-the-art approaches. It is impressive that our approach could significantly increase BLUE-1 and F1 scores, which demonstrates an accuracy improvement for visual concepts in image captioning.

Hyperspectral Image Classification Based on 3D Convolution Joint Attention Mechanism

The high data dimension of hyperspectral images causes difficulties in the classification process.To improve the accuracy of classification,a hyperspectral image classification method using a 3D convolution joint attention mechanism is proposed.First,by pairing the center pixel with other pixels adjacent to it,it can form multiple sets of new pixel-pairs,and the neighborhood correlation between the pixels can be fully utilized.Then,the pixel-pairs are classified into the 3D convolution joint attention mechanism network framework,which can selectively learn the features in the hyperspectral image.Finally,the pixel label is obtained through the voting strategy.An experiment was carried out on two real hyperspectral image datasets.The results show that the proposed method fully exploits the spectral–spatial features of hyperspectral images,and this can effectively improve the classification accuracy.

Knowledge-Grounded Response Generation with Deep Attentional Latent-Variable Model

End-to-end dialogue generation has achieved promising results without using handcrafted features and attributes specific to each task and corpus. However, one of the fatal drawbacks in such approaches is that they are unable to generate informative utterances, so it limits their usage from some real-world conversational applications. In order to tackle this issue, this paper attempts to generate diverse and informative responses with a variational generation model, which contains a joint attention mechanism conditioning on the information from both dialogue contexts and extra knowledge. The experiments on benchmark DSTC7 data show that the proposed method generates responses with more grounded knowledge and improve the diversity of generated language.

Совместное внимание и чувствительность к ориентирующему направлению взгляда у детей с атипичным развитием

The article is devoted to the study of the relationship between the mechanism of joint attention and the subsequent normative and atypical development of the child. It is shown that the ability to establish joint attention based on the direction of the interlocutor's gaze is a precursor of the emergence of social and cognitive abilities. The study examined the differences between typical developmental children and atypical development control groups in their ability to be sensitive to social behavioral signals, namely the orientation direction of the interlocutor's gaze. On a sample of 120 children with typical development, mental retardation, hearing impairment, speech disorders and visual impairment, a number of tasks were developed to assess the ability to use the character's direction of sight in the picture to determine its intentions. The task measured such a skill of joint attention as the child’s ability to calculate and report what the other person was looking at and what he intended to choose. We also used tasks with a central hint with an explicit indication of an arrow and a peripheral hint to check the recognition of social and non-social orienting signals. Based on the comparison of contrasting groups, the variability of the formation and age-related changes in skills of joint attention, or rather the presence of differences in recognition, synthesis and interpretation orienting social information coming from eye contact. It was found that children with atpic development have a low level of "downward" joint attention. Impaired joint attention may be one of the earliest signs in children with atypical development at a later age. The results showed that deficiencies in the behavior of joint attention vary depending on mental age and level of development, a sensory defect, so the level of development can really affect the ability of children to joint attention.

Joint Attention Mechanisms for Monocular Depth Estimation With Multi-Scale Convolutions and Adaptive Weight Adjustment

Monocular depth estimation is a fundamental problem for various vision applications, and is therefore gaining increasing attention in the field of computer vision. Though a great improvement has been made thanks to the rapid progress of deep convolutional neural networks, depth estimation of the object at finer details remains an unsatisfactory issue, especially in complex scenes that has rich structure information. In this article, we proposed a deep end-to-end learning framework with the combination of multi-scale convolutions and joint attention mechanisms to tackle this challenge. Specifically, we firstly elaborately designed a lightweight up-convolution to generate multi-scale feature maps. Then we introduced an attention-based residual block to aggregate different feature maps in joint channel and spatial dimension, which could enhance the discriminant ability of feature fusion at finer details. Furthermore, we explored an effective adaptive weight adjustment strategy for the loss function to further improve the performance, which adjusts the weight of each loss term during training without additional hyper-parameters. The proposed framework was evaluated using challenging NYU Depth v2 and KITTI datasets. Experimental results demonstrated that the proposed approach is superior to most of the state-of-the-art methods.

Снижение распознавания ориентирующего значения взгляда как нарушение совместного внимания у дошкольников

The article discusses the role of the model of mental and separately the mechanism of joint attention in the normative age development of preschool children. It is shown that children with a lack of joint attention may experience difficulties in acquiring a wide range of developmental skills, and the ability to use the line of sight is part of a general mechanism that will further allow understanding and understanding the meaning of social information, the child’s ability to accumulate normal social experience. The hypothesis was verified that the developmental deficit of the mental model in children may be due to a decrease in the level of intellect: the difficulties of joint attention lead to an unformed component of the “objective-reflexive-normative thinking”. On a sample of 493 children of preschool age (typically developing and with mental retardation), a methodology was tested that assesses children’s understanding of the intentions and desires of others in the direction of gaze. It was found that the differences between typically developing children, children with lower bound and delayed age development can be tied to the fact that children who participate in joint attention can contribute to having more conditions to expand social learning opportunities. Changes in the accuracy of identifying the direction of sight shows the dynamics of the cognitive development of the child, which in comparison makes it possible to assess the characteristic differences not only in pathology, but at a decrease in the overall level of age development.

Чувствительность к ориентирующему значению направления взгляда партнера у дошкольников с задержкой психического развития

Introduction. The relevance of the study of communication between the mechanism of joint attention and the subsequent normal and abnormal development of the child has been substantiated. It has been shown that a child’s ability to initiate and react to a joint attention of an ontogenetically precedes emergence of social and cognitive abilities. It has been determined a role of the joint attention in the child's ability o integrate information on himself, on other person and ability to unite his own intentions and other person's intentions. The problem of need for solution to a question of how the cognitive development pf the child interferes with development of social experience of the child has been raised.
 Methods. Sample group was made by 514 children of preschool age, including the children with developmental disorders which characteristic of different forms of mental retardation. In the section there are descriptions of some techniques: 1. “Wrong opinion test”, “Sally-Ann”; the task “What does Charlie want?”, etc. for assessment of children's understanding of intentions, desires, the interests of others according to behavioural manifestations. 2. The developed task for assessment of ability of using the gaze direction of the character on the picture for the purpose of determination of its intentions. 3. Neuropsychological tests for a research of the block of reception and processing of information.
 Results. Potential mechanisms of integration disorder of the joint attention, decrease in recognition of the focusing value of the gaze are revealed. The variability of formation and age changes of joint attention skills, to be exact, existence of differences in recognition, synthesis and interpretation of the focusing social information proceeding from eye contact is shown. It is revealed that children with a developmental delay have a low level of the "descending" joint attention. The parallelism of early deficiency of joint attention and difficulties of processing of information is revealed.
 Discussion. The data display synchronism of cognitive development and formation of the mechanism of the joint attention. The conclusion is drawn that difficulties of integrative function of processing of social information can be combined with difficulties of synthesis of separate information signs in a Gestalt, complex information fusion, subject and symbolical orientation in space.

The Neural Mechanisms of Joint Attention

Joint attention behaviors involve sharing attention with others to an object or event by means of eye-gazing or pointing, which form the common basis for communication. There are two types of these behaviors: responding to joint attention (RJA) and initiating joint attention (IJA). RJA is the ability to follow the gaze of others, suggesting reception of a social signal from others; IJA is the ability to voluntarily direct the attention of others, to share the experience of an object or event, suggesting transmission of a social signal to others. Infants experience these roles (as signal receiver and signal transmitter) throughout the first year of life and learn social cognitive skills. Recent neuroimaging studies indicate that joint attention is supported by widely distributed neural systems with nodes in the dorsomedial prefrontal cortex, the orbitofrontal cortex and insula, the anterior and posterior cingulate cortex, the superior temporal cortex, the precuneus and parietal cortex, and the amygdala and striatum.

DeepAffinity: interpretable deep learning of compound-protein affinity through unified recurrent and convolutional neural networks.

Drug discovery demands rapid quantification of compound-protein interaction (CPI). However, there is a lack of methods that can predict compound-protein affinity from sequences alone with high applicability, accuracy and interpretability. We present a seamless integration of domain knowledges and learning-based approaches. Under novel representations of structurally annotated protein sequences, a semi-supervised deep learning model that unifies recurrent and convolutional neural networks has been proposed to exploit both unlabeled and labeled data, for jointly encoding molecular representations and predicting affinities. Our representations and models outperform conventional options in achieving relative error in IC50 within 5-fold for test cases and 20-fold for protein classes not included for training. Performances for new protein classes with few labeled data are further improved by transfer learning. Furthermore, separate and joint attention mechanisms are developed and embedded to our model to add to its interpretability, as illustrated in case studies for predicting and explaining selective drug-target interactions. Lastly, alternative representations using protein sequences or compound graphs and a unified RNN/GCNN-CNN model using graph CNN (GCNN) are also explored to reveal algorithmic challenges ahead. Data and source codes are available at https://github.com/Shen-Lab/DeepAffinity. Supplementary data are available at Bioinformatics online.

Joint Attention Mechanism for Person Re-Identification

Although person re-identification (ReID) has drawn increasing research attention due to its potential to address the problem of analysis and processing of massive monitoring data, it is very challenging to learn discriminative information when the people in the images are occluded, in large pose variations or from different perspectives. To address this problem, we propose a novel joint attention person ReID (JA-ReID) architecture. The idea is to learn two complementary feature representations by combining a soft pixel-level attention mechanism and a hard region-level attention mechanism. The soft pixel-level attention mechanism learns a discriminative embedding for the fine-grained information by exploring the salient parts in the feature maps. The hard region-level attention mechanism conducts uniform partitions on the convolutional feature maps for learning local features. We have achieved competitive results in three popular benchmarks, including Market1501, DukeMTMC-reID, and CUHK03. The experimental results verify the adaptability of the joint attention mechanism to non-rigid deformation of the human body, which can effectively improve the accuracy of ReID.

Image To Latex with DenseNet Encoder and Joint Attention

Mathematical formula structural analysis usually converts mathematical formulas in images into Latex codes. It has been named as Image2Latex by OpenAi. At present, many researchers use the model in the field of image captioning for image2latex and have achieved good results. In this paper, we propose some improvements to the baseline model which is a sequence-to-sequence model used in image caption. We improve the encoder by employing densely connected convolutional network (DenseNet) because it can strengthen feature extraction and facilitate gradient propagation. We propose to use a more effective joint attention mechanism which include both spatial attention and channel-wise attention to solve this problem. We conducted experiments on the dataset im2latex-100k. Experimental results showed that our model improved the performance of formula analysis.

Topic Aware Neural Response Generation

We consider incorporating topic information into a sequence-to-sequence framework to generate informative and interesting responses for chatbots. To this end, we propose a topic aware sequence-to-sequence (TA-Seq2Seq) model. The model utilizes topics to simulate prior human knowledge that guides them to form informative and interesting responses in conversation, and leverages topic information in generation by a joint attention mechanism and a biased generation probability. The joint attention mechanism summarizes the hidden vectors of an input message as context vectors by message attention and synthesizes topic vectors by topic attention from the topic words of the message obtained from a pre-trained LDA model, with these vectors jointly affecting the generation of words in decoding. To increase the possibility of topic words appearing in responses, the model modifies the generation probability of topic words by adding an extra probability item to bias the overall distribution. Empirical studies on both automatic evaluation metrics and human annotations show that TA-Seq2Seq can generate more informative and interesting responses, significantly outperforming state-of-the-art response generation models.

The cognitive and neural mechanisms of joint attention: a second person approach

The cognitive and neural mechanisms of joint attention: a second person approach

Language origin from simulation of language evolution: Comment on “Modeling the cultural evolution of language” by Luc Steels

The study on language evolution clarifies the process and mechanism of emergence and cultural evolution of human language. By simulating Language Games Steels [3] demonstrates that not only general features of human language such as sharing vocabularies in a population but also characteristics studied in linguistics such as case systems and agreement can emerge through cultural evolution. In the field of the evolution of language, the language origin has also been explored. It helps understanding the biological evolutionary processes of cognitive and physical mechanisms necessary for evolution of human language. Steels [3] emphasizes that general cognitive abilities are sufficient for the emergence of basic features of human language through cultural evolution. Although such findings are critically important, we cannot say that the problem of language origin is solved. The general human cognitive abilities are not the same as those of non-human animals, while there may be continuity. For example, Tomasello [4] says that joint attention including three-term relationships is necessary for human communication, but chimpanzees can understand only two-term relationships. The success of communication requires mutual understanding of intention to communicate [4], which is suggested to be fundamental for formation of human communication system [1]. Sharing vocabulary as linguistic knowledge, which is described by Naming Game simulations, is necessary but insufficient. How can we progress in the study of language origin with modeling and simulation? Simulation studies of language evolution have clarified required cognitive mechanisms for the emergence and cultural evolution of linguistic features. Steels indicates three essential functions to achieve shared vocabulary: invention, adoption, and alignment. An iterated learning model [2] demonstrates that generalization learning is insufficient and some cognitive biases are essential for unidirectional meaning change observed in grammaticalization, a characteristic language change phenomenon. Simulations demonstrate that more specific mechanisms are required for specific linguistic characters including case systems and alignment [3]. Cognitive mechanisms of joint attention have also been studied. These cognitive mechanisms and biases are the candidates of dispositions peculiar to human linguistic communication. The modeling studies including not only computer simulation but also mathematical, robotic, and experimental approaches have illustrated the conditions, dispositions, and mechanisms required for the emergence of characteristics peculiar to human language. These findings advance the exploration of language origins through simulations in more

Investigating joint attention mechanisms through spoken human–robot interaction

Referential gaze during situated language production and comprehension is tightly coupled with the unfolding speech stream ( Griffin, 2001; Meyer, Sleiderink, & Levelt, 1998; Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1995). In a shared environment, utterance comprehension may further be facilitated when the listener can exploit the speaker’s focus of (visual) attention to anticipate, ground, and disambiguate spoken references. To investigate the dynamics of such gaze-following and its influence on utterance comprehension in a controlled manner, we use a human–robot interaction setting. Specifically, we hypothesize that referential gaze is interpreted as a cue to the speaker’s referential intentions which facilitates or disrupts reference resolution. Moreover, the use of a dynamic and yet extremely controlled gaze cue enables us to shed light on the simultaneous and incremental integration of the unfolding speech and gaze movement. We report evidence from two eye-tracking experiments in which participants saw videos of a robot looking at and describing objects in a scene. The results reveal a quantified benefit-disruption spectrum of gaze on utterance comprehension and, further, show that gaze is used, even during the initial movement phase, to restrict the spatial domain of potential referents. These findings more broadly suggest that people treat artificial agents similar to human agents and, thus, validate such a setting for further explorations of joint attention mechanisms.

Grasping the other’s attention: The role of animacy in action cueing of joint attention

The current experiment investigates the role of animacy on grasp-cueing effects as investigated in joint attention research. In a simple detection task participants responded to the colour change of one of two objects of identical size. Before the target onset, we presented a cueing stimulus consisting of either two human hands with a small and a large grip aperture (animate condition) or two comparable U-shaped figures with small and large aperture (inanimate condition). Depending on the size of the objects and the arrangement of the apertures (i.e., large aperture to the left and small aperture to the right or vice versa), either the left or right object matched the grasping hand or U-shapes. Our data show that biological grasping actions modulate the observer’s attention whereas the perception of inanimate stimuli does not result in a comparable cueing effect. This strong impact of animacy on attentional priming suggests that grasp cueing represents a marker of a joint attention mechanism that involves spontaneous simulation of the observed motor behaviour.

When one sees what the other hears: Crossmodal attentional modulation for gazed and non-gazed upon auditory targets

Three experiments investigated the nature of visuo-auditory crossmodal cueing in a triadic setting: participants had to detect an auditory signal while observing another agent’s head facing one of the two laterally positioned auditory sources. Experiment 1 showed that when the agent’s eyes were open, sounds originating on the side of the agent’s gaze were detected faster than sounds originating on the side of the agent’s visible ear; when the agent’s eyes were closed this pat-tern of responses was reversed. Two additional experiments showed that the results were sensitive to whether participants could infer a hearing function on the part of the agent. When no ear was depicted on the agent, only a gaze-side advantage was observed (Experiment 2), but when the agent’s ear was covered (Experiment 3), an ear side advantage was observed only when hearing could still be inferred (i.e., wearing the hat) but not when hearing was inferred to be diminished (i.e., wearing a helmet). The findings are discussed in the context of inferential and simulation processes and joint attention mechanisms.

Is there a direct link between gaze perception and joint attention behaviours? Effects of gaze contrast polarity on oculomotor behaviour

Previous studies have found that attention is oriented in the direction of other people's gaze suggesting that gaze perception is related to the mechanisms of joint attention. However, the role of the perception of gaze direction on joint attention has been challenged. We investigated the effects of disrupting gaze perception on the orienting of observers' attention, in particular, whether orienting to gaze direction is affected by the disruptive effect of negative contrast polarity on gaze perception. A dynamic distracting gaze was presented to observers performing an endogenous saccadic task. Gaze perception was manipulated by reversing the contrast polarity between the sclera and the iris. With positive display polarity, eye movement recordings showed shorter saccadic latencies when the direction of the instructed saccade matched the direction of the distracting gaze, and a substantial number of erroneous saccades towards the direction of the perceived gaze when the latter did not match the instruction. Crucially, such effects were not found when gaze contrast polarity was reversed and gaze perception was impaired. These results extend previous studies by demonstrating the existence of a direct link between joint attention and the perception of gaze direction, and show how orienting of attention to other people's gaze can be suppressed.

Grasp cueing and joint attention

We studied how two different hand posture cues affect joint attention in normal observers. Visual targets appeared over lateralized objects, with different delays after centrally presented hand postures. Attention was cued by either hand direction or the congruency between hand aperture and object size. Participants pressed a button when they detected a target. Direction cues alone facilitated target detection following short delays but aperture cues alone were ineffective. In contrast, when hand postures combined direction and aperture cues, aperture congruency effects without directional congruency effects emerged and persisted, but only for power grips. These results suggest that parallel parameter specification makes joint attention mechanisms exquisitely sensitive to the timing and content of contextual cues.

Gaze cueing of attention: visual attention, social cognition, and individual differences.

During social interactions, people's eyes convey a wealth of information about their direction of attention and their emotional and mental states. This review aims to provide a comprehensive overview of past and current research into the perception of gaze behavior and its effect on the observer. This encompasses the perception of gaze direction and its influence on perception of the other person, as well as gaze-following behavior such as joint attention, in infant, adult, and clinical populations. Particular focus is given to the gaze-cueing paradigm that has been used to investigate the mechanisms of joint attention. The contribution of this paradigm has been significant and will likely continue to advance knowledge across diverse fields within psychology and neuroscience.