Contextual Modules Research Articles

Cross-Sentence Gloss Consistency for Continuous Sign Language Recognition

Continuous sign language recognition (CSLR) aims to recognize gloss sequences from continuous sign videos. Recent works enhance the gloss representation consistency by mining correlations between visual and contextual modules within individual sentences. However, there still remain much richer correlations among glosses across different sentences. In this paper, we present a simple yet effective Cross-Sentence Gloss Consistency (CSGC), which enforces glosses belonging to a same category to be more consistent in representation than those belonging to different categories, across all training sentences. Specifically, in CSGC, a prototype is maintained for each gloss category and benefits the gloss discrimination in a contrastive way. Thanks to the well-distinguished gloss prototype, an auxiliary similarity classifier is devised to enhance the recognition clues, thus yielding more accurate results. Extensive experiments conducted on three CSLR datasets show that our proposed CSGC significantly boosts the performance of CSLR, surpassing existing state-of-the-art works by large margins (i.e., 1.6% on PHOENIX14, 2.4% on PHOENIX14-T, and 5.7% on CSL-Daily).

Improving Text Recognition in Natural Images Using Contextual Modules and Transformer-based Decoding

Improving Text Recognition in Natural Images Using Contextual Modules and Transformer-based Decoding

EPMF: Efficient Perception-Aware Multi-Sensor Fusion for 3D Semantic Segmentation.

We study multi-sensor fusion for 3D semantic segmentation that is important to scene understanding for many applications, such as autonomous driving and robotics. For example, for autonomous cars equipped with RGB cameras and LiDAR, it is crucial to fuse complementary information from different sensors for robust and accurate segmentation. Existing fusion-based methods, however, may not achieve promising performance due to the vast difference between the two modalities. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) to effectively exploit perceptual information from two modalities, namely, appearance information from RGB images and spatio-depth information from point clouds. To this end, we first project point clouds to the camera coordinate using perspective projection. In this way, we can process both inputs from LiDAR and cameras in 2D space while preventing the information loss of RGB images. Then, we propose a two-stream network that consists of a LiDAR stream and a camera stream to extract features from the two modalities, separately. The extracted features are fused by effective residual-based fusion modules. Moreover, we introduce additional perception-aware losses to measure the perceptual difference between the two modalities. Last, we propose an improved version of PMF,i.e., EPMF, which is more efficient and effective by optimizing data pre-processing and network architecture under perspective projection. Specifically, we propose cross-modal alignment and cropping to obtain tight inputs and reduce unnecessary computational costs. We then explore more efficient contextual modules under perspective projection and fuse the LiDAR features into the camera stream to boost the performance of the two-stream network. Extensive experiments on benchmark data sets show the superiority of our method. For example, on nuScenes test set, our EPMF outperforms the state-of-the-art method,i.e., RangeFormer, by 0.9% in mIoU. Compared to PMF, EPMF also achieves 2.06× acceleration with 2.0% improvement in mIoU. Our source code is available at https://github.com/ICEORY/PMF.

Sparse Coding Inspired LSTM and Self-Attention Integration for Medical Image Segmentation.

Accurate and automatic segmentation of medical images plays an essential role in clinical diagnosis and analysis. It has been established that integrating contextual relationships substantially enhances the representational ability of neural networks. Conventionally, Long Short-Term Memory (LSTM) and Self-Attention (SA) mechanisms have been recognized for their proficiency in capturing global dependencies within data. However, these mechanisms have typically been viewed as distinct modules without a direct linkage. This paper presents the integration of LSTM design with SA sparse coding as a key innovation. It uses linear combinations of LSTM states for SA's query, key, and value (QKV) matrices to leverage LSTM's capability for state compression and historical data retention. This approach aims to rectify the shortcomings of conventional sparse coding methods that overlook temporal information, thereby enhancing SA's ability to do sparse coding and capture global dependencies. Building upon this premise, we introduce two innovative modules that weave the SA matrix into the LSTM state design in distinct manners, enabling LSTM to more adeptly model global dependencies and meld seamlessly with SA without accruing extra computational demands. Both modules are separately embedded into the U-shaped convolutional neural network architecture for handling both 2D and 3D medical images. Experimental evaluations on downstream medical image segmentation tasks reveal that our proposed modules not only excel on four extensively utilized datasets across various baselines but also enhance prediction accuracy, even on baselines that have already incorporated contextual modules. Code is available at https://github.com/yeshunlong/SALSTM.

Real-time Context-Aware Multimodal Network for Activity and Activity-Stage Recognition from Team Communication in Dynamic Clinical Settings.

In clinical settings, most automatic recognition systems use visual or sensory data to recognize activities. These systems cannot recognize activities that rely on verbal assessment, lack visual cues, or do not use medical devices. We examined speech-based activity and activity-stage recognition in a clinical domain, making the following contributions. (1) We collected a high-quality dataset representing common activities and activity stages during actual trauma resuscitation events-the initial evaluation and treatment of critically injured patients. (2) We introduced a novel multimodal network based on audio signal and a set of keywords that does not require a high-performing automatic speech recognition (ASR) engine. (3) We designed novel contextual modules to capture dynamic dependencies in team conversations about activities and stages during a complex workflow. (4) We introduced a data augmentation method, which simulates team communication by combining selected utterances and their audio clips, and showed that this method contributed to performance improvement in our data-limited scenario. In offline experiments, our proposed context-aware multimodal model achieved F1-scores of 73.2±0.8% and 78.1±1.1% for activity and activity-stage recognition, respectively. In online experiments, the performance declined about 10% for both recognition types when using utterance-level segmentation of the ASR output. The performance declined about 15% when we omitted the utterance-level segmentation. Our experiments showed the feasibility of speech-based activity and activity-stage recognition during dynamic clinical events.

Fully Contextual Network for Hyperspectral Scene Parsing

In this article, we propose fully contextual networks (FullyContNets) for hyperspectral scene parsing. Different from the previous approaches that leveraging the local information, the proposed methods can effectively capture the more generic nonlocal contexts. To this end, we first propose the scale attention module (SAM) that can adaptively aggregate the multiple features through obtaining the interfeature dependencies of multiscale with self-attention mechanism, where the weights are determined by measuring the similarity between features. What is more, two fully contextual modules (FCMs) called pyramid fully contextual module (Pyramid-FCM) and atrous spatial pyramid fully contextual module (ASP-FCM) are separately developed to obtain the contextual information that simultaneously lying across positions, channels, and features when combining the intrafeature information aggregation algorithms with SAM on the foundation of existing multiscale modules, such as pyramid pooling (PP) in PSPNet and atrous spatial pyramid pooling (ASPP) in DeeplabV3. We design four schemes for FCMs to obtain more effective contexts. The corresponding FullyContNet-Pyramid and FullyContNet-ASP are separately constructed based on the Pyramid-FCM or ASP-FCM. There are extensive quantitative and qualitative experiments are conducted, depicting the capability of SAM and FCMs and demonstrating the competitiveness of proposed networks on four public hyperspectral scenes when comparing with the current state-of-the-art approaches.

Preserving details in semantics-aware context for scene parsing

Great success of scene parsing (also known as, semantic segmentation) has been achieved with the pipeline of fully convolutional networks (FCNs). Nevertheless, there are a lot of segmentation failures caused by large similarities between local appearances. To alleviate the problem, most of existing methods attempt to improve the global view of FCNs by introducing different contextual modules. Though the reconstructed high resolution output of these methods is of rich semantics, it cannot faithfully recover the fine image details owing to lack of desired precise low-level information. To overcome the problem, we propose to improve the spatial decoding process through embedding possibly lost low-level information in a principled way. To this end, we make the following three contributions. First, we propose a semantics conformity module to make low-level features variations agnostic. Second, we introduce semantics into the conformed low level features through guidance from semantically aware features. Finally, we institute the availability of various possible contextual features at feature fusion to enrich context information. The proposed approach demonstrates competitive performance on challenging PASCAL VOC 2012, Cityscapes, and ADE20K benchmarks in comparison to the state-of-the-art methods.

Contextual deconvolution network for semantic segmentation

In this paper, we propose a Contextual Deconvolution Network (CDN) and focus on context association in decoder network. Specifically, in upsampling path, we introduce two types of contextual modules to model the interdependencies of features in channel and spatial dimensions respectively. The channel contextual module captures image-level semantic information by aggregating the feature maps across spatial dimensions, and clarifies global ambiguity of features. Meanwhile, the spatial contextual module obtains patch-level semantic context by learning a spatial weight map, and enhance the feature discrimination. We embed the two contextual modules into individual components of the decoder network, thus improving the representation power and gaining more precise segment results. Thorough evaluations are performed on four challenging datasets, i.e., PASCAL VOC 2012, ADE20K, PASCAL-Context and Cityscapes dataset. Our approach achieves competitive performance with state-of-the-art models on PASCAL VOC 2012,ADE20K and Cityscapes dataset, and new state-of-the-art performance on PASCAL-Context dataset.

Deep Learning-Based Caution Area Traffic Prediction with Automatic Identification System Sensor Data.

In a crowded harbor water area, it is a major concern to control ship traffic for assuring safety and maximizing the efficiency of port operations. Vessel Traffic Service (VTS) operators pay much attention to caution areas like ship route intersections or traffic congestion area in which there are some risks of ship collision. They want to control the traffic of the caution area at a proper level to lessen risk. Inertial ship movement makes swift changes in direction and speed difficult. It is hence important to predict future traffic of the caution area earlier on so as to get enough time for control actions on ship movements. In the harbor area, VTS stations collect a large volume of Automatic Identification Service (AIS) sensor data, which contain information about ship movement and ship attributes. This paper proposes a new deep neural network model called Ship Traffic Extraction Network (STENet) to predict the medium-term traffic and long-term traffic of the caution area. The STENet model is trained with AIS sensor data. The STENet model is organized into a hierarchical architecture in which the outputs of the movement and contextual feature extraction modules are concatenated and fed into a prediction module. The movement module extracts the features of overall ship movements with a convolutional neural network. The contextual modules consist of five separated fully-connected neural networks, each of which receives an associated attribute. The separation of feature extraction modules at the front phase helps extract the effective features by preventing unrelated attributes from crosstalking. To evaluate the performance of the proposed model, the developed model is applied to a real AIS sensor dataset, which has been collected over two years at a Korean port called Yeosu. In the experiments, four methods have been compared including two new methods: STENet and VGGNet-based models. For the real AIS sensor dataset, the proposed model has shown 50.65% relative performance improvement on average for the medium-term predictions and 57.65% improvement on average for the long-term predictions over the benchmark method, i.e., the SVR-based method.

Query-driven approach of contextual ontology module learning using web snippets

The main objective of this work is to automatically build ontology modules that cover search terms of users in ontology-based question answering on the Web. Indeed, some arising approaches of ontology module extraction aim at solving the problem of identifying ontology fragment candidates that are relevant for the application. The main problem is that these approaches consider only the input of predefined ontologies, instead of the underlying semantics represented in texts. This work proposes an approach of contextual ontology module learning covering particular search terms by analyzing past user queries and by searching for web snippets provided by the traditional search engines. The obtained contextual modules will be used for query reformulation. The proposal has been evaluated on the ground of two criteria: the semantic cotopy measure of discovered ontology modules and the precision measure of the search results obtained by using the resulted ontology modules for query reformulation. The experiments have been carried out according to two case studies: an open domain web search and the medical digital library "PubMed".

Exploring the use of contextual modules for understanding and supporting collaborative learning activities: An empirical study

We report three student groups? collaboration experiences in a semester-long classroom project. The project included both tasks that required completion in virtual group workspace and activities that could be carried out in the physical world environment. We observed different collaboration patterns among the groups with respect to building and maintaining social relationships, submitting individual work to the group, and scheduling group meetings. We use Bereiter?s two contextual modules, intentional learning and schoolwork, to help us understand the observed patterns and suggest that the group leader?s contextual module plays a significant role in all members? group learning experiences and outcomes. We propose design implications that are intended for encouraging learning-based (as opposed to work-based) practices in virtual group environments.

Curriculum for Individuals with a Developmental Disability: An Overview

A curriculum that is functional for individuals who are considered developmentally disabled was developed at Tohatchi Special Education and Training Center, located within the Navajo Reservation in New Mexico. The purpose of the curriculum is to provide learning experiences for these individuals with a focus on survival skills. The curriculum is mainly divided into five parts: (a) overview of the contextual framework, (b) curriculum usage, (c) information modules, (d) gross and fine motor performance, and (e) language/communication components.

Aspects of an Educational Learning Theory

Contemporary cognitive science has created the possibility of an educational learning theory closely related to existing cognitive theories but operative at a higher level of description. Issues that must be addressed in developing such a theory are: How much of the external world should be included in cognitive descriptions, how to avoid degenerate functionalism, what needs explaining by an educational learning theory, and what its units of analysis should be. It is proposed that a constitutive problem for educational learning theory is the explanation of difficult learning. A unit of analysis called a contextual module is proposed and applied to this problem. An advantage of contextual modules is that they treat environmental situatedness as an emergent property resulting from reasonably well understood processes.

Aspects of an Educational Learning Theory

Contemporary cognitive science has created the possibility of an educational learning theory closely related to existing cognitive theories but operative at a higher level of description. Issues that must be addressed in developing such a theory are: How much of the external world should be included in cognitive descriptions, how to avoid degenerate functionalism, what needs explaining by an educational learning theory, and what its units of analysis should be. It is proposed that a constitutive problem for educational learning theory is the explanation of difficult learning. A unit of analysis called a contextual module is proposed and applied to this problem. An advantage of contextual modules is that they treat environmental situatedness as an emergent property resulting from reasonably well understood processes.