Commonsense Knowledge Research Articles

Principle and Application for Rumination Computing Algorithms

To fully analyze, mine, and utilize the information and knowledge implied in problem resolving use cases, this paper proposed the autonomous learning method based on machine inducting, hypothesis formulating, and result verifying, which was similar to the biological process of cows ruminating, called rumination computing. Firstly, after inducting and summarizing over 1080 mathematic application problem, the system architecture and general algorithm for humanoid automatic resolving mathematic application problems were represented, which typically included functional modules such as commonsense knowledge base, domain knowledge base, and local knowledge base, preprocessing, word segmentation and part of speech tagging, semantic framework matching, global semantic analyzing, thinking mechanism implementing, etc. Secondly, after the use case solutions were approved, three typical rumination computing modes, including vocabulary sequence, semantic relationship, and computing action, were introduced based on the correct results, resolving steps, and basic rumination actions. The rumination computing step plan was formulated, new knowledge was obtained from the commonsense and results verification, so the continuous autonomous learning loop for machine thinking was formed. Detailed explanations were provided for the three core algorithms implemented (rumination framework algorithm, rumination semantic algorithm, rumination action algorithm). Then, by specific mathematic application problem humanoid resolving user cases, the above three types of rumination computing modes were illumined.

CoMaSa:Context Multi-aware Self-attention for emotional response generation

Dialogue generation is an important research direction in natural language generation, and generating utterances with appropriate contextual emotion is a challenging task. The previous work incorporates commonsense knowledge as an auxiliary information source by simply concatenating or adding it to the embeddings of the dialogue context. They did not consider updating the commonsense knowledge from the perspective of context semantic information or selecting context from the perspective of commonsense knowledge semantic information. At the same time, they did not extract emotional information from dialogue context and commonsense knowledge to enhance the embeddings of the dialogue text. These defects may lead to the dialogue model’s inability to judge the semantic consistency between context and generated utterance, and result in the generation of utterance unassociated with the dialogue context and the corresponding context emotion being untrue. Our goal is to build an emotional dialogue generation model that can deeply integrate commonsense knowledge with dialogue context and achieve bidirectional semantic interaction and enhancement. Therefore, in this paper, we propose a context multi-aware self-attention emotion dialogue generation model that models from multiple aspects such as the semantic information, emotional information of context and external knowledge, it effectively alleviates the problems of generating utterance unrelated to the context and disordered emotional expressions in utterance. First, we adopt the transformer encoder as the basic framework and carefully construct a context-aware encoder capable of deeply extracting semantic information. Specifically, we ingeniously integrate different levels of positional information to simulate the natural flow of conversation between speakers. Then, we effectively integrate these positional information with the basic word embedding vectors, and enrich the dialogue’s embedding representation. Lastly, by introducing the attention mechanism, the model can precisely capture and extract context information closely related to the current situation and achieve a deep understanding and accurate grasp of the dialogue content. Second, we consider that the relationship between context and commonsense knowledge is not just a unidirectional selection but a more intimate bidirectional interaction. Therefore, we designed a bidirectional interaction attention component for context and external knowledge. Specifically, we use the method of iterative similarity matrices to calculate the bidirectional similarity relationship between context and external knowledge. Then, using the attention algorithm, we calculate the connectivity from context to commonsense knowledge and from commonsense knowledge to context. Finally, by combining the semantic embeddings of both aspects, we achieve bidirectional updating of context and external knowledge. Third, we establish an emotion-aware decoder based on multi-task learning, which includes an emotion classifier and an utterance generator. In the emotion classifier, we establish an auxiliary task for emotion recognition to help the model understand the process of emotional transition. We use the output of the emotion recognition model as an additional supervisory signal to better control the generation of emotional responses. In the utterance generator, we integrate the output of the emotion classifier with the interaction vector from the bidirectional interaction attention component. This fusion allows us to generate utterance that aligns with the emotional context. Our experiments on DailyDialog and ESConv datasets show that CoMaSa outperforms the baselines in terms of Perplexity, Distinct-1, Distinct-2, and human evaluation. The experiment demonstrates the effectiveness of our Context Multi-aware Self-attention model in the task of emotion generation.

CIRG-SL: Commonsense Inductive Relation Graph framework with Soft Labels for Empathetic Response Generation

Empathy, the ability to understand and respond to others’ emotions, is critical in dialogue systems, particularly for applications such as psychological counselling and casual conversation. However, existing approaches often struggle to accurately capture emotional transitions in user utterances and dialogue contexts across multiple levels of granularity. Additionally, the orthogonality of one-hot emotion representations overlooks the subtle relationships between emotion categories. To overcome these limitations, we propose a novel framework: the commonsense inductive relation graph with fine emotional soft labels for empathetic response generation. This framework refines the representation of user utterances by incorporating commonsense knowledge to derive external emotional cues, which are integrated into an adaptive inductive relation graph. This graph explicitly captures and models dynamic emotional trajectories throughout the dialogue. Additionally, we propose a fine-grained soft label construction method that more accurately reflects the nuanced distribution of emotions, replacing the rigid one-hot encoding with a continuous representation that better captures inter-emotion relationships. Our approach achieves strong performance on the benchmark EmpatheticDialogues dataset, with a perplexity score of 27.85, Dist-1 of 1.81, Dist-2 of 6.58, and a dialogue emotion accuracy of 55.65%. These results highlight the effectiveness of our framework in capturing subtle emotions and generating contextually rich, empathetic responses.

A fine-grained deconfounding study for knowledge-based visual dialog

Knowledge-based Visual Dialog is a challenging vision-language task, where an agent engages in dialog to answer questions with humans based on the input image and corresponding commonsense knowledge. The debiasing methods based on causal graphs have gradually sparked much attention in the field of Visual Dialog (VD), yielding impressive achievements. However, existing studies focus on the coarse-grained deconfounding, which lacks a principled analysis of the bias. In this paper, we propose a fined-grained study of deconfounding on: (1) We define the confounder from two perspectives. The first is user preference (denoted as Uh), derived from human-annotated dialog history, which may introduce spurious correlations between questions and answers. The second is commonsense language bias (denoted as Uc), where certain words appear so frequently in the retrieved commonsense knowledge that the model tends to memorize these patterns, thereby establishing spurious correlations between the commonsense knowledge and the answers. (2) Given that the current question directly influences answer generation, we further decompose the confounders into Uh1,Uh2 and Uc1,Uc2, based on their relevance to the current question. Specifically, Uh1 and Uc1 represent dialog history and high-frequency words that are highly correlated with the current question, while Uh2 and Uc2 are sampled from dialog history and words with low relevance to the current question. Through a comprehensive evaluation and comparison of all components, we demonstrate the necessity of jointly considering both Uh and Uc. Fine-grained deconfounding, particularly with respect to the current question, proves to be more effective. Ablation studies, quantitative results, and visualizations further confirm the effectiveness of the proposed method.

Validating pretrained language models for content quality classification with semantic-preserving metamorphic relations

Context:Utilizing pretrained language models (PLMs) has become common practice in maintaining the content quality of question-answering (Q&A) websites. However, evaluating the effectiveness of PLMs poses a challenge as they tend to provide local optima rather than global optima. Objective:In this study, we propose using semantic-preserving Metamorphic Relations (MRs) derived from Metamorphic Testing (MT) to address this challenge and validate PLMs. Methods:To validate four selected PLMs, we conducted an empirical experiment using a publicly available dataset comprising 60000 data points. We defined three groups of Metamorphic Relations (MRGs), consisting of thirteen semantic-preserving MRs, which were then employed to generate “Follow-up” testing datasets based on the original “Source” testing datasets. The PLMs were trained using a separate training dataset. A comparison was made between the predictions of the four trained PLMs for “Source” and “Follow-up” testing datasets in order to identify instances of violations, which corresponded to inconsistent predictions between the two datasets. If no violation was found, it indicated that the PLM was insensitive to the associate MR; thereby, the MR can be used for validation. In cases where no violation occurred across the entire MRG, non-violation regions were identified and supported simulation metamorphic testing. Results:The results of this study demonstrated that the proposed MRs could effectively serve as a validation tool for content quality classification on Stack Overflow Q&A using PLMs. One PLM did not violate the “Uppercase conversion” MRG and the “Duplication” MRG. Furthermore, the absence of violations in the MRGs allowed for the identification of non-violation regions, confirming the ability of the proposed MRs to support simulation metamorphic testing. Conclusion:The experimental findings indicate that the proposed MRs can validate PLMs effectively and support simulation metamorphic testing for PLMs. However, further investigations are required to enhance the semantic comprehension and common sense knowledge of PLMs and explore highly informative statistical patterns of PLMs, in order to improve their overall performance.

Triple confidence-aware encoder–decoder model for commonsense knowledge graph completion

Triple confidence-aware encoder–decoder model for commonsense knowledge graph completion

Learning 3D human–object interaction graphs from transferable context knowledge for construction monitoring

We propose a novel framework for detecting 3D human–object interactions (HOI) in construction sites and a toolkit for generating construction-related human–object interaction graphs. Computer vision methods have been adopted for construction site safety surveillance in recent years. The current computer vision methods rely on videos and images, with which safety verification is performed on common-sense knowledge, without considering 3D spatial relationships among the detected instances. We propose a new method to incorporate spatial understanding by directly inferring the interactions from 3D point cloud data. The proposed model is trained on a 3D construction site dataset generated from our crafted simulation toolkit. The model achieves 54.11% mean interaction over union (mIOU) and 72.98% average mean precision(mAP) for the worker–object interaction relationship recognition. The model is also validated on PiGraphs, a benchmarking dataset with 3D human–object interaction types, and compared against other existing 3D interaction detection frameworks. It was observed that it achieves superior performance from the state-of-the-art model, increasing the interaction detection mAP by 17.01%. Besides the 3D interaction model, we also simulate interactions from industrial surveillance footage using MoCap and physical constraints, which will be released to foster future studies in the domain.

A survey on knowledge-enhanced multimodal learning

Multimodal learning has been a field of increasing interest, aiming to combine various modalities in a single joint representation. Especially in the area of visiolinguistic (VL) learning multiple models and techniques have been developed, targeting a variety of tasks that involve images and text. VL models have reached unprecedented performances by extending the idea of Transformers, so that both modalities can learn from each other. Massive pre-training procedures enable VL models to acquire a certain level of real-world understanding, although many gaps can be identified: the limited comprehension of commonsense, factual, temporal and other everyday knowledge aspects questions the extendability of VL tasks. Knowledge graphs and other knowledge sources can fill those gaps by explicitly providing missing information, unlocking novel capabilities of VL models. At the same time, knowledge graphs enhance explainability, fairness and validity of decision making, issues of outermost importance for such complex implementations. The current survey aims to unify the fields of VL representation learning and knowledge graphs, and provides a taxonomy and analysis of knowledge-enhanced VL models.

The Stanford Dependency Relations for Commonsense Knowledge Representation of Winograd Schema Challenge (WSC)

The Stanford Dependency Relations for Commonsense Knowledge Representation of Winograd Schema Challenge (WSC)

PrimeNet: A Framework for Commonsense Knowledge Representation and Reasoning Based on Conceptual Primitives

PrimeNet: A Framework for Commonsense Knowledge Representation and Reasoning Based on Conceptual Primitives

Generative commonsense knowledge subgraph retrieval for open-domain dialogue response generation

Grounding on a commonsense knowledge subgraph can help the model generate more informative and diverse dialogue responses. Prior Traverse-based works explicitly retrieve a subgraph from the external knowledge base (eKB). Notably, the available knowledge is strictly restricted by the eKB. To break this restriction, Generative Retrieval methods externalize knowledge from the language model. However, they always generate boring knowledge due to their one-pass externalization procedure. This work proposes a novel TiLM Traverse in Language Model (TiLM), which uses three ‘Chain-of-Thought’ sub-tasks, i.e., Query Entity Production, Topic Entity Prediction, and Knowledge Subgraph Completion, to build a high-quality knowledge subgraph to ground the next Response Generation without explicitly accessing the eKB in inference. Experimental results on both Chinese and English datasets demonstrate TiLM’s outstanding performance even only with a small scale of parameters.

ImageSchemaNet: A knowledge graph for embodied commonsense knowledge

ImageSchemaNet: A knowledge graph for embodied commonsense knowledge

Sensitivity analysis of slope stability based on eXtreme gradient boosting and SHapley Additive exPlanations: An exploratory study

Slope instability through can cause catastrophic consequences, so slope stability analysis has been a key topic in the field of geotechnical engineering. Traditional analysis methods have shortcomings such as high operational difficulty and time-consuming, for this reason many researchers have carried out slope stability analysis based on AI. However, the current relevant studies only judged the importance of each factor and did not specifically quantify the correlation between factors and slope stability. For this purpose, this paper carried out a sensitivity analysis based on the XGBoost and SHAP. The sensitivity analysis results of SHAP were also validated using GeoStudio software. The selected influence factors included slope height (H), slope angle (β), unit weight (γ), cohesion (c), angle of internal friction (φ) and pore water pressure coefficient (ru). The results showed that c and γ were the most and least important influential parameters, respectively. GeoStudio simulation results showed a negative correlation between γ, β, H, ru and slope stability, while a positive correlation between c, φ and slope stability. However, for real data, SHAP misjudged the correlation between γ and slope stability. Because current AI lacked common sense knowledge and, leading SHAP unable to effectively explain the real mechanism of slope instability. For this reason, this paper overcame this challenge based on the priori data-driven approach. The method provided more reliable and accurate interpretation of the results than a real sample, especially with limited or low-quality data. In addition, the results of this method showed that the critical values of c, φ, β, H, and ru in slope destabilization are 18 Kpa, 28°, 32°, 30 m, and 0.28, respectively. These results were closer to GeoStudio simulations than real samples.

EmpCI: Empathetic response generation with common sense and empathetic intent

Empathy plays an important role in human conversations as an ability that enables individuals to understand the emotions and situations of others. Integrating empathy into dialogue systems is a crucial step in making them humanized. Relevant psychological studies have shown that a complete, high-quality empathetic dialogue should consist of the following two stages: (1) Empathetic Perception: the listener needs to perceive the emotional state of the speaker from both cognitive and affective aspects; (2) Empathetic Expression: the appropriate expression is chosen to respond to the perceived information. However, many existing studies on empathetic response generation only focus on one of these stages, resulting in incomplete and insufficiently empathetic responses. To this end, we propose the EmpCI, a two-stage empathetic response generation model that utilizes commonsense knowledge and mixed empathetic intent, respectively. Specifically, we use commonsense knowledge in the first stage to enhance the model’s perception of the user’s emotion and introduce mixed empathetic intent in the second stage to generate responses with appropriate expressions for the perceived information. Finally, we evaluated the EmpCI on the EmpatheticDialogues dataset, and extensive experiment results show that the proposed model outperforms the baselines in both perceiving users’ emotions and generating empathetic responses.

Implicit and explicit commonsense for multi-sentence video captioning

Existing dense or paragraph video captioning approaches rely on holistic representations of videos, possibly coupled with learned object/action representations, to condition hierarchical language decoders. However, they fundamentally lack the commonsense knowledge of the world required to reason about progression of events, causality, and even the function of certain objects within a scene. To address this limitation we propose a novel video captioning Transformer-based model, that takes into account both implicit (visuo-lingual and purely linguistic) and explicit (knowledge-base) commonsense knowledge. We show that these forms of knowledge, in isolation and in combination, enhance the quality of produced captions. Further, inspired by imitation learning, we propose a new task of instruction generation, where the goal is to produce a set of linguistic instructions from a video demonstration of its performance. We formalize the task using the ALFRED dataset generated using an AI2-THOR environment. While instruction generation is conceptually similar to paragraph captioning, it differs in the fact that it exhibits stronger object persistence, as well as spatially-aware and causal sentence structure. We show that our commonsense knowledge enhanced approach produces significant improvements on this task (up to 57% in METEOR and 8.5% in CIDEr), as well as the state-of-the-art result on more traditional video captioning in the ActivityNet Captions dataset.

SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments

Semantic reasoning and dynamic planning capabilities are crucial for an autonomous agent to perform complex navigation tasks in unknown environments. It requires a large amount of common-sense knowledge, that humans possess, to succeed in these tasks. We present SayNav, a new approach that leverages human knowledge from Large Language Models (LLMs) for efficient generalization to complex navigation tasks in unknown large-scale environments. SayNav uses a novel grounding mechanism, that incrementally builds a 3D scene graph of the explored environment as inputs to LLMs, for generating feasible and contextually appropriate high-level plans for navigation. The LLM-generated plan is then executed by a pre-trained low-level planner, that treats each planned step as a short-distance point-goal navigation sub-task. SayNav dynamically generates step-by-step instructions during navigation and continuously refines future steps based on newly perceived information. We evaluate SayNav on multi-object navigation (MultiON) task, that requires the agent to utilize a massive amount of human knowledge to efficiently search multiple different objects in an unknown environment. We also introduce a benchmark dataset for MultiON task employing ProcTHOR framework that provides large photo-realistic indoor environments with variety of objects. SayNav achieves state-of-the-art results and even outperforms an oracle based baseline with strong ground-truth assumptions by more than 8% in terms of success rate, highlighting its ability to generate dynamic plans for successfully locating objects in large-scale new environments. The code, benchmark dataset and demonstration videos are accessible at https://www.sri.com/ics/computer-vision/saynav.

The War of Vladimir. Nationalism, Narcissism, and Childhood Battles

Abstract The war in Ukraine has had enormous media coverage in the West, generating a wide-ranging debate concerning its origins, responsibilities, and consequences. The debate in the media raises an interesting question seldom explored in the study of nations and nationalism, specifically, the scarce incidence or penetration by expert knowledge into the theories of common sense and profane knowledge. In fact, despite the current consolidation in the scientific literature of the constructionist and modernist paradigm – of a sociological nature – other visions and interpretations of the past continue to be widely accepted and popularized beyond academia. This article analyzes the weight and influence of psychology, psychiatry, and psychoanalysis in the social representation of nationalism. The persistence of psychopathological language in the profane interpretation of the conflict is explored using the discourse of the press and, in particular, the opinion columns published in various Western countries during the first weeks of the war. The article shows that the criticism and denunciation of nationalism are still being formulated nowadays with the voices of psychology and psychiatry, the psychodynamic vision of the authoritarian, and the vulgarized portrayal of Adolf Hitler.

Exploring Alternative Approaches to Language Modeling for Learning from Data and Knowledge

Despite their extensive application in language understanding tasks, large language models (LLMs) still encounter challenges including hallucinations - occasional fabrication of information - and alignment issues - lack of associations with human-curated world models (e.g., intuitive physics or common-sense knowledge). Moreover, the black-box nature of LLMs presents significant obstacles in training them effectively to achieve desired behaviors. In particular, modifying the concept embedding spaces of LLMs can be highly intractable. This process involves analyzing the implicit impact of such adjustments on the myriad parameters within LLMs and the resulting inductive biases. We propose a novel architecture that wraps powerful function approximation architectures within an outer, interpretable read-out layer. This read-out layer can be scrutinized to explicitly observe the effects of concept modeling during the training of the LLM. Our method stands in contrast with gradient-based implicit mechanisms, which depend solely on adjustments to the LLM parameters and thus evade scrutiny. By conducting extensive experiments across both generative and discriminative language modeling tasks, we evaluate the capabilities of our proposed architecture relative to state-of-the-art LLMs of similar sizes. Additionally, we offer a qualitative examination of the interpretable read-out layer and visualize the concepts it captures. The results demonstrate the potential of our approach for effectively controlling LLM hallucinations and enhancing the alignment with human expectations.

Acquiring and modeling abstract commonsense knowledge via conceptualization

Conceptualization, or viewing entities and situations as instances of abstract concepts in mind and making inferences based on that, is a vital component in human intelligence for commonsense reasoning. Despite recent progress in artificial intelligence to acquire and model commonsense attributed to neural language models and commonsense knowledge graphs (CKGs), conceptualization is yet to be introduced thoroughly, making current approaches ineffective to cover knowledge about countless diverse entities and situations in the real world. To address the problem, we thoroughly study the role of conceptualization in commonsense reasoning, and formulate a framework to replicate human conceptual induction by acquiring abstract knowledge about events regarding abstract concepts, as well as higher-level triples or inferences upon them. We then apply the framework to ATOMIC, a large-scale human-annotated CKG, aided by the taxonomy Probase. We annotate a dataset on the validity of contextualized conceptualizations from ATOMIC on both event and triple levels, develop a series of heuristic rules based on linguistic features, and train a set of neural models to generate and verify abstract knowledge. Based on these components, a pipeline to acquire abstract knowledge is built. A large abstract CKG upon ATOMIC is then induced, ready to be instantiated to infer about unseen entities or situations. Finally, we empirically show the benefits of augmenting CKGs with abstract knowledge in downstream tasks like commonsense inference and zero-shot commonsense QA.

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge