Commonsense Reasoning Research Articles

Open-Ethical AI: Advancements in Open-Source Human-Centric Neural Language Models

This survey summarizes the most recent methods for building and assessing helpful, honest, and harmless neural language models, considering small, medium, and large-size models. Pointers to open-source resources that help to align pre-trained models are given, including methods that use parameter-efficient techniques, specialized prompting frameworks, adapter modules, case-specific knowledge injection, and adversarially robust training techniques. Special care is given to evidencing recent progress on value alignment, commonsense reasoning, factuality enhancement, and abstract reasoning of language models. Most reviewed works in this survey publicly shared their code and related data and were accepted in world-leading Machine Learning venues. This work aims to help researchers and practitioners accelerate their entrance into the field of human-centric neural language models, which might be a cornerstone of the contemporary and near-future industrial and societal revolution.

GPT-4 and Beyond : Advancements in AI Language Models

This comprehensive article explores the groundbreaking advancements and implications of GPT-4, the latest iteration in the Generative Pre-trained Transformer series. It delves into GPT-4's enhanced comprehension and contextual awareness, multilingual proficiency, expanded knowledge base, multimodal capabilities, and improved safety and ethical considerations. The article provides quantifiable improvements and practical implications across various domains, including natural language processing, machine translation, research assistance, education, and content creation. It also addresses the challenges and limitations of the model, such as biases and computational requirements, while discussing future directions in AI development. The piece concludes with projections for future AI capabilities, emphasizing more human-like reasoning, handling of complex tasks, and seamless integration with other systems, while also highlighting the ethical considerations and challenges that accompany these advancements.

Language-based reasoning graph neural network for commonsense question answering

Language model (LM) has played an increasingly important role in the common-sense understanding and reasoning in the CSQA task (Common Sense Question Answering). However, due to the amount of model parameters, increasing training data helps little in further improving model performance. Introducing external knowledge through graph neural networks (GNNs) proves positive in boosting performance, but exploiting different knowledge sources and capturing contextual information between text and knowledge inside remains a challenge. In this paper, we propose LBR-GNN, a Language-Based Reasoning Graph Neural Network method to address these problems, by representing the question with each answer and external knowledge using a language model and predicting the reasoning score with a designed language-based GNN. Our LBR-GNN will first regulate external knowledge into a consistent textual form and encode it using a standard LM to capture the contextual information. Then, we build a graph neural network using the encoded information, especially the language-level edge representation. Finally, we design a novel edge aggregation method to select the edge information for GNN update and the language-guided GNN reasoning. We assess the performance of LBR-GNN across the CommonsenseQA, CommonsenseQA-IH, and OpenBookQA datasets. Our evaluation reveals a performance boost of more than 5% compared to the state-of-the-art methods on the CSQA dataset, achieved with a similar number of additional parameters.

Foundation models in robotics: Applications, challenges, and the future

We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning, while vision-language models enable open-vocabulary visual recognition. However, significant open research challenges remain, particularly around the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution. In this survey, we study recent papers that have used or built foundation models to solve robotics problems. We explore how foundation models contribute to improving robot capabilities in the domains of perception, decision-making, and control. We discuss the challenges hindering the adoption of foundation models in robot autonomy and provide opportunities and potential pathways for future advancements. The GitHub project corresponding to this paper can be found here: https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models .

Systematic exploration and in-depth analysis of ChatGPT architectures progression

The fast evolution of artificial intelligence frameworks has resulted in the creation of increasingly sophisticated large language models (LLM), ChatGPT being the most famous one. This study paper dives into this LLM with a case study of ChatGPT’s architecture and provides a thorough comparative analysis of its numerous versions, tracking its history from its conception to its most recent incarnations. This research intends to give a full knowledge of the model’s history by investigating the underlying mechanisms and enhancements provided in each edition. The comparative analysis covers key aspects such as model size, training data, fine-tuning techniques, and performance metrics. Furthermore, this study evaluates the limits of ChatGPT in its many incarnations. These limitations include common sense reasoning difficulties, biased replies, verbosity, sensitivity to input wording, and others. Each constraint is investigated for potential remedies and workarounds. This research article also provides a complete analysis of the ChatGPT architecture and its progress through multiple iterations. It gives vital insights for academics, developers, and users wanting to harness the promise of ChatGPT while managing its restrictions by exploring both the model’s strengths and limitations. The distinctiveness of this paper rests in its comprehensive assessment of ChatGPT’s architectural development and its practical strategy for resolving the myriad difficulties in producing cohesive and contextually relevant replies.

A Reasoning and Value Alignment Test to Assess Advanced GPT Reasoning

In response to diverse perspectives on artificial general intelligence (AGI), ranging from potential safety and ethical concerns to more extreme views about the threats it poses to humanity, this research presents a generic method to gauge the reasoning capabilities of artificial intelligence (AI) models as a foundational step in evaluating safety measures. Recognizing that AI reasoning measures cannot be wholly automated, due to factors such as cultural complexity, we conducted an extensive examination of five commercial generative pre-trained transformers (GPTs), focusing on their comprehension and interpretation of culturally intricate contexts. Utilizing our novel “Reasoning and Value Alignment Test,” we assessed the GPT models’ ability to reason in complex situations and grasp local cultural subtleties. Our findings have indicated that, although the models have exhibited high levels of human-like reasoning, significant limitations remained, especially concerning the interpretation of cultural contexts. This article also explored potential applications and use-cases of our Test, underlining its significance in AI training, ethics compliance, sensitivity auditing, and AI-driven cultural consultation. We concluded by emphasizing its broader implications in the AGI domain, highlighting the necessity for interdisciplinary approaches, wider accessibility to various GPT models, and a profound understanding of the interplay between GPT reasoning and cultural sensitivity.

Two semantic interpretations of probabilities in description logics of typicality

Abstract In this work we focus on extensions of Description Logics (DLs) of typicality by means of probabilities. We introduce a novel extension of the logic of typicality $\mathcal{ALC}+\mathbf{T}_{\mathbf{R}}$, able to represent and reason about typical properties and defeasible inheritance in DLs. The novel logic ($\mathcal{ALC}\mathbf{T}^{\textsf{P}}$: Typical $\mathcal{ALC}$ with Probabilities as Proportions) allows inclusions of the form $\mathbf{T}(C) \sqsubseteq ^{p} D$, with probability $p$ representing a proportion, meaning that “all the typical $C$s are $D$s, and the probability that a $C$ element is not a $D$ element is $1-p$”. We also compare and confront this novel logic with a similar one already presented in the literature ($\mathbf{T}^{\textsf{CL}}$, introduced in Lieto and Pozzato (2020, J. Exp. Theor. Artif. Intell., 32, 769–804)), inspired by the DISPONTE semantics and that allows inclusions of the form $p \:: \ \mathbf{T}(C) \sqsubseteq D$ with probability $p$, where $p$ represents a degree of belief, whose meaning is that “we believe with a degree $p$ that typical $C$s’ are also $D$s.”. We then show that the proposed $\mathcal{ALC}\mathbf{T}^{\textsf{P}}$ extension (like the previous $\mathbf{T}^{\textsf{CL}}$) can be applied in order to tackle a specific and challenging problem in the field of common-sense reasoning, namely the combination of prototypical concepts, that have been shown to be problematic to model for other symbolic approaches like fuzzy logic. We show that, for the proposed extension, the complexity of reasoning remains ExpTime-complete as for the underlying standard monotonic DL $\mathcal{ALC}$.

Robust Commonsense Reasoning Against Noisy Labels Using Adaptive Correction.

Commonsense reasoning based on knowledge graphs (KGs) is a challenging task that requires predicting complex questions over the described textual contexts and relevant knowledge about the world. However, current methods typically assume clean training scenarios with accurately labeled samples, which are often unrealistic. The training set can include mislabeled samples, and the robustness to label noises is essential for commonsense reasoning methods to be practical, but this problem remains largely unexplored. This work focuses on commonsense reasoning with mislabeled training samples and makes several technical contributions: 1) we first construct diverse augmentations from knowledge and model, and offer a simple yet effective multiple-choice alignment method to divide the training samples into clean, semi-clean, and unclean parts; 2) we design adaptive label correction methods for the semi-clean and unclean samples to exploit the supervised potential of noisy information; and 3) finally, we extensively test these methods on noisy versions of commonsense reasoning benchmarks (CommonsenseQA and OpenbookQA). Experimental results show that the proposed method can significantly enhance robustness and improve overall performance. Furthermore, the proposed method is generally applicable to multiple existing commonsense reasoning frameworks to boost their robustness. The code is available at https://github.com/xdxuyang/CR_Noisy_Labels.

When large language models meet personalization: perspectives of challenges and opportunities

The advent of large language models marks a revolutionary breakthrough in artificial intelligence. With the unprecedented scale of training and model parameters, the capability of large language models has been dramatically improved, leading to human-like performances in understanding, language synthesizing, common-sense reasoning, etc. Such a major leap forward in general AI capacity will fundamentally change the pattern of how personalization is conducted. For one thing, it will reform the way of interaction between humans and personalization systems. Instead of being a passive medium of information filtering, like conventional recommender systems and search engines, large language models present the foundation for active user engagement. On top of such a new foundation, users’ requests can be proactively explored, and users’ required information can be delivered in a natural, interactable, and explainable way. For another thing, it will also considerably expand the scope of personalization, making it grow from the sole function of collecting personalized information to the compound function of providing personalized services. By leveraging large language models as a general-purpose interface, the personalization systems may compile user’s requests into plans, calls the functions of external tools (e.g., search engines, calculators, service APIs, etc.) to execute the plans, and integrate the tools’ outputs to complete the end-to-end personalization tasks. Today, large language models are still being rapidly developed, whereas the application in personalization is largely unexplored. Therefore, we consider it to be right the time to review the challenges in personalization and the opportunities to address them with large language models. In particular, we dedicate this perspective paper to the discussion of the following aspects: the development and challenges for the existing personalization system, the newly emerged capabilities of large language models, and the potential ways of making use of large language models for personalization.

Studying and improving reasoning in humans and machines

In the present study, we investigate and compare reasoning in large language models (LLMs) and humans, using a selection of cognitive psychology tools traditionally dedicated to the study of (bounded) rationality. We presented to human participants and an array of pretrained LLMs new variants of classical cognitive experiments, and cross-compared their performances. Our results showed that most of the included models presented reasoning errors akin to those frequently ascribed to error-prone, heuristic-based human reasoning. Notwithstanding this superficial similarity, an in-depth comparison between humans and LLMs indicated important differences with human-like reasoning, with models’ limitations disappearing almost entirely in more recent LLMs’ releases. Moreover, we show that while it is possible to devise strategies to induce better performance, humans and machines are not equally responsive to the same prompting schemes. We conclude by discussing the epistemological implications and challenges of comparing human and machine behavior for both artificial intelligence and cognitive psychology.

ASMR: Aggregated Semantic Matching Retrieval Unleashing Commonsense Ability of LLM through Open-Ended Question Answering

Commonsense reasoning refers to the ability to make inferences, draw conclusions, and understand the world based on general knowledge and commonsense. Whether Large Language Models (LLMs) have commonsense reasoning ability remains a topic of debate among researchers and experts. When confronted with multiple-choice commonsense reasoning tasks, humans typically rely on their prior knowledge and commonsense to formulate a preliminary answer in mind. Subsequently, they compare this preliminary answer to the provided choices, and select the most likely choice as the final answer. We introduce Aggregated Semantic Matching Retrieval (ASMR) as a solution for multiple-choice commonsense reasoning tasks. To mimic the process of humans solving commonsense reasoning tasks with multiple choices, we leverage the capabilities of LLMs to first generate the preliminary possible answers through open-ended question which aids in enhancing the process of retrieving relevant answers to the question from the given choices. Our experiments demonstrate the effectiveness of ASMR on popular commonsense reasoning benchmark datasets, including CSQA, SIQA, and ARC (Easy and Challenge). ASMR achieves state-of-the-art (SOTA) performance with a peak of +15.3% accuracy improvement over the previous SOTA on SIQA dataset.

Modeling Patterns for Neural-Symbolic Reasoning Using Energy-based Models

Neural-symbolic (NeSy) AI strives to empower machine learning and large language models with fast, reliable predictions that exhibit commonsense and trustworthy reasoning by seamlessly integrating neural and symbolic methods. With such a broad scope, several taxonomies have been proposed to categorize this integration, emphasizing knowledge representation, reasoning algorithms, and applications. We introduce a knowledge representation-agnostic taxonomy focusing on the neural-symbolic interface capturing methods that reason with probability, logic, and arithmetic constraints. Moreover, we derive expressions for gradients of a prominent class of learning losses and a formalization of reasoning and learning. Through a rigorous empirical analysis spanning three tasks, we show NeSy approaches reach up to a 37% improvement over neural baselines in a semi-supervised setting and a 19% improvement over GPT-4 on question-answering.

GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction

Predicting pedestrian behavior is the key to ensure safety and reliability of autonomous vehicles. While deep learning methods have been promising by learning from annotated video frame sequences, they often fail to fully grasp the dynamic interactions between pedestrians and traffic, crucial for accurate predictions. These models also lack nuanced common sense reasoning. Moreover, the manual annotation of datasets for these models is expensive and challenging to adapt to new situations. The advent of Vision Language Models (VLMs) introduces promising alternatives to these issues, thanks to their advanced visual and causal reasoning skills. To our knowledge, this research is the first to conduct both quantitative and qualitative evaluations of VLMs in the context of pedestrian behavior prediction for autonomous driving. We evaluate GPT-4V(ision) on publicly available pedestrian datasets: JAAD and WiDEVIEW. Our quantitative analysis focuses on GPT-4V's ability to predict pedestrian behavior in current and future frames. The model achieves a 57% accuracy in a zero-shot manner, which, while impressive, is still behind the state-of-the-art domain-specific models (70%) in predicting pedestrian crossing actions. Qualitatively, GPT-4V shows an impressive ability to process and interpret complex traffic scenarios, differentiate between various pedestrian behaviors, and detect and analyze groups. However, it faces challenges, such as difficulty in detecting smaller pedestrians and assessing the relative motion between pedestrians and the ego vehicle.

Acquiring and modeling abstract commonsense knowledge via conceptualization

Conceptualization, or viewing entities and situations as instances of abstract concepts in mind and making inferences based on that, is a vital component in human intelligence for commonsense reasoning. Despite recent progress in artificial intelligence to acquire and model commonsense attributed to neural language models and commonsense knowledge graphs (CKGs), conceptualization is yet to be introduced thoroughly, making current approaches ineffective to cover knowledge about countless diverse entities and situations in the real world. To address the problem, we thoroughly study the role of conceptualization in commonsense reasoning, and formulate a framework to replicate human conceptual induction by acquiring abstract knowledge about events regarding abstract concepts, as well as higher-level triples or inferences upon them. We then apply the framework to ATOMIC, a large-scale human-annotated CKG, aided by the taxonomy Probase. We annotate a dataset on the validity of contextualized conceptualizations from ATOMIC on both event and triple levels, develop a series of heuristic rules based on linguistic features, and train a set of neural models to generate and verify abstract knowledge. Based on these components, a pipeline to acquire abstract knowledge is built. A large abstract CKG upon ATOMIC is then induced, ready to be instantiated to infer about unseen entities or situations. Finally, we empirically show the benefits of augmenting CKGs with abstract knowledge in downstream tasks like commonsense inference and zero-shot commonsense QA.

Moving to Advanced Research on Human Machine Interface Design

This poster paper will discuss a variety of items related to Human-Machine Teaming and research in support in-creasing control of autonomous machines present in phys-ical problem domains of interest. Many military tasks can be decomposed into their primary elements – intelligence preparation, reconnaissance, movement, maneuver, fires, and and support across the combat domains of interest – air, land, sea, etc. There are an increasing number of au-tonomous and semi-autonomous ground-based systems available, such as the Multi-Utility Tactical Transport (MUTT) Unmanned Ground Vehicle, for movement of materials, or Quadrupedal Unmanned Ground Vehicle (QUGV) for the disposal of explosive ordinance, comple-mented by aerial platforms considering of a wide variety of Unmanned Aerial Systems (UAS) for the gathering and transmitting of information, held together by a common backbone and network. The preponderance of new systems and capabilities brings new issues, one of which is critical to research - how to control and manage a large number of systems. Commercial systems of significantly less capability, such as light-up drone shows, involve approximately 2 people per 100 drones; meaning that a single controller for a sin-gle drone is simply a non-starter. How can research be applied to scale the complexity of operations upwards without additional demands of personnel? This poster discusses several portions of early-stage research into a variety of applications, including: Reasoning about mixed-team processes, includ-ing real/synthetic teammates, and how infor-mation gathered about the use of synthetic teammates within simulation can be transitioned into utilization for a robotic teammate Embedding affective information into dialogue channels in order to save cognitive bandwidth Repurposing of foundational dialogue models for specific tasks and purposes Utilizing psychological research to design sys-tems in order to infer user intent Theory of mind research for autonomous systems regarding their human operators Simulated environments and agents in order to test the simulations in representative areas. Common-Sense reasoning augmented by Large Language Model technology for instructing ro-botic platforms The following things will be discussed at the poster ses-sion, representing a portfolio of ongoing work addressing the problems of large-scale multi-agent command and control. While the near-term application of these tech-nologies is into military problem domains; the poster presentation is cleared for public release.

Natural Language Reasoning, A Survey

This survey paper proposes a clearer view of natural language reasoning in the field of Natural Language Processing (NLP), both conceptually and practically. Conceptually, we provide a distinct definition for natural language reasoning in NLP, based on both philosophy and NLP scenarios, discuss what types of tasks require reasoning, and introduce a taxonomy of reasoning. Practically, we conduct a comprehensive literature review on natural language reasoning in NLP, mainly covering classical logical reasoning, natural language inference, multi-hop question answering, and commonsense reasoning. The paper also identifies and views backward reasoning, a powerful paradigm for multi-step reasoning, and introduces defeasible reasoning as one of the most important future directions in natural language reasoning research. We focus on single-modality unstructured natural language text, excluding neuro-symbolic research and mathematical reasoning.

Temporal validity reassessment: commonsense reasoning about information obsoleteness

It is useful for machines to know whether text information remains valid or not for various applications including text comprehension, story understanding, temporal information retrieval, and user state tracking on microblogs as well as via chatbot conversations. This kind of inference is still difficult for current models, including also large language models, as it requires temporal commonsense knowledge and reasoning. We approach in this paper the task of Temporal Validity Reassessment, inspired by traditional natural language reasoning to determine the updates of the temporal validity of text content. The task requires judgment whether actions expressed in a sentence are still ongoing or rather completed, hence, whether the sentence still remains valid or has become obsolete, given the presence of context in the form of a supplementary content such as a follow-up sentence. We first construct our own dataset for this task and train several machine learning models. Then we propose an effective method for learning information from an external knowledge base that gives information regarding temporal commonsense knowledge. Using our prepared dataset, we introduce a machine learning model that incorporates the information from the knowledge base and demonstrate that incorporating external knowledge generally improves the results. We also experiment with different embedding types to represent temporal commonsense knowledge as well as with data augmentation methods to increase the size of our dataset.

ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI

Abstract Mastering commonsense understanding and reasoning is a pivotal skill essential for conducting engaging conversations. While there have been several attempts to create datasets that facilitate commonsense inferences in dialogue contexts, existing datasets tend to lack in-depth details, restate information already present in the conversation, and often fail to capture the multifaceted nature of commonsense reasoning. In response to these limitations, we compile a new synthetic dataset for commonsense reasoning in dialogue contexts using GPT, ℂonvoSense, that boasts greater contextual novelty, offers a higher volume of inferences per example, and substantially enriches the detail conveyed by the inferences. Our dataset contains over 500,000 inferences across 12,000 dialogues with 10 popular inference types, which empowers the training of generative commonsense models for dialogue that are superior in producing plausible inferences with high novelty when compared to models trained on the previous datasets. To the best of our knowledge, ℂonvoSense is the first of its kind to provide such a multitude of novel inferences at such a large scale.

Evaluating ChatGPT: Strengths and Limitations in NLP Problem Solving

This paper critically analyzes ChatGPT’s problem-solving performance on a range of natural language processing (NLP) tasks. Using a comparative methodology, it compares ChatGPT’s performance with that of its predecessor, GPT-3.5, in seven different domains: summarization, named entity recognition, arithmetic, natural language inference, symbolic and logical reasoning, question answering, conversation, and arithmetic. The process entails a methodical assessment highlighting ChatGPT’s replies’ quantitative and qualitative elements. The findings show that although ChatGPT performs very well on math and question-answering tasks, it struggles with summarization and commonsense reasoning. The conversation sheds light on the subtleties of these findings while considering the applications and development of AI. The article concludes that although ChatGPT is a significant progress in natural language processing, its uneven problem-solving performance highlights the need for continued development and optimization of artificial intelligence models. This study aids in understanding the state and promise of AI-driven language models in challenging problem-solving situations.

Unveiling the Cognitive Capacity of ChatGPT: Assessing its Human-Like Reasoning Abilities

Unveiling the Cognitive Capacity of ChatGPT: Assessing its Human-Like Reasoning Abilities