Linguistic Instruction Research Articles

Implicit and explicit commonsense for multi-sentence video captioning

Existing dense or paragraph video captioning approaches rely on holistic representations of videos, possibly coupled with learned object/action representations, to condition hierarchical language decoders. However, they fundamentally lack the commonsense knowledge of the world required to reason about progression of events, causality, and even the function of certain objects within a scene. To address this limitation we propose a novel video captioning Transformer-based model, that takes into account both implicit (visuo-lingual and purely linguistic) and explicit (knowledge-base) commonsense knowledge. We show that these forms of knowledge, in isolation and in combination, enhance the quality of produced captions. Further, inspired by imitation learning, we propose a new task of instruction generation, where the goal is to produce a set of linguistic instructions from a video demonstration of its performance. We formalize the task using the ALFRED dataset generated using an AI2-THOR environment. While instruction generation is conceptually similar to paragraph captioning, it differs in the fact that it exhibits stronger object persistence, as well as spatially-aware and causal sentence structure. We show that our commonsense knowledge enhanced approach produces significant improvements on this task (up to 57% in METEOR and 8.5% in CIDEr), as well as the state-of-the-art result on more traditional video captioning in the ActivityNet Captions dataset.

Natural language instructions induce compositional generalization in networks of neurons

A fundamental human cognitive feat is to interpret linguistic instructions in order to perform novel tasks without explicit task experience. Yet, the neural computations that might be used to accomplish this remain poorly understood. We use advances in natural language processing to create a neural model of generalization based on linguistic instructions. Models are trained on a set of common psychophysical tasks, and receive instructions embedded by a pretrained language model. Our best models can perform a previously unseen task with an average performance of 83% correct based solely on linguistic instructions (that is, zero-shot learning). We found that language scaffolds sensorimotor representations such that activity for interrelated tasks shares a common geometry with the semantic representations of instructions, allowing language to cue the proper composition of practiced skills in unseen settings. We show how this model generates a linguistic description of a novel task it has identified using only motor feedback, which can subsequently guide a partner model to perform the task. Our models offer several experimentally testable predictions outlining how linguistic information must be represented to facilitate flexible and general cognition in the human brain.

DSG-GAN: Multi-turn text-to-image synthesis via dual semantic-stream guidance with global and local linguistics

Multi-turn text-to-image synthesis task aims to manipulate desired visual content according to the user's intention step by step, which has recently attracted a lot of research interest in the community of language and vision. Different from traditional text-to-image synthesis, multi-turn text-to-image synthesis is more challenging as 1) it needs to continuously recognize the user's intention from spoken instruction and perceive the visual information from the source image; 2) it requires reasoning about the position, appearance, and characteristics of fresh modifications in target images as well as connecting objects in instructions with visual components in source images. To deal with this issue, in this paper, we propose a Dual Semantic-stream Guidance with global and local linguistics Generative Adversarial Network (DSG-GAN), which reasons and learns the user's intention from text description and iteratively manipulates visual information. Specifically, we design a novel dual semantic-stream discriminator, which combines with a hierarchical instruction encoder to evaluate the logic and quality between human intention in linguistic instruction and generates visual content from the perspective of global and fine-grained consistency matching. Meanwhile, the discriminator's backpropagation gradient is used to optimize the instruction encoder, which incentivizes it to purify the user's intention into global and local information that is consistent with the manipulation's visual representation. Extensive experiments show that even when producing high-resolution images and making deep iterative turns, our method performs significantly better due to local fine-grained linguistic information being combined with cross-modal correlation.

Actional Atomic-Concept Learning for Demystifying Vision-Language Navigation

Vision-Language Navigation (VLN) is a challenging task which requires an agent to align complex visual observations to language instructions to reach the goal position. Most existing VLN agents directly learn to align the raw directional features and visual features trained using one-hot labels to linguistic instruction features. However, the big semantic gap among these multi-modal inputs makes the alignment difficult and therefore limits the navigation performance. In this paper, we propose Actional Atomic-Concept Learning (AACL), which maps visual observations to actional atomic concepts for facilitating the alignment. Specifically, an actional atomic concept is a natural language phrase containing an atomic action and an object, e.g., ``go up stairs''. These actional atomic concepts, which serve as the bridge between observations and instructions, can effectively mitigate the semantic gap and simplify the alignment. AACL contains three core components: 1) a concept mapping module to map the observations to the actional atomic concept representations through the VLN environment and the recently proposed Contrastive Language-Image Pretraining (CLIP) model, 2) a concept refining adapter to encourage more instruction-oriented object concept extraction by re-ranking the predicted object concepts by CLIP, and 3) an observation co-embedding module which utilizes concept representations to regularize the observation representations. Our AACL establishes new state-of-the-art results on both fine-grained (R2R) and high-level (REVERIE and R2R-Last) VLN benchmarks. Moreover, the visualization shows that AACL significantly improves the interpretability in action decision. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/VLN-AACL.

Active exploration based on information gain by particle filter for efficient spatial concept formation

Autonomous robots need to learn the categories of various places by exploring their environments and interacting with users. However, preparing training datasets with linguistic instructions from users is time-consuming and labor-intensive. Moreover, effective exploration is essential for appropriate concept formation and rapid environmental coverage. To address this issue, we propose an active inference method, referred to as spatial concept formation with information gain-based active exploration (SpCoAE) that combines sequential Bayesian inference using particle filters and information gain-based destination determination in a probabilistic generative model. This study interprets the robot's action as a selection of destinations to ask the user, ‘What kind of place is this?’ in the context of active inference. This study provides insights into the technical aspects of the proposed method, including active perception and exploration by the robot, and how the method can enable mobile robots to learn spatial concepts through active exploration. Our experiment demonstrated the effectiveness of the SpCoAE in efficiently determining a destination for learning appropriate spatial concepts in home environments.

Embedding Explicit Linguistic Instruction in an SRSD Writing Intervention

Teaching linguistic aspects relevant to text construction is an essential component of any thorough writing instruction program, despite the conflicting evidence regarding its effectiveness. In this study, 889 second- and fourth-grade students were assigned to one of three conditions: Self-Regulated Development (SRSD), SRSD-connectors (SRSD-C), and business-as-usual (BAU). The experimental conditions addressed planning and self-regulation strategies to write opinion essays, but only the SRSD condition included explicit teaching of connectors (e.g., because) and discourse markers (e.g., In conclusion). Children in both experimental conditions outscored children in the BAU condition across grades and outcome variables. In addition, the SRSD condition showed larger effect sizes on Grade 2 children’s gains in text quality, number of genre-appropriate elements, and number of connectors than the SRSD-C condition. The study provides evidence of the effectiveness of explicitly teaching functionally motivated linguistic representations within a SRSD program. Theoretical and educational implications are discussed.

Enhancing language skills through English novel instruction

The use of literature, particularly novels, is crucial in English-as-Foreign-Language (EFL) classrooms for language skill development. Education today emphasizes holistic learning, fostering independence, and personality development. This approach requires rethinking traditional teaching methods and considering interdisciplinary aspects. English holds a pivotal role in creativity and communication fields, making it essential for fostering intercultural competence, identity development, altered perspectives, and tolerance. Literature greatly contributes to these goals. The study focuses on the impact of teaching novels in EFL classrooms, determining suitable texts for language learning. Novels are unanimously recognized as an effective tool for instilling reading habits in students. The primary objective of this study is to investigate the novel's potential to enhance students’ language skills, focusing on vocabulary expansion and grammatical structure acquisition. Additionally, the study aims to identify shortcomings in current teaching methods employed for novels in classrooms. Through statistical analysis of tests conducted, this research successfully demonstrates that when novels are methodically and earnestly integrated into linguistic instruction, they significantly contribute to learners’ language development. Despite the importance of novels in language education, their full potential remains untapped due to inadequate teaching approaches. The study reveals a notable disparity between students’ pre-test and post-test scores, highlighting the impactful role of comprehensive and effective novel-based language teaching methods.

Русский как иностранный (РКИ): несогласованность в субстантивных словосочетаниях с предлогами – трудности и подходы к решениям

The article discusses complexity of syncretic-character relationships between the constituents of noun phrases with prepositional postmodification in Russian language, considering their contradictory interpretations in Russian grammar guides’ analysis when applied in Russian-as-a-foreign-language teaching. The experience with beginners and pre-intermediate learners of different ages indicates the practical necessity of analyzing such units regarding their multi-aspect syntactic potential, which is often incompletely treated by contemporary linguistic theory instructions.

Ethereum smart contracts: Analysis and statistics of their source code and opcodes

Smart contracts are programs that are used for verifying and enforcing the terms of an agreement. Ethereum is an extensively used platform that can be used to execute smart contracts. These can be defined by using high-level languages such as Solidity, Vyper and Bamboo that are compiled into low-level machine instructions, i.e., bytecode. These instructions define a Turing complete language and are often represented by using a readable format that is referred to as opcodes. Here, we present a novel, more comprehensive, study that aims at gaining a precise understanding on how programmers use the linguistic instructions supported by Ethereum. More precisely, the contracts’ source code has been also analysed, in order to understand the high-level control structures and core instructions that are used for smart contract definitions, and how they are reflected on the opcode level. We have gathered ten of thousands of verified Ethereum smart contracts that have been written by using the Solidity language. This study can lay the groundwork for defining new formalisms and new domain specific languages (DSL). These can support the users in the development of decentralised applications (DAPPs).

The effect of linguistic comprehension instruction on generalized language and reading comprehension skills: A systematic review.

The effect of linguistic comprehension instruction on generalized language and reading comprehension skills: A systematic review.

Is Language Necessary for the Social Transmission of Lithic Technology?

AbstractRecently, a growing number of studies have considered the role of language in the social transmission of tool-making skill during human evolution. In this article, I address this question in light of a new theory of language and its evolution, and review evidence from anthropology and experimental archaeology related to it. I argue that the specific function of language—the instruction of imagination—is not necessary for the social transmission of tool-making skill. Evidence from hunter-gatherer ethnographies suggests that social learning relies mainly on observation, participation, play, and experimentation. Ethnographies of traditional stone cultures likewise describe group activities with simple, context-bound interactions embedded in the here and now. Experiments comparing gestural and verbal teaching of tool-making skills also demonstrate that language is not necessary for that process. I conclude that there is no convincing evidence that language played an important role in the social transmission of lithic technology, although the possibility that linguistic instruction was involved as part of the social interactions accompanying tool-making cannot be excluded.

ENGLISH FOR SPECIFIC PURPOSES AS A LINGUISTIC RESPONSE TO GLOBALIZATION

English for Specific Purposes is one of the most recent strains in English language teaching that appeared in the second half of the previous century following the wider societal trends subsumed under the term globalization. ESP is a narrowly, domain focused linguistic instruction in professional communication, the main goal being functional interaction. Language therefore becomes an instrument for achieving higher performativity. In such an educational setting, its core characteristic of being an ontology is disregarded. Thus, arguments will be presented here examining hidden cultural issues in the syllabus of English for Specific Purposes course. It will argue against the proposition that such syllabus is culturally void being supposedly focused on expert language only and for critical approaches to be integrated in this approach to English language instruction.

Paired Recurrent Autoencoders for Bidirectional Translation Between Robot Actions and Linguistic Descriptions

We propose a novel deep learning framework for bidirectional translation between robot actions and their linguistic descriptions. Our model consists of two recurrent autoencoders (RAEs). One RAE learns to encode action sequences as fixed-dimensional vectors in a way that allows the sequences to be reproduced from the vectors by its decoder. The other RAE learns to encode descriptions in a similar way. In the learning process, in addition to reproduction losses, we create another loss function whereby the representations of an action and its corresponding description approach each other in the latent vector space. Across the shared representation, the trained model can produce a linguistic description given a robot action. The model is also able to generate an appropriate action by receiving a linguistic instruction, conditioned on the current visual input. Visualization of the latent representations shows that the robot actions are embedded in a semantically compositional way in the vector space by being learned jointly with descriptions.

Critical SFL Praxis With Bilingual Youth: Disciplinary Instruction in a Third Space

ABSTRACTThe authors explore the theoretical and pedagogical premises of their critical systemic functional linguistics approach, which they developed to challenge the deficit positioning of bilingual students in the Southeast of the United States. As multilingual educators from postcolonial Ireland and India, the theoretical framework has helped the authors conceptualize and design interventions that support the incorporation of the linguistic and cultural repertoires of students while acknowledging that disciplinary literacy needs an explicit form of linguistic instruction. Key tenets of this approach include fostering a third space where voices, languages, and registers of students and teachers interweave to make and challenge disciplinary knowledge domains. The authors’ pedagogical and professional work, which they illustrate through discussion of data from a larger ethnographic study, has encouraged their students to switch among registers and languages while appropriating academic discourses in a high-stakes testing culture. It has also supported bilingual students in challenging normative discourses about immigration and other issues related to social inequity.

Multicomponent Linguistic Awareness Intervention for At-Risk Kindergarteners

This study investigated the feasibility of multicomponent linguistic awareness intervention on early literacy skills in at-risk kindergarteners. Seventeen students, including native Spanish-speaking English language learners ( n = 10) and native English speakers ( n = 7), participated in a 6-week small-group therapy program, for a total of 12 intervention hours. Students received therapy in one of the following: phonological awareness and letter knowledge; morphological awareness; or a three-pronged intervention that addressed all three areas. Students demonstrated moderate to large gains in word reading ( d = 1.79–2.19), phonological awareness ( d = 0.73–1.59), morphological awareness ( d = 0.57–3.96), and morphological spelling ( d = 0.77–3.0). Analyses revealed no significant differences based on the type of intervention received. These results provide preliminary evidence for the feasibility of three-pronged linguistic awareness instruction for kindergarten students at risk for later reading difficulties, including English language learners, in intensive intervention programs.

Dynamical Integration of Language and Behavior in a Recurrent Neural Network for Human-Robot Interaction.

To work cooperatively with humans by using language, robots must not only acquire a mapping between language and their behavior but also autonomously utilize the mapping in appropriate contexts of interactive tasks online. To this end, we propose a novel learning method linking language to robot behavior by means of a recurrent neural network. In this method, the network learns from correct examples of the imposed task that are given not as explicitly separated sets of language and behavior but as sequential data constructed from the actual temporal flow of the task. By doing this, the internal dynamics of the network models both language–behavior relationships and the temporal patterns of interaction. Here, “internal dynamics” refers to the time development of the system defined on the fixed-dimensional space of the internal states of the context layer. Thus, in the execution phase, by constantly representing where in the interaction context it is as its current state, the network autonomously switches between recognition and generation phases without any explicit signs and utilizes the acquired mapping in appropriate contexts. To evaluate our method, we conducted an experiment in which a robot generates appropriate behavior responding to a human’s linguistic instruction. After learning, the network actually formed the attractor structure representing both language–behavior relationships and the task’s temporal pattern in its internal dynamics. In the dynamics, language–behavior mapping was achieved by the branching structure. Repetition of human’s instruction and robot’s behavioral response was represented as the cyclic structure, and besides, waiting to a subsequent instruction was represented as the fixed-point attractor. Thanks to this structure, the robot was able to interact online with a human concerning the given task by autonomously switching phases.

Comparing two types of explicit pronunciation instructions on second language accentedness

Jang, Juhyun and Junkyu Lee. 2015. Comparing two types of explicit pronunciation instructions on second language accentedness. Linguistic Research 32 (Special Edition), 15-32. This study aims to compare the effect of two types of explicit pronunciation instructions, i.e., explicit prosody (EP) instruction and explicit segmental (ES) instruction, on second language (L2) accentedness. This study used a pretest-treatment-posttest design. Sixty two Korean EFL 5 th and 6 th graders of comparable English proficiency were randomly assigned into three groups (i.e., EP instruction group, ES instruction group, and Control group). Prior to pronunciation-focused activities, English teachers taught explicit linguistic information to the EP instruction group and the ES instruction group. The control group participated in a meaning-oriented activity in which explicit linguistic instruction was not provided. The instructions lasted for five weeks. The improvement of accentedness by each group was estimated by the performance of a reading task in which participants were asked to read aloud a series of narrative sentences. From the recorded data of the learners, foreign accents and pitch accents were analyzed into the sequential phases: a pretest, an immediate posttest, and a delayed posttest. Repeated measure ANOVA analyses revealed that the EP instruction group yielded a strong improvement of L2 accentedness (i.e., foreign accent and prosodic grouping with accent types: pitch accent, phrase accent, and intonational boundary) while the ES instruction group showed the positive results in relation to L2 accentendess as well. (Hankuk University of Foreign Studies)

Causal learning from probabilistic events in 24-month-olds: an action measure.

How do young children learn about causal structure in an uncertain and variable world? We tested whether they can use observed probabilistic information to solve causal learning problems. In two experiments, 24-month-olds observed an adult produce a probabilistic pattern of causal evidence. The toddlers then were given an opportunity to design their own intervention. In Experiment 1, toddlers saw one object bring about an effect with a higher probability than a second object. In Experiment 2, the frequency of the effect was held constant, though its probability differed. After observing the probabilistic evidence, toddlers in both experiments chose to act on the object that was more likely to produce the effect. The results demonstrate that toddlers can learn about cause and effect without trial-and-error or linguistic instruction on the task, simply by observing the probabilistic patterns of evidence resulting from the imperfect actions of other social agents. Such observational causal learning from probabilistic displays supports human children's rapid cultural learning.

Teaching Mathematics through VerbalLinguistic Intelligence

This paper is based on a quasiexperiment conducted last SY 2009-2010. From six sections of PreCollege Algebra course offered during first semester in Silliman University, one section was randomly chosen to be taught using verballinguistic approach and another section was chosen as the control. The control section made use of the traditional lecture method. Preests were given to both sections as well as posttests. The topics covered were (a) operations on algebraic expressions, and (b) factoring. There were 37 students in the experimental group and 33 in the control group. After the data had been analyzed using the method of Analysis of Covariance, the computed Fvalue for the pretest is 26.276 with pvalue of 0.00 and the Fvalue for treatment between VerbalLinguistic and Control is 5.817 with pvalue of 0.019. The first pvalue of 0.00 signified that there is a relationship between pretest results and posttest results and hence, the difference in the means between the control and experimental groups is due to teaching methods. The second pvalue of 0.019 on treatment implies that the group that received using the methods of Verballinguistic instruction performed better than those that received using the traditional lecture method.

Educational implications of the deficit/deprivation hypothesis in L2 situations: a case of Zimbabwe

The paper discussed Basil Bernstein’s language deprivation theory which presupposes that working class children have limited language competences as a result of their social and economic background. First, it established the anomaly in the use of some key terms in the theory such as deficit/deprivation, ‘elaborated’ and ‘restricted’ codes which are prejudicial and not truly reflective of what Bernstein actually meant. Then, it emerged that the education system, especially in the Zimbabwean ESL situation generally alienates children from financially challenged backgrounds. In the system, what came out as the most injurious are the examination system and the instructional methods used. Finally, the paper recommended a total overhaul of the whole system, starting from teachers’ training where more emphasis should be on cooperative and/or communicative language learning rather than ‘commandist’ linguistic instruction that is individualistic, resulting in the production of half-baked mimetic language graduates whose language codes remain ‘restricted’ even after ‘passing’ their L2 examinations. Key words: Deficit theory, restricted codes, elaborated codes, language web, implicit.