Frame Representation Research Articles

Query-aware multi-scale proposal network for weakly supervised temporal sentence grounding in videos

Recently, weakly supervised temporal sentence grounding in videos (TSGV) has attracted extensive attention because it does not require precise start-end time annotations during training, and it can quickly retrieve interesting segments according to user needs. In weakly supervised TSGV, query reconstruction (QR)-based methods are the current mainstream, and the quality of proposals determines their performance. QR-based methods have two problems in proposal quality. First, a multi-modal global token is usually mapped to proposals with limited duration diversity, making it difficult to capture relevant segments at varying durations in real scenarios. Additionally, Gaussian functions are typically used to generate relatively fixed weights for frames within proposals, which weigh the original video features to generate proposal-specific features. This results in query-irrelevant frames affecting the discrimination of the proposal features. In this study, we propose a query-aware multi-scale proposal network (QMN). Initially, pre-trained encoders are used to extract video and query features. Subsequently, a multi-scale proposal generation module is designed to refine video features guided by queries and diversify the duration of the proposal. This module performs multi-modal interaction and multi-scale modeling to obtain proposals of different durations. Furthermore, to extract discriminative proposal features and enhance the modeling of proposal frame correlation, a query-aware weight generator is constructed to learn frame weights to suppress query-irrelevant frame representations through contrastive learning. Finally, the masked query is reconstructed using the proposal features to select the best proposal. The effectiveness of the proposed QMN is verified through experiments on the Charades-STA and ActivityNet-Captions datasets.

Bidirectional temporal and frame-segment attention for sparse action segmentation of figure skating

Temporal action segmentation is a task for understanding human activities in long-term videos. Most of the efforts have been focused on dense-frame action, which relies on strong correlations between frames. However, in the figure skating scene, technical actions are sparsely shown in the video. This brings new challenges: a large amount of redundant temporal information leads to weak frame correlation. To end this, we propose a Bidirectional Temporal and Frame-Segment Attention Module (FSAM). Specifically, we propose an additional reverse-temporal input stream to enhance frame correlation, learned by fusing bidirectional temporal features. In addition, the proposed FSAM contains a Multi-stage segment-aware GCN and decoder interaction module, aiming to learn the correlation between segment features across time domains and integrate embeddings between frame and segment representations. To evaluate our approach, we propose the Figure Skating Sparse Action Segmentation (FSSAS) dataset: The dataset comprises 100 samples of the Olympic figure skating final and semi-final competition, with more than 50 different men and women athletes. Extensive experiments show that our method achieves an accuracy of 87.75 and an edit score of 90.18 on the FSSAS dataset.

Extracting Semantic Frames from Specialized Corpora for Lexicographic Purposes

This paper focuses on Frame-based Terminology (FBT), a theoretical framework based on cognitive premises that underpin the representation of specialized objects and events. It demonstrates that specialized semantic frames can be inferred from lexical schemas and semantic annotation. This research used corpus techniques to collect data for the representation of multilingual semantic frames. The research focuses on deforestation within the domain of Environmental Science. The methodology involves the encapsulation of corpus queries and semantic annotation to guide the creation of specialized frames associated with the deforestation event. Although the creation of frame-based terminological resources can be time-consuming, our results show that it is facilitated by the use, of e-tools that extract the arguments of specialized predicates, which are often immersed in complex syntactic constructions.

Frame representation of quantum systems with finite-dimensional Hilbert space

Abstract There exist many attempts to define a Wigner function for quantum systems with finite-dimensional Hilbert space, each of them coming with its advantages and limitations. The existing finite versions have simple definitions, but they are based only on the existence of a formal analogy with the continuous-variable Wigner function and do not allow an intuitive state analysis. The continuous versions have more complicated definitions, but they are closer to the original Wigner function and allow a visualization of the quantum states. The version based on the concept of tight frame we present is finite, but it has certain properties and applications similar to those of continuous versions. It allows us to present a new graphical representation of qubit states, and to define new parameters concerning them. An important advantage of frame representation follows from the use of redundant information. The values taken by the frame version of Wigner function are not independent. They have to satisfy a large number of mathematical relations, useful in error detection and correction.

An empirical study of excitation and aggregation design adaptions in CLIP4Clip for video–text retrieval

CLIP4Clip model transferred from the CLIP has been the de-factor standard to solve the video clip retrieval task from frame-level input, triggering the surge of CLIP4Clip-based models in the video–text retrieval domain. In this work, we rethink the inherent limitation of widely-used mean pooling operation in the frame features aggregation and investigate the adaptions of excitation and aggregation design for discriminative video representation generation. We present a novel excitation-and-aggregation design, including (1) The excitation module is available for capturing non-mutually-exclusive relationships among frame features and achieving frame-wise features recalibration, and (2) The aggregation module is applied to learn exclusiveness used for frame representations aggregation. Similarly, we employ the cascade of sequential module and aggregation design to generate discriminative video representation in the sequential type. Besides, we adopt the excitation design in the tight type to obtain representative frame features for multi-modal interaction. The proposed modules are evaluated on three benchmark datasets of MSR-VTT, ActivityNet and DiDeMo, achieving MSR-VTT (43.9 R@1), ActivityNet (44.1 R@1) and DiDeMo (31.0 R@1). They outperform the CLIP4Clip results by +1.2% (+0.5%), +4.5% (+1.9%) and +9.5% (+2.7%) relative (absolute) improvements, demonstrating the superiority of our proposed excitation and aggregation designs. We hope our work will serve as an alternative for frame representations aggregation and facilitate future research.

Some Remarks on Projective Representations of Compact Groups and Frames

Some Remarks on Projective Representations of Compact Groups and Frames

DigCode—A generic mid-air gesture coding method on human-computer interaction

With high flexibility and rich semantic expressiveness, mid-air gesture interaction is an important part of natural human-computer interaction (HCI) and has broad application prospects. However, there is no unified representation frame for designing, recording, investigating and comparing HCI mid-air gestures. Therefore, this paper proposes an interpretable coding method, DigCode, for HCI mid-air gestures. DigCode converts the unstructured continuous actions into structured discrete string encoding. From the perspective of human cognition and expression, the research employed psychophysical methods to divide gesture actions into discrete intervals, defined the coding rules of representation in letters and numbers, and developed automated programs to enable encoding and decoding by using gesture sensors. The coding method can cover the existing representations of HCI mid-air gestures by considering human understanding and computer recognition and can be applied to HCI mid-air gesture design and gesture library construction.

Terminology in the domain of seafood: A comparative analysis Germany-Spain

In the last few decades, the study of terminology has undergone a cognitive shift that has led to the development of several approaches that study the social, linguistic, and cognitive dimension of terms, such as Communicative Theory of Terminology (CTT) and Frame-Based Terminology (FBT). CTT was developed in the early 1990s and argues that the study of terminology should be based on a communicative perspective, taking into account aspects such as the communicators and the context of communication. FBT has been developed from 2007 and uses certain aspects of Frame Semantics to conceptualise specialised domains and create non-language-specific representations through the analysis of the domain event and on the study of the behaviour of the terminological units in texts. The two theories share many of the same premises and propose the representation of the concepts of a domain in an ontology. FBT also proposes a representation in frames. We explore how these two methods of domain representation can be used to represent the terminology of the domain of seafood in Germany and Spain.

Leveraging Knowledge Management Techniques for Developing Multimedia Exercise Guides for Elderly Fall Prevention

In this research study, the primary objective was to develop a comprehensive exercise program specifically designed for elderly individuals. The focus was on evaluating various exercise postures and validating their impact on muscle groups. By integrating knowledge management systems with knowledge engineering methodologies, the aim was to optimize the design of exercise postures and promote optimal health outcomes for the elderly. Divided into three distinct experiments, the study employed a systematic approach to acquire, represent, and validate knowledge related to exercise postures for the elderly population. The use of knowledge management systems and engineering methodologies facilitated the design of effective exercise postures tailored to meet the unique needs and capabilities of elderly individuals. Experiment I focused on knowledge acquisition through structured interviews with physical therapists. The acquired knowledge was used to screen and prioritize exercise postures suitable for elderly individuals. Expert recommendations and analysis were employed to select a set of exercise postures. Using a matrix combination approach, 189 possible exercise postures were generated by combining aerobic and dance postures. Through a screening process, 52 postures were selected as suitable for elderly individuals. Experiment II utilized kinesthetic representation techniques to visually represent the 52 selected exercise postures for the elderly. Additionally, frame representation was employed to capture muscle specifications associated with each posture. The representation design was validated by physical exercise experts. In the first step of Experiment III, a total of 52 exercise postures were implemented and evaluated with elderly participants. The implementation and validation process aimed to identify the best and most appropriate postures for the elderly, considering factors such as satisfaction levels, difficulty levels, and safety considerations. Through this rigorous evaluation, the initial selection of 52 postures was narrowed down to a final set of 21 suitable postures. The validation results provided valuable insights into the effectiveness of the exercise postures and their impact on the elderly participants. It ensured that the chosen postures were not only effective in promoting optimal health outcomes but also minimized the risk of injury. The iterative assessment and refinement process contributed to the development of an evidence-based exercise program specifically tailored to the unique needs and capabilities of elderly individuals. For the second step of Experiment III, the effects of the selected exercise postures on different muscle groups were validated. Three physical exercise experts evaluated the impact on upper limb, trunk, and lower limb muscles. Specific muscle groups, such as brachioradialis, deltoid, quadriceps, and hamstring, were found to be strongly focused on during the exercises, while trunk muscles were rated as poor overall. The evaluation involved implementing and evaluating a total of 52 exercise postures with elderly participants. The selection of the final 21 postures was based on the evaluation results and provided insights into satisfaction levels, difficulty levels, and safety considerations for each exercise posture. The validation outcomes showed a high level of agreement among the experts, ranging from 79% to 91%. For conclusion, this research aimed to enhance exercise programs for the elderly by developing more effective methodologies for designing exercise postures. By considering the specific requirements of the elderly population and utilizing knowledge management systems, the study successfully created exercise postures that maximize health benefits and overall well-being. Through the implementation of knowledge engineering methodologies and the utilization of knowledge management systems, the research optimized the design of exercise postures for elderly individuals. The division of the study into three experiments enabled a comprehensive analysis of the acquired knowledge, leading to the development of exercise postures specifically tailored to the needs of the elderly population.

No More Shortcuts: Realizing the Potential of Temporal Self-Supervision

Self-supervised approaches for video have shown impressive results in video understanding tasks. However, unlike early works that leverage temporal self-supervision, current state-of-the-art methods primarily rely on tasks from the image domain (e.g., contrastive learning) that do not explicitly promote the learning of temporal features. We identify two factors that limit existing temporal self-supervision: 1) tasks are too simple, resulting in saturated training performance, and 2) we uncover shortcuts based on local appearance statistics that hinder the learning of high-level features. To address these issues, we propose 1) a more challenging reformulation of temporal self-supervision as frame-level (rather than clip-level) recognition tasks and 2) an effective augmentation strategy to mitigate shortcuts. Our model extends a representation of single video frames, pre-trained through contrastive learning, with a transformer that we train through temporal self-supervision. We demonstrate experimentally that our more challenging frame-level task formulations and the removal of shortcuts drastically improve the quality of features learned through temporal self-supervision. Our extensive experiments show state-of-the-art performance across 10 video understanding datasets, illustrating the generalization ability and robustness of our learned video representations. Project Page: https://daveishan.github.io/nms-webpage.

Stitching Segments and Sentences towards Generalization in Video-Text Pre-training

Video-language pre-training models have recently achieved remarkable results on various multi-modal downstream tasks. However, most of these models rely on contrastive learning or masking modeling to align global features across modalities, neglecting the local associations between video frames and text tokens. This limits the model’s ability to perform fine-grained matching and generalization, especially for tasks that selecting segments in long videos based on query texts. To address this issue, we propose a novel stitching and matching pre-text task for video-language pre-training that encourages fine-grained interactions between modalities. Our task involves stitching video frames or sentences into longer sequences and predicting the positions of cross-model queries in the stitched sequences. The individual frame and sentence representations are thus aligned via the stitching and matching strategy, encouraging the fine-grained interactions between videos and texts. in the stitched sequences for the cross-modal query. We conduct extensive experiments on various benchmarks covering text-to-video retrieval, video question answering, video captioning, and moment retrieval. Our results demonstrate that the proposed method significantly improves the generalization capacity of the video-text pre-training models.

3sG: Three‐stage guidance for indoor human action recognition

AbstractInference using skeleton to steer RGB videos is applicable to fine‐grained activities in indoor human action recognition (IHAR). However, existing methods that explore only spatial alignment are prone to bias, resulting in limited performance. The authors propose a Three‐stage Guidance (3sG) framework, leveraging skeleton knowledge to promote RGB in three stages. First, a soft shading image is proposed for alleviating background noise in videos, allowing the network to directly focus more on the motion region. Second, the authors propose to extract RGB frames of interest to reduce the computational effort. Furthermore, to explore more fully the complementary information between skeletons and RGB, the skeleton is coupled to the frame representation in a different spatial–temporal sharing pattern. Third, the global skeleton and skeleton‐guided RGB features are fed into the shared classifiers, which approximate the logit distributions of the two to enhance the performance in RGB unimodal. Finally, a fusion strategy that utilizes two learnable parameters to adaptively integrate the skeleton with the RGB is proposed. 3sG outperforms the state‐of‐the‐art results on the Toyota Smarthome dataset while it is more efficient than similar methods on the NTU RGB+D dataset.

A structure theorem for generalized-noncontextual ontological models

It is useful to have a criterion for when the predictions of an operational theory should be considered classically explainable. Here we take the criterion to be that the theory admits of a generalized-noncontextual ontological model. Existing works on generalized noncontextuality have focused on experimental scenarios having a simple structure: typically, prepare-measure scenarios. Here, we formally extend the framework of ontological models as well as the principle of generalized noncontextuality to arbitrary compositional scenarios. We leverage a process-theoretic framework to prove that, under some reasonable assumptions, every generalized-noncontextual ontological model of a tomographically local operational theory has a surprisingly rigid and simple mathematical structure — in short, it corresponds to a frame representation which is not overcomplete. One consequence of this theorem is that the largest number of ontic states possible in any such model is given by the dimension of the associated generalized probabilistic theory. This constraint is useful for generating noncontextuality no-go theorems as well as techniques for experimentally certifying contextuality. Along the way, we extend known results concerning the equivalence of different notions of classicality from prepare-measure scenarios to arbitrary compositional scenarios. Specifically, we prove a correspondence between the following three notions of classical explainability of an operational theory: (i) existence of a noncontextual ontological model for it, (ii) existence of a positive quasiprobability representation for the generalized probabilistic theory it defines, and (iii) existence of an ontological model for the generalized probabilistic theory it defines.

AMPS: Predicting popularity of short-form videos using multi-modal attention mechanisms in social media marketing environments

Emerging as a dominant content format amid the shift from television to mobile, short-form videos wield immense potential across diverse domains. However, the scarcity of datasets and established metrics for their popularity evaluation poses a challenge in accurately reflecting their real-world distribution. In response, our work introduces a dataset and pioneers a cumulative distribution function-based standard tailored specifically for short-form videos. Our model, AMPS (Attention-based Multi-modal Popularity prediction model of Short-form videos) is designed to effectively forecast the popularity of these videos. Considering YouTube Shorts, typically confined to under one minute, our research capitalizes on complete video frames for a holistic prediction of popularity. AMPS harnesses BiLSTM with Self-Attention and Co-Attention mechanisms, enabling a deeper understanding of intra-modal and inter-modal relationships across various modalities. Leveraging full video frame representation, our model significantly enhances prediction accuracy. Comprehensive evaluations against baseline models and machine learning algorithms consistently showcase AMPS' superiority in metrics like G-Mean, average F1-score, and Accuracy. Furthermore, when compared with other open social media datasets, our dataset coupled with AMPS consistently outperforms, affirming its robustness and reliability. Additionally, ablation studies underscore the effectiveness of AMPS' architecture and highlight the significance of each modality in predicting popularity.

Temporal Action Segmentation: An Analysis of Modern Techniques.

Temporal action segmentation (TAS) in videos aims at densely identifying video frames in minutes-long videos with multiple action classes. As a long-range video understanding task, researchers have developed an extended collection of methods and examined their performance using various benchmarks. Despite the rapid growth of TAS techniques in recent years, no systematic survey has been conducted in these sectors. This survey analyzes and summarizes the most significant contributions and trends. In particular, we first examine the task definition, common benchmarks, types of supervision, and prevalent evaluation measures. In addition, we systematically investigate two essential techniques of this topic, i.e., frame representation and temporal modeling, which have been studied extensively in the literature. We then conduct a thorough review of existing TAS works categorized by their levels of supervision and conclude our survey by identifying and emphasizing several research gaps.

Towards liveness: collective memory and reproductions of studio reverberation

ABSTRACT Through a consideration of their reverberant characteristics, this article places representations of recording studios as an active cultural process that plays out through the recorded sounds such spaces become associated with. Like other heritage sites, the work to define, maintain, and allocate certain structures as unique or significant relies on the cultural process of collective memory. By focusing on the concept of sonic recollection, the reconfiguration of unique tracking room environments for “in the box” digital audio workstation software, exemplified through reverberation, is shown to draw from a network of people, sounds, and technologies that cumulatively reflect various stages of a recording studio’s trajectory. By uncovering this network – encompassing well-known and unnamed engineers, famous to merely aspiring recording artists, established or defunct studios and software companies, and technologies such as convolution reverb, all of which articulate one another in digital audio workstation software – it suggests that the theoretical underpinnings of collective memory studies are a useful tool of analysis to more fully frame reconstructions or representations of physical spaces undertaken through recorded sound, and that reverberation is an ideal metaphor for the work of collective memory.

Theorizing the concept of social tie using frames

In classical Social Network Analysis (SNA), what counted as a “social tie” was fixed by available data collection methods. The emergence of large-scale unobtrusive data collection techniques has sparked renewed interest in the very idea of what counts as a “social tie.” Importantly, there has been an acknowledgment that the core issues raised by these developments are primarily conceptual. As a result, there is renewed interest in developing a scientifically grounded characterization of what is arguably the most central concept in social network analysis. This paper contributes to this conceptual effort. I rely on a technique of conceptual representation borrowed from cognitive psychology and cognitive linguistics in which frames for concepts are represented as directed graphs linking attributes to values. I show how the frame representation helps clarify the sort of claims that network theories make (e.g., imposing restrictions on attributes and values), how it helps specify both intra and inter-conceptual relations, how it illuminates seldom noted inter-theoretical commonalities and contrasts, and how it helps avoid common conceptual pitfalls.

The Ubiquity of Asia: Cai Guo-Qiang's Fireworks and Spectral Marxism

Asia is everywhere. Asia feels ubiquitous, from soft to hard power; from high to low culture; and from the Americas to Africa. In the American cycle of capital accumulation, under Giovanni Arrighi's schema, the overrepresentation of the United States was marked through representation-based and symbolic notions like Disneyfication (Jean Baudrillard) and “the ascendancy of whiteness” (Rey Chow). Under our contemporary Asian cycle of capital accumulation, this geographic area's affective ubiquity appears much like that of the US by saturating global public sentiment. However, considering that Asia's ubiquity emerges across sites like China, South Korea, India, and Japan, among others, what might we use to capture this sense of affective intensity and spread beyond representation-based frames? More specifically, how do we contend with geographic space within racial capital today, and how might we update our models for understanding race and capital so as to fully grapple with this transitional moment from American empire to Global Asia? Under the American cycle, representational frames like Orientalism and white supremacy emerged to understand Asian racialization. Can these frames still help explain the ubiquity of Asia today? This essay explores how we might discuss the subject beyond representation and in light of the ubiquity of Asia and what methods we deploy for Marxist analysis. This essay focuses less on Asian bodies and Asian spaces and turns toward aesthetic objects and practices that engage the affective sublime. More specifically, it examines the aesthetics of fireworks for what they might offer as a lexicon to contend with the transition in cycles of capital accumulation and changes in our ideas of space and race. The article thus turns to the artist Cai Guo-Qiang for insight on the subject amid contemporary racial capital and the methods available for critique.

Hybrid NVH modeling approach: High quality of NVH results enables psychoacoustic analysis of numerical computations

In the field of acoustics, the human perception of sound is often more important than the overall sound pressure level. The human perception is highly individual and is linked to how the user's expectations of the sound and the actual perceived sound match. Thus, generalization is difficult. However, psychoacoustic parameters that are developed and validated by means of jury tests are an established tool in the field of experimental measurements to quantify the human perception for specific acoustic phenomena like tonality, roughness, and loudness. Since the precision of numerical simulations results is continuously improving, these psychoacoustics parameters can be applied as well to the results of numerical simulation. In this Paper, the advantages of this approach are shown using the example of a numerical model of an e-bike. E-bikes have become increasingly popular in recent years, used for commuting, transporting loads or, in the case of e-mountain bikes, just for fun. As already mentioned, the acceptance of a sound is strongly linked to the expectations of the user. Since the electric drive unit of an e-bike is added to an acoustically known system, the user expects the sound to be quiet rather than dominant. Different e-bike analyses are presented using numerical and experimental methods. The numerical model used in this project predicts the sound generated by the electrical drive unit and radiated by the frame. For the calculation of airborne noise the distribution of velocities on the surface of the frame is used as an input, which is taken from a previous simulation of structure-borne noise. The model to simulate the structure-borne noise consists of a detailed representation of the carbon frame and simplified representations of different components such as battery, fork etc. With the help of the numerical model, different combinations of frame and electrical drive can be analyzed and compared by psychoacoustic parameters. The validity of the approach is shown by means of data from experimental measurements.

THE FRAME STRUCTURE OF THE CONCEPT “SECRET” IN ENGLISH LITERARY DISCOURSE

The article is focused on the study of the frame structure of the concept of secret based on English-language literary fiction. A concept is a mental representation of knowledge about a certain object, phenomenon, feature, or action. There are different types of knowledge representation structures: mental model, scheme, image, frame, scenario, gestalt. It is the frame that allows us to cognitively represent knowledge about the world. A frame is a structure of information used to represent a stereotypical situation, which is why this study focuses on the frame representation of the analyzed concept. The analysis of the dictionary definitions of the concept of secret has enabled us to identify the following conceptual features: information known to a narrow circle, confidentiality (verbalized by the lexemes — known by few people; shared confidentially with a few; should not be told to others; kept hidden); method for achieving a goal/success (method for doing sth well; the best way of achieving something); something mysterious, incomprehensible/something that is difficult to explain (mystery; cannot be explained/understood; unrevealed). Various aspects of the concept of secret are represented by subject frames (possessing one’s own or someone else’s secret; concealing or disclosing a secret; the existence of a secret in a certain social setting; discussing a secret; a secret as a surprise, or, by contrast, as shameful information), action frames (to report a secret; to hide a secret; to demand to reveal a secret; to promise to keep a secret), as well as by possessive frames (to possess a secret). The analysis of the characters’ speech in English-language novels has been helpful to unveil the following frame units of the concept secret: on the part of the subject: keeping a secret (the need to keep a secret, the protection of a secret by lying) and disclosure of a secret (due to situational necessity, one’s own emotional nature, the desire to show off one’s own knowledge, accidental disclosure of a secret, the use of “secret” knowledge as hints or warnings); on the part of the object: the demand to disclose a secret (asking the interlocutor for a secret and the requirement to avoid secrets) and the promise to keep a secret.