Video Modeling Research Articles

The Short Video Platform Recommendation Mechanism Based on the Improved Neural Network Algorithm to the Mainstream Media

Currently, the capacity of short videos continues to increase. Video manufacturers hope to enhance user experience and stickiness through recommendation algorithms, while users seek personalized videos to save time and money. Therefore, in order to address the data sparsity and high-dimensional feature extraction, this study proposes a novel short video platform recommendation model. The proposed method utilizes the term frequency inverse document frequency algorithm for text mining, and combines error back propagation neural network for learning to explore the potential connection between users and videos. This method combines natural language processing and image analysis in deep learning to construct accurate user and video models, deeply explore user interests, and improve the accuracy and effectiveness of recommendation systems for user preferences. The research results showed that the recommendation accuracy of this method was 66% and 70% respectively, and the prediction accuracy was 73.50% and 88% respectively. When Num=128, 200 data points were recommended within 0.3678s. The proposed algorithm outperforms the other three methods in terms of recommendation accuracy compared with the ItemCF and UserCF algorithms. This is because the method uses an approach based on image and user vector, combined with relevant features of the video and user. The proposed method can deeply explore the relevant features of videos and users, overcoming the data scarcity in previous collaborative screening, and guiding video recommendation on practical media platforms.

Video Modelling Module in Improving Self-Help Skills of Children with Autism Spectrum Disorder

This study aims to validate a video modelling module for children with Autism Spectrum Disorder (ASD) in Malaysia to improve their self-help skills. The module was developed using video modelling techniques to teach these skills successfully. Focus group discussion (FGD) was conducted to gather expert opinions on the module's content and structures. The results were analyzed through content analysis, revealing that the module achieved its objectives with minimal improvements needed before validation. The module can be widely used to improve self-help skills among children with ASD in Malaysia. The study's findings will be used to enhance the module further.

Fusion Attention for Action Recognition: Integrating Sparse-Dense and Global Attention for Video Action Recognition.

Conventional approaches to video action recognition perform global attention over the entire video patches, which may be ineffective due to the temporal redundancy of video frames. Recent works on masked video modeling adopt a high-ratio tube masking and reconstruction strategy as a pre-training method to mitigate the problem of focusing on spatial features well but not on temporal features. Inspired by this pre-training method, we propose Fusion Attention for Action Recognition (FAR), which fuses the sparse-dense attention patterns specialized for temporal features with global attention during fine-tuning. FAR has three main components: head-split sparse-dense attention (HSDA), token-group interaction, and group-averaged classifier. First, HSDA splits the head of multi-head self-attention to fuse global and sparse-dense attention. The sparse-dense attention is divided into groups of tube-shaped patches to focus on temporal features. Second, token-group interaction is used to improve information exchange between divided patch groups. Finally, the group-averaged classifier uses spatio-temporal features from different patch groups to improve performance. The proposed method uses the weight parameters that are pre-trained with VideoMAE and MVD, and achieves higher performance (+0.1-0.4%) with less computation than models fine-tuned with global attention on Something-Something V2 and Kinetics-400. Moreover, qualitative comparisons show that FAR captures temporal features quite well in highly redundant video frames. The FAR approach demonstrates improved action recognition with efficient computation, and exploring its adaptability across different pre-training methods presents an interesting direction for future research.

The effectiveness of perspective video modelling training on one‐stage word problem‐solving skills in children with autism spectrum disorder

AbstractSingle‐subject research can provide adequate justification when developing successful evidence‐based educational practices for children with autism spectrum disorder (ASD), which can help such students thrive academically. This research investigates whether point‐of‐view video modelling effectively improved the word problem‐solving addition performances of primary‐school‐aged students with ASD. The effectiveness of the intervention on each participant's ability to solve single‐digit addition by a single digit, effectively view the video on a tablet and generalise a learned skill was investigated by utilising a design that included several participant baselines. All participants saw an improvement in their ability to solve simple addition problems due to using point‐of‐view video modelling. Between the baseline and intervention phases, each participant's rate of digits correctly entered per minute and total number of steps completed significantly increased. A generalisation phase was performed at home. After receiving the intervention, people with ASD could independently complete word problems involving addition with a single digit. The findings suggest that this technology could practically support the education of the growing number of children and young people with ASD by mollifying the particular learning obstacles their impairment brings.

Beyond Granularity: Enhancing Continuous Sign Language Recognition with Granularity-Aware Feature Fusion and Attention Optimization

The advancement of deep learning techniques has significantly propelled the development of the continuous sign language recognition (cSLR) task. However, the spatial feature extraction of sign language videos in the RGB space tends to focus on the overall image information while neglecting the perception of traits at different granularities, such as eye gaze and lip shape, which are more detailed, or posture and gestures, which are more macroscopic. Exploring the efficient fusion of visual information of different granularities is crucial for accurate sign language recognition. In addition, applying a vanilla Transformer to sequence modeling in cSLR exhibits weak performance because specific video frames could interfere with the attention mechanism. These limitations constrain the capability to understand potential semantic characteristics. We introduce a feature fusion method for integrating visual features of disparate granularities and refine the metric of attention to enhance the Transformer’s comprehension of video content. Specifically, we extract CNN feature maps with varying receptive fields and employ a self-attention mechanism to fuse feature maps of different granularities, thereby obtaining multi-scale spatial features of the sign language framework. As for video modeling, we first analyze why the vanilla Transformer failed in cSLR and observe that the magnitude of the feature vectors of video frames could interfere with the distribution of attention weights. Therefore, we utilize the Euclidean distance among vectors to measure the attention weights instead of scaled-dot to enhance dynamic temporal modeling capabilities. Finally, we integrate the two components to construct the model MSF-ET (Multi-Scaled feature Fusion–Euclidean Transformer) for cSLR and train the model end-to-end. We perform experiments on large-scale cSLR benchmarks—PHOENIX-2014 and Chinese Sign Language (CSL)—to validate the effectiveness.

Virtual reality interview with feedback framework for situational practice of gaze among autistic adults

BackgroundAutistic individuals commonly seek employment; however, only a small fraction are in the workforce. In part, this might be due to mismatches between their social patterns of attention and gaze, and society’s normative expectations during interviews. MethodTo help mitigate such disadvantages through a solo situational practice tool, we present a framework that consists of a virtual reality (VR) based job interview simulation and a coaching component. Employing data visualization, video modeling, and VR role-play of the targeted behavior, the coaching support could be done in a self-deliverable practice manner. A participatory design session with two autistic design partners was important in the co-creation of the feedback methods, making them easier to understand, and including positive reinforcement. ResultsFourteen autistic individuals used the VR job interview simulation tool. Eleven received the gaze analysis and support stage and participated in a second VR simulation session. Preliminary results were positive, in that participant scores on average eye contact duration, average time without eye contact, and percentage of eye contact while listening and while speaking generally approached the corresponding medians of the non-autistic reference dataset. Participants were surveyed about the utility of the tool after the second feedback session and two months later. All survey respondents perceived the tool to be useful and the provided feedback to be helpful in daily social interactions. ConclusionsThis study provides insights towards the development of a VR job interview simulation and feedback framework that can enable solo situational practice of gaze and common interview questions.

Teaching a Physical Activity to a Young Child With Autism Using the Three-Tier Video Modeling Model: A Pilot Study

The Three-Tier Video Modeling intervention model was derived from the theory of Response to Intervention. This pilot study aimed to demonstrate the application of the Three-Tier Video Modeling model with a top-down approach in teaching physical activity to a 4-year-old boy with autism spectrum disorder and his typically developing peers. The study utilized a single case-changing conditions design and measured children’s performance in the target physical activity. The results showed that the boy with autism spectrum disorder acquired the target skill and consequently generalized it to the natural environment, and all peers mastered the skill after intervention. Teachers and parents also reported a more inclusive learning environment. The potential implications of this novel approach were discussed.

Effects of Curriculum-linked Social Story Intervention with Video Self-modeling on Academic Engagement Behaviors of a High School Student with Autism Spectrum Disorders

Effects of Curriculum-linked Social Story Intervention with Video Self-modeling on Academic Engagement Behaviors of a High School Student with Autism Spectrum Disorders

Training status in evidence-based practices among in-service special education teachers and its association with teaching in autism educational settings

The present study investigates the adequacy of training, perceived knowledge, and the frequency of use of evidence-based practices (EBPs) among in-service special education teachers and its association with their teaching practices in autism educational settings in Jordan. Additionally, it analyzes the influence of key variables, including gender, age, level of educational qualification, number of years teaching in special education, number of years teaching students with autism, number of students with autism in the classroom, and number of in-service professional development programs attended. Using an analytical descriptive approach utilizing survey-based research, a convenience sample of 121 teachers from seven specialized autism centers reported on their training adequacy, knowledge, and frequency of use of a set of 23 EBPs. The results showed that the adequacy of in-service training was higher than that of pre-service training. Moreover, there was a positive correlation between in-service training, teachers’ knowledge, and the use of EBPs. The number of in-service professional development training programs emerged as the only variable that significantly influenced the teachers’ responses regarding the adequacy of their in-service training. The multiple regression analysis revealed that in-service training programs and years of teaching students with autism were the only predictors of teachers’ knowledge and use of EBPs. The most reported EBPs were reinforcement, task analysis, and modeling, whereas the least reported were video modeling, social narratives, and functional communication. Further enhancements in teacher training in EBPs are needed to improve teacher training programs in Jordan.

M-adapter: Multi-level image-to-video adaptation for video action recognition

With the growing size of visual foundation models, training video models from scratch has become costly and challenging. Recent attempts focus on transferring frozen pre-trained Image Models (PIMs) to video fields by tuning inserted learnable parameters such as adapters and prompts. However, these methods require saving PIM activations for gradient calculations, leading to limited savings of GPU memory. In this paper, we propose a novel parallel branch that adapts the multi-level outputs of the frozen PIM for action recognition. It avoids passing gradients through the PIMs, thus naturally owning much lower GPU memory footprints. The proposed adaptation branch consists of hierarchically combined multi-level output adapters (M-adapters), comprising a fusion module and a temporal module. This design digests the existing discrepancies between the pre-training task and the target task with lower training costs. We show that when using larger models or on scenarios with higher demands for temporal modelling, the proposed method performs better than those with the full-parameter tuning manner. Finally, despite only tuning fewer parameters, our method achieves superior or comparable performance against current state-of-the-art methods.

The Effectiveness of Point-of-View Video Modeling in Improving Symbolic Play Skills of Children with Autism Spectrum Disorder

The Effectiveness of Point-of-View Video Modeling in Improving Symbolic Play Skills of Children with Autism Spectrum Disorder

Video Modeling to Support Social Communication Goals of Autistic Adults: A Tutorial.

Decades of research have shown video modeling to be an effective tool for teaching and supporting a variety of skills in autistic children. More recently, video modeling has emerged as an effective support for autistic adults, with much of the literature focused on vocational success through support of language skills. The purpose of this tutorial is to provide speech-language pathologists, autistic adults, and other team members with evidence-based guidelines for use of video modeling to support success with social communication across settings. This evidence-based tutorial draws from the literature on video modeling for autistic adults in the area of social communication, presenting empirically supported guidelines for speech-language pathologists considering video modeling as a tool to support social communication within this population and their interaction partners. This tutorial presents an evidence-based, step-by-step guide to the planning, creation, and use of video models with and by autistic adults, along with important additional considerations based on relevant literature. Video modeling is a social communication support that is backed by the scientific literature as an efficacious tool for use by autistic adults. This tutorial will guide speech-language pathologists' use of this evidence-based tool as they work alongside autistic adults and others to help meet social communication goals across settings.

Learning to enhance areal video captioning with visual question answering

ABSTRACT The utilization of Unmanned Aerial Vehicles (UAV) in remote sensing (RS) has witnessed a significant surge, offering valuable insights into Earth dynamics and human activities. However, this has led to a substantial increase in the volume of video data, rendering manual screening and analysis impractical. Consequently, there is a pressing need for the development of automated interpretation models for these aerial videos. In this paper, we propose a novel approach that leverages visual dialogue to enhance aerial video captioning. Our model adopts an encoder-decoder architecture, integrating a Visual Question Answering (VQA) task before the captioning task. The VQA task aims to enrich the captioning process by soliciting additional information about the image content. Specifically, our video encoder utilizes ViT-L/16, while the decoder employs Generative Pre-trained Transformer-2 (Distill-GPT-2). To validate our model, we introduce a novel benchmark dataset named CapERA-VQA, comprising videos accompanied by sets of questions, answers, and captions. Through experimental validation, we demonstrate the effectiveness of our proposed approach in enhancing the automated captioning of aerial videos.

Application of transformer architectures in generative video modeling for neurosurgical education.

This article explores the potential impact of OpenAI's Sora, a generative video modeling technology, on neurosurgical training. It evaluates how such technology could revolutionize the field by providing realistic surgical simulations, thereby enhancing the learning experience and proficiency in complex procedures for neurosurgical trainees. The study examines the incorporation of this technology into neurosurgical education by leveraging transformer architecture and processing of video and image data. It involves compiling a neurosurgical procedure dataset for model training, aiming to create accurate, high-fidelity simulations. Our findings indicate significant potential applications in neurosurgical training, including immersive simulations for skill development and exposure to diverse surgical scenarios. The technology also promises to transform assessment and feedback, introducing a standardized, objective way to measure and improve trainee competencies. Integrating generative video modeling technology into neurosurgical education marks a progressive step toward enhancing training methodologies. Despite challenges in technical, ethical, and practical domains, continuous development and evaluation could lead to substantial advancements in surgical education, preparing neurosurgeons more effectively for their demanding roles.

The Effectiveness of Teaching High-Risk Cooking Skills with Video Models to Individuals with Intellectual Disability

ABSTRACT The education of individuals with intellectual disability should include professional and safety skills to become qualified for jobs and increase employment rates. The purpose of this study is to examine the effectiveness of teaching these individuals high-risk cooking skills with video models in order to learn to work as assistant cooks. The results show that this approach was effective. In follow-up after one, two, and four weeks, all three subjects retained the nine skills taught by video and generalized them to different settings.

Beyond PhacoTrainer: Deep Learning for Enhanced Trabecular Meshwork Detection in MIGS Videos.

The purpose of this study was to develop deep learning models for surgical video analysis, capable of identifying minimally invasive glaucoma surgery (MIGS) and locating the trabecular meshwork (TM). For classification of surgical steps, we had 313 video files (265 for cataract surgery and 48 for MIGS procedures), and for TM segmentation, we had 1743 frames (1110 for TM and 633 for no TM). We used transfer learning to update a classification model pretrained to recognize standard cataract surgical steps, enabling it to also identify MIGS procedures. For TM localization, we developed three different models: U-Net, Y-Net, and Cascaded. Segmentation accuracy for TM was measured by calculating the average pixel error between the predicted and ground truth TM locations. Using transfer learning, we developed a model which achieved 87% accuracy for MIGS frame classification, with area under the receiver operating characteristic curve (AUROC) of 0.99. This model maintained a 79% accuracy for identifying 14 standard cataract surgery steps. The overall micro-averaged AUROC was 0.98. The U-Net model excelled in TM segmentation with an Intersection over union (IoU) score of 0.9988 and an average pixel error of 1.47. Building on prior work developing computer vision models for cataract surgical video, we developed models that recognize MIGS procedures and precisely localize the TM with superior performance. Our work demonstrates the potential of transfer learning for extending our computer vision models to new surgeries without the need for extensive additional data collection. Computer vision models in surgical videos can underpin the development of systems offering automated feedback for trainees, improving surgical training and patient care.

The use of video-based interventions to teach activity of daily living to children with autism spectrum disorder–scoping review

The study aimed to delve into the successful implementation of Video-Based Intervention (VBI) in improving activity of daily living skills (ADL) among children with autism spectrum disorder (ASD). Three primary electronic databases were utilised to perform this review: WOS, Scopus and Google Scholar. The process primarily used the research methodologies by Arksey and O'Malley as a framework. In the course of performing the scoping review, three themes emerged: firstly, a wide range of VBI techniques can be employed to enhance the skills of children with ASD; secondly, when VBI is coupled with adequate implementation support specifically tailored for children with ASD, it can effectively improve their daily living abilities. Essentially, the findings of this study support that VBI implementations can promote desired skills in ADL among children with ASD. Furthermore, additional implementation support complements video modeling, improving skill acquisition in ASD. This research enhances the existing knowledge base dedicated to enhancing the quality of life for children with ASD, providing valuable insights for educators, clinicians, and families involved in their care.

Emboldened in the rap "game": How severely stigmatized video models navigate disrespect and vulnerability to workplace mistreatment.

Moral stigma attached to an occupation can scar workers through discrediting, shaming, and denying respect. It can also open the door to interpersonal mistreatment, but little is known about how morally stigmatized workers navigate anticipated disrespect to potentially avoid harm. We explore this issue in a study of an occupation carrying severe moral stigma and where disrespect and workplace mistreatment are pervasive: models in hip-hop and rap music videos. Through analyses of 71 interviews with 48 video models and 19 industry informants, field observations, and archival data, we show how severe moral stigma and industry constraints promote generalized disrespect of video models (i.e., denial of worth to all role occupants) and, thus, each model's personal vulnerability to mistreatment. Two distinct groups of models emerged from our analysis-those who viewed themselves as emboldened in their role identity and those who did not-and this emboldened role identity was associated with differing perceptions of their personal vulnerability to mistreatment and their behaviors to mitigate it. The first group of models, those reporting an emboldened role identity, perceived their vulnerability to mistreatment as controllable. They strategically used both assertive behaviors (that earned respect from others) and passive behaviors (that avoided disrespect from others) to mitigate mistreatment. By contrast, the second group perceived their vulnerability to mistreatment as uncontrollable and reported using only passive behaviors (to avoid disrespect) when mistreatment was imminent. We discuss theoretical and practical implications of our findings, advancing knowledge of dirty work, workplace mistreatment, respect dynamics, and identity. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation

The action anticipation task refers to predicting what action will happen based on observed videos, which requires the model to have a strong ability to summarize the present and then reason about the future. Experience and common sense suggest that there is a significant correlation between different actions, which provides valuable prior knowledge for the action anticipation task. However, previous methods have not effectively modeled this underlying statistical relationship. To address this issue, we propose a novel end-to-end video modeling architecture that utilizes attention mechanisms, named Anticipation via Recognition and Reasoning (ARR). ARR decomposes the action anticipation task into action recognition and sequence reasoning tasks, and effectively learns the statistical relationship between actions by next action prediction (NAP). In comparison to existing temporal aggregation strategies, ARR is able to extract more effective features from observable videos to make more reasonable predictions. In addition, to address the challenge of relationship modeling that requires extensive training data, we propose an innovative approach for the unsupervised pre-training of the decoder, which leverages the inherent temporal dynamics of video to enhance the reasoning capabilities of the network. Extensive experiments on the Epic-kitchen-100, EGTEA Gaze+, and 50salads datasets demonstrate the efficacy of the proposed methods. The code is available at https://github.com/linuxsino/ARR .

Using Video Modeling Plus Feedback to Teach Vocational Social Skills to Employment-Aged Autistic Youth

Using Video Modeling Plus Feedback to Teach Vocational Social Skills to Employment-Aged Autistic Youth