Video Frames Research Articles

Motion-Aware Dynamic Graph Neural Network for Video Compressive Sensing.

Video snapshot compressive imaging (SCI) utilizes a 2D detector to capture sequential video frames and compress them into a single measurement. Various reconstruction methods have been developed to recover the high-speed video frames from the snapshot measurement. However, most existing reconstruction methods are incapable of efficiently capturing long-range spatial and temporal dependencies, which are critical for video processing. In this paper, we propose a flexible and robust approach based on the graph neural network (GNN) to efficiently model non-local interactions between pixels in space and time regardless of the distance. Specifically, we develop a motion-aware dynamic GNN for better video representation, i.e., represent each node as the aggregation of relative neighbors under the guidance of frame-by-frame motions, which consists of motion-aware dynamic sampling, cross-scale node sampling, global knowledge integration, and graph aggregation. Extensive results on both simulation and real data demonstrate both the effectiveness and efficiency of the proposed approach, and the visualization illustrates the intrinsic dynamic sampling operations of our proposed model for boosting the video SCI reconstruction results. The code and model will be released.

Detection and identification of un-uniformed shape text from blurred video frames

<p><span lang="EN-GB">The identification and recognition of text from video frames have received a lot of attention recently, that makes many computer vision-based applications conceivable. In this study, we modify the picture mask and the original identification of the mask region convolution neural network and permit detection in three levels, including holistic, sequence, and at the level of pixels. To identify the texts and determine the text forms, semantics at the pixel and holistic levels can be used. With masking and detection, existences of the character and the word are separated and recognised. In addition, text detection using the results of 2-D feature space instance segmentation is done. Moreover, we explore text recognition using an attention-based optical character recognition (OCR) method with mask</span><span lang="EN-US"> r</span><span lang="EN-GB">egion convolution neural networks (R-CNN) to address and detect the problem of smaller and blurrier texts at the sequential level. Using attribute maps of the word occurrences in sequence to seq, the OCR method calculates the character sequence. At last, a fine-grained learning strategy is proposed to constructs models at word level using the annotated datasets, resulting in the training of a more precise and reliable model. The well-known benchmark datasets ICDAR 2013 and ICDAR 2015 are used to test our suggested methodology.</span></p>

Framework for abnormal event detection and tracking based on effective sparse factorization strategy

The idea of tracking video objects has evolved to facilitate the area of surveillance systems. However, most current research efforts lie in speedy abnormal event detection and tracking of objects of interest tracking. However, the primary challenge is dealing with complex video structures' inherent redundancy. The existing research models for video tracking are more inclined towards improving accuracy. In contrast, the consideration of a more significant proportion of mobile object dynamics, e.g. abnormal events, in motion over the crowded video frame sequence is mainly overlooked, which is essential to study a specific movement pattern of the object of interest appearing in the frame sequence concerning the cost of computation factors. The study thereby introduces a unique strategy of speedy abnormal event detection and tracking, which facilitates video tracking to assess a specific pattern of object of interest movement over complex and crowded video scenes, considering a unique learning-based approach. The extensive simulation outcome further shows that the proposed tracking model accomplishes better tracking accuracy yet retains an optimized computation cost compared to the baseline studies. The computation of video tracking also accomplishes higher detection rates even in the challenging constraints of partial/complete occlusion, illumination variation and background clutter.

Internet of things assisted deep learning enabled driver drowsiness monitoring and alert system using CNN-LSTM framework

Abstract Driver fatigue has been generally viewed as a critical road safety factor and has been cited for contributing to a good percentage of traffic accidents. Developing systems to monitor and alert drowsy drivers is essential in reducing incidents. This research proposes a robust framework for driver drowsiness detection using a CNN-LSTM architecture developed by fusing facial landmark analysis with multiple aspect ratios. It uses, as key metrics in detecting drowsiness, the Eye Aspect Ratio (EAR), Pupil Circularity (PUC), the Mouth Aspect Ratio (MAR), and the Mouth over Eye Aspect Ratio (MOE). CNN-LSTM had been trained on YawDD, NITYMD, FL3D, and custom datasets. Data augmentation techniques such as flipping, scaling, shearing, rotation, brightness and contrast adjustment are used to generalize under different illumination conditions and driver postures. The system is implemented on NVIDIA’s 128-core Jetson Nano GPU platform and does real-time processing of video frames captured by a CSI camera. It detects eye closure and yawning as symptoms of driver fatigue and immediately raises an alert through vibrations in the seatbelt and pre-recorded voice messages. The Internet connectivity allows remote monitoring via mobile applications, making it safer by ensuring that alerts reach the driver and passengers. This CNN-LSTM model has been carefully tested for various scenarios, including day and night conditions, proving its effectiveness. The proposed framework indicated excellent performance concerning accuracy (98%), precision (95%), recall (93%), F1 score (94%), and AUC (99%), thereby revealing its efficiency in practical scenarios. By incorporating EAR, MAR, PUC, and MOE for the early detection of drowsiness in this CNN-LSTM architecture, the system will be able to alert the driver ahead of time so that necessary precautions can be taken to avoid an accident. The proposed approach enhances the driver's safety and is a scalable solution that adapts to different environments and populations.

Reduction of Vision-Based Models for Fall Detection

Due to the limitations that falls have on humans, early detection of these becomes essential to avoid further damage. In many applications, various technologies are used to acquire accurate information from individuals such as wearable sensors, environmental sensors or cameras, but all of these require high computational resources in many cases, delaying the response of the entire system. The complexity of the models used to process the input data and detect these activities makes them almost impossible to complete on devices with limited resources, which are the ones that could offer an immediate response avoiding unnecessary communications between sensors and centralized computing centers. In this work, we chose to reduce the models to detect falls using images as input data. We proceeded to use image sequences as video frames, using data from two open source datasets, and we applied the Sparse Low Rank Method to reduce certain layers of the Convolutional Neural Networks that were the backbone of the models. Additionally, we chose to replace a convolutional block with Long Short Term Memory to consider the latest updates of these data sequences. The results showed that performance was maintained decently while significantly reducing the parameter size of the resulting models.

STA-net: a deblurring network combined with spatiotemporal information for zinc froth flotation

Abstract The clarification of blurry frames in froth flotation videos is crucial for the identification and fault diagnosis of zinc flotation conditions. Irregular motion blur represents the primary blur phenomenon in froth flotation videos. Existing methods for motion blur removal have not effectively integrated spatio-temporal information, overlooking the utilization of clear frames, thus failing to achieve satisfactory results in the removal of froth flotation video blur frames. Therefore, this paper proposes a Spatio Temporal Awareness network (STA-net) capable of utilizing the temporal and spatial information from consecutive frames to naturally reconstruct froth images with clarity. This network consists of two sub-networks: the Temporal network (T-net) and the Spatio network (S-net). T-net extracts dynamic information between adjacent clear frames and blurry frames, while S-net captures spatial information within frames and enhances spatial texture details for the coarse images reconstructed based on the temporal information extracted by T-net in a layer-by-layer manner. Additionally, considering that blurry image datasets from actual froth flotation processes often lack corresponding clear images as training labels for the network, a method based on probability density functions (PDF) for generating blurry froth image datasets based clear images is proposed. This method extracts motion vectors from froth images to create blurry froth images that closely resemble the actual blurry conditions of froth images. The deblurring performance of STA-net has been validated in practical froth flotation processes.

Glottic opening detection using deep learning for neonatal intubation with video laryngoscopy.

This study aimed to develop an artificial intelligence (AI) method to augment video laryngoscopy (VL) by automating the detection of the glottic opening in neonates, as a step toward future studies on improving intubation outcomes. A deep learning model, YOLOv8, was trained on 1623 video frames from 84 neonatal intubations to detect the glottic opening and evaluated using 14-fold cross-validation on metrics like precision and recall. Additionally, it was compared with 25 medical providers of varied intubation experience to assess its relative performance. The model demonstrated a precision of 80.8% and a recall of 75.3% in identifying the glottic opening, detecting it 0.31 s faster than the average medical provider. It performed comparably or better than novice and intermediate providers, and slightly slower than experts. AI-powered tools can aid VL by providing real-time guidance, potentially enhancing neonatal intubation safety and efficiency for less experienced users.

An innovative traffic flow detection model based on temporal video frame analysis and grayscale aggregation quantification

AbstractCurrent traffic status detection methods heavily rely on historical traffic flow data and vehicle counts. However, these methods often fail to meet the stringent real‐time requirements of state detection, especially on edge devices with limited computing resources.To address these challenges, this study develops a traffic alert model using temporal video frame analysis and grayscale aggregation quantization techniques. Initially, the model uses distance mapping between pixel features and frames of road traffic videos to construct a comprehensive road environment and vehicle segmentation model. The model also establishes a mapping between pixel equidistant lines and actual distances, enabling precise congestion detection. This approach significantly reduces costs associated with traditional traffic detection methods as it does not rely on historical data. Performance evaluation using fixed‐point road monitoring data indicates that the proposed model outperforms traditional traffic state detection models, with a performance improvement of approximately 4.7% to 9.5%. Additionally, the model improves computing resource efficiency by approximately 72.5% and demonstrates substantial real‐time detection capabilities.

Facial Movements Extracted from Video for the Kinematic Classification of Speech

Speech Sound Disorders (SSDs) are prevalent communication problems in children that pose significant barriers to academic success and social participation. Accurate diagnosis is key to mitigating life-long impacts. We are developing a novel software solution—the Speech Movement and Acoustic Analysis Tracking (SMAAT) system to facilitate rapid and objective assessment of motor speech control issues underlying SSD. This study evaluates the feasibility of using automatically extracted three-dimensional (3D) facial measurements from single two-dimensional (2D) front-facing video cameras for classifying speech movements. Videos were recorded of 51 adults and 77 children between 3 and 4 years of age (all typically developed for age) saying 20 words from the mandibular and labial-facial levels of the Motor-Speech Hierarchy Probe Wordlist (MSH-PW). Measurements around the jaw and lips were automatically extracted from the 2D video frames using a state-of-the-art facial mesh detection and tracking algorithm, and each individual measurement was tested in a Leave-One-Out Cross-Validation (LOOCV) framework for its word classification performance. Statistics were evaluated at the α=0.05 significance level and several measurements were found to exhibit significant classification performance in both the adult and child cohorts. Importantly, measurements of depth indirectly inferred from the 2D video frames were among those found to be significant. The significant measurements were shown to match expectations of facial movements across the 20 words, demonstrating their potential applicability in supporting clinical evaluations of speech production.

Noise & mottle suppression methods for cumulative Cherenkov images of radiation therapy delivery

Purpose.Cherenkov imaging during radiotherapy provides a real time visualization of beam delivery on patient tissue, which can be used dynamically for incident detection or to review a summary of the delivered surface signal for treatment verification. Very few photons form the images, and one limitation is that the noise level per frame can be quite high, and mottle in the cumulative processed images can cause mild overall noise. This work focused on removing or suppressing noise via image postprocessing.Approach.Images were analyzed for peak-signal-to-noise and spatial frequencies present, and several established noise/mottle reduction algorithms were chosen based upon these observations. These included total variation minimization (TV-L1), non-local means filter (NLM), block-matching 3D (BM3D), alpha (adaptive) trimmed mean (ATM), and bilateral filtering. Each were applied to images acquired using a BeamSite camera (DoseOptics) imaged signal from 6x photons from a TrueBeam linac delivering dose at 600 MU min-1incident on an anthropomorphic phantom and tissue slab phantom in various configurations and beam angles. The standard denoised images were tested for PSNR, noise power spectrum (NPS) and image sharpness.Results.The average peak-signal-to-noise ratio (PSNR) increase was 17.4% for TV-L1. NLM denoising increased the average PSNR by 19.1%, BM3D processing increased it by12.1% and the bilateral filter increased the average PSNR by 19.0%. Lastly, the ATM filter resulted in the lowest average PSNR increase of 10.9%. Of all of these, the NLM and bilateral filters produced improved edge sharpness with, generally, the lowest NPS curve.Conclusion.For cumulative image Cherenkov data, NLM and the bilateral filter yielded optimal denoising with the TV-L1 algorithm giving comparable results. Single video frame Cherenkov images exhibit much higher noise levels compared to cumulative images. Noise suppression algorithms for these frame rates will likely be a different processing pipeline involving these filters incorporated with machine learning.

Enhancing Video Anomaly Detection with Improved UNET and Cascade Sliding Window Technique

Computer vision video anomaly detection still needs to be improved, especially when identifying images with unusual motions or objects. Current approaches mainly concentrate on reconstruction and prediction methods, and unsupervised video anomaly detection faces difficulties because there are not enough tagged abnormalities, which reduces accuracy. This paper presents a novel framework called the Improved UNET (I-UNET), designed to counteract overfitting by addressing the need for complex models that can extract subtle information from video anomalies. Video frame noise can be eliminated by preprocessing the frames with a Weiner filter. Moreover, the system uses Convolution Long Short-Term Memory (ConvLSTM) layers to smoothly integrate temporal and spatial data into its encoder and decoder portions, improving the accuracy of anomaly identification. The Cascade Sliding Window Technique (CSWT) is used post-processing to identify anomalous frames and generate anomaly scores. Compared to baseline approaches, experimental results on the UCF, UCSDped1, and UCSDped2 datasets demonstrate notable performance gains, with 99% accuracy, 90.8% Area Under Curve (AUC), and 10.9% Equal Error Rate (EER). This study provides a robust and accurate framework for video anomaly detection with the highest accuracy rate.

Take good care of your fish: fish re-identification with synchronized multi-view camera system

IntroductionFish re-identification (re-ID) is of great significance for fish monitoring and can contribute to aquaculture and fish breeding. Synchronizing information from different cameras is beneficial for optimizing re-ID performance.MethodsWe constructed the first underwater fish re-identification benchmark dataset (FS48) under three camera conditions. FS48 encompasses 48 different fish identities, 10,300 frames, and 39,088 bounding boxes, covering various lighting conditions and background environments. Additionally, we developed the first robust and accurate fish re-identification baseline, FSNet, which fuses information from three camera positions by extracting features from synchronized video frames of each position and combining the synchronized information.ResultsThe experimental results show that FS48 is universal and of high quality. FSNet has an effective network design and demonstrates good performance, achieving better re-identification performance by combining information from three positions, helping improve overall re-test accuracy, and evaluating the effectiveness of re-identification among detectors.DiscussionOur dataset will be released upon acceptance of this paper, which is expected to further promote the development of underwater fish re-identification.

Computer-Vision-Aided Deflection Influences Line Identification of Concrete Bridge Enhanced by Edge Detection and Time-Domain Forward Inference

To enhance the accuracy and efficiency of the deflection response measurement of concrete bridges with a non-contact scheme and address the ill-conditioned nature of the inverse problem in influence line (IL) identification, this study introduces a computer-vision-aided deflection IL identification method that integrates edge detection and time-domain forward inference (TDFI). The methodology proposed in this research leverages computer vision technology with edge detection to surpass traditional contact-based measurement methods, greatly enhancing the operational efficiency and applicability of IL identification and, in particular, addressing the challenge of accurately measuring small deflections in concrete bridges. To mitigate the limitations of the Lucas–Kanade (LK) optical flow method, such as unclear feature points within the camera’s field of view and occasional point loss in certain video frames, an edge detection technique is employed to identify maximum values in the first-order derivatives of the image, creating virtual tracking points at the bridge edges through image processing. By precisely defining the bridge boundaries, only the essential structural attributes are preserved to enhance the reliability of minimal deflection deformations under vehicular loads. To tackle the ill-posed nature of the inverse problem, a TDFI model is introduced to identify IL, recursively capturing the static bridge response generated by the bridge under the influence of successive axles of a multi-axle vehicle. The IL is then computed by dividing the response by the weight of the preceding axle. Furthermore, an axle weight ratio reduction coefficient is proposed to mitigate noise amplification issues, ensuring that the weight of the preceding axle surpasses that of any other axle. To validate the accuracy and robustness of the proposed method, it is applied to numerical examples of a simply supported concrete beam, indoor experiments on a similar beam, and field tests on a three-span continuous concrete beam bridge.

ACORN+: Adaptive Compression-Reconstruction for Device-Cloud Collaboration Video Services

With the improvement of edge-based autonomous systems such as mobile Industrial IoT(IIoT) networks, edge devices can capture and upload videos with increasing bitrates. Massive edge-computing end nodes are eager for adequate multimedia data to satisfy the requirements of real-time video services. However, existing encoding standards for video services in Web 2.0 are specifically designed for something other than IoT video streaming. We have improved our Adaptive Compression-Reconstruction framework ACORN to obtain ACORN+, based on compressed sensing and recent advances in deep learning. At end nodes, we compress multiple sequential video frames into a single frame to reduce video volume. Given that multiple kinds of intelligent tasks are expected to be finished on the device side, we also designed a device-cloud collaboration scheme where deep learning-based algorithms can be executed on both the device and server sides. Experiments reveal that video analytics can be conducted on compressed frames. Taking action recognition as a device-cloud collaboration use case, we find ACORN+ obtains more than 3x speedup on compressed frames. The reconstruction algorithm in ACORN+ is with 1- 4 dB improvements. Moreover, the encoding time cost and the encoded video volume are reduced by more than 4x under the ACORN+ framework. 1

Nose-Driven Cursor Control: An Assistive System for Disabled Individuals Using OCRM Similarity Tracking

This paper introduces a visual mouse system designed to assist disabled individuals in controlling computer mouse functions, such as cursor movement and clicking actions, using simple body gestures. Specifically, the system tracks the user’s nose tip to execute mouse commands, providing an untouchable alternative for desktop interaction. The main goal is to create an accessible, low-cost solution using widely available hardware to make computing more inclusive for individuals with physical disabilities. Using a camera, the system captures and translates user head movements into mouse gestures on the screen. The proposed methodology is built around visual tracking, isolating a sample region from each video frame and then identifying that region in subsequent frames through Optimal Cost Region Matching (OCRM). This similarity measure ensures accurate and seamless tracking of the user's movement, providing an effective control mechanism. Experimental results demonstrate that the OCRM-based visual mouse system outperforms existing Camera Mouse applications regarding reliability and efficiency. The proposed approach delivers a flexible, cost-effective, and consistent solution for enhancing computer accessibility for disabled users.

Vapor Channel Oscillations in Laser Lithotripsy.

Laser-based endoscopic procedures present special challenges to deliver energy for ablation or coagulation of target tissues. When optical fiber-target quasi-contact (< 0.5 mm distance) cannot be maintained or is undesirable, the creation of intervening vapor bubbles and channels provide for the necessary transmission of laser energy to the target. This work investigates the characteristics and the dynamics of vapor channels that directly affect ablation efficiency and ablation rate and are known to effect stone movement, all of which impact procedure efficiency and safety. A simplified, experimental model for thulium fiber laser (1940 nm) lithotripsy consists of a water-filled cuvette and a vertically oriented laser fiber (200 μm core diameter) with its tip at 9 mm for "quasi-free" bubble generation and at vapor channel working distances 1-5 mm from and centered on the transparent cuvette bottom simulating a target's surface. Laser power transmission is recorded and synchronized with video frames from a high-speed camera (24,260 frames per second) to capture the induced vapor channels' and bubbles' development. Laser-induced channel transmission from 0% to 100% for 1, 2, and 3 mm fiber-target distances undergoes oscillations with average periods of 0.32, 0.64, and 1.0 ms, respectively, for 500 W laser output power. For fixed fiber-target distances of 0.5, 1, and 2 mm, the variation of these average oscillation frequencies across laser powers from 500 to 1000 W is much smaller, not exceeding 14%. For fiber-target distances in the range of 1-5 mm, the fraction of the 500 W laser's total pulse energy delivered to the target for 1, 2, and 3 ms pulses linearly decreases from 0.78 to less than 0.2. The channel and bubble dynamics begin with a spherical seed bubble expansion centered on the distal fiber tip that evolves into a pear shape whose surface exhibits periodic irregularities attributable to laser beam interruption by water droplets within the developing bubble. The study of laser-induced channel oscillations provides quantitative information relating fiber-target distance to channel oscillation frequency and energy transmission onto a target. These oscillations directly effect ablation efficiency and ablation rates that are important parameters for the optimization of a procedure's safety and duration. Insights that may lead to further reduction in retropulsion are also presented. Lasers Surg. Med. 00:00-00, 2024. © 2024 Wiley Periodicals LLC.

INTELLIGENT VIDEO ANALYSIS TECHNOLOGY FOR AUTOMATIC FIRE CONTROL TARGET RECOGNITION BASED ON MACHINE LEARNING

Context. Target recognition is a priority in military affairs. This task is complicated by the fact that it is necessary to recognize moving objects, different terrain and landscape create obstacles for recognition. Combat actions can take place at different times of the day, accordingly, it is necessary to take into account the perspective of lighting and general lighting. It is necessary to detect the object in the video by segmenting the video frames, recognize and classify. Objective of the study is to develop a technology for the analysis of the development of a technology for recognizing targets in real time as a component of the fire control system, due to the use of artificial intelligence, YOLO and machine learning. Method. The article develops a video stream analysis technology for automatic target recognition of the fire control system based on machine learning. The paper proposes the development of a target recognition module as a component of the fire control system within the framework of the proposed information technology using artificial intelligence. The YOLOv8 pattern recognition model family was used to develop the target recognition module. The methods used during the study of the formed dataset. – Bounding Box: Noise – Up to 15% of pixels (limiting frame: adding salt and pepper noise to the image – up to 15% of pixels). – Bounding Box: Blur – Up to 2.5px (bounding box: adding Gaussian blur to the image – up to 2.5 pixels). – Cutout – 3 boxes with 10% size each (cut out a part of the image – 3 boxes of 10% size each). – Brightness Between –25% and +25% (changing the brightness of the image to increase the resistance of the model to changes in lighting and camera settings – from –25% to +25%). – Rotation – Between –15 and +15 (rotation of the image object – clockwise or counterclockwise by degrees from –15 to +15). – Flip – Horizontal (flip the image object horizontally). Results. The data is collected from open sources, in particular, from videos posted in open sources on the YouTube platform. The main task of data preprocessing is the classification of three classes of objects on video or in real time – APC, BMP and TANK. The dataset is formed using the Roboflow platform based on the labeling tools and subsequently the augmentation tools. The dataset consists of 1193 unique images – approximately equally for each class. The training was conducted using Google Colab resources. It took 100 epochs to train the model. Conclusions. Analysis is performed according to mAP50 (average precision as 0.85), mAP50-95 (0.6), precision (0.89) and recall (0.75). Big losses are due to the fact that the background was not taken into account during the research – training the module on the basis of confirmed data (images) of the background without technology

RoadSitu: Leveraging Road Video Frame Extraction and Three-Stage Transformers for Situation Recognition

Situation recognition is an crucial problem in scene understanding, activity understanding, and action reasoning as it provides a structured representation of the main activity depicted in the image.Semantic role labeling is crucial to situation recognition, which is challenging because a single action can have multiple meanings and purposes depending on its context. Understanding images beyond the highlighted actions requires inferences about the context of the scene, the objects, and their role in the captured event. Recently, situation recognition (SR) has been introduced, which jointly derives a collection of the action (activity), meaning-role, and noun (entities) pairs in the form of moving images. To label these frames as action frames, we must assign nouns (entities) to the role based on the content of the observed image. One of the main challenges is managing the complex dependencies between the assigned roles (nouns) and the predicted action, as the correct role assignment often depends on the accuracy of the action prediction. We introduce, RoadSitu, a road situation recognition that involves generating a structured summary of what is happening in a road scenario using an action and the semantic roles played by agents from a video frame. The action can describe a diverse set of situations, and the same agent can play various roles depending on the situation depicted in the video frame. Therefore, a situation recognition model needs to understand the context of each video frame and the visual-linguistic meaning of the semantic roles of that particular frame. One of the main challenges in this work is the complex task of annotating video frames with semantic roles and handling the structured dependencies between the assigned roles (nouns) and the predicted action (activity). Additionally, the sparsity of meaningful semantic information within road scenarios poses further difficulties. To overcome these challenges, we introduce a novel approach where action recognition and noun estimation work together interactively to form structured summaries of each situation. In experiments using a road video dataset obtained from a South Korean company, RoadSitu achieved significant improvements across various performance metrics, with a Top-1 verb accuracy of 43.46%, Top-5 verb accuracy of 72.48%, and value accuracy of 34.21%, outperforming baseline models such as GSRTR and JSL by 2.4% and 3.86% in Top-1 verb accuracy, respectively. These results demonstrate the effectiveness of our model in handling complex road scenarios.

Text–video retrieval re-ranking via multi-grained cross attention and frozen image encoders

State-of-the-art methods for text–video retrieval generally leverage CLIP embeddings and cosine similarity for efficient retrieval. Meanwhile, recent advancements in cross-attention techniques introduce transformer decoders to facilitate attention computation between text queries and visual tokens extracted from video frames, enabling a more comprehensive interaction between textual and visual information. In this study, we combine the advantages of both approaches and propose a fine-grained re-ranking approach incorporating a multi-grained text–video cross attention module. Specifically, the re-ranker enhances the top K similar candidates identified by the cosine similarity network. To explore video and text interactions efficiently, we introduce frame and video token selectors to obtain salient visual tokens at both frame and video levels. Then, a multi-grained cross-attention mechanism is applied between text and visual tokens at these levels to capture multimodal information. To reduce the training overhead associated with the multi-grained cross-attention module, we freeze the vision backbone and only train the multi-grained cross attention module. This frozen strategy allows for scalability to larger pre-trained vision models such as ViT-G, leading to enhanced retrieval performance. Experimental evaluations on text–video retrieval datasets showcase the effectiveness and scalability of our proposed re-ranker combined with existing state-of-the-art methodologies.

DFCNet +: Cross-modal dynamic feature contrast net for continuous sign language recognition

In sign language communication, the combination of hand signs and facial expressions is used to convey messages in a fluid manner. Accurate interpretation relies heavily on understanding the context of these signs. Current methods, however, often focus on static images, missing the continuous flow and the story that unfolds through successive movements in sign language. To address this constraint, our research introduces the Dynamic Feature Contrast Net Plus (DFCNet+), a novel model that incorporates both dynamic feature extraction and cross-modal learning. The dynamic feature extraction module of DFCNet+ uses dynamic trajectory capture to monitor and record motion across frames and apply key features as an enhancement tool that highlights pixels that are critical for recognizing important sign language movements, allowing the model to follow the temporal variation of the signs. In the cross-modal learning module, we depart from the conventional approach of aligning video frames with textual descriptions. Instead, we adopt a gloss-level alignment, which provides a more detailed match between the visual signals and their corresponding text glosses, capturing the intricate relationship between what is seen and the associated text. The enhanced proficiency of DFCNet+ in discerning inter-frame details translates to heightened precision on benchmarks such as PHOENIX14, PHOENIX14-T and CSL-Daily. Such performance underscores its advantage in dynamic feature capture and inter-modal learning compared to conventional approaches to sign language interpretation. Our code is available at https://github.com/fyzjut/DFCNet_Plus.