Recognition Datasets Research Articles

Hybrid attentive prototypical network for few-shot action recognition

Most previous few-shot action recognition works tend to process video temporal and spatial features separately, resulting in insufficient extraction of comprehensive features. In this paper, a novel hybrid attentive prototypical network (HAPN) framework for few-shot action recognition is proposed. Distinguished by its joint processing of temporal and spatial information, the HAPN framework strategically manipulates these dimensions from feature extraction to the attention module, consequently enhancing its ability to perform action recognition tasks. Our framework utilizes the R(2+1)D backbone network, coupling the extraction of integrated temporal and spatial features to ensure a comprehensive understanding of video content. Additionally, our framework introduces the novel Residual Tri-dimensional Attention (ResTriDA) mechanism, specifically designed to augment feature information across the temporal, spatial, and channel dimensions. ResTriDA dynamically enhances crucial aspects of video features by amplifying significant channel-wise features for action distinction, accentuating spatial details vital for capturing the essence of actions within frames, and emphasizing temporal dynamics to capture movement over time. We further propose a prototypical attentive matching module (PAM) built on the concept of metric learning to resolve the overfitting issue common in few-shot tasks. We evaluate our HAPN framework on three classical few-shot action recognition datasets: Kinetics-100, UCF101, and HMDB51. The results indicate that our framework significantly outperformed state-of-the-art methods. Notably, the 1-shot task, demonstrated an increase of 9.8% in accuracy on UCF101 and improvements of 3.9% on HMDB51 and 12.4% on Kinetics-100. These gains confirm the robustness and effectiveness of our approach in leveraging limited data for precise action recognition.

Feature extractor optimization for discriminative representations in Generalized Category Discovery

Generalized Category Discovery (GCD) task involves transferring knowledge from labeled known categories to recognize both known and novel categories within an unlabeled dataset. A significant challenge arises from the lack of prior information for novel categories. To address this, we develop a feature extractor that can learn discriminative features for both known and novel categories. Our approach leverages the observation that similar samples often belong to the same class. We construct a similarity matrix and employ similarity contrastive loss to increase the similarity between similar samples in the feature space. Additionally, we incorporate cluster labels to further refine the feature extractor, utilizing K-means clustering to assign these labels to unlabeled data, providing valuable supervision. Our feature extractor is optimized through the utilization of instance-level contrastive learning and class-level contrastive learning constraints. These constraints promote similarity maximization in both the instance space and the label space for instances sharing the same pseudo-labels. These three components complement each other, facilitating the learning of discriminative representations for both known and novel categories. Through comprehensive evaluations of generic image recognition datasets and challenging fine-grained datasets, we demonstrate that our proposed method achieves state-of-the-art performance.

Cucumber Disease Image Classification with A Model Combining LBP and VGG-16 Features

Cucumber (Cucumis sativus) is a significant horticultural crop worldwide, highly valued for both fresh consumption and processing. However, cucumber cultivation faces challenges due to diseases that can substantially reduce yield and quality. Diseases like leaf spots, stem wilt, and fruit rot are caused by pathogens including viruses, bacteria, and fungi. Traditionally, disease detection in cucumbers is performed manually, which is time-consuming and inefficient. Therefore, developing machine vision-based models using Deep Learning (DL) and Machine Learning (ML) for early disease detection through image analysis is crucial for assisting farmers. While many studies on plant disease classification using various DL and ML models show optimal results, research on cucumbers has mostly focused on leaf diseases. This study aims to optimize cucumber disease image classification by developing a model that combines Local Binary Pattern (LBP) texture features and VGG-16 convolutional features. The dataset used, Cucumber Disease Recognition Dataset consists of 8 classes of cucumber plant disease images covering leaves, stems, and fruits. This study classifies cucumber plant disease images using Random Forest (RF) combined with LBP texture features and VGG-16 visual features and compares its performance with models using VGG-16, LBP+RF, and VGG-16+RF on the same dataset. The results show that the proposed model achieved a precision of 84.7%, recall of 84%, F1-Score of 83.8%, and accuracy of 84%. These results outperform the comparative models, demonstrating the effectiveness of the combined approach in classifying cucumber plant diseases.

A radial basis deformable residual convolutional neural model embedded with local multi-modal feature knowledge and its application in cross-subject classification

In the spatial cognition and emotion recognition tasks based on electroencephalography (EEG), the signal information representation of the single modal is incomplete because of the significant inter-subject differences in the EEG signals, resulting in low generalization performance of classification models. To address this issue, we propose a radial basis deformable residual convolutional neural networks model embedded with local multi-modal feature knowledge (RBDRCNN-LMFK). The RBDRCNN leverages Euclidean distance alignment, deformable convolution, depth-separable convolution, and residual connection modules for effective EEG feature extraction. The embedded local feature knowledge algorithm enables the effective combination of multi-modal information. To further validate the effectiveness of the algorithm, we used the left-one-subject-out cross-validation algorithm on the virtual city walking (VCW), SJTU Emotion EEG Dataset IV (SEED IV), and Building block gesture recognition (BBGR) datasets for spatial cognition and emotion recognition tasks. The average accuracy of the VCW was 97.84 % using the kNN classifier, surpassing the Concate fusion method’s 96.62 %. The average accuracy for the SEED IV dataset was 87.56 %, higher than the Concate Fusion’s 69.97 %. On the BBGR dataset, the kNN classifier achieved an average accuracy of 87.80 %, compared to 85.31 % with the Concate fusion. The results show that the model enhances the recognition of biologically significant features in EEG signals by embedding local features and increases the correlation between different brain regions associated with the task. This work illustrates a promising direction of using deep learning models to discover effective task-related features from highly diverse EEG signals and enhance brain regions’ correlation through multi-modal knowledge embedding.

A new dataset for video-based cow behavior recognition

A new video based multi behavior dataset for cows, CBVD-5, is introduced in this paper. The dataset includes five cow behaviors: standing, lying down, foraging,rumination and drinking. The dataset comprises 107 cows from the entire barn, maintaining an 80% stocking density. Monitoring occurred over 96 h for these 20-month-old cows, considering varying light conditions and nighttime data to ensure standardization and inclusivity.The dataset consists of ranch monitoring footage collected by seven cameras, including 687 video segment samples and 206,100 image samples, covering five daily behaviors of cows. The data collection process entailed the deployment of cameras, hard drives, software, and servers for storage. Data annotation was conducted using the VIA web tool, leveraging the video expertise of pertinent professionals. The annotation coordinates and category labels of each individual cow in the image, as well as the generated configuration file, are also saved in the dataset. With this dataset,we propose a slowfast cow multi behavior recognition model based on video sequences as the baseline evaluation model. The experimental results show that the model can effectively learn corresponding category labels from the behavior type data of the dataset, with an error rate of 21.28% on the test set. In addition to cow behavior recognition, the dataset can also be used for cow target detection, and so on.The CBVD-5 dataset significantly influences dairy cow behavior recognition, advancing research, enriching data resources, standardizing datasets, enhancing dairy cow health and welfare monitoring, and fostering agricultural intelligence development. Additionally, it serves educational and training needs, supporting research and practical applications in related fields. The dataset will be made freely available to researchers world-wide.

MSAPersonality: a modern standard Arabic dataset for personality recognition

Automatic personality recognition is a task that attempts to automatically infer personality traits from a variety of data sources, including Text. Our words, whether spoken or written, reveal a lot about who we are. As people speak different languages, each with its own set of characteristics and level of complexity, identifying their personalities automatically might be language-dependent. This task requires an annotated text corpus with personality traits. However, the lack of corpora for languages other than English makes the task extremely challenging. We concentrated our efforts in this paper on the Arabic language in particular because it is understudied and lacks a corpus, despite being one of the most widely spoken languages in the world. Our primary goal was constructing our “MSAPersonality” dataset, which consists of 267 texts in modern standard Arabic that have been annotated with the Big Five personality traits. To evaluate the dataset and its potential for classification and regression, we used text preprocessing techniques, feature extraction, and machine learning algorithms. We obtained promising experimental results. Therefore, further research into predicting personality from Arabic text can be conducted.

CNN-LSTM for MFCC-based Speech Recognition on Smart Mirrors for Edge Computing Command

Smart mirrors are conventional mirrors that are augmented with embedded system capabilities to provide comfort and sophistication for users, including introducing the speech command function. However, existing research still applies the Google Speech API, which utilizes the cloud and provides sub-optimal processing time. Our research aim is to design speech recognition using Mel-frequency cepstral coefficients (MFCC) and convolutional neural network–long short-term memory (CNN-LSTM) to be applied to smart mirror edge devices for optimum processing time. Our first step was to download a synthetic speech recognition dataset consisting of waveform audio files (WAVs) from Kaggle, which included the utterances “left,” “right,” “yes,” “no,” “on,” and “off. ” We then designed speech recognition by involving Fourier transformation and low-pass filtering. We benchmark MFCC with linear predictive coding (LPC) because both are feature extraction methods on speech datasets. Then, we benchmarked CNN-LSTM with LSTM, simple recurrent neural network (RNN), and gated recurrent unit (GRU). Finally, we designed a smart mirror system complete with GUI and functions. The test results show that CNN-LSTM performs better than the three other methods with accuracy, precision, recall, and an f1-score of 0.92. The speech command with the best precision is "no," with a value of 0.940. Meanwhile, the command with the best recall is "off," with a value of 0.963. On the other hand, the speech command with the worst precision and recall is "other," with a value of 0.839. The contribution of this research is a smart mirror whose speech commands are carried out on the edge device with CNN-LSTM.

Overview of Pest Detection and Recognition Algorithms

Detecting and recognizing pests are paramount for ensuring the healthy growth of crops, maintaining ecological balance, and enhancing food production. With the advancement of artificial intelligence technologies, traditional pest detection and recognition algorithms based on manually selected pest features have gradually been substituted by deep learning-based algorithms. In this review paper, we first introduce the primary neural network architectures and evaluation metrics in the field of pest detection and pest recognition. Subsequently, we summarize widely used public datasets for pest detection and recognition. Following this, we present various pest detection and recognition algorithms proposed in recent years, providing detailed descriptions of each algorithm and their respective performance metrics. Finally, we outline the challenges that current deep learning-based pest detection and recognition algorithms encounter and propose future research directions for related algorithms.

Robust 3D Point Cloud Recognition: Enhancing Robustness with GPT-4 and CLIP Integration

In recent years, deep neural networks have achieved significant success in 3D point cloud recognition tasks. However, these models still demonstrate substantial performance challenges in the presence of data corruption. It is crucial to improve the robustness and generalization ability of the model. In this work, we propose a novel framework that combines GPT and CLIP models to enhance the robustness of existing point cloud classification models, which has two main modules: the Text-Image Fusion Module, which includes a GPT-Driven TextGen Processor and FocalView Projection, and the Dual-Path Intelligent Adapter Module. First, the GPT-Driven TextGen Processor leverages GPT-4’s capabilities to generate detailed textual descriptions tailored to point cloud intricacies. FocalView Projection dynamically selects viewpoints based on attention maps, enhancing two-dimensional representations of three-dimensional point clouds. Secondly, the Dual-Path Intelligent Adapter Module achieves fine-tuning and feature adaptation by combining internal and external adapters. Additionally, during the fine-tuning process, we employ a variant of Projected Gradient Descent (PGD) adversarial training, named VPGD, to increase the model’s resilience to adversarial perturbations. Our approach has achieved state-of-the-art results on robust 3D points cloud recognition datasets such as ModelNet40-C and ScanObjectNN-C.

Paired competing neurons improving STDP supervised local learning in Spiking Neural Networks.

Direct training of Spiking Neural Networks (SNNs) on neuromorphic hardware has the potential to significantly reduce the energy consumption of artificial neural network training. SNNs trained with Spike Timing-Dependent Plasticity (STDP) benefit from gradient-free and unsupervised local learning, which can be easily implemented on ultra-low-power neuromorphic hardware. However, classification tasks cannot be performed solely with unsupervised STDP. In this paper, we propose Stabilized Supervised STDP (S2-STDP), a supervised STDP learning rule to train the classification layer of an SNN equipped with unsupervised STDP for feature extraction. S2-STDP integrates error-modulated weight updates that align neuron spikes with desired timestamps derived from the average firing time within the layer. Then, we introduce a training architecture called Paired Competing Neurons (PCN) to further enhance the learning capabilities of our classification layer trained with S2-STDP. PCN associates each class with paired neurons and encourages neuron specialization toward target or non-target samples through intra-class competition. We evaluate our methods on image recognition datasets, including MNIST, Fashion-MNIST, and CIFAR-10. Results show that our methods outperform state-of-the-art supervised STDP learning rules, for comparable architectures and numbers of neurons. Further analysis demonstrates that the use of PCN enhances the performance of S2-STDP, regardless of the hyperparameter set and without introducing any additional hyperparameters.

Impact of face swapping and data augmentation on sign language recognition

AbstractThis study addresses the challenge of improving communication between the deaf and hearing community by exploring different sign language recognition (SLR) techniques. Due to privacy issues and the need for validation by interpreters, creating large-scale sign language (SL) datasets can be difficult. The authors address this by presenting a new Spanish isolated sign language recognition dataset, CALSE-1000, consisting of 5000 videos representing 1000 glosses, with various signers and scenarios. The study also proposes using different computer vision techniques, such as face swapping and affine transformations, to augment the SL dataset and improve the accuracy of the model I3D trained using them. The results show that the inclusion of these augmentations during training leads to an improvement in accuracy in top-1 metrics by up to 11.7 points, top-5 by up to 8.8 points and top-10 by up to 9 points. This has great potential to improve the state of the art in other datasets and other models. Furthermore, the analysis confirms the importance of facial expressions in the model by testing with a facial omission dataset and shows how face swapping can be used to include new anonymous signers without the costly and time-consuming process of recording.

Ensemble transfer learning using MaizeSet: A dataset for weed and maize crop recognition at different growth stages

Maize holds significant importance as a staple food source globally. Increasing maize yield requires the effective removal of weeds from maize fields, as they pose a detrimental threat to the growth of maize plants. In recent years, there has been a drive towards Precision Agriculture (PA), involving the integration of farming methods with artificial intelligence and advanced automation techniques. In the realm of PA, deep learning techniques present a promising solution for addressing the complex challenge of classifying maize plants and weeds. In this work, a deep learning method based on transfer learning and ensemble techniques is developed. The proposed method is implementable on any number of existing CNN models irrespective of their architecture and complexity. The developed ensemble model is trained and tested on our custom-built dataset, namely MaizeSet, comprising 3330 images of maize plants and weeds under varying environmental conditions. The performance of the ensemble model is compared against individual pre-trained VGG16 and InceptionV3 models using two experiments: the identification of weeds and maize plants, and the identification of the various vegetative growth stages of maize plants. VGG16 attained an accuracy of 83% in Experiment 1 and 71% in Experiment 2, while InceptionV3 showcased improved performance, boasting an accuracy of 98% in Experiment 1 and 81% in Experiment 2. With the proposed ensemble approach, VGG16 when combined with InceptionV3, achieved an accuracy of 90% for Experiment 1 and 80% for Experiment 2. The findings demonstrate that integrating a sub-optimal pre-defined classifier, specifically VGG16, with a more proficient model like InceptionV3, yields enhanced performance across various analytical metrics. This underscores the efficacy of ensemble techniques in the context of maize classification and analogous applications within the agricultural domain.

Fall Detection Method for Infrared Videos Based on Spatial-Temporal Graph Convolutional Network.

The timely detection of falls and alerting medical aid is critical for health monitoring in elderly individuals living alone. This paper mainly focuses on issues such as poor adaptability, privacy infringement, and low recognition accuracy associated with traditional visual sensor-based fall detection. We propose an infrared video-based fall detection method utilizing spatial-temporal graph convolutional networks (ST-GCNs) to address these challenges. Our method used fine-tuned AlphaPose to extract 2D human skeleton sequences from infrared videos. Subsequently, the skeleton data was represented in Cartesian and polar coordinates and processed through a two-stream ST-GCN to recognize fall behaviors promptly. To enhance the network's recognition capability for fall actions, we improved the adjacency matrix of graph convolutional units and introduced multi-scale temporal graph convolution units. To facilitate practical deployment, we optimized time window and network depth of the ST-GCN, striking a balance between model accuracy and speed. The experimental results on a proprietary infrared human action recognition dataset demonstrated that our proposed algorithm accurately identifies fall behaviors with the highest accuracy of 96%. Moreover, our algorithm performed robustly, identifying falls in both near-infrared and thermal-infrared videos.

Generative Adversarial Networks for SAR Automatic Target Recognition and Classification Models Enhanced Explainability: Perspectives and Challenges

Generative adversarial networks (or GANs) are a specific deep learning architecture often used for different usages, such as data generation or image-to-image translation. In recent years, this structure has gained increased popularity and has been used in different fields. One area of expertise currently in vogue is the use of GANs to produce synthetic aperture radar (SAR) data, and especially expand training datasets for SAR automatic target recognition (ATR). In effect, the complex SAR image formation makes these kind of data rich in information, leading to the use of deep networks in deep learning-based methods. Yet, deep networks also require sufficient data for training. However, contrary to optical images, we generally do not have a substantial number of available SAR images because of their acquisition and labelling cost; GANs are then an interesting tool. Concurrently, how to improve explainability for SAR ATR deep neural networks and how to make their reasoning more transparent have been increasingly explored as model opacity deteriorates trust of users. This paper aims at reviewing how GANs are used with SAR images, but also giving perspectives on how GANs could be used to improve interpretability and explainability of SAR classifiers.

Football teaching and training based on video surveillance using deep learning.

The objective performance evaluation of an athlete is essential to allow detailed research into elite sports. The automatic identification and classification of football teaching and training exercises overcome the shortcomings of manual analytical approaches. Video monitoring is vital in detecting human conduct acts and preventing or reducing inappropriate actions in time. The video's digital material is classified by relevance depending on those individual actions. The research goal is to systematically use the data from an inertial measurement unit (IMU) and data from computer vision analysis for the deep Learning of football teaching motion recognition (DL-FTMR). There has been a search for many libraries. The studies included have examined and analyzed training through profound model construction learning methods. Investigations show the ability to distinguish the efficiency of qualified and less qualified officers for sport-specific video-based decision-making assessments. Video-based research is an effective way of assessing decision-making due to the potential to present changing in-game decision-making scenarios more environmentally friendly than static picture printing. The data showed that the filtering accuracy of responses is improved without losing response time. This observation indicates that practicing with a video monitoring system offers a play view close to that seen in a game scenario. It can be an essential way to improve the perception of selection precision. This study discusses publicly accessible training datasets for Human Activity Recognition (HAR) and presents a dataset that combines various components. The study also used the UT-Interaction dataset to identify complex events. Thus, the experimental results of DL-FTMR give a performance ratio of 94.5%, behavior processing ratio of 92.4%, athletes energy level ratio of 92.5%, interaction ratio of 91.8%, prediction ratio of 92.5%, sensitivity ratio of 93.7%, and the precision ratio of 94.86% compared to the optimized convolutional neural network (OCNN), Gaussian Mixture Model (GMM), you only look once (YOLO), Human Activity Recognition- state-of-the-art methodologies (HAR-SAM). This finding proves that exercising a video monitoring system that provides a play view similar to that seen in a game scenario can be a valuable technique to increase selection accuracy perception.

GR‐Former: Graph‐reinforcement transformer for skeleton‐based driver action recognition

AbstractIn in‐vehicle driving scenarios, composite action recognition is crucial for improving safety and understanding the driver's intention. Due to spatial constraints and occlusion factors, the driver's range of motion is limited, thus resulting in similar action patterns that are difficult to differentiate. Additionally, collecting skeleton data that characterise the full human posture is difficult, posing additional challenges for action recognition. To address the problems, a novel Graph‐Reinforcement Transformer (GR‐Former) model is proposed. Using limited skeleton data as inputs, by introducing graph structure information to directionally reinforce the effect of the self‐attention mechanism, dynamically learning and aggregating features between joints at multiple levels, the authors’ model constructs a richer feature vector space, enhancing its expressiveness and recognition accuracy. Based on the Drive & Act dataset for composite action recognition, the authors’ work only applies human upper‐body skeleton data to achieve state‐of‐the‐art performance compared to existing methods. Using complete human skeleton data also has excellent recognition accuracy on the NTU RGB + D‐ and NTU RGB + D 120 dataset, demonstrating the great generalisability of the GR‐Former. Generally, the authors’ work provides a new and effective solution for driver action recognition in in‐vehicle scenarios.

Sign Tone: A Deep Learning-Based Deaf Companion System for Two Way Communication Between Deaf and Non-Deaf Individuals

Communication is essential to express and receive information, knowledge, ideas, and views among people, but it has been quite a while to be an obstruction for people with hearing and mute disabilities. Sign language is one method of communicating with deaf people. Though there is sign language to communicate with non-sign people it is difficult for everyone to interpret and understand. The performance of existing sign language recognition approaches is typically limited. Developing an assistive device that will translate the sign language to a readable format will help the deaf-mutes to communicate with ease to the common people. Recent advancements in the development of deep learning, deep neural networks, especially Temporal convolutional networks (TCNs) have provided solutions to the communication of deaf and mute individuals. In this project, the main objective is to design Deaf Companion System for that to develop SignNet Model to provide two-way communication of deaf individuals and to implement an automatic speaking system for deaf and mute people. It provides two-way communication for all classes of people (deaf-and-mute, hard of hearing, visually impaired, and non-signers) and can be scaled commercially. The proposed system, consists of three modules; the sign recognition module (SRM) that recognizes the signs of a deaf individual using TCN, the speech recognition using Hidden Marko Model and synthesis module (SRSM) that processes the speech of a non-deaf individual and converts it to text, and an Avatar module (AM) to generate and perform the corresponding sign of the non-deaf speech, which were integrated into the sign translation companion system called deaf companion system to facilitate the communication from the deaf to the hearing and vice versa. The proposed model is trained on Indian Sign Language. Then developed a web-based user interface to deploy SignNet Model for ease of use. Experimental results on MNIST sign language recognition datasets validate the superiority of the proposed framework. The TCN model gives an accuracy of 98.5%..

A survey for table recognition based on deep learning

Tables, as an important information interaction media containing much structured information, are popularly adopted in our life. There are a lot of table recognition methods for understanding and applying these information. Among them, table recognition based on deep learning is the mainstream. Therefore, we conducted this survey to summarize the latest research on table recognition based on deep learning in recent years. Table recognition can be divided into three sub-tasks, namely table detection, table structure recognition and table content recognition, so we firstly introduce the importance and significance of table recognition research and analyze the current challenges faced by each sub-task. Then, we list the existing public table recognition datasets and introduce commonly used evaluation metrics. Furthermore, we review deep learning-based solutions proposed in recent years for three different sub-tasks and reclassify them. Especially for table structure recognition, we reclassify them according to their annotation schemes. Finally, we report the performance of some excellent methods, such as CascadeTabNet, TSRFormer, SEM, etc, on public datasets, and describe the future research directions, including table recognition in natural scenes, question and answer about tables, visual information extraction, domain adaptation etc. In summary, this survey reviews the data, evaluations, methods, and development directions of table recognition. Table understanding and domain adaptation need as well to be focused on. The former can predict the relationships between two table elements, which helps us to understand the table. The latter can enhance the generalization ability of table recognition methods, thereby obtaining more robust recognition performance. Both are important development directions for table recognition in the future.

Intuitionistic fuzzy generalized eigenvalue proximal support vector machine

Generalized eigenvalue proximal support vector machine (GEPSVM) has attracted widespread attention due to its simple architecture, rapid execution, and commendable performance. GEPSVM gives equal significance to all samples, thereby diminishing its robustness and efficacy when confronted with real-world datasets containing noise and outliers. In order to reduce the impact of noises and outliers, we propose a novel intuitionistic fuzzy generalized eigenvalue proximal support vector machine (IF-GEPSVM). The proposed IF-GEPSVM assigns the intuitionistic fuzzy score to each training sample based on its location and surroundings in the high-dimensional feature space by using a kernel function. The solution of the IF-GEPSVM optimization problem is obtained by solving a generalized eigenvalue problem. Further, we propose an intuitionistic fuzzy improved generalized eigenvalue proximal support vector machine (IF-IGEPSVM) by solving standard eigenvalue decomposition resulting in simpler optimization problems with less computation cost which leads to an efficient intuitionistic fuzzy-based model. We conduct a comprehensive evaluation of the proposed IF-GEPSVM and IF-IGEPSVM models on UCI and KEEL benchmark datasets. Moreover, to evaluate the robustness of the proposed IF-GEPSVM and IF-IGEPSVM models, label noise is introduced into some UCI and KEEL datasets. The experimental findings showcase the superior generalization performance of the proposed IF-GEPSVM and IF-IGEPSVM models when compared to the existing baseline models, both with and without label noise. Our experimental results, supported by rigorous statistical analyses, confirm the superior generalization abilities of the proposed IF-GEPSVM and IF-IGEPSVM models over the baseline models. Furthermore, we implement the proposed IF-GEPSVM and IF-IGEPSVM models on the USPS recognition dataset, yielding promising results that underscore the models’ effectiveness in practical and real-world applications. The source code of the proposed IF-GEPSVM and IF-IGEPSVM models are available at https://github.com/mtanveer1/IF-GEPSVM.

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data.

Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods.