Frame Classification Research Articles

Lightweight Indoor Positioning System Based on Multiple Self-Learning Features and Key Frame Classification

Abstract. Traditional indoor positioning technologies mostly require advanced installation of hardware devices, resulting in high costs and long-term maintenance. With advancements in image recognition and deep learning technologies, indoor visual positioning based on image recognition has become increasingly mature. This method offers the benefits of low cost and does not require additional hardware installation. However, it still has inherent defects, such as cumbersome data collection, complex algorithms, and universality. To minimize indoor information pre-collection cost, improve versatility, and enable rapid deployment in low-performance mobile devices, this paper proposes a lightweight indoor positioning system based on multiple self-learning features and key frame classification. The system is divided into two stages: preprocessing and real-time positioning. In the preprocessing stage, image information is collected for the entire indoor environment, and a key-frame recognizer is trained based on the image information. Simultaneously, an environmental feature information database is established. In the real-time positioning stage, the system first uses mobile devices such as smartphones to obtain real-time video streams. A key frame recognizer based on convolutional neural networks identifies key frames in each video stream frame, thereby obtaining approximate positions for rough positioning. Second, feature points are identified in each frame of the video stream and matched with feature points with location information in the built environmental feature information database to calculate precise positions for fine positioning. It has significant optimizations compared with conventional visual solutions in terms of preprocessing data collection, algorithm performance consumption, and versatility.

Beyond PhacoTrainer: Deep Learning for Enhanced Trabecular Meshwork Detection in MIGS Videos.

The purpose of this study was to develop deep learning models for surgical video analysis, capable of identifying minimally invasive glaucoma surgery (MIGS) and locating the trabecular meshwork (TM). For classification of surgical steps, we had 313 video files (265 for cataract surgery and 48 for MIGS procedures), and for TM segmentation, we had 1743 frames (1110 for TM and 633 for no TM). We used transfer learning to update a classification model pretrained to recognize standard cataract surgical steps, enabling it to also identify MIGS procedures. For TM localization, we developed three different models: U-Net, Y-Net, and Cascaded. Segmentation accuracy for TM was measured by calculating the average pixel error between the predicted and ground truth TM locations. Using transfer learning, we developed a model which achieved 87% accuracy for MIGS frame classification, with area under the receiver operating characteristic curve (AUROC) of 0.99. This model maintained a 79% accuracy for identifying 14 standard cataract surgery steps. The overall micro-averaged AUROC was 0.98. The U-Net model excelled in TM segmentation with an Intersection over union (IoU) score of 0.9988 and an average pixel error of 1.47. Building on prior work developing computer vision models for cataract surgical video, we developed models that recognize MIGS procedures and precisely localize the TM with superior performance. Our work demonstrates the potential of transfer learning for extending our computer vision models to new surgeries without the need for extensive additional data collection. Computer vision models in surgical videos can underpin the development of systems offering automated feedback for trainees, improving surgical training and patient care.

Noise robustness evaluation of image processing algorithms for eye blinking detection

Robust algorithms for eye blinking detection are required due to the effects of noisy environments and varying light conditions on image-based detection methods. This paper compares five non-supervised image-based algorithms for eye blinking detection, evaluating their robustness to additive Gaussian noise. The algorithms were tested on a video dataset acquired using a smartphone and an ophthalmology chin rest. Through Monte Carlo simulation that introduces Gaussian noise at different intensities, we evaluate the algorithms’ precision, sensitivity, and F1-scores for frame classification, True Positive Rate (TPR), False Discovery Rate (FDR) and F1-score for the event detector. The results of experimental tests reveal significant variations in algorithms’ performance with increasing noise levels. Notably, the Image Correlation (IC) algorithm demonstrates superior eye blinking detection capabilities under various noise conditions, emerging as the most robust algorithm among those tested. This distinction highlights the potential of IC for reliable blink detection in noisy environments.

CANSat-IDS: An adaptive distributed Intrusion Detection System for satellites, based on combined classification of CAN traffic

The increasing dependence on satellite technology for critical applications, such as telecommunications, Earth observation, and navigation, underscores the need for robust security measures to safeguard these assets from potential cyber threats. Moreover, as many satellite systems rely on the Controller Area Network (CAN) protocol for efficient data exchange among onboard subsystems, they become prime targets for cyberattacks. While contributions present various options for detecting attacks in the CAN bus, no one proposes an architecture suitable for satellite systems. To address this concern, this paper presents a novel approach to develop an adaptive distributed Intrusion Detection System (IDS) for satellites, which integrates machine and deep learning techniques for the classification of CAN frames. This system is specifically designed to overcome the inherent power and computational challenges of satellite operations by executing time-based anomaly detection on board, and content-based detection at the ground segment. To evaluate the effectiveness of the proposed solution, experiments are conducted using representative Datasets. The obtained results demonstrate that the distributed IDS presented in this research offers a promising solution to improve the security of satellite systems by achieving high detection rates ranging from 91.12% to 99.86% (F1-score).

METHOD FOR IDENTIFYING THE EQUIVALENT FRAME OF A SOLUTION BY ANALYZING TITRATION VIDEO IMAGES USING AN AUTOMATIC APPROACH

Determination of equivalence in inorganic chemistry can be achieved by various methods, such as colorimetric titration, conductimetric titration and pH-metric titration. However, these traditional methods often require repetition to guarantee reliable results, increasing the time and cost of the experiment. In response to this limitation, some authors have suggested a semi-automatic approach based on colorimetric titration, although this also has subjective aspects. The aim of this article is to propose a new method based on intuitive observation to identify the equivalent frame in a titration video. This approach relies on the use of the KNN classifier to automate the detection of the reference frame within a predefined confidence interval. To facilitate this automation, a dataset was built up from experiments at the Groupe Chimie de lEau et Substances Naturelles (GCESNA) laboratory at the Institut National Polytechnique HouphouetBoigny (INP-HB).The performance of the KNN algorithm was effectively assessed by evaluating it against the performance indicators of precision, recall and F-measure. The results obtained are as follows: Precision: 92.2%, Recall: 91.7% and F-measure: 92.0%. These experimental results demonstrate the effectiveness of the KNN algorithm in frame classification.

Automated tear film break-up time measurement for dry eye diagnosis using deep learning

In the realm of ophthalmology, precise measurement of tear film break-up time (TBUT) plays a crucial role in diagnosing dry eye disease (DED). This study aims to introduce an automated approach utilizing artificial intelligence (AI) to mitigate subjectivity and enhance the reliability of TBUT measurement. We employed a dataset of 47 slit lamp videos for development, while a test dataset of 20 slit lamp videos was used for evaluating the proposed approach. The multistep approach for TBUT estimation involves the utilization of a Dual-Task Siamese Network for classifying video frames into tear film breakup or non-breakup categories. Subsequently, a postprocessing step incorporates a Gaussian filter to smooth the instant breakup/non-breakup predictions effectively. Applying a threshold to the smoothed predictions identifies the initiation of tear film breakup. Our proposed method demonstrates on the evaluation dataset a precise breakup/non-breakup classification of video frames, achieving an Area Under the Curve of 0.870. At the video level, we observed a strong Pearson correlation coefficient (r) of 0.81 between TBUT assessments conducted using our approach and the ground truth. These findings underscore the potential of AI-based approaches in quantifying TBUT, presenting a promising avenue for advancing diagnostic methodologies in ophthalmology.

Detecting video forgery: A machine learning approach for consistency analysis of video frames

Video forgery, a prevalent concern in today’s digital age, involves the deliberate manipulation of video content, often carried out using sophisticated video editing software. In response to this challenge, the need for an automated approach to detect forged video footage has become increasingly pressing. Our proposed methodology addresses this need by employing a multi-faceted strategy. It begins with the classification of video frames as either originating from genuine sources or having undergone manipulation. To assess the authenticity, the Δ r ¯ s metric is applied to evaluate the coherence of frame sequences. Additionally, we’ve harnessed the power of machine learning, training a model on a diverse dataset, namely the VIFFD dataset. This robust machine learning approach, particularly the suggested Support Vector Machine (SVM) method, consistently achieves an impressive average accuracy of 94.4%, showcasing its potential as a dependable and effective solution for video forgery detection. In an era where the trustworthiness of video content is of paramount importance, our method emerges as a pivotal safeguard, contributing significantly to the preservation of the integrity and credibility of visual media.

Testing the effects of moral intensity news frames on consumer boycott intention

PurposeThis exploratory research examined how emphasizing a brand’s unethical behaviour through high moral intensity news framing influences consumer boycott intention.Design/methodology/approachThe hypotheses were tested and validated using two experimental studies that expose customers of real retail and personal care product brands to news articles that have high and low moral intensity news frames.FindingsThe results showed high moral intensity news framing’s positive effect on consumer boycott intention. The frame’s influence is moderated by moral awareness and partially mediated by perceived moral intensity and moral judgement. The findings suggest that consumers’ perception of the frame and their attitude towards the brand will have a substantial role in boycott intention.Practical implicationsThese research outcomes aid in the understanding of news framing effects on boycott intention, providing both insights for consumer activists and managerial implications for stewards of brands.Originality/valueWhile previous research have examined the impact of news frames on the typical audience, there has been relatively little focus on news framing’s impact on consumers and their decision to boycott brands. This study addresses this gap by applying the work on emphasis framing to a consumer decision-making context. It also introduces moral intensity framing to the news frame classification. In addition, this study expands current conceptualizations of individual ethical decision-making to help explain consumer boycott intent.

Adapting and scaling a proven diabetes prevention program across 11 worksites in India: the INDIA-WORKS trial.

Structured lifestyle change education reduces the burden of cardiometabolic diseases such as diabetes. Delivery of these programs at worksites could overcome barriers to program adoption and improve program sustainability and reach; however, tailoring to the worksite setting is essential. The Integrating Diabetes Prevention in Workplaces (INDIA-WORKS) study tested the implementation and effectiveness of a multi-level program for reducing cardiometabolic disease risk factors at 11 large and diverse worksites across India. Herein, we describe and classify program adaptations reported during in-depth interviews and focus group discussions with worksite managers, program staff, and peer educators involved in program delivery, and program participants and drop-outs. We used thematic analysis to identify key themes in the data and classified reported program adaptations using the FRAME classification system. Adaptations were led by worksite managers, peer educators, and program staff members. They occurred both pre- and during program implementation and were both planned (proactive) and unplanned (proactive and reactive). The most frequently reported adaptations to the individual-level intervention were curriculum changes to tailor lessons to the local context, make the program more appealing to the workers at the site, or add a wider variety of exercise options. Other content adaptations included improvements to the screening protocol, intervention scheduling, and outreach plans to tailor participant recruitment and retention to the sites. Environment-level content adaptations included expanding or leveraging healthy food and exercise options at the worksites. Challenges to adaptation included scheduling and worksite-level challenges. Participants discussed the need to continue adapting the program in the future to continue making it relevant for worksite settings and engaging for employees. This study describes and classifies site-specific modifications to a structured lifestyle change education program with worksite-wide health improvements in India. This adds to the literature on implementation adaptation in general and worksite wellness in India, a country with a large and growing workforce with, or at risk of, serious cardiometabolic diseases. This information is key for program scale-up, dissemination, and implementation in other settings. Clinicaltrials.gov NCT02813668. Registered June 27, 2016.

Federated Deep Learning for Wireless Capsule Endoscopy Analysis: Enabling Collaboration Across Multiple Data Centers for Robust Learning of Diverse Pathologies

Wireless capsule endoscopy (WCE) is a revolutionary diagnostic method for small bowel pathology. The manual perusal of the resulting lengthy and redundant videos is cumbersome. Automated analysis of WCE video frames is an intricate data modeling task because of the diverse representations of anomalies caused by inappropriate capture conditions. Deep neural networks require training to learn diverse pathological manifestations utilizing heterogeneous data collected from multiple institutions. However, the accessibility of WCE data poses privacy concerns for multiple centers. The efficient learning of heterogeneous data distributed over multiple institutions in a privacy-preserving fashion has become a challenge hampering the adoption of AI-based diagnoses in clinical practice. Prior studies have contrived extensive data augmentation and the generation of synthetic images from the same center. However, models trained at one center are at risk of a lack of generalization for a global deployment. Federated learning (FL) is a novel paradigm in which models learn from distributed data and share knowledge without accessing the data themselves. This study proposes an FL framework for multiple anomaly classifications of WCE frames, elaborating on the potential of collaborative learning from multiple data centers on the edge. Our empirical results prove that the proposed decentralized approach can learn the generalized features of WCE frames. Validating heterogeneous test sets revealed a 10–12% improvement in performance for decentralized models based on FL compared to the best-case performance of centralized models, demonstrating the potential of the federated framework to support multiple anomaly classification of WCE frames while preserving data privacy across various clinical setups.

An implementation of intelligent YOLOv3-based anomaly detection model from crowded video scenarios with optimized ensemble pattern extraction

ABSTRACT The anomaly or abnormality detection in crowded scenes helps in identifying the violence and protecting the people from severe damage. Thus, there is a need to detect the anomalies with the classifier for learning information along with the usage of huge architectures. A new anomaly detection model is implemented in this model. The collected data is fed to optimal ensemble pattern extraction scheme through techniques like Local binary patterns (LBP), Local Gradient Pattern (LGP), and Local Tetra Pattern (LTrP). The weights are tuned by a new hybrid Spiral Search-based Black Widow Glowworm Swarm Optimization (SS-BWGSO) for getting the optimal ensemble patterns. Next, anomaly frame classification is carried out by optimized VGG16+ResNet technique, where the hyperparameters of VGG16 and ResNet are tuned by SS-BWGSO algorithm. Finally, anomaly detection is performed by the YOLOV3 classifier. Throughout the result analysis the higher performance of the designed technique is observed over the classical methods.

Glitch in the matrix: A large scale benchmark for content driven audio–visual forgery detection and localization

Most deepfake detection methods focus on detecting spatial and/or spatio-temporal changes in facial attributes and are centered around the binary classification task of detecting whether a video is real or fake. This is because available benchmark datasets contain mostly visual-only modifications present in the entirety of the video. However, a sophisticated deepfake may include small segments of audio or audio–visual manipulations that can completely change the meaning of the video content. To addresses this gap, we propose and benchmark a new dataset, Localized Audio Visual DeepFake (LAV-DF), consisting of strategic content-driven audio, visual and audio–visual manipulations. The proposed baseline method, Boundary Aware Temporal Forgery Detection (BA-TFD), is a 3D Convolutional Neural Network-based architecture which effectively captures multimodal manipulations. We further improve (i.e. BA-TFD+) the baseline method by replacing the backbone with a Multiscale Vision Transformer and guide the training process with contrastive, frame classification, boundary matching and multimodal boundary matching loss functions. The quantitative analysis demonstrates the superiority of BA-TFD+ on temporal forgery localization and deepfake detection tasks using several benchmark datasets including our newly proposed dataset. The dataset, models and code are available at https://github.com/ControlNet/LAV-DF.

QuEHRy: a question answering system to query electronic health records.

We propose a system, quEHRy, to retrieve precise, interpretable answers to natural language questions from structured data in electronic health records (EHRs). We develop/synthesize the main components of quEHRy: concept normalization (MetaMap), time frame classification (new), semantic parsing (existing), visualization with question understanding (new), and query module for FHIR mapping/processing (new). We evaluate quEHRy on 2 clinical question answering (QA) datasets. We evaluate each component separately as well as holistically to gain deeper insights. We also conduct a thorough error analysis for a crucial subcomponent, medical concept normalization. Using gold concepts, the precision of quEHRy is 98.33% and 90.91% for the 2 datasets, while the overall accuracy was 97.41% and 87.75%. Precision was 94.03% and 87.79% even after employing an automated medical concept extraction system (MetaMap). Most incorrectly predicted medical concepts were broader in nature than gold-annotated concepts (representative of the ones present in EHRs), eg, Diabetes versus Diabetes Mellitus, Non-Insulin-Dependent. The primary performance barrier to deployment of the system is due to errors in medical concept extraction (a component not studied in this article), which affects the downstream generation of correct logical structures. This indicates the need to build QA-specific clinical concept normalizers that understand EHR context to extract the "relevant" medical concepts from questions. We present an end-to-end QA system that allows information access from EHRs using natural language and returns an exact, verifiable answer. Our proposed system is high-precision and interpretable, checking off the requirements for clinical use.

Conceptual framework and documentation standards of cystoscopic media content for artificial intelligence

BackgroundThe clinical documentation of cystoscopy includes visual and textual materials. However, the secondary use of visual cystoscopic data for educational and research purposes remains limited due to inefficient data management in routine clinical practice. MethodsA conceptual framework was designed to document cystoscopy in a standardized manner with three major sections: data management, annotation management, and utilization management. A Swiss-cheese model was proposed for quality control and root cause analyses. We defined the infrastructure required to implement the framework with respect to FAIR (findable, accessible, interoperable, reusable) principles. We applied two scenarios exemplifying data sharing for research and educational projects to ensure compliance with FAIR principles. ResultsThe framework was successfully implemented while following FAIR principles. The cystoscopy atlas produced from the framework could be presented in an educational web portal; a total of 68 full-length qualitative videos and corresponding annotation data were sharable for artificial intelligence projects covering frame classification and segmentation problems at case, lesion, and frame levels. ConclusionOur study shows that the proposed framework facilitates the storage of visual documentation in a standardized manner and enables FAIR data for education and artificial intelligence research.

A Novel Method Using Deep Learning Technique for Automatic Grading and Classification of Interferometry Video Frames for Dry Eye Analysis

A Novel Method Using Deep Learning Technique for Automatic Grading and Classification of Interferometry Video Frames for Dry Eye Analysis

Classification of endoscopic image and video frames using distance metric-based learning with interpolated latent features.

The online version contains supplementary material available at 10.1007/s11042-023-14982-1.

Effectiveness of transferring ultrasound deep learning models from adults to pediatrics for frame based pneumonia classification

Hand-held ultrasound devices are emerging as a promising intervention to aid in diagnosing deadly early childhood pneumonia in the developing world. Lung ultrasound (LUS) data, however, can be difficult to read and interpret accurately, and thus require trained professionals. A variety of deep learning (DL) models have been developed to aid professionals in this task, but the difficulty curating quality training datasets limits the generalization capabilities of these models. To combat this data scarcity, we utilized a variety of DL models trained on LUS data collected from adult COVID-19 patients in Italy and were able to apply transfer learning to increase pneumonia related imaging pattern classification in LUS data from children and infants in rural Zambia. In this regard, we found that transfer learning significantly increased frame classification F1 performance as compared to training models from scratch. These findings indicate the promising potential for developing generalizable AI for LUS classification in diverse contexts given limited data. This analysis demonstrates that DL models effectively transfer their capabilities when fine tuned on a new population demographics, allowing the possibility to clinically deploy these models without the need to acquire large amounts of new data.

Application of Artificial Intelligence to the Monitoring of Medication Adherence for Tuberculosis Treatment in Africa: Algorithm Development and Validation.

Artificial intelligence (AI) applications based on advanced deep learning methods in image recognition tasks can increase efficiency in the monitoring of medication adherence through automation. AI has sparsely been evaluated for the monitoring of medication adherence in clinical settings. However, AI has the potential to transform the way health care is delivered even in limited-resource settings such as Africa. We aimed to pilot the development of a deep learning model for simple binary classification and confirmation of proper medication adherence to enhance efficiency in the use of video monitoring of patients in tuberculosis treatment. We used a secondary data set of 861 video images of medication intake that were collected from consenting adult patients with tuberculosis in an institutional review board-approved study evaluating video-observed therapy in Uganda. The video images were processed through a series of steps to prepare them for use in a training model. First, we annotated videos using a specific protocol to eliminate those with poor quality. After the initial annotation step, 497 videos had sufficient quality for training the models. Among them, 405 were positive samples, whereas 92 were negative samples. With some preprocessing techniques, we obtained 160 frames with a size of 224 × 224 in each video. We used a deep learning framework that leveraged 4 convolutional neural networks models to extract visual features from the video frames and automatically perform binary classification of adherence or nonadherence. We evaluated the diagnostic properties of the different models using sensitivity, specificity, F1-score, and precision. The area under the curve (AUC) was used to assess the discriminative performance and the speed per video review as a metric for model efficiency. We conducted a 5-fold internal cross-validation to determine the diagnostic and discriminative performance of the models. We did not conduct external validation due to a lack of publicly available data sets with specific medication intake video frames. Diagnostic properties and discriminative performance from internal cross-validation were moderate to high in the binary classification tasks with 4 selected automated deep learning models. The sensitivity ranged from 92.8 to 95.8%, specificity from 43.5 to 55.4%, F1-score from 0.91 to 0.92, precision from 88% to 90.1%, and AUC from 0.78 to 0.85. The 3D ResNet model had the highest precision, AUC, and speed. All 4 deep learning models showed comparable diagnostic properties and discriminative performance. The findings serve as a reasonable proof of concept to support the potential application of AI in the binary classification of video frames to predict medication adherence.

Coding infant engagement in the Face-to-Face Still-Face paradigm using deep neural networks

BackgroundThe Face-to-Face Still-Face (FFSF) task is a validated and commonly used observational measure of mother-infant socio-emotional interactions. With the ascendence of deep learning-based facial emotion recognition, it is possible that common complex tasks, such as the coding of FFSF videos, could be coded with a high degree of accuracy by deep neural networks (DNNs). The primary objective of this study was to test the accuracy of four DNN image classification models against the coding of infant engagement conducted by two trained independent manual raters. Methods68 mother-infant dyads completed the FFSF task at three timepoints. Two trained independent raters undertook second-by-second manual coding of infant engagement into one of four classes: 1) positive affect, 2) neutral affect, 3) object/environment engagement, and 4) negative affect. ResultsTraining four different DNN models on 40,000 images, we achieved a maximum accuracy of 99.5% on image classification of infant frames taken from recordings of the FFSF task with a maximum inter-rater reliability (Cohen's κ-value) of 0.993. LimitationsThis study inherits all sampling and experimental limitations of the original study from which the data was taken, namely a relatively small and primarily White sample. ConclusionsBased on the extremely high classification accuracy, these findings suggest that DNNs could be used to code infant engagement in FFSF recordings. DNN image classification models may also have the potential to improve the efficiency of coding all observational tasks with applications across multiple fields of human behavior research.

Spatiotemporal Edges for Arbitrarily Moving Video Classification in Protected and Sensitive Scenes

Classification of arbitrary moving objects including vehicles and human beings in a real environment (such as protected and sensitive areas) is challenging due to arbitrary deformation and directions caused by shaky camera and wind. This work aims at adopting a spatiotemporal approach for classifying arbitrarily moving objects. The intuition to propose the approach is that the behavior of the arbitrary moving objects caused by wind and shaky camera is inconsistent and unstable, while, for static objects, the behavior is consistent and stable. The proposed method segments foreground objects from background using the frame difference between median frame and individual frame. This step outputs several different foreground information. The method finds static and dynamic edges by subtracting Canny of foreground information from the Canny edges of respective input frames. The ratio of the number of static and dynamic edges of each frame is considered as features. The features are normalized to avoid the problems of imbalanced feature size and irrelevant features. For classification, the work uses 10-fold cross-validation to choose the number of training and testing samples, and the random forest classifier is used for the final classification of frames with static objects and arbitrarily moving objects. For evaluating the proposed method, we construct our own dataset, which contains video of static and arbitrarily moving objects caused by shaky camera and wind. The results on the video dataset show that the proposed method achieves the state-of-the-art performance (76% classification rate) which is 14% better than the best existing method.