Frame Level Research Articles

A Review on Slit Steel Shear Walls: Development, Implementation, and Performance

Over centuries, natural phenomena have claimed a large number of lives and caused large amounts of damage. However, they were the reason for the development of knowledge and science. In engineering, there is no doubt that earthquakes were the driving force behind many design philosophies and advanced technologies. In this regard, steel plate shear walls (SPSWs) represent one of the technologies employed in both new constructions and existing structures to bolster lateral load resistance. In addition to its high elastic stiffness and strength, SPSWs often experience significant pinching during their hysteretic response unless heavily stiffened. Thus, numerous investigations have been performed recently to enhance the seismic behavior of SPSWs during severe ground shaking. The shear walls of steel plates have been studied in various ways, including stiffened and unstiffened steel plates, low yield strength steel plates, SPSWs perforated with circular holes or vertical slits, steel walls with stiffened and unstiffened openings. A stiffened SPSW panel dissipates significantly more energy than an unstiffened panel, as well as exhibiting a ductile and stable behavior. In view of its high elongation, the SPSW with low yield point has the best damping capacity. It has been demonstrated that steel walls perforated with circular holes have improved energy dissipation, in addition to allowing utilities to pass through their openings. Vertical slits on steel plate shear walls produce an exceptionally full hysteresis loop with improved performances. Slit steel walls (SSWs) can sustain a drift of over 3% while experiencing minimal hysteresis degradation. Additionally, a frame with a SSW can endure up to 6% drift without significant damage, surpassing expected ductility levels of a special moment frame, particularly with efficient slit geometry.

Safeguarding media integrity: A hybrid optimized deep feature fusion based deepfake detection in videos

The surge in deepfake techniques allows even non-experts to create deceptive videos using only source and target images, posing risks such as impersonation, character defamation, and spreading misinformation on various platforms. Existing methods often prove insufficient, particularly when faced with videos generated through diverse deepfake techniques that feature variations in lighting conditions and ethnicities. This paper introduces a novel approach called Hybrid Optimized Deep Feature Fusion-based Deepfake Detection (HODFF-DD) in videos, employing a spotted hyena optimizer. The HODFF-DD framework is designed to be resilient across ethnicities and varying lighting conditions, capable of detecting deepfake videos created by different techniques. The architecture comprises two integral components: a customized model incorporating InceptionResNetV1 and InceptionResNetV2, and a Bidirectional Long-Short Term Memory (BiLSTM). In this framework, faces extracted from videos undergo feature extraction at the frame level using our custom model comprising InceptionResNetV1 and InceptionResNetV2. The resulting sequences of features are then utilized to train a temporally aware BiLSTM for binary classification between real and fake videos. To optimize network weights, a bio-inspired spotted hyena optimizer is applied during the training process. Evaluation on diverse datasets, including the Kaggle dataset, FaceForensics++ (FF++) encompassing videos manipulated using various techniques (DeepFakes, FaceSwap, Face2Face, FaceShifter, and NeuralTextures), and the FakeAVCeleb dataset, demonstrates the effectiveness of our presented approach. Our method achieves an accuracy exceeding 90 % on subsets like DeepFakes, FaceSwap, and Face2Face.

A Framework for Video Summarization using Visual Attention Technique

Objectives: To develop an efficient Video Summarization technique that aims to utilize the saliency map for mimicking the human way of selecting the important events in the given video. Methods: This paper proposes Histogram based Weighted Fusion (HWF) algorithm that uses spatial and temporal saliency maps to act as guidance in creating the summary of the video. The spatial saliency score and temporal saliency score obtained from the corresponding saliency maps are fused using the proposed HWF algorithm to obtain the frame level importance score. It tries to depict the visual attention of the human brain when watching a particular video. Findings: The experimental results show that the proposed HWF algorithm performs better than the state-of-the-art methods. Novelty: The use of Histogram intersection and the incorporation of the exponential function as the weight for the combined feature enhance the summarization ability of the proposed model. Keywords: Video Summarization, Saliency Map, Histogram intersection, Contrast sensitivity function, Attention curves

Leveraging Frame- and Feature-Level Progressive Augmentation for Semi-supervised Action Recognition

Semi-supervised action recognition is a challenging yet prospective task due to its low reliance on costly labeled videos. One high-profile solution is to explore frame-level weak/strong augmentations for learning abundant representations, inspired by the FixMatch framework dominating the semi-supervised image classification task. However, such a solution mainly brings perturbations in terms of texture and scale, leading to the limitation in learning action representations in videos with spatiotemporal redundancy and complexity. Therefore, we revisit the creative trick of weak/strong augmentations in FixMatch, and then propose a novel Frame- and Feature-level augmentation FixMatch (dubbed as F 2 -FixMatch) framework to learn more abundant action representations for being robust to complex and dynamic video scenarios. Specifically, we design a new Progressive Augmentation (P-Aug) mechanism that implements the weak/strong augmentations first at the frame level, and further implements the perturbation at the feature level, to obtain abundant four types of augmented features in broader perturbation spaces. Moreover, we present an evolved Multihead Pseudo-Labeling (MPL) scheme to promote the consistency of features across different augmented versions based on the pseudo labels. We conduct extensive experiments on several public datasets to demonstrate that our F 2 -FixMatch achieves the performance gain compared with current state-of-the-art methods. The source codes of F 2 -FixMatch are publicly available at https://github.com/zwtu/F2FixMatch.

DLLBVS: Design of a High-Efficiency Deep Learning based Low-BER Video Streaming Model for High-Noise Wireless Networks

Design of video streaming models for high-noise networks is a multimodal process that requires efficient channel modelling, noise analysis, error-specific stream control, intelligent data augmentation, etc. This article proposes design of a high-efficiency deep learning based low-BER (Bit Error Rate) Video streaming model for high-noise wireless networks. The proposed model initially uses a Long-Short-Term Memory (LSTM) block for estimation of frame level features, and then uses Orthogonal Frequency Division Multiple Access (OFDMA) based modulation platform for transmission and reception of processed video frames. The OFDMA model is cascaded with a chaotic communication module, which assists in improving data fidelity under different noise conditions. The chaotic communication module is optimized using Grey Wolf Optimizer (GWO), which assists in setting up its hyperparameters for better efficiency under real-time communication scenarios. Video frames were pre-processed using Dual Neural Networks that assisted in estimation of differential frame information sets. Due to estimation of this differential frame information, streaming speed is improved, which assists in increasing number of frames transmitted per second, thereby improving streaming performance for different video types. This frame information is further processed by an iterative Gated Recurrent Unit (GRU) based VARMAx Model, which assists in predicting frame removal instances, thus assisting in faster & low error communications. The GRU VARMAx Model optimizes data flow, thus reducing congestion and improving throughput. The model was tested under AWGN (Additive While Gaussian Noise), Rayleigh, & Rician channel types, and its efficiency was compared with standard streaming techniques in terms of communication delay, Bit Error Rate, Peak to Average Power Ratio (PAPR), throughput, communication jitter and computational complexity levels. Based on this comparison it was observed that the proposed model showcased 3.5% lower communication delay, 8.3% lower BER, 5.9% lower PAPR, 10.5% higher throughput, 3.9% lower jitter and 6.4% lower computational complexity, which makes it highly useful for a wide variety of real-time streaming scenarios.

SF-TMN: SlowFast temporal modeling network for surgical phase recognition.

Automatic surgical phase recognition is crucial for video-based assessment systems in surgical education. Utilizing temporal information is crucial for surgical phase recognition; hence, various recent approaches extract frame-level features to conduct full video temporal modeling. For better temporal modeling, we propose SlowFast temporal modeling network (SF-TMN) for offline surgical phase recognition that can achieve not only frame-level full video temporal modeling but also segment-level full video temporal modeling. We employ a feature extraction network, pretrained on the target dataset, to extract features from video frames as the training data for SF-TMN. The Slow Path in SF-TMN utilizes all frame features for frame temporal modeling. The Fast Path in SF-TMN utilizes segment-level features summarized from frame features for segment temporal modeling. The proposed paradigm is flexible regarding the choice of temporal modeling networks. We explore MS-TCN and ASFormer as temporal modeling networks and experiment with multiple combination strategies for Slow and Fast Paths. We evaluate SF-TMN on Cholec80 and Cataract-101 surgical phase recognition tasks and demonstrate that SF-TMN can achieve state-of-the-art results on all considered metrics. SF-TMN with ASFormer backbone outperforms the state-of-the-art Swin BiGRU by approximately 1% in accuracy and 1.5% in recall on Cholec80. We also evaluate SF-TMN on action segmentation datasets including 50salads, GTEA, and Breakfast, and achieve state-of-the-art results. The improvement in the results shows that combining temporal information from both frame level and segment level by refining outputs with temporal refinement stages is beneficial for the temporal modeling of surgical phases.

Transfer Learning-Based B-Line Assessment of Lung Ultrasound for Acute Heart Failure

ObjectiveB-lines assessed by lung ultrasound (LUS) outperform physical exam, chest radiograph, and biomarkers for the associated diagnosis of acute heart failure (AHF) in the emergent setting. The use of LUS is however limited to trained professionals and suffers from interpretation variability. The objective was to utilize transfer learning to create an AI-enabled software that can aid novice users to automate LUS B-line interpretation. MethodsData from an observational AHF LUS study provided standardized cine clips for AI model development and evaluation. A total of 49,952 LUS frames from 30 patients were hand scored and trained on a convolutional neural network (CNN) to interpret B-lines at the frame level. A random independent evaluation set of 476 LUS clips from 60 unique patients assessed model performance. The AI models scored the clips on both a binary and ordinal 0–4 multiclass assessment. ResultsA multiclassification AI algorithm had the best performance at the binary level when applied to the independent evaluation set, AUC of 0.967 (95% CI 0.965–0.970) for detecting pathologic conditions. When compared to expert blinded reviewer, the 0–4 multiclassification AI algorithm scale had a reported linear weighted kappa of 0.839 (95% CI 0.804–0.871). ConclusionsThe multiclassification AI algorithm is a robust and well performing model at both binary and ordinal multiclass B-line evaluation. This algorithm has the potential to be integrated into clinical workflows to assist users with quantitative and objective B-line assessment for evaluation of AHF.

A Light Weight Depthwise Separable Layer Optimized CNN Architecture for Object-Based Forgery Detection in Surveillance Videos

Abstract The present era is at the peak of technological advancement in image and video processing techniques, with user-friendly accessible tools/techniques. This immersive technology development makes video forensics enormously challenging. Specifically, the passive approaches to object-based forgeries in videos are crucial for legal and judicial matters. Hence, to ensure the integrity of the videos, a scientific, statistical and passive investigation of videos is required to maintain the spatial and temporal information content. This paper aims to develop a passive approach for digging out the forgery traces by applying the motion residue windowing technique for object removal forgery in surveillance videos. The novel max averaging windowing techniques improve visual imprints of the object removal forgery in the videos from the existing methods in the literature. A deep learning approach is the next step for achieving forgery detection in surveillance videos. The proposed lightweight depth-separable layer-optimized CNN has fast execution speed, optimized in terms of parameters without compromising the desired accuracy. This network is trained at a frame level with 98.60% testing accuracy, followed by a pipeline architecture of the proposed model for detection of forgery at video level with 99.01% accuracy. The suggested model works better than current models regarding post-processing operations, compression rates, forged video detection accuracy, precision, recall and F1 score.

A Smart Ad Display System

This paper proposes a smart ad display system to provide personalized delivery of video ads. The proposed system records consumers’ facial expression and eye gaze stream data as they watch an ad and analyzes data at the frame level. The recognized facial expression and detected eye gaze are matched to the corresponding frame of the video ad, thereby linking facial expressions to specific visual objects appearing in the ad. By tracking a consumer’s facial expressions in response to various visual objects in real time, the system learns the consumer’s individual preferences toward different ads, searches the ad pool, and selects and subsequently displays a new ad that is most likely to elicit positive attitudinal and behavioral responses. We demonstrate the feasibility and effectiveness of the proposed system with two empirical studies. The results show that by tracking a consumer’s facial responses to only one ad or even part of an ad, our proposed system is able to make reasonably accurate inferences about a consumer’s ad preferences, with or without using information about other consumers. These inferences are used to make personalized recommendations that help enhance consumers’ ad viewing experiences and elicit favorable responses.

A Light-Weight Autoregressive CNN-Based Frame Level Transducer Decoder for End-to-End ASR

A convolutional neural network (CNN) transducer decoder was proposed to reduce the decoding time of an end-to-end automatic speech recognition (ASR) system while maintaining accuracy. The CNN of 177 k parameters and a kernel size of 6 generates the probabilities of the current token at the token level, at the token transition of the output token sequence. Two probabilities of the current token, one from the encoder and the other from the CNN are added to the frame level to reduce the decoding step to the number of input frames. An encoder composed of an 18-layer conformer was combined with the proposed decoder for training with the Librispeech data set. The forward-backward algorithm was used for training. The space and re-appearance tokens are added to the 300-word piece tokens to represent the token string. A space token appears at a frame between two words. A comparison with the autoregressive decoders such as transformer and RNN-T decoders demonstrates that this work provides comparable WERs with much less decoding time. A comparison with non-autoregressive decoders such as CTC indicates that this work enhanced WERs.

Unified Spatio-Temporal Attention Models for Advanced Human Action Recognition & Detection

Abstract: This research paper introduces innovative approaches to enhance human action analytics in computer vision through the development of spatial and temporal attention models. The first model is based on Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units, aiming to extract discriminative spatial and temporal features from skeleton data for improved human action recognition and detection. The model selectively focuses on discriminative joints within each frame, utilizing a regularized cross-entropy loss for effective training. Additionally, a joint training strategy is proposed to optimize the learning process. The second model, a spatio-temporal attention (STA) network, is designed to address the limitations of existing 3D Convolutional Neural Networks (3D CNNs) in treating all video frames equally. The STA network characterizes beneficial information at both the frame and channel levels, leveraging differences in spatial and temporal dimensions to enhance the learning capability of 3D convolutions. The proposed STA method can be seamlessly integrated into state-of-the-art 3D CNN architectures for video action detection and recognition. Evaluation of diverse datasets, including SBU Kinect Interaction, NTU RGB+D, PKU-MMD, UCF-101, HMDB-51, and THUMOS 2014, demonstrates the effectiveness of both proposed models in achieving state-of-the-art performance for action recognition and detection tasks. The spatial and temporal attention models contribute significantly to capturing discriminative features, showcasing their potential in advancing the field of human action analytics in computer vision.

P487 Automated Scoring of Patient Endoscopy Videos Using Deep Learning Techniques: A Promising Approach for Clinical Trials in Inflammatory Bowel Disease

Abstract Background Accurate assessment of patient endoscopy videos is crucial for various clinical trials, including those related to inflammatory bowel diseases (IBD). However, manual scoring by trained local and central readers can lead to discrepancies, inconsistencies, and logistical challenges. To address these issues, we devised a novel approach utilizing deep learning techniques to automate the analysis, parsing, and scoring of patient endoscopy videos. Our objectives were to reduce human error/variability, accelerate endoscopic scoring by enhancing read accuracy, and lower clinical trial cost. Methods We developed and trained deep learning models using a large dataset of labeled patient endoscopy images. The models were designed to predict the Mayo endoscopic score (MES), which serves as a standardized measure of disease severity in IBD. Our approach developed convolutional neural networks (CNNs) to extract features from raw video data and processed them into meaningful patterns. The models were optimized using transfer learning and validated on external endoscopy video datasets through rigorous testing protocols. Results Using an iterative process, we developed two sets (total 4) of proof-of-concept (POC) Artificial Intelligence (AI) models using frame level datasets from endoscopy videos. First, quality was assessed by the blur detection model, currently performing at > 90% accuracy, and the bowel preparation model, performing at > 95% accuracy compared to the labels provided by human annotators. Second, MES sub-score prediction currently performs with >75% accuracy for 4-level Mayo sub-score prediction and >90% accuracy in predicting advanced MES scores (2 or higher compared with consensus score derived from 3 annotators). Conclusion The presented study highlights the potential of deep learning techniques in revolutionizing the assessment of patient endoscopy videos in clinical trials for IBD. Automating the endoscopic scoring process enables faster, more accurate, and cost-efficient evaluations, ultimately leading to better detection of patient outcomes. Future research directions involve further refining the models to enhance their performance and expand their application to other gastrointestinal diseases.

Media framing and trust: A randomized survey experiment on attitudes towards the unvaccinated during the COVID-19 pandemic

This research examines the influence of news media and trust in institutions on how vaccinated individuals in Finland perceived the unvaccinated during the COVID-19 pandemic. The study employed a randomized survey experiment involving 455 participants and incorporated episodic and thematic news framings on vaccine side effects alongside a non-treatment control group. Episodic framing, which presented personal stories, was found to significantly reduce the inclination to impose additional restrictions on the unvaccinated and lessen the perception of them as an outgroup. The converse effect was shown in thematic framing, which focused on statistics and expert opinions. The study also underscored the pivotal role of trust in institutions: lower trust increases the influence of episodic framing, while strong political trust can mitigate the effects of news framing. This investigation provides vital insights into the interplay between media framing and trust levels in shaping public attitudes towards health risks and non-adherence during a health emergency.

Derivation of Key Characteristics of Columns in Frames with Sway

The paper paper examines columns of elastic frames that are able to displace laterally under axial load. Columns on the same level (story) of such frames may contribute to lateral support (stability) or require lateral support (supported columns). In the former case, maximum moments will occur at column ends, but may occur between column ends in the latter case. The column response, including the formation of moments and shears, is often complicated, often calling for approximate analysis methods and information that presently may not be readily available. Main attention of the paper is paid to providing results that will be helpful in providing, in a rather simple manner, improved understanding of column mechanics, and that may be helpful also as a supplement to full second-order analyses. Towards this goal, the major objective of the paper is threefold: (i) to identify characteristic points, or behavioral“landmarks” in the axial load-moment solution space, through the study of rotationally restrained elastic columns and two-column panels with sidesway; (ii) to derive simple, novel, closed-form expressions for these, thereby providing a tool simplifying the establishment of the variation of end and maximum moments versus axial load; (iii) to identify to what extent isolated (single) column analyses may adequately represent horizontally interacting columns, and potential unwinding, in frames.

Semantic pattern recognition of special English words based on data corpus technology

Abstract This paper first researches special English vocabulary teaching defines special English vocabulary definitions and characteristics based on special English usage, and proposes strategies for learning special English vocabulary. The study involves studying the semantic recognition of special English using data corpus technology. Based on data corpus technology, the semantic structure of English vocabulary is analyzed according to Chinese analysis and word vectors, and to facilitate the extraction of English semantic features, IOB2 labeled training corpus is needed. Then, the CRFs network algorithm in the data corpus technology is used to construct the special English semantic recognition model, the lexical annotation set is refined on the preprocessing, and at the same time, the semantic features are added to the recognition model of CRFs mainly used to recognize the noun phrases of gerund class and the experimental analysis is carried out on the recognition of the semantic patterns of the special English vocabulary. The results show that among high school students with high English semantic recognition ability, the correlation coefficient between their aggregates and reading level is 0.753, the correlation coefficient between collocated phrases and reading level is 0.772, and there is a high correlation between the ability to recognize aggregates, collocated phrases, and sentence framing phrases and reading level. This study is designed to improve the efficiency of English reading and communication and has a facilitatory effect on recognizing English lexical semantic patterns.

Spatio-temporal human action localization in indoor surveillances

Spatio-temporal action localization is a crucial and challenging task in the field of video understanding. Existing benchmarks for spatio-temporal action detection are limited by factors such as incomplete annotations, high-level non-universal actions, and uncommon scenarios. To address these limitations and facilitate research in real-world security applications, we introduce a novel human-centric dataset for spatio-temporal localization of atomic actions in indoor surveillance settings, termed as HIA (Human-centric Indoor Actions). The HIA dataset is constructed by selecting 30 atomic action classes, compiling 100 surveillance videos, and annotating 219,225 frames with 370,937 bounding boxes. The primary characteristics of HIA include (1) accurate spatio-temporal annotations for atomic actions, (2) human-centric annotations at the frame level, (3) temporal linking of persons across discontinuous tracks, and (4) utilization of indoor surveillance videos. Our HIA, with its realistic settings in indoor surveillance scenes and comprehensive annotations, presents a valuable and novel challenge to the spatio-temporal action localization domain. To establish a benchmark, we evaluate various methods and provide an in-depth analysis of the HIA dataset. The HIA dataset will be made available soon, and we anticipate that it will serve as a standard and practical benchmark for the research community.

Memory Network With Pixel-Level Spatio-Temporal Learning for Visual Object Tracking

Making full use of temporal and spatial information is critical to cope with the appearance changes of objects in visual object tracking. However, existing methods in the tracking field, which employ a memory network at frame level to learn this information, bring redundancy and cannot build long-term relationships among historical frames due to the limited memory size. In this paper, we propose a novel memory network, Pixel-level Spatio-Temporal Memory (PSTM), which organizes object features in an efficient way to leverage temporal and spatial context information. Specifically, PSTM is constructed and updated by a memory writer, which includes a pixel-level updating strategy to maintain the temporal consistency and dynamically memorize the noteworthy variations. Furthermore, in order to exploit relationships between the object and search region and precisely estimate the state of the object, we propose a memory reader, Pixel-wise Matching and Refinement module (PMR), and model spatial context without a complex manual-designed mechanism. Comprehensive experiments and comparisons on challenging large-scale benchmarks, including GOT-10k, TrackingNet, LaSOT, OTB2015, VOT2020, and NfS, have demonstrated the effectiveness of our proposed method, which performs favorably against state-of-the-art trackers.

Constructional change and frameelement selection

Abstract While most work in diachronic construction grammar considers individual constructional changes, these changes share an understudied relationship with broader changes at the level of the semantic frame. Literary corpus data shows that in the Spending frame in French, between the 12th and 20th centuries, the selection rates of four alternating frame elements shifted: Purpose, Cause of expense, Goods, and Seller. This frame-level shift is examined alongside related constructional changes including lexicalization, pejoration, and constructional loss. The analysis suggests that constructional change does not proceed independently but is driven by shifts in framing due in part to sociocultural changes.

Ductile failure behaviour characterisation of HRT with GISSMO and crashworthiness optimisation design of VRB-HRT

In this paper, a generalised incremental stress state dependent damage model (GISSMO) is used to accurately characterise the ductile failure behaviour of hot formed steel under collision, which is the excellent material selection to improve the crashworthiness and lightweight level of car body frames. The deformation patterns, structural failure patterns, and key crashworthiness indexes of B1500HS hot-rolled tube (HRT) are validated by the comparative researches between experiments and simulations. To fully exploit the lightweight potential of HRT, a variable-thickness rolled blanks hot rolled tube (VRB-HRT) is proposed to perform its crashworthiness optimisation design with considering the ductile failure by the extended hybrid cellular automaton for VRB thin-walled structures (eHCA-VRB) algorithm. It is found that the optimal case of VRB-HRT with considering the ductile failure is easier to meet the actual performance constraints, even though possesses a lower weight reduction ratio. Therefore, the ductile failure of VRB-HRT should be considered to guarantee the structural reliability by a more reasonable optimal design.

Cascade Multi-Level Transformer Network for Surgical Workflow Analysis.

Surgical workflow analysis aims to recognise surgical phases from untrimmed surgical videos. It is an integral component for enabling context-aware computer-aided surgical operating systems. Many deep learning-based methods have been developed for this task. However, most existing works aggregate homogeneous temporal context for all frames at a single level and neglect the fact that each frame has its specific need for information at multiple levels for accurate phase prediction. To fill this gap, in this paper we propose Cascade Multi-Level Transformer Network (CMTNet) composed of cascaded Adaptive Multi-Level Context Aggregation (AMCA) modules. Each AMCA module first extracts temporal context at the frame level and the phase level and then fuses frame-specific spatial feature, frame-level temporal context, and phase-level temporal context for each frame adaptively. By cascading multiple AMCA modules, CMTNet is able to gradually enrich the representation of each frame with the multi-level semantics that it specifically requires, achieving better phase prediction in a frame-adaptive manner. In addition, we propose a novel refinement loss for CMTNet, which explicitly guides each AMCA module to focus on extracting the key context for refining the prediction of the previous stage in terms of both prediction confidence and smoothness. This further enhances the quality of the extracted context effectively. Extensive experiments on the Cholec80 and the M2CAI datasets demonstrate that CMTNet achieves state-of-the-art performance.