Raw Video Research Articles

Markov Progressive Framework, a Universal Paradigm for Modeling Long Videos.

The computational complexity of video models increases linearly with the square number of frames. Thus, constrained bycomputational resources, training video models to learn long-term temporal semantics end-to-end is quite a challenge. Currently, the main-stream method is to split a raw video into clips, leading to incomplete fragmentary temporal information flow and failure of modeling long-term semantics. In this paper, we design the Markov Progressive framework (MaPro), a theoretical framework consisting of the progressive modeling method and a paradigm model tailored for it. Thecore idea of MaPro is to find a paradigm model consisting of proposed Markov operators which can be trained in multiple sequential steps and ensure that the multi-step progressive modeling is equivalent to the conventional end-to-endmodeling. By training the paradigm model under the progressive method, we are able to model long videos end-to-endwith limited resources and ensure the effective transmission of long-term temporal information. We provide implementations of this theoretical system on the mainstream CNN- and Transformer-based models, where they are modified to conform to the Markov paradigm. As a general and robust training method, we experimentally demonstrate that it yields significant performance improvements on different backbones and datasets. As an illustrative example, the proposed method improves the SlowOnly network by 4.1 mAP on Charades and 2.5 top-1 accuracy on Kinetics. And for TimeSformer, MaPro improves its performance on Kinetics by 2.0 top-1 accuracy. Importantly, all these improvements areachieved with a little parameter and computation overhead.

Armed boundary sabotage: A case study of human malicious behaviors identification with computer vision and explainable reasoning methods

Nowadays, the technologies in computer vision (CV) are labor-saving and convenient to identify human malicious behaviors. However, they usually fail to consider the robustness, generalization and interpretability of calculation frameworks. In this paper, a very common but sometimes difficult-to-detect case research called armed boundary sabotage is conducted, which is achieved by computer vision module (CVM) and reasoning module (RM). Among them, CVM is used for extracting the key information from raw videos, while RM is applied to obtain the final reasoning results. Considering the transient and confusing properties in such scenarios, a specific human-object interaction analysis process with soft constraint is proposed in CVM. In addition, two reasoning methods which are data-based reasoning method and language-based reasoning methods are implemented in RM. The results show that the human-object interaction analysis process with soft constraint prove to be effective and practical, while the optimal testing accuracy achieves 0.7871. Furthermore, the two proposed reasoning methods are promising for identification of human malicious behaviors. Among them, the advanced language-based reasoning method outperforms others, with highest precision value of 0.8750 and perfect recall value of 1.0000. Besides, these proposals are also verified to be high-performance in other external intrusion scenarios of our previous work. Finally, our research also obtain state-of-the-art results by comparing with other related works.

Comparative Analysis of Power Consumption and Resource Utilization in Open-Source and Proprietary Media Players while using Raw Videos

This study evaluates and compares the power consumption and resource utilization of open-source and proprietary media players during the playback of a large raw video file. Using real-time monitoring tools like HWiNFO, key metrics such as GPU power consumption, CPU power consumption, memory usage, and CPU usage percentage were collected and analyzed. The experiment was conducted on a system powered by a 12th Gen Intel(R) Core(TM) i7-12700H processor, and the media players were tested with a 2-minute, 14-second raw video file in .MOV format. A statistical analysis using t-tests was performed to assess the significance of the differences between the two categories. The results indicated that open-source media players generally exhibit lower GPU and CPU power consumption, with a potential for saving energy. Long-term power consumption analysis further demonstrated that users could achieve significant energy savings by opting for open-source media players, making them more suitable for energy-conscious environments. These findings highlight the trade-offs between power efficiency and performance while playing raw videos.

Empirical Studies on Merging Characteristics in Temporary Highway Work Zones with Different Traffic States and Layouts

Vehicles conduct mandatory lane changes to merge into the open lanes in temporary highway work zones, and the merging characteristics have been widely studied over the past decades. Unlike research on other traffic facilities, the data and scenarios of highway work zones are often lacking and monotonous. This study presented a vehicle trajectory dataset in highway work zones. The dataset contains ten work zone sites and three different layouts. After transforming raw videos into high-resolution trajectories, we selected eight areas and divided them into three scenarios to investigate the merging characteristics in highway work zones with different traffic states and layouts. In work zones with varying traffic conditions, we found opposite merging locations and lane speed distribution trends. In work zones with varying layouts, we found different merging location distributions, lane speed distributions, and impacts on the following vehicle’s speed. The relationship between the gap and merge speed was consistent with previous studies.

Impact of Wireless Network Packet Loss on Real-Time Video Streaming Application: A Comparative Study of H.265 and H.266 Codecs

The transmission of real-time videos over wireless networks is prone to the negative consequences of packet loss and delay, which can have a potential effect on the video quality during streaming. These impairments can lead to interruptions, buffering, and degradation of visual and auditory elements, resulting in an unsatisfactory user experience. In this paper, we aim to address the challenges associated with packet loss and delay parameters in wireless networks, and propose an approach to alleviate their impact on real-time video transmission. The proposed approach involves utilizing the H.265/H.266 video coding standards. For Versatile Video Coding (VVC), a patch support for VVdeC and VVenC to FFmpeg (Fast Forward Moving Picture Expert Group) is added. As a result, FFmpeg is used to encode, stream and decode all videos. Raw videos of 2K qualities are encoded based on the adaptive quantization (QP) for the above-mentioned codecs. By selecting optimal transmission data based on various network conditions, this approach enhances the Quality of Experience (QoE) for end-users while minimizing resource usage in the wireless network. Furthermore, the proposed approach selects the codec standards according to their bitrates and frame rates. Simulation results indicate that the proposed approach has a significant improvement for real-time video streaming over wireless networks to satisfy the end user experience. The approach also outperforms other related work by gaining a PSNR of +12 dB for H.265 and +13 dB for H.266 when the network packet loss is 1%.

465 Comparative analysis of semantic segmentation and deep regression models with supervised pre-training for accurate prediction of pig body weight from video data: Insights from industry-scale datasets

Abstract Accurate pig body weight (BW) measurement is essential for producers as it is related to pig growth, health, and marketing, yet conventional manual weighing methods are time-consuming and may cause potential stress to the animals. Although there is a growing trend towards the adoption of three-dimensional cameras coupled with computer vision techniques for pig BW estimation, their validation using industry-scale data are still limited. Among the prevailing methodologies, semantic segmentation and supervised pre-training regression are prominent models currently used. Therefore, the objectives of this study were: 1) to estimate pig BW from repeatedly measured video data obtained from a commercial setting, 2) to compare the performance of the two image analysis methods: thresholding segmentation and deep regression approaches, and 3) to evaluate the predictive ability of BW estimation models. An Intel RealSense D435 camera was installed in the commercial farm to collect top-view videos of 540 pigs biweekly at five different time points over three months. At the same time, manually measured BW records were collected using a digital weighing system. We used an automated video conversion pipeline and fine-tuned YOLOv8 to pre-process the raw depth videos. Subsequently, we acquired a total of 151,756 depth images and depth map files. Adaptive thresholding was applied to segment the pig body from the background. Four image-derived biometric features, including dorsal length, abdominal width, height, and volume, were estimated from the segmented images and fitted using ordinary least squares and random forest models. We applied transfer learning by initializing the weights of five deep learning models, including ResNet50, Xception, EfficientNetV2S, ConvNeXtBase, and Vision Transformer, with pre-trained weights from ImageNet. We then fine-tuned these models on the pig depth images. The last layer of each model was adapted to linear regression, enabling direct estimation of BW for each image without the need for additional image pre-processing steps. We employed random repeated subsampling cross-validation, dividing 80% of the pigs for training and 20% for testing, to evaluate prediction performance at each time point. The best prediction coefficients of determination and mean absolute percentage error for each time point were 0.76, 0.86, 0.90, 0.83, 0.90, and 4.57%, 3.76%, 3.01%, 3.41%, 4.84%, respectively. On average, the Xception model resulted in the best prediction coefficient of determination and mean absolute percentage error of 0.90 and 3.01%. Our results suggest that deep learning-based supervised learning models improve the prediction performance of pig BW from industry-scale depth video data.

A novel approach for long-term secure storage of domain independent videos

Long-term protection of multimedia contents is a complex task, especially when the video has critical elements. It demands sophisticated technology to ensure confidentiality. In this paper, we propose a blended approach which uses proactive visual cryptography scheme along with video summarization techniques to circumvent the aforementioned issues. Proactive visual cryptography is used to protect digital data by updating periodically or renewing the shares, which are stored in different servers. And, video summarization schemes are useful in various scenarios where memory is a major concern. We use a domain independent scheme for summarizing videos and is applicable to both edited and unedited videos. In our scheme, the visual continuity of the raw video is preserved even after summarization. The original video can be reconstructed through the shares using auxiliary data, which was generated during video summarization phase. The mathematical studies and experimental results demonstrate the applicability of our proposed method.

Multimodal Convolutional Neural Network-Based Algorithm for Real-Time Detection and Differentiation of Malignant and Inflammatory Biliary Strictures in Cholangioscopy: a Proof of Concept Study with Video

Deep learning algorithms gained attention for detection (computer-aided detection [CADe]) of biliary tract cancer in digital single-operator cholangioscopy (dSOC). We developed a multimodal convolutional neural network (CNN) for detection (CADe), characterization and discriminating (computer-aided diagnosis [CADx]) between malignant, inflammatory, and normal biliary tissue in raw dSOC videos. In addition, clinical metadata were included in the CNN algorithm to overcome limitations of image-only models. Based on dSOC videos and images of 111 patients (total of 15,158 still frames), a real-time CNN-based algorithm for CADe and CADx was developed and validated. We established an image-only model and metadata injection approach. In addition, frame-wise and case-based predictions on complete dSOC video sequences were validated. Model embeddings were visualized, and class activation maps highlighted relevant image regions. The concatenation-based CADx approach achieved a per-frame area under the receiver-operating characteristic curve of .871, sensitivity of .809 (95% CI, .784-.832), specificity of .773 (95% CI, .761-.785), positive predictive value of .450 (95% CI, .423-.467), and negative predictive value of .946 (95% CI, .940-.954) with respect to malignancy on 5715 test frames from complete videos of 20 patients. For case-based diagnosis using average prediction scores, 6 of 8 malignant cases and all 12 benign cases were identified correctly. Our algorithm distinguishes malignant and inflammatory bile duct lesions in dSOC videos, indicating the potential of CNN-based diagnostic support systems for both CADe and CADx. The integration of non-image data can improve CNN-based support systems, targeting current challenges in the assessment of biliary strictures.

AI-enhanced real-time cattle identification system through tracking across various environments

Video-based monitoring is essential nowadays in cattle farm management systems for automated evaluation of cow health, encompassing body condition scores, lameness detection, calving events, and other factors. In order to efficiently monitor the well-being of each individual animal, it is vital to automatically identify them in real time. Although there are various techniques available for cattle identification, a significant number of them depend on radio frequency or visible ear tags, which are prone to being lost or damaged. This can result in financial difficulties for farmers. Therefore, this paper presents a novel method for tracking and identifying the cattle with an RGB image-based camera. As a first step, to detect the cattle in the video, we employ the YOLOv8 (You Only Look Once) model. The sample data contains the raw video that was recorded with the cameras that were installed at above from the designated lane used by cattle after the milk production process and above from the rotating milking parlor. As a second step, the detected cattle are continuously tracked and assigned unique local IDs. The tracked images of each individual cattle are then stored in individual folders according to their respective IDs, facilitating the identification process. The images of each folder will be the features which are extracted using a feature extractor called VGG (Visual Geometry Group). After feature extraction task, as a final step, the SVM (Support Vector Machine) identifier for cattle identification will be used to get the identified ID of the cattle. The final ID of a cattle is determined based on the maximum identified output ID from the tracked images of that particular animal. The outcomes of this paper will act as proof of the concept for the use of combining VGG features with SVM is an effective and promising approach for an automatic cattle identification system

Demixing fluorescence time traces transmitted by multimode fibers

Optical methods based on thin multimode fibers (MMFs) are promising tools for measuring neuronal activity in deep brain regions of freely moving mice thanks to their small diameter. However, current methods are limited: while fiber photometry provides only ensemble activity, imaging techniques using of long multimode fibers are very sensitive to bending and have not been applied to unrestrained rodents yet. Here, we demonstrate the fundamentals of a new approach using a short MMF coupled to a miniscope. In proof-of-principle in vitro experiments, we disentangled spatio-temporal fluorescence signals from multiple fluorescent sources transmitted by a thin (200 µm) and short (8 mm) MMF, using a general unconstrained non-negative matrix factorization algorithm directly on the raw video data. Furthermore, we show that low-cost open-source miniscopes have sufficient sensitivity to image the same fluorescence patterns seen in our proof-of-principle experiment, suggesting a new avenue for novel minimally invasive deep brain studies using multimode fibers in freely behaving mice.

Assessment of colonoscopy skill using machine learning to measure quality: Proof-of-concept and initial validation.

Background and study aims Low-quality colonoscopy increases cancer risk but measuring quality remains challenging. We developed an automated, interactive assessment of colonoscopy quality (AI-CQ) using machine learning (ML). Methods Based on quality guidelines, metrics selected for AI development included insertion time (IT), withdrawal time (WT), polyp detection rate (PDR), and polyps per colonoscopy (PPC). Two novel metrics were also developed: HQ-WT (time during withdrawal with clear image) and WT-PT (withdrawal time subtracting polypectomy time). The model was pre-trained using a self-supervised vision transformer on unlabeled colonoscopy images and then finetuned for multi-label classification on another mutually exclusive colonoscopy image dataset. A timeline of video predictions and metric calculations were presented to clinicians in addition to the raw video using a web-based application. The model was externally validated using 50 colonoscopies at a second hospital. Results The AI-CQ accuracy to identify cecal intubation was 88%. IT ( P = 0.99) and WT ( P = 0.99) were highly correlated between manual and AI-CQ measurements with a median difference of 1.5 seconds and 4.5 seconds, respectively. AI-CQ PDR did not significantly differ from manual PDR (47.6% versus 45.5%, P = 0.66). Retroflexion was correctly identified in 95.2% and number of right colon evaluations in 100% of colonoscopies. HQ-WT was 45.9% of, and significantly correlated with ( P = 0.85) WT time. Conclusions An interactive AI assessment of colonoscopy skill can automatically assess quality. We propose that this tool can be utilized to rapidly identify and train providers in need of remediation.

DiffRank: Enhancing efficiency in discontinuous frame rate analysis for urban surveillance systems

Urban public safety management relies heavily on video surveillance systems, which provide crucial visual data for resolving a wide range of incidents and controlling unlawful activities. Traditional methods for target detection predominantly employ a two-stage approach, focusing on precision in identifying objects such as pedestrians and vehicles. These objects, typically sparse in large-scale, lower-quality surveillance footage, induce considerable redundant computation during the initial processing stage. This redundancy constrains real-time detection capabilities and escalates processing costs. Furthermore, transmitting raw images and videos laden with superfluous information to centralized back-end systems significantly burdens network communications and fails to capitalize on the computational resources available at diverse surveillance nodes. This study introduces DiffRank, a novel preprocessing method for fixed-angle video imagery in urban surveillance. The method strategically generates candidate regions during preprocessing, thereby reducing redundant object detection and improving the efficiency of the detection algorithm. Drawing upon change detection principles, a background feature learning approach utilizing shallow features has been developed. This approach prioritizes learning the characteristics of fixed-area backgrounds over direct background identification. As a result, alterations in ROI are efficiently discerned using computationally efficient shallow features, markedly accelerating the generation of proposed Regions of Interest (ROIs) and diminishing the computational demands for subsequent object detection and classification. Comparative analysis on various public and private datasets illustrates that DiffRank, while maintaining high accuracy, substantially outperforms existing baselines in terms of speed, particularly with larger image sizes (e.g., an improvement exceeding 300 % at 1920×1080 resolution). Moreover, the method demonstrates enhanced robustness compared to baseline methods, efficiently disregarding static targets like mannequins in display windows. The advancements in candidate area preprocessing enable a balanced approach between detection accuracy and overall detection speed, making the algorithm highly applicable for real-time on-site analysis in edge computing scenarios and cloud-edge collaborative computing environments.

Astrocytic Calcium Signaling Toolkit (astroCaST): efficient analysis of dynamic astrocytic calcium events.

The Astrocytic Calcium Signaling Toolkit (astroCaST) is a novel solution to a longstanding challenge in neuroscience research: the specialized analysis of astrocytic calcium events within fluorescence time-series imaging. Distinct from existing neuron-centric tools, astroCaST is adept at detecting and clustering astrocytic calcium events based on their unique spatiotemporal characteristics, thus filling a gap in astrocytic research methodologies. This toolkit not only facilitates the detection of such events but also extends its utility to provide comprehensive end-to-end analysis. This feature is absent in most tools targeting astrocytic activity. AstroCaST's development was motivated by the critical need for dedicated software that supports researchers in transitioning from raw video data to insightful experimental conclusions, efficiently managing large-scale datasets without compromising computational speed. It offers a user-friendly interface that caters to both novice and expert users, incorporating both a graphical user interface (GUI) for detailed explorations and a command-line interface (CLI) for extensive analyses. Expected outcomes from utilizing astroCaST include the ability to process and analyze a significantly larger volume of data. This enables a more profound and comprehensive analysis than previously possible, addressing the demands of large-scale astrocytic studies. In summary, astroCaST aims to advance astrocytic calcium imaging analysis, offering a tailored, efficient, and comprehensive toolset that enhances our understanding of astrocytic functions and their implications in neuroscience.

Temporal cues enhanced multimodal learning for action recognition in RGB-D videos

Action recognition is an important and active research direction in computer vision, where temporal modeling is critical for action representation. Generally, unimodal methods that use only RGB or skeleton modality for human action recognition have their limitations, e.g., information redundancy/environment noise of RGB video modality, and spatial interaction deficiency of skeleton modality. In this paper, we present a novel multimodal learning approach based on RGB and skeleton modalities for action recognition in RGB-D videos. Specifically, we (1) transfer skeleton knowledge to RGB video for effective video compression, which produces the informative action image from raw RGB video, (2) introduce the temporal cues enhancement module to adequately learn the spatiotemporal representation for action classification, and (3) propose a multi-level multimodal co-learning framework for human action recognition in RGB-D videos. Experimental results on NTU RGB+D, PKU-MMD, and N-UCLA datasets demonstrate the effectiveness of the proposed multimodal learning method.

Nefis: A network coding based flexible device-to-device video streaming scheme

Wireless device-to-device (D2D) communication has empowered efficient and convenient video sharing among neighboring devices. However, the mobility and diversity of user devices, combined with dynamically shifting channel conditions, make these processes susceptible to environmental interference. In this paper, we introduce an adaptive network coding scheme tailored for D2D raw source video streaming in the scalable video coding (SVC) format. Video frames are partitioned into equal-sized fragments and encoded into streaming packets with the addition of certain redundancy. To address the observed deficiency where decoding throughput of coded video frames is generally lower than that of the encoding operation on mobile device platforms, we introduce a parallel decoding algorithm based on LU decomposition. Furthermore, we propose a flexible adjustment mechanism, named Nefis, which dynamically adapts the network coding redundancy and stream resolution based on the statistical model established to predict current network conditions and video streaming quality. Implementation on Android platform demonstrates that Nefis reduce redundancy in bandwidth usage during streaming process and enhance resilience to network dynamics. Experimental results also conclusively demonstrate the feasibility and advantages of network coding technology in D2D streaming applications. Nevertheless, achieving these advantages requires carefully designed encoding and decoding mechanisms.

A deep learning-based system for accurate detection of anatomical landmarks in colon environment

Colonoscopy is a standard imaging tool for examining the lower gastrointestinal tract of patients to capture lesion areas. However, if a lesion area is found during the colonoscopy process, it is difficult to record its location relative to the colon for subsequent therapy or recheck without any reference landmark. Thus, automatic detection of biological anatomical landmarks is highly demanded to improve clinical efficiency. In this article, we propose a novel deep learning-based approach to detect biological anatomical landmarks in colonoscopy videos. First, raw colonoscopy video sequences are pre-processed to reject interference frames. Second, a ResNet-101-based network is used to detect three biological anatomical landmarks separately to obtain the intermediate detection results. Third, to achieve more reliable localization, we propose to post-process the intermediate detection results by identifying the incorrectly predicted frames based on their temporal distribution and reassigning them back to the correct class. Finally, the average detection accuracy reaches 99.75%. Meanwhile, the average intersection over union of 0.91 shows a high degree of similarity between our predicted landmark periods and ground truth. The experimental results demonstrate that our proposed model can accurately detect and localize biological anatomical landmarks from colonoscopy videos.

INNBC DApp, a decentralized application to permanently store biomedical data on a modern, proof-of-stake (POS), blockchain such as BNB Smart Chain

BackgroundA blockchain can be described as a distributed ledger database where, under a consensus mechanism, data are permanently stored in records, called blocks, linked together with cryptography. Each block contains a cryptographic hash function of the previous block, a timestamp, and transaction data, which are permanently stored in thousands of nodes and never altered. This provides a potential real-world application for generating a permanent, decentralized record of scientific data, taking advantage of blockchain features such as timestamping and immutability.ImplementationHere, we propose INNBC DApp, a Web3 decentralized application providing a simple front-end user interface connected with a smart contract for recording scientific data on a modern, proof-of-stake (POS) blockchain such as BNB Smart Chain. Unlike previously proposed blockchain tools that only store a hash of the data on-chain, here the data are stored fully on-chain within the transaction itself as “transaction input data”, with a true decentralized storage solution. In addition to plain text, the DApp can record various types of files, such as documents, images, audio, and video, by using Base64 encoding. In this study, we describe how to use the DApp and perform real-world transactions storing different kinds of data from previously published research articles, describing the advantages and limitations of using such a technology, analyzing the cost in terms of transaction fees, and discussing possible use cases.ResultsWe have been able to store several different types of data on the BNB Smart Chain: raw text, documents, images, audio, and video. Notably, we stored several complete research articles at a reasonable cost. We found a limit of 95KB for each single file upload. Considering that Base64 encoding increases file size by approximately 33%, this provides us with a theoretical limit of 126KB. We successfully overcome this limitation by splitting larger files into smaller chunks and uploading them as multi-volume archives. Additionally, we propose AES encryption to protect sensitive data. Accordingly, we show that it is possible to include enough data to be useful for storing and sharing scientific documents and images on the blockchain at a reasonable cost for the users.ConclusionINNBC DApp represents a real use case for blockchain technology in decentralizing biomedical data storage and sharing, providing us with features such as immutability, timestamp, and identity that can be used to ensure permanent availability of the data and to provide proof-of-existence as well as to protect authorship, a freely available decentralized science (DeSci) tool aiming to help bring mass adoption of blockchain technology among the scientific community.

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Current methods for assessing individual well-being in team collaboration at the workplace often rely on manually collected surveys. This limits continuous real-world data collection and proactive measures to improve team member workplace satisfaction. We propose a method to automatically derive social signals related to individual well-being in team collaboration from raw audio and video data collected in teamwork contexts. The goal was to develop computational methods and measurements to facilitate the mirroring of individuals’ well-being to themselves. We focus on how speech behavior is perceived by team members to improve their well-being. Our main contribution is the assembly of an integrated toolchain to perform multi-modal extraction of robust speech features in noisy field settings and to explore which features are predictors of self-reported satisfaction scores. We applied the toolchain to a case study, where we collected videos of 20 teams with 56 participants collaborating over a four-day period in a team project in an educational environment. Our audiovisual speaker diarization extracted individual speech features from a noisy environment. As the dependent variable, team members filled out a daily PERMA (positive emotion, engagement, relationships, meaning, and accomplishment) survey. These well-being scores were predicted using speech features extracted from the videos using machine learning. The results suggest that the proposed toolchain was able to automatically predict individual well-being in teams, leading to better teamwork and happier team members.

Lossy compression of infrared videos in WEST (W Environment in Steady-state Tokamak).

During the 2019 C4 experimental campaign of the WEST (W Environment in Steady-state Tokamak) (France), the infrared diagnostic produced more than seven terabytes of uncompressed video data. Constraints on the computer infrastructure required for storage, backup, and especially offline access to infrared videos made the use of a compression algorithm mandatory. This paper proposes an innovative method to compress infrared videos with controlled temperature precision. This compression method is based on a controlled averaging of the video that maximizes the compression potential of standard lossless video codecs such as H264/AVC or HEVC. The combination of the loss introduction algorithm and the H264/AVC lossless video codec obtains the best compression ratio in the range of 8 to 41 with a maximum temperature error of 2 °C. This method also outperforms the JPEG-LS algorithm in terms of compression ratio and image quality for the same temperature precision.

Sign Language Word Detection Using LRCN

Sign language is the most effective communication for deaf or hard-of-hearing people. Specialized training is required to understand sign language, and as such, people without disabilities around them cannot communicate effectively. The main objective of this study is to develop a mechanism for streamlining the deep learning model for sign language recognition by utilizing the 30 most prevalent words in our everyday lives. The dataset was designed through 30 ASL (American Sign Language) words consisting of custom-processed video sequences, which consist of 5 subjects and 50 sample videos for each class. The CNN model can be applied to video frames to extract spatial properties. Using CNN’s acquired data, the LSTM model may then predict the action being performed in the video. We present and evaluate the results of two separate datasets—the Pose dataset and the Raw video dataset. The dataset was trained with the Long-term Recurrent Convolutional Network (LRCN) approach. Finally, a test accuracy of 92.66% was reached for the raw dataset, while 93.66% for the pose dataset.