Image Frames Research Articles

Techniques for Detecting the Start and End Points of Sign Language Utterances to Enhance Recognition Performance in Mobile Environments

Recent AI-based technologies in mobile environments have enabled sign language recognition, allowing deaf individuals to communicate effectively with hearing individuals. However, varying computational performance across different mobile devices can result in differences in the number of image frames extracted in real time during sign language utterances. The number of extracted frames is a critical factor influencing the accuracy of sign language recognition models. If the number of extracted frames is too small, the performance of the sign language recognition model may decline. Additionally, detecting the start and end points of sign language utterances is crucial for improving recognition accuracy, as the period before the start point and after the end point often involves no action being performed. These parts do not capture the unique characteristics of each sign language. Therefore, this paper proposes a technique to dynamically adjust the sampling rate based on the number of frames extracted in real time during sign language utterances in mobile environments, with the aim of accurately detecting the start and end points of the sign language. Experiments were conducted to compare the proposed technique with the fixed sampling rate method and with the no-sampling method as a baseline. Our findings show that the proposed dynamic sampling rate adjustment method improves performance by up to 83.64% in top-5 accuracy and by up to 66.54% in top-1 accuracy compared to the fixed sampling rate method. The performance evaluation results underscore the effectiveness of our dynamic sampling rate adjustment approach in enhancing the accuracy and robustness of sign language recognition systems across different operational conditions.

Deep Learning-Enabled Optimization of Polyp Frame Detection and Segmentation

Polyp segmentation is a method for identifying and characterizing polyps, or abnormal tissue development, as seen in frame images from medical procedures such as endoscopic or colonoscopy frame images. Polyp segments are reliant on high-quality imaging data, which can be intermittently accessible, causing lower detection and precision in segmentation. To overcome these challenges, we propose a novel deep learning (DL) method called intelligent chameleon swarm optimized-adaptive fully convolutional neural network (ICSO-AFConvNet) for polyp segmentation and frame detection. Initially, we gather the dataset named (KVasir-SEG), which contains a thousand frame images, and then contrast limited adaptive histogram equalization (CLAHE) technique, is used to preprocess the data. Experimental findings on the KVasir-SEG dataset reveal that ICSO-AFConvNet outperforms previous approaches in measures such as Dice, Interception over union (IoU), Recall and Precision. These findings show that the suggested ICSO-AFConvNET technique has tough generalization power and learning ability, making it an appealing alternative for practical applications with significant data variances.

Real-Time Facial Cover Detection System based Deep Convolution Neural Network with Proposed Activation Function

The ongoing global contagion has highlighted the importance of effective preventive measures such as wearing face masks in public spaces. In this study, we suggest a deep learning-based approach for real-time facial covering detection to aid in enforcing mask-wearing protocols. Our system utilizes deep learning networks (CNNs) to automatically detect whether individuals in images or video streams are wearing mask or not. The suggested system includes of 3 main stages: face detection, facial cover detection, and real-time monitoring. Firstly, faces are localized in the input image or video frame using a proposed face detection model. Then, the detected faces are fed into a proposed CNN model for mask classification, which determines whether each face is covered with a mask or not. Finally, the system will provide real-time monitoring and alerts authorities or stakeholders about non-compliance with mask wearing guidelines. We appraise the execution of our system on publicly available datasets and demonstrate its effectiveness in accurately detecting face masks in various scenarios. Additionally, we discuss the challenges and limitations of deploying such as system in real-world settings, including issues related to privacy, bias, and scalability. Overall, our proposed facial covering detection system offers a viable solution for automated monitoring and enforcement of face mask policies, contributing to public health efforts in mitigating the spread of contagious diseases.

Faint Space Target Information Extraction Based on Small Aperture Telescope in Complex Background

AbstractThere are many problems in space debris monitoring with ground‐based telescopes, such as too many stars in the same field of view, uneven background and optical distortion in the optical system. We propose a two‐stage weak debris detection algorithm. In the first stage, wavelet transform is used to extract different components of three frames of images, and the median of corresponding components of the images is taken respectively to eliminate the influence of stars. In the second stage, an improved version of the faint space target extraction based on principal component analysis. The algorithm uses a smooth‐detection idea to extract target information. Based on a 150 mm aperture telescope, we improved the existing method of faint space debris extraction based on principal component analysis by introducing the smooth‐detection idea, and transformed the target detection problem into the separation problem of sparse matrix and low‐rank matrix. We applied a certain preprocessing consisting of wavelet‐based star removal and median pre‐filtering to keep as little noise and other contaminants as possible. After experimental measurements by observers, the algorithm demonstrated advanced detection capabilities on multiple indicators.

A novel technique to resolve directional ambiguity for Particle Streak Velocimetry

In this study, a novel streak direction-resolving algorithm is introduced to determine the direction of the flow field for a single image frame with multiple streaks. The streak direction was resolved by varying its intensity from 100% to 50% of the total intensity of the streak image. This technique was applied to two main types of flows parallel and Hill’s vortex flows, parallel flows were divided into further two types constant velocity parallel flow and accelerating parallel flow. The purpose of using different types of flows was to test the robustness of the algorithm with easy and complex flows. The performance of the algorithm was checked by the angular deviation between the true and least-square fitted velocity vectors. The number of correct synthetic streak directions was measured with the success rate in percentage. A high success rate means low angular deviation and a high number of velocity vectors with correct direction was obtained. In this research, four different types of image formats were considered DP (double precision), 16-bit (without noise), 16-bit (1.0% noise), 16-bit (5.0% noise), and 8-bit (without noise) and the best results were obtained for DP and 16-bit image formats. The results of the parallel flows indicated a 100% streak direction success rate, Hill’s vortex was a type of complex flow therefore, the algorithm hard to resolve some streak directions due to very low velocities (less than 0.1 px (pixel)/interval) and very high-velocity gradients (greater than 52 px/interval). The observation shows that Hill’s vortex synthetic streak images were resolved with a success rate of 92.76% which means that the majority of the synthetic streaks were resolved. Experimental analysis was also done by using the PDMS microchannel setup. Intensity variation streaks were recorded with long camera exposure time and for maintaining the intensity variation, as the LED switched on maximum illumination was started, and continuous decrement in illumination was set by switching off at 50% of the total illumination intensity. The best results were achieved with a 94% success rate. Therefore, the proposed novel approach can be used with less expensive hardware for image processing with a single image frame and is useful for multiple applications.

Automated analysis of a novel object recognition test in mice using image processing and machine learning

The novel object recognition test (NORT) is one of the most commonly employed behavioral tests in experimental animals designed to evaluate an animal's interest in and recognition of novelty. However, manual procedures, which rely on researchers’ observations, prevent high throughput analysis. In this study, we developed an automated analysis method for NORT utilizing machine learning-assisted exploratory behavior detection. We recorded the exploratory behavior of the mice using a video camera. The coordinates of the mouse nose and tail base in recorded video files were detected using a pre-trained machine learning model, DeepLabCut. Each video was then segmented into frame images, which were categorized into "exploratory,” or "non-exploratory" frames based on manual observation. Mouse feature vectors were calculated as vectors from the nose to the vertices of the object and were utilized for SVM training. The trained SVM effectively detected exploratory behaviors, showing a strong correlation with human observer assessments. Upon application to NORT, the duration of mouse exploratory behavior towards objects predicted by the SVM exhibited a significant correlation with the assessments made by human observers. The novelty discrimination index derived from the SVM predictions also aligned well with that from human observations.

A Nonintrusive Optical Approach to Characterize Heliostats in Utility-Scale Power Tower Plants: Camera Position Sensitivity Analysis

Abstract Optics plays a major role in the effectiveness of concentrating solar power (CSP) technologies. The nonintrusive optical (NIO) approach developed by the National Renewable Energy Laboratory uses uncrewed aircraft system (UAS)-based imaging to survey heliostats in a commercial-scale power tower CSP plant and characterize their optical errors. The image processing algorithm uses photogrammetry to calculate the camera position for each image frame, and the accuracy of the estimated optical errors is highly sensitive to the calculated camera position accuracy. In this study, we simulate a series of case studies in python to examine the impact of different parameters of the sensitivity of the camera calculation, including the number of facet corners used as control points for the photogrammetric calculation, precision error in the detected pixel locations of the facet corners in the image, and precision error of the canting and mounting positions of the facets of the heliostat. The case studies consider heliostat geometry based on three commercial designs to serve as representative examples of different possible sizes of heliostats that the NIO method could be applied to. The results show that increasing the number of control points can improve accuracy for heliostats with many facets, pixel precision has a significantly larger impact on camera calculation accuracy than facet canting and mounting errors, and the camera distance and focal length must be chosen to ensure adequate pixel accuracy on the heliostat surface depending on the size of heliostat. Based on the results, recommendations for the allowable values of each parameter are provided to achieve the required NIO optical error estimation accuracy depending on the size of heliostat.

Deep learning for automated encrustation detection in sewer inspection

Rapid urbanization and population growth in recent decades have placed significant pressure on urban cities to rely heavily on underground infrastructure, such as sewers and tunnels, to maintain the provision of essential services. These sewers, typically having a limited lifespan of 50 to 100 years, are prone to various forms of defects. While prior research has primarily addressed common sewer defect like crack, root intrusion, and infiltration among others, the challenge of encrustation—the formation of hard deposits within sewer systems—has received less attention. This study presents a pioneering deep-learning approach to detect encrustation in sewers by leveraging survey videos from 14 different sewers in the United Kingdom. Our work marks the first effort to develop models specifically for detecting encrustation using deep learning techniques, as previous studies have focused on other types of deposits such as settled and attached deposits. By converting the videos into sequential image frames, we subjected them to thorough analysis and several image pre-processing techniques. Our contributions include the development and comparison of different classification models using backbone CNN networks such as AlexNet, VGG16, EfficientNet, and VGG19 to classify encrustation. Notably, this study provides the first metric-based comparison of these backbone networks to identify the most effective model for encrustation detection. The results demonstrate an impressive 96 % accuracy using the deep architecture of VGG19. Beyond accuracy, this research explores the impact of data augmentation and network dropout on reducing overfitting and enhancing model performance. Additionally, we analyze the time complexities associated with training models with and without data augmentation, providing valuable insights into the efficiency of our approach.

SwinTExCo: Exemplar-based video colorization using Swin Transformer

Video colorization represents a compelling domain within the field of Computer Vision. The traditional approach in this field relies on Convolutional Neural Networks (CNNs) to extract features from each video frame and employs a recurrent network to learn information between video frames. While demonstrating considerable success in colorization, most traditional CNNs suffer from a limited receptive field size, capturing local information within a fixed-sized window. Consequently, they struggle to directly grasp long-range dependencies or pixel relationships that span large image or video frame areas. To address this limitation, recent advancements in the field have leveraged Vision Transformer (ViT) and their variants to enhance performance. This article introduces Swin Transformer Exemplar-based Video Colorization (SwinTExCo), an end-to-end model for the video colorization process that incorporates the Swin Transformer architecture as the backbone. The experimental results demonstrate that our proposed method outperforms many other state-of-the-art methods in both quantitative and qualitative metrics. The achievements of this research have significant implications for the domain of documentary and history video restoration, contributing to the broader goal of preserving cultural heritage and facilitating a deeper understanding of historical events through enhanced audiovisual materials.

Vibration displacement measurement of bridge structural models using image super-resolution reconstruction and visual object detection network

Abstract Visual vibration measurement has emerged in the field of structural health monitoring in recent years, but it still has some shortcomings in terms of resolution, recognition rate and real-time performance. Considering the three aspects of recovering high-frequency image details, improving the compactness of the target bounding box, and reducing the computational time, we use the constructed image super-resolution reconstruction model and target detection model to measure the vibration displacement of the bridge structural model. First, we integrate the Transformer module into the Unet network with a simple structure. The Swin and Global Transformer Unet (SGTU) module constructed in this form can reduce the computational cost while reconstructing the large-resolution feature map target, and it can sharply edge information of the vibration target. We use the framework of the YOLOv5 algorithm as the backbone, and use the GhostBottleneck (GB) module to reduce the time for convolution operations to generate similar features. In addition, the proposed DWCBottleneck (DWCB) fusion module is also able to achieve high-level semantic fusion and network depth expansion with minimal computational cost. Finally, the center point offset of the bounding box predicted by the model can be used to obtain the displacement offset of the object in the image sequence. The position information of the target in the first frame image is used as the reference frame for calculating the offset, and the vibration displacement of the flexible structure in the image coordinate system is obtained by calculating the deviation of the displacement between the remaining frames and the first frame. We perform qualitative and quantitative comparisons in three aspects: video super-resolution reconstruction, visual detection robustness, and sensor vibration measurement displacement using a homemade vibration image dataset. The time-frequency domain displacement curves regressed by the visual vibration measurement algorithm are compared with the curves acquired after accelerometer acquisition, indicating the necessity of super-resolution reconstruction in visual vibration measurement.

Nested object detection using mask R-CNN: application to bee and varroa detection

AbstractIn this paper, we address an essential problem related to object detection and image processing: detecting objects potentially nested in other ones. This problem exists particularly in the beekeeping sector: detecting varroa parasites on bees. Indeed, beekeepers must ensure the level of infestation of their apiaries by the varroa parasite which settles on the backs of bees. As far as we know, there is no yet a published approach to deal with nested object detection using only one neural network trained on two different datasets. We propose an approach that fills this gap. Therefore, we improve the accuracy and the efficiency of bee and varroa detection task. Our work is based on deep learning, more precisely Mask R-CNN neural network. Instead of segmenting detected objects (bees), we segment internal objects (varroas). We add a branch to Faster R-CNN to segment internal objects. We extract relevant features for internal object segmentation and suggest efficient method for training the neural network on two different datasets. Our experiments are based on a set of images of bee frames, containing annotated bees and varroa mites. Due to differences in occurrence rates, two different sets were created. After carrying out experiments, we ended up with a single neural network capable of detecting two nested objects without decreasing accuracy compared to two separate neural networks. Our approach, compared to traditional separate neural networks, improves varroa detection accuracy by 1.9%, reduces infestation level prediction error by 0.22%, and reduces execution time by 28% and model memory by 23%. In our approach, we extract Res4 (a layer of the ResNet neural network) features for varroa segmentation, which improves detection accuracy by 11% compared to standard FPN extraction. Thus, we suggest a new approach that detects nested objects more accurately than two separate network approaches.

Image Framing of Indonesia's Constitutional Court in Tempo.co News (18-27 October 2023)

The Constitutional Court of the Republic of Indonesia (MK) plays a crucial role in reviewing laws against the 1945 Constitution and resolving election disputes, ensuring adherence to constitutional principles. Tempo.co, a prominent media outlet, has extensively covered the MK, especially following the recent controversy over the Court's ruling on age limits for presidential and vice-presidential candidates. This research examines how Tempo.co frames its coverage of the MK using Robert N. Entman’s framing analysis model, alongside agenda-setting theory and media reality construction theory. Applying a qualitative methodology, the study analyzes Tempo.co’s news articles from October 18 to October 27, 2023. The analysis reveals that Tempo.co employs various framing techniques consistent with Entman's model, which influences public perception through agenda-setting by emphasizing specific aspects of the MK’s activities. Additionally, the coverage constructs a particular narrative about the MK's role and decisions. To ensure data validity, thorough checks were performed. Findings indicate that Tempo.co’s headlines generally maintain neutrality and do not exhibit overt political bias, focusing on the institution’s functions. This research offers insights into how media framing, through Entman's model, agenda-setting, and reality construction, shapes the portrayal of significant constitutional issues and affects public understanding of the MK.

The real-time shadow detection of the PV module by computer vision based on histogram matching and gamma transformation method

Solar energy plays an important role in renewable energy generation, with the advantages of low pollution, easy installation, and relatively easy access. However, photovoltaic (PV) modules are susceptible to cause localized shading from external factors such as leaves in the canopy, surrounding buildings, etc., which would affect power generation efficiency and even pose safety risks. Existing methods cannot perform well in real-time conditions. This paper proposes a real-time shading monitoring method for the PV module based on computer vision. The gamma transform and histogram matching were adopted to enhance key features and adjust the global gamut strength distribution in the image of the PV module; then the gray-level slicing method finished the segmentation to detect the shadow from the video. All processing can be realized in the real-time monitor camera and the detection results can be displayed on the HMI in PC with high efficiency and low cost. According to tests in the practical complex environment, the method can have enough detection performance and high real-time performance with an accuracy of 0.98, and the F0.5 and F2 values are 0.87 and 0.85, respectively. The metrics of the proposed method are higher than those of the existing Canny detection method, the Random Forest detection method, and the CNN detection method. In addition, the average time required by the proposed method to process a frame is 0.721 s. In addition, the average time required by the method to process an image frame is 0.721 s, which has good real-time performance.

An Integrated Framework of Critical Metaphor Analysis and Multi-level View with an Application to Metaphors for the Vietnam War

Since Conceptual Metaphor Theory (CMT) (Lakoff & Johnson, 1980) was introduced, methods of studying conceptual metaphor have kept improving to respond to methodological criticisms. Critical Metaphor Analysis (CMA) proposed by Charteris-Black (2004) has been considered as a “thought-provoking contribution” to metaphor analysis (Deignan, 2005) when approaching metaphor from various perspectives: critical discourse analysis, corpus analysis, pragmatics and cognitive linguistics. CMA is originally applied to one conceptual level in metaphor – domain. However, this paper argues that CMA can be exploited at four conceptual levels in Multi-level View of conceptual metaphor (Kövecses, 2017b) - image schema, domain, frame, and mental space. The combined framework of Critical Metaphor Analysis – CMA (Charteris-Black, 2004) and Multi-level View of conceptual metaphor – MLV (Kövecses, 2017b) can gain deeper insights into ideologies motivating metaphorical concepts for the Vietnam war as well as elucidate the conceptual structure of metaphor via the four levels. Hence, this combination fills the gap of lacking a framework with optimal balance of semantic, pragmatic, cognitive and critical dimensions. It also features the intriguing relationship between ideologies and conceptual structure, i.e., ideologies are embedded in all the four conceptual levels and systematically develop with increasing specificity from image schema to domain, frame and mental space. The focus of this paper is on our argument for an integrated framework of conceptual metaphor and the newspaper articles written by American war correspondents during the Vietnam war are used for illustration of how the integrated framework can help us better understand the conceptual metaphors in the articles.

A Trusted Supervision Paradigm for Autonomous Driving Based on Multimodal Data Authentication

At the current stage of autonomous driving, monitoring the behavior of safety stewards (drivers) is crucial to establishing liability in the event of an accident. However, there is currently no method for the quantitative assessment of safety steward behavior that is trusted by multiple stakeholders. In recent years, deep-learning-based methods can automatically detect abnormal behaviors with surveillance video, and blockchain as a decentralized and tamper-resistant distributed ledger technology is very suitable as a tool for providing evidence when determining liability. In this paper, a trusted supervision paradigm for autonomous driving (TSPAD) based on multimodal data authentication is proposed. Specifically, this paradigm consists of a deep learning model for driving abnormal behavior detection based on key frames adaptive selection and a blockchain system for multimodal data on-chaining and certificate storage. First, the deep-learning-based detection model enables the quantification of abnormal driving behavior and the selection of key frames. Second, the key frame selection and image compression coding balance the trade-off between the amount of information and efficiency in multiparty data sharing. Third, the blockchain-based data encryption sharing strategy ensures supervision and mutual trust among the regulatory authority, the logistic platform, and the enterprise in the driving process.

Deep learning-based prediction of nodal metastasis in lung cancer using endobronchial ultrasound

ObjectiveEndobronchial ultrasound-guided transbronchial needle aspiration is a vital tool for mediastinal and hilar lymph node staging in patients with lung cancer. Despite its high diagnostic performance and safety, it has a limited negative predictive value. Our objective was to evaluate the diagnostic performance of deep learning-based prediction of lung cancer lymph node metastases using convolutional neural networks developed from automatically extracted images of endobronchial ultrasound videos without supervision of the lymph node location. MethodsPatient and lymph node data were collected from a single-center database. The diagnosis of metastasis was confirmed with endobronchial ultrasound-guided transbronchial needle aspiration and/or surgically resected specimens; the diagnosis of normal lymph node was confirmed with surgically resected specimens only. An annotation system facilitated automated image extraction from endobronchial ultrasound videos. Image frames were randomly selected and split into training and validation datasets on a per-patient basis. A deep learning model with convolutional neural networks, SqueezeNet, was used for image classification via transfer learning based on pretraining from ImageNet. Adaptive moment estimation and stochastic gradient descent were applied as optimizers. ResultsSqueezeNet, with adaptive moment estimation, achieved a sensitivity, specificity, accuracy, positive predictive value, and negative predictive value of 96.7% each after 300 epochs, whereas SqueezeNet with stochastic gradient descent achieved 91.1% each. However, SqueezeNet with stochastic gradient descent demonstrated more stable performance than with adaptive moment estimation. ConclusionsDeep learning-based image classification using convolutional neural networks showed promising diagnostic accuracy for lung cancer nodal metastasis. Future clinical trials are warranted to validate the algorithm's efficacy in a prospective, large-cohort study.

Image-driven prediction of fatigue crack growth in metal materials via spatiotemporal neural network

This study proposes an image-driven model based on the SimVP spatiotemporal neural network (STNN) to predict the fatigue crack growth (FCG) in aluminum alloys. This methodology represents a novel usage of STNNs for FCG analysis. It does not require repetitive modeling, extensive computations, or conventional mechanical assumptions. The datasets used during this study were gathered from fatigue experiments with a variety of crack positions, angles, and load levels; they contained a total of 17,925 image frames obtained from DIC measurements. Subsequently, the displacement fields were interpolated onto uniform grids and then augmented, so they could be fitted into an STNN. The proposed method was validated using specimens with edge and central cracks subjected to loads equal to 15.0 % and 20.0 % of the ultimate load. The generalization capability of the proposed method was studied by predicting the FCG under load levels and crack angles outside the training set. In addition, its predictive capability was investigated for both short and long step sizes by employing datasets in which the image data were collected at varying intervals. The overall structural similarity index measurement values were greater than 0.968, and the root mean square errors were held within 0.025 mm. The predicted displacement fields, crack lengths, and crack growth rates agreed well with experimental measurements.

Deep learning dives: Predicting anxiety in zebrafish through novel tank assay analysis

Behavior is fundamental to neuroscience research, providing insights into the mechanisms underlying thoughts, actions and responses. Various model organisms, including mice, flies, and fish, are employed to understand these mechanisms. Zebrafish, in particular, serve as a valuable model for studying anxiety-like behavior, typically measured through the novel tank diving (NTD) assay. Traditional methods for analyzing NTD assays are either manually intensive or costly when using specialized software. To address these limitations, it is useful to develop methods for the automated analysis of zebrafish NTD assays using deep-learning models. In this study, we classified zebrafish based on their anxiety levels using DeepLabCut. Subsequently, based on a training dataset of image frames, we compared deep-learning models to identify the model best suited to classify zebrafish as anxious or non anxious and found that specific architectures, such as InceptionV3, are able to effectively perform this classification task. Our findings suggest that these deep learning models hold promise for automated behavioral analysis in zebrafish, offering an efficient and cost-effective alternative to traditional methods.

A systematic review of the application of machine learning techniques to ultrasound tongue imaging analysis.

B-mode ultrasound has emerged as a prevalent tool for observing tongue motion in speech production, gaining traction in speech therapy applications. However, the effective analysis of ultrasound tongue image frame sequences (UTIFs) encounters many challenges, such as the presence of high levels of speckle noise and obscured views. Recently, the application of machine learning, especially deep learning techniques, to UTIF interpretation has shown promise in overcoming these hurdles. This paper presents a thorough examination of the existing literature, focusing on UTIF analysis. The scope of our work encompasses four key areas: a foundational introduction to deep learning principles, an exploration of motion tracking methodologies, a discussion of feature extraction techniques, and an examination of cross-modality mapping. The paper concludes with a detailed discussion of insights gleaned from the comprehensive literature review, outlining potential trends and challenges that lie ahead in the field.

Non-cooperative target recognition and relative motion estimation with inertial measurement unit assistance

This study investigated the problems of non-cooperative target recognition and relative motion estimation during spacecraft rendezvous maneuvers. A structure integrating an Inertial Measurement Unit (IMU) and a visual camera was presented. The angular velocity output of the IMU was used to calculate the motion trajectories of star points in multiple image frames, which can highlight the motion of non-cooperative targets with respect to the image background to improve the probability of target recognition. To solve the problem of target misidentification caused by new star points entering the field of view, a target-tracking link based on IMU prediction was introduced to track the position of the target in the image. Furthermore, a measurement model was constructed using the line-of-sight vector generated from target recognition, and the relative motion state was estimated using a Huber-based non-linear filter. Semi-physical and numerical simulations were performed to evaluate the effectiveness and efficiency of the proposed method.