Sequence Of Image Frames Research Articles

Deep learning for automated encrustation detection in sewer inspection

Rapid urbanization and population growth in recent decades have placed significant pressure on urban cities to rely heavily on underground infrastructure, such as sewers and tunnels, to maintain the provision of essential services. These sewers, typically having a limited lifespan of 50 to 100 years, are prone to various forms of defects. While prior research has primarily addressed common sewer defect like crack, root intrusion, and infiltration among others, the challenge of encrustation—the formation of hard deposits within sewer systems—has received less attention. This study presents a pioneering deep-learning approach to detect encrustation in sewers by leveraging survey videos from 14 different sewers in the United Kingdom. Our work marks the first effort to develop models specifically for detecting encrustation using deep learning techniques, as previous studies have focused on other types of deposits such as settled and attached deposits. By converting the videos into sequential image frames, we subjected them to thorough analysis and several image pre-processing techniques. Our contributions include the development and comparison of different classification models using backbone CNN networks such as AlexNet, VGG16, EfficientNet, and VGG19 to classify encrustation. Notably, this study provides the first metric-based comparison of these backbone networks to identify the most effective model for encrustation detection. The results demonstrate an impressive 96 % accuracy using the deep architecture of VGG19. Beyond accuracy, this research explores the impact of data augmentation and network dropout on reducing overfitting and enhancing model performance. Additionally, we analyze the time complexities associated with training models with and without data augmentation, providing valuable insights into the efficiency of our approach.

A systematic review of the application of machine learning techniques to ultrasound tongue imaging analysis.

B-mode ultrasound has emerged as a prevalent tool for observing tongue motion in speech production, gaining traction in speech therapy applications. However, the effective analysis of ultrasound tongue image frame sequences (UTIFs) encounters many challenges, such as the presence of high levels of speckle noise and obscured views. Recently, the application of machine learning, especially deep learning techniques, to UTIF interpretation has shown promise in overcoming these hurdles. This paper presents a thorough examination of the existing literature, focusing on UTIF analysis. The scope of our work encompasses four key areas: a foundational introduction to deep learning principles, an exploration of motion tracking methodologies, a discussion of feature extraction techniques, and an examination of cross-modality mapping. The paper concludes with a detailed discussion of insights gleaned from the comprehensive literature review, outlining potential trends and challenges that lie ahead in the field.

Individual honey bee tracking in a beehive environment using deep learning and Kalman filter

The honey bee is the most essential pollinator and a key contributor to the natural ecosystem. There are numerous ways for thousands of bees in a hive to communicate with one another. Individual trajectories and social interactions are thus complex behavioral features that can provide valuable information for an ecological study. To study honey bee behavior, the key challenges that have resulted from unreliable studies include complexity (high density of similar objects, small objects, and occlusion), the variety of background scenes, the dynamism of individual bee movements, and the similarity between the bee body and the background in the beehive. This study investigated the tracking of individual bees in a beehive environment using a deep learning approach and a Kalman filter. Detection of multiple bees and individual object segmentation were performed using Mask R-CNN with a ResNet-101 backbone network. Subsequently, the Kalman filter was employed for tracking multiple bees by tracking the body of each bee across a sequence of image frames. Three metrics were used to assess the proposed framework: mean average precision (mAP) for multiple-object detection and segmentation tasks, CLEAR MOT for multiple object tracking tasks, and MOTS for multiple object tracking and segmentation tasks. For CLEAR MOT and MOTS metrics, accuracy (MOTA and MOTSA) and precision (MOTP and MOTSP) are considered. By employing videos from a custom-designed observation beehive, recorded at a frame rate of 30 frames per second (fps) and utilizing a continuous frame rate of 10 fps as input data, our system displayed impressive performance. It yielded satisfactory outcomes for tasks involving segmentation and tracking of multiple instances of bee behavior. For the multiple-object segmentation task based on Mask R-CNN, we achieved a 0.85 mAP. For the multiple-object-tracking task with the Kalman filter, we achieved 77.48% MOTA, 79.79% MOTSP, and 79.56% recall. For the overall system for multiple-object tracking and segmentation tasks, we achieved 77.00% MOTSA, 75.60% MOTSP, and 80.30% recall.

Identity-Aware and Shape-Aware Propagation of Face Editing in Videos.

The development of deep generative models has inspired various facial image editing methods, but many of them are difficult to be directly applied to video editing due to various challenges ranging from imposing 3D constraints, preserving identity consistency, ensuring temporal coherence, etc. To address these challenges, we propose a new framework operating on the StyleGAN2 latent space for identity-aware and shape-aware edit propagation on face videos. In order to reduce the difficulties of maintaining the identity, keeping the original 3D motion, and avoiding shape distortions, we disentangle the StyleGAN2 latent vectors of human face video frames to decouple the appearance, shape, expression, and motion from identity. An edit encoding module is used to map a sequence of image frames to continuous latent codes with 3D parametric control and is trained in a self-supervised manner with identity loss and triple shape losses. Our model supports propagation of edits in various forms: I. direct appearance editing on a specific keyframe, II. implicit editing of face shape via a given reference image, and III. existing latent-based semantic edits. Experiments show that our method works well for various forms of videos in the wild and outperforms an animation-based approach and the recent deep generative techniques.

Streamline video-based automatic fabric pattern recognition using Bayesian-optimized convolutional neural network

Examining fabric weave patterns (FWPs) is connected to image-based surface texture feature (STF) acquisition, which can be difficult due to the structural complexity of woven fabrics. Randomly capturing static images may not correlate with the entire STF of a fabric. Traditionally, FWPs analysis is conducted by human vision, which causes an intensive cognitive load. Ultimately, the human vision-based cognitive load leads to ineffective quality inspection and error-prone FWPs analysis results. Given the above challenges, this study proposes a new streamlined video-based FWPs recognition method by incorporating Bayesian-optimized convolutional neural network (Bayes Opt-CNN). Essentially, this method is capable of leveraging the spatiotemporal features of the fabric’s intricate surface structure. In this study, to validate the effectiveness of the proposed method, seven types of fabric structures were captured as streamline videos, which were then converted into sequences of image frames. Subsequently, the Bayesian optimization process was introduced to select the best hyperparameters during CNN-based supervised learning for pattern recognition. The evaluation demonstrates that the proposed method outperforms the benchmark method for identifying FWPs.

A modulized lane‐follower for driverless vehicles using multi‐frame

AbstractAs a fundamental function, lane following plays an important role for driverless vehicles. Unfortunately, lane followers generally confront great difficulty in lane line missed situations caused by vague line, shadows etc. However, for most lane line missed situation, clues of the line may be hidden in prior view of it. Consequently, a lane follower called UNL Lane Follower, which contains two deep learning network modules is proposed. The first module is a lane line detection model called UNET_CLB. Here, the sequence of image frames is utilised rather than only the current frame to deal with the missing lane lines. The second module is a lane‐following model called LSTM_DTS, which combines a deep learning attention mechanism (temporal attention network and spatial attention network) with a recurrent neural network. As a result, the proposed UNL Lane Follower produces smoother driving behaviour, especially when a lane line is temporally missed. For better explain ability, the role of each part of the network structure is analysed and explained intuitively. As a modularised network, the UNET_CLB is firstly trained and tested on the TuSimple dataset and CULane dataset. The LSTM_DTS lane follow is then trained and tested on our actual lane following dataset. Finally, the UNL Lane Follower is trained and tested as a whole in a simulation running on Webots, after importing the weight of the two modules trained separately. All testing results showed that the UNL Lane Follower can provide better robustness and accuracy for lane line following mission in the line missed situations.

Moving Target Detection Algorithm for Dynamic Image Sequences on The Basis of Artificial Neural Network

<p>Due to the large changes in dynamic image sequence frames and the complex detection scene, it is difficult to accurately detect moving objects. Therefore, the study proposes a moving target detection algorithm based on artificial neural network. First, the algorithm performs standardized grayscale processing and gamma correction processing on the dynamic image to eliminate the noise interference of the dynamic image. After that, the model calculates the gradient of the dynamic image in order to complete the feature extraction of the dynamic image. Then, according to the result of hog feature extraction, the study adopts the inter-frame calculation method to update the background of the dynamic image. Finally, the principle and structure of the neural network are analyzed experimentally, and a channel attention mechanism is introduced to train dynamic image sequences to obtain MTD results. Experimental results show that the proposed algorithm achieves higher accuracy in MTD than conventional detection algorithms. The calculation efficiency of the algorithm in this paper has significant advantages, and the average detection time is 3.69515ms, which can meet the real-time requirements of MTD.</p> <p> </p>

Surrounding-aware representation prediction in Birds-Eye-View using transformers.

Birds-Eye-View (BEV) maps provide an accurate representation of sensory cues present in the surroundings, including dynamic and static elements. Generating a semantic representation of BEV maps can be a challenging task since it relies on object detection and image segmentation. Recent studies have developed Convolutional Neural networks (CNNs) to tackle the underlying challenge. However, current CNN-based models encounter a bottleneck in perceiving subtle nuances of information due to their limited capacity, which constrains the efficiency and accuracy of representation prediction, especially for multi-scale and multi-class elements. To address this issue, we propose novel neural networks for BEV semantic representation prediction that are built upon Transformers without convolution layers in a significantly different way from existing pure CNNs and hybrid architectures that merge CNNs and Transformers. Given a sequence of image frames as input, the proposed neural networks can directly output the BEV maps with per-class probabilities in end-to-end forecasting. The core innovations of the current study contain (1) a new pixel generation method powered by Transformers, (2) a novel algorithm for image-to-BEV transformation, and (3) a novel network for image feature extraction using attention mechanisms. We evaluate the proposed Models performance on two challenging benchmarks, the NuScenes dataset and the Argoverse 3D dataset, and compare it with state-of-the-art methods. Results show that the proposed model outperforms CNNs, achieving a relative improvement of 2.4 and 5.2% on the NuScenes and Argoverse 3D datasets, respectively.

USER AUTHENTICATION USING EYE-BLINK PASSWORD

Personal identification numbers (PINS) are used to authenticate a user for security purpose. In order to authenticate himself the user should physically input the PIN, which might be susceptible to password cracking via shoulder surfing or thermal tracking. Authenticating a PIN with hands-off eye blinks PIN entry techniques, on the opposite hand, offers a safer password entry option by not leaving any physical footprints behind. User authentication using eye-blink password refers to finding the attention blinks in sequential image frames, and generation of PIN. This paper presents an application with three layer authentication system that are face detection, PIN entry by eye blink and OTP (One Time Password) to avoid thermal tracking attacks and shoulder surfing.

Video logo detection by Deep-Transfer Active Learning

Brand logo detection is a special aspect of machine vision. However, Video logo detection benchmarks are scarce in the public domain. We exploit the power of a deep convolutional neural network (DCNN) and leverage established datasets related to existing applications to develop a deep-transfer active-learning (DTAL) algorithm to select the most valuable samples so that the smallest number possible needs to be labeled to achieve maximum performance improvements for video object detection model training. By exploiting the possible shared deep feature space between static and video datasets through transfer learning based on highly adaptable DCNN features, DTAL implements diversity-based active learning to select the most informative samples from a sequence of similar image frames for video object detection. We successfully apply the new DTAL algorithm to implement active learning for logo detection from live streaming sports videos as well as pedestrian and face detection from video data. We show that DTAL is a better active-learning method than state-of-the-art deep-learning-based active-learning object detection techniques. We also contribute one of the largest video-based logo resources, the Sports Match Video Logo (SMVL) dataset, to facilitate general logo detection research using transfer- and active-learning applications for video object detection.

A constructive deep convolutional network model for analyzing video-to-image sequences

This paper proposes a new technique to add sign language as a special and real-time feature. In addition to gestures, sign language also uses grammatical rules, linguistic frameworks, etc. We require a method that can extract meaningful sentences from non-manual features. Utilizing techniques from computer graphics, neurologic music therapy, and NN-based image/video formation, this is accomplished. Our goal is to use this to process dynamic images for output generation and real-time classification. Current CNN-based techniques operate by taking the entire video as input, dividing it into layers for the classifier to work on, and then combining and providing the output to the user. Here, Convolutional Deep VGG-16 (CDVGG-16) classifiers adopted for sign feature learning, which is iteratively trained and tested. Their architecture consists of blocks, where each block is composed of 2D Convolution and Max Pooling layers. We prefer VGG-16 over VGG-19 in order to improve feature extraction and decrease overfitting. The captured sign language video processing with some pre-processing and classification steps leads to the extraction of sign language feature space assisting deaf people. As a positive approach, the Sign Language Recognition (SLR) modeling framework with deep learning is introduced as it includes various notable advantages. There are three essential phases for provisioning the SLR framework: (1) data acquisition, (2) image pre-processing, and (3) CDVGG-16 classifier. The simulation is done with the MATLAB 2020a environment with various performance metrics like accuracy, precision, F-measure, recall and running time. The promising results show that real-world SLR applications can be formed using the suggested SLR paradigm. The proposed CDVGG-16 gives 95.6% prediction accuracy, 85.6% precision, 99.9% recall, 96.8% F-measure and 0.125 s running time. For improved performance, our proposed method employs normalization and examines sequential image frames. The results and accuracy indicate that it is a suitable method for video-image classification, as expected, and that deep layers and data augmentation can be used to address any overfitting and underfitting issues. Instead of being a literal translation frame by frame, the filter size is adjusted to ensure that output is translated appropriately given the context.

A Generic Approach towards Amharic Sign Language Recognition

In the day-to-day life of communities, good communication channels are crucial for mutual understanding. The hearing-impaired community uses sign language, which is a visual and gestural language. In terms of orientation and expression, it is separate from written and spoken languages. Despite the fact that sign language is an excellent platform for communication among hearing-impaired persons, it has created a communication barrier between hearing-impaired and non-disabled people. To address this issue, researchers have proposed sign language to text translation systems for English and other European languages as a solution. The goal of this research is to design and develop an Amharic digital text converter system using Ethiopian sign language. The proposed system was created with the help of two key deep learning algorithms: a pretrained deep learning model and a Long Short-Term Memory (LSTM). The LSTM was used to extract sequence information from a sequence of image frames of a specific sign language, while the pretrained deep learning model was used to extract features from single frame images. The dataset used to train the algorithms was gathered in video format from Addis Ababa University. Prior to feeding the obtained dataset to the deep learning models, data preprocessing activities such as cleaning and video to image frame segmentation were conducted. The system was trained, validated, and tested using 80%, 10%, and 10% of the 2475 images created during the preprocessing step. Two pretrained deep learning models, EfficientNetB0 and ResNet50, were used in this investigation, and they attained an accuracy of 72.79%. In terms of precision and f1-score, ResNet50 outperformed EfficientNetB0. For the proposed system, a graphical user interface prototype was created, and the best performing model was chosen and implemented. The proposed system can be utilized as a starting point for other researchers to improve upon, based on the outcomes of the experiment. More high-quality training datasets and high-performance training machines, such as GPU-enabled computers, can be added to the system to improve it.

3D vision‐based out‐of‐plane displacement quantification for steel plate structures using structure‐from‐motion, deep learning, and point‐cloud processing

AbstractIn this paper, a novel accurate and economical 3D computer vision‐based framework is proposed to quantify out‐of‐plane displacements of steel plate structures. First, a sequence of image frames of the steel plate structures of interest is collected. Second, using image association, structure‐from‐motion, and multi‐view stereo algorithms, a 3D point cloud of the steel plate structures and their surroundings is created. Third, an efficient 3D object detection method based on convolutional neural networks is developed and implemented to identify the steel plate structures in the 3D point cloud. Last, the out‐of‐plane displacements of the steel plate structures are quantified using point cloud postprocessing algorithms. The proposed framework has been implemented on a steel plate damper and a full‐scale steel corrugated plate wall panel, which are commonly used in structural and earthquake engineering applications. The results indicate the developed framework can successfully localize the steel plate components in the 3D scene and accurately quantify the out‐of‐plane structural displacements with an average accuracy of ∼1 mm. The implementation shows the proposed framework can accurately and efficiently quantify the out‐of‐plane displacements of steel plate structures in realistic engineering applications.

A NOVEL WOOD FEATURE EXTRACTION METHOD BASED ON IMPROVED BLOCKED HIGHER-ORDER LOCAL AUTO-CORRELATION

Traditionally, HLAC (Higher-order Local Auto-Correlation) algorithm was used to extract texture features of wood images. However, heavy memory consumption and complexity of high-order mask pattern were common in HLAC. A novel feature extraction strategy based on improved blocked higher-order local auto-correlation (IBHLAC) is proposed to circumvent these problems. Initially, sequences of the whole wood image frames, which are the grayscale treatment, were being divided into series of subdivisions vertically and horizontally. Additionally,to enhance auto-correlation ability of the proposed method, different high-order patterns of masks were rebuilt based on zero-order mask by introducing the morphology and affine transformation. Finally, time-consumption and memory occupation of related four methods were compared. Experiment results indicated IBHLAC costs less time and fewer memory consumption on the wood texture database compared with other methods, which reveal that IBHLAC is efficient.

Fast and flexible processing of large FRET image stacks using the FRET-IBRA toolkit.

Ratiometric time-lapse FRET analysis requires a robust and accurate processing pipeline to eliminate bias in intensity measurements on fluorescent images before further quantitative analysis can be conducted. This level of robustness can only be achieved by supplementing automated tools with built-in flexibility for manual ad-hoc adjustments. FRET-IBRA is a modular and fully parallelized configuration file-based tool written in Python. It simplifies the FRET processing pipeline to achieve accurate, registered, and unified ratio image stacks. The flexibility of this tool to handle discontinuous image frame sequences with tailored configuration parameters further streamlines the processing of outliers and time-varying effects in the original microscopy images. FRET-IBRA offers cluster-based channel background subtraction, photobleaching correction, and ratio image construction in an all-in-one solution without the need for multiple applications, image format conversions, and/or plug-ins. The package accepts a variety of input formats and outputs TIFF image stacks along with performance measures to detect both the quality and failure of the background subtraction algorithm on a per frame basis. Furthermore, FRET-IBRA outputs images with superior signal-to-noise ratio and accuracy in comparison to existing background subtraction solutions, whilst maintaining a fast runtime. We have used the FRET-IBRA package extensively to quantify the spatial distribution of calcium ions during pollen tube growth under mechanical constraints. Benchmarks against existing tools clearly demonstrate the need for FRET-IBRA in extracting reliable insights from FRET microscopy images of dynamic physiological processes at high spatial and temporal resolution. The source code for Linux and Mac operating systems is released under the BSD license and, along with installation instructions, test images, example configuration files, and a step-by-step tutorial, is freely available at github.com/gmunglani/fret-ibra.

Learning meta-adversarial features via multi-stage adaptation network for robust visual object tracking

Visual tracking is a crucial research topic in computer vision, which aims to locate any object as precisely as possible over a sequence of image frames. However, the existing trackers often suffer from the object drifting problem due to the difficulty of adapting to complex environments. In this paper, we propose a novel multi-stage adaptation network (MAN), including the meta-adaptation, feature adaptation, and location adaptation sub-networks, to improve the adaptability and robustness of tracking. Specifically, the meta-adaptation sub-network takes advantage of meta-learning to enhance the generalization ability for the new tracking sequence. The feature adaptation sub-network exploits an adversarial attention mask module and a multi-level and multi-scale meta-classifier module for improving the robustness and discriminative ability. Moreover, the location adaptation sub-network can refine the tracking location to avoid the drifting problem. The three sub-networks can benefit from each other and are strategically integrated in a whole framework. Extensive experimental results demonstrate that the proposed tracker outperforms the state-of-the-art methods on several challenging datasets, including OTB50, OTB2013, OTB100, UAV123, UAV20L, NfS, LaSOT, VOT2016, and VOT2018.

Computing driver tiredness and fatigue in automobile via eye tracking and body movements

The aim of this paper is to classify the driver tiredness and fatigue in automobile via eye tracking and body movements using deep learning based Convolutional Neural Network (CNN) algorithm. Vehicle driver face localization serves as one of the most widely used real-world applications in fields like toll control, traffic accident scene analysis, and suspected vehicle tracking. The research proposed a CNN classifier for simultaneously localizing the region of human face and eye positioning. The classifier, rather than bounding rectangles, gives bounding quadrilaterals, which gives a more precise indication for vehicle driver face localization. The adjusted regions are preprocessed to remove noise and passed to the CNN classifier for real time processing. The preprocessing of the face features extracts connected components, filters them by size, and groups them into face expressions. The employed CNN is the well-known technology for human face recognition. One we aim to extract the facial landmarks from the frames, we will then leverage classification models and deep learning based convolutional neural networks that predict the state of the driver as 'Alert' or 'Drowsy' for each of the frames extracted. The CNN model could predict the output state labels (Alert/Drowsy) for each frame, but we wanted to take care of sequential image frames as that is extremely important while predicting the state of an individual. The process completes, if all regions have a sufficiently high score or a fixed number of retries are exhausted. The output consists of the detected human face type, the list of regions including the extracted mouth and eyes with recognition reliability through CNN with an accuracy of 98.57% with 100 epochs of training and testing.

Design of a Low-Power Embedded System Based on a SoC-FPGA and the Honeybee Search Algorithm for Real-Time Video Tracking.

Video tracking involves detecting previously designated objects of interest within a sequence of image frames. It can be applied in robotics, unmanned vehicles, and automation, among other fields of interest. Video tracking is still regarded as an open problem due to a number of obstacles that still need to be overcome, including the need for high precision and real-time results, as well as portability and low-power demands. This work presents the design, implementation and assessment of a low-power embedded system based on an SoC-FPGA platform and the honeybee search algorithm (HSA) for real-time video tracking. HSA is a meta-heuristic that combines evolutionary computing and swarm intelligence techniques. Our findings demonstrated that the combination of SoC-FPGA and HSA reduced the consumption of computational resources, allowing real-time multiprocessing without a reduction in precision, and with the advantage of lower power consumption, which enabled portability. A starker difference was observed when measuring the power consumption. The proposed SoC-FPGA system consumed about 5 Watts, whereas the CPU-GPU system required more than 200 Watts. A general recommendation obtained from this research is to use SoC-FPGA over CPU-GPU to work with meta-heuristics in computer vision applications when an embedded solution is required.

Flow visualization by Matlab® based image analysis of high-speed polymer melt extrusion film casting process for determining necking defect and quantifying surface velocity profiles

Abstract The primary objective of this research paper is to detect and quantify the necking defect and surface velocity profiles in high-speed polymer melt extrusion film casting (EFC) process using Matlab® based image processing techniques. Extrusion film casting is an industrially important manufacturing process and is used on an industrial scale to produce thousands of kilograms of polymer films/sheets and coated products. In this research, the necking defect in an EFC process has been studied experimentally and the effects of macromolecular architecture such as long chain branching (LCB) on the extent of necking have been determined using image processing methodology. The methodology is based on the analysis of a sequence of image frames taken with the help of a commercial CCD camera over a specific target area of the EFC process. The image sequence is then analyzed using Matlab® based image processing toolbox wherein a customized algorithm is written and executed to determine the edges of the extruded molten polymeric film to quantify the necking defect. Alongwith the necking defect, particle tracking velocimetry (PTV) technique is also used in conjunction with the Matlab® software to determine the centerline and transverse velocity profiles in the extruded molten film. It is concluded from this study that image processing techniques provide valuable insights into quantifying both the necking defect and the associated velocity profiles in the molten extruded film.

A micro-vibration apparatus for dynamic mechanical analysis of ultrathin polymer films.

We present the designand validation of an apparatus, which is developed based on micro-vibration in cyclic bulge and contraction, to determine dynamic mechanical properties of ultrathin polymer films. By controlling air pressure acting on a polymer membrane, the apparatus exerts simple harmonic or other type periodic stresses on the film, and the resultant real-time deformation of the freely standing film is recorded by using a high-speed CCD camera. From the image frame sequences and the gas pressure data, the real-time stress and strain of the polymer film are attained. Consequently, the dynamic mechanical properties, including biaxial storage and loss moduli, and loss factor can be determined for polymer films with thickness down to 20nm. This apparatus could also be used to determine other mechanical properties such as fatigue and yield for nanoconfined films of soft matter.