Low Inference Time Research Articles

Real-time crop row detection using computer vision- application in agricultural robots.

The goal of achieving autonomous navigation for agricultural robots poses significant challenges, mostly arising from the substantial natural variations in crop row images as a result of weather conditions and the growth stages of crops. The processing of the detection algorithm also must be significantly low for real-time applications. In order to address the aforementioned requirements, we propose a crop row detection algorithm that has the following features: Firstly, a projective transformation is applied to transform the camera view and a color-based segmentation is employed to distinguish crop and weed from the background. Secondly, a clustering algorithm is used to differentiate between the crop and weed pixels. Lastly, a robust line-fitting approach is implemented to detect crop rows. The proposed algorithm is evaluated throughout a diverse range of scenarios, and its efficacy is assessed in comparison to four distinct existing solutions. The algorithm achieves an overall intersection over union (IOU) of 0.73 and exhibits robustness in challenging scenarios with high weed growth. The experiments conducted on real-time video featuring challenging scenarios show that our proposed algorithm exhibits a detection accuracy of over 90% and is a viable option for real-time implementation. With the high accuracy and low inference time, the proposed methodology offers a viable solution for autonomous navigation of agricultural robots in a crop field without damaging the crop and thus can serve as a foundation for future research.

Real-Time Navigation Roads: Lightweight and Efficient Convolutional Neural Network (LE-CNN) for Arabic Traffic Sign Recognition in Intelligent Transportation Systems (ITS)

Smart cities are now embracing the new frontier of urban living, with advanced technology being used to enhance the quality of life for residents. Many of these cities have developed transportation systems that improve efficiency and sustainability, as well as quality. Integrating cutting-edge transportation technology and data-driven solutions improves safety, reduces environmental impact, optimizes traffic flow during peak hours, and reduces congestion. Intelligent transportation systems consist of many systems, one of which is traffic sign detection. This type of system utilizes many advanced techniques and technologies, such as machine learning and computer vision techniques. A variety of traffic signs, such as yield signs, stop signs, speed limits, and pedestrian crossings, are among those that the traffic sign detection system is trained to recognize and interpret. Ensuring accurate and robust traffic sign recognition is paramount for the safe deployment of self-driving cars in diverse and challenging environments like the Arab world. However, existing methods often face many challenges, such as variability in the appearance of signs, real-time processing, occlusions that can block signs, low-quality images, and others. This paper introduces an advanced Lightweight and Efficient Convolutional Neural Network (LE-CNN) architecture specifically designed for accurate and real-time Arabic traffic sign classification. The proposed LE-CNN architecture leverages the efficacy of depth-wise separable convolutions and channel pruning to achieve significant performance improvements in both speed and accuracy compared to existing models. An extensive evaluation of the LE-CNN on the Arabic traffic sign dataset that was carried out demonstrates an impressive accuracy of 96.5% while maintaining superior performance with a remarkably low inference time of 1.65 s, crucial for real-time applications in self-driving cars. It achieves high accuracy with low false positive and false negative rates, demonstrating its potential for real-world applications like autonomous driving and advanced driver-assistance systems.

An Ensemble Approach for Robust Automated Crack Detection and Segmentation in Concrete Structures.

To prevent potential instability the early detection of cracks is imperative due to the prevalent use of concrete in critical infrastructure. Automated techniques leveraging artificial intelligence, machine learning, and deep learning as the traditional manual inspection methods are time-consuming. The existing automated concrete crack detection algorithms, despite recent advancements, face challenges in robustness, particularly in precise crack detection amidst complex backgrounds and visual distractions, while also maintaining low inference times. Therefore, this paper introduces a novel ensemble mechanism based on multiple quantized You Only Look Once version 8 (YOLOv8) models for the detection and segmentation of cracks in concrete structures. The proposed model is tested on different concrete crack datasets yielding enhanced segmentation results with at least 89.62% precision and intersection over a union score of 0.88. Moreover, the inference time per image is reduced to 27 milliseconds which is at least a 5% improvement over other models in the comparison. This is achieved by amalgamating the predictions of the trained models to calculate the final segmentation mask. The noteworthy contributions of this work encompass the creation of a model with low inference time, an ensemble mechanism for robust crack segmentation, and the enhancement of the learning capabilities of crack detection models. The fast inference time of the model renders it appropriate for real-time applications, effectively tackling challenges in infrastructure maintenance and safety.

A Strong and Simple Deep Learning Baseline for BCI Motor Imagery Decoding.

We propose EEG-SimpleConv, a straightforward 1D convolutional neural network for Motor Imagery decoding in BCI. Our main motivation is to propose a simple and performing baseline that achieves high classification accuracy, using only standard ingredients from the literature, to serve as a standard for comparison. The proposed architecture is composed of standard layers, including 1D convolutions, batch normalisations, ReLU activation functions and pooling functions. EEG-SimpleConv architecture is accompanied by a straightforward and tailored training routine, which is subjected to an extensive ablation study to quantify the influence of its components. We evaluate its performance on four EEG Motor Imagery datasets, including simulated online setups, and compare it to recent Deep Learning and Machine Learning approaches. EEG-SimpleConv is at least as good or far more efficient than other approaches, showing strong knowledge-transfer capabilities across subjects, at the cost of a low inference time. We believe that using standard components and ingredients can significantly help the adoption of Deep Learning approaches for BCI. We make the code of the models and the experiments accessible.

Robust spacecraft relative pose estimation via CNN-aided line segments detection in monocular images

Autonomous spacecraft relative navigation via monocular images became a hot topic in the past few years and, recently, received a further push thanks to the constantly growing field of artificial neural networks and the publication of several spaceborne image datasets. Despite the proliferation of spacecraft relative-state initialization algorithms developed, most architectures adopt computationally expensive solutions relying on convolutional neural networks (CNNs) that provide accurate output at the cost of a high computational burden that seems unfeasible for current spaceborne hardware. The paper addresses this issue by proposing a novel pose initialization algorithm based on lightweight CNNs. Inspired by previous state-of-the-art algorithms, the developed architecture leverages a fast and accurate target detection CNN followed by a line segment detection CNN capable of running with low inference time on mobile devices. The line segments and their junctions are grouped into complex geometrical groups, reducing the solution search space, and subsequently, they are adopted to extract the final pose estimate. As a main outcome, the analyses demonstrate that the lightweight architecture developed scores high accuracy in the pose estimation task, with a mean estimation error of less than 10 cm in translation and 2.5°in rotation. The baseline algorithm scores a mean SLAB error of 0.04552 with a standard deviation of 0.22972 in the test dataset. Detailed analyses demonstrate that the uncertainties on the overall pose score are driven mainly by errors in the relative attitude, which gives the highest contribution to the pose error metric adopted. The analyses on the error distributions point out that the uncertainties on the estimated relative position are higher in the camera boresight axis direction. Concerning the relative attitude, the algorithm proposed has higher uncertainties in estimating directions of the target x and y axes due to ambiguities related to the target geometry. Notably, the target detection CNN trained in this work outperforms the previous top scores in the benchmark dataset. The performances of the proposed algorithm have been investigated further by analyzing the effects on the accuracy due to the relative distance and the presence of background in the images. Lastly, the paper delves into the possibility of adopting a sub-portion of the 2D-to-3D match matrix made by the most complex perceptual groups identified that positively affects the overall run-time, pointing out the performances in terms of accuracy of the estimates and providing a comparison of both the baseline and the reduced match matrix versions against state-of-the-art algorithms concerning relative position and attitude errors and solution availability, highlighting the high accuracy and solution availability of the proposed architectures.

Smart Buildings: Water Leakage Detection Using TinyML.

The escalating global water usage and the increasing strain on major cities due to water shortages highlights the critical need for efficient water management practices. In water-stressed regions worldwide, significant water wastage is primarily attributed to leakages, inefficient use, and aging infrastructure. Undetected water leakages in buildings' pipelines contribute to the water waste problem. To address this issue, an effective water leak detection method is required. In this paper, we explore the application of edge computing in smart buildings to enhance water management. By integrating sensors and embedded Machine Learning models, known as TinyML, smart water management systems can collect real-time data, analyze it, and make accurate decisions for efficient water utilization. The transition to TinyML enables faster and more cost-effective local decision-making, reducing the dependence on centralized entities. In this work, we propose a solution that can be adapted for effective leakage detection in real-world scenarios with minimum human intervention using TinyML. We follow an approach that is similar to a typical machine learning lifecycle in production, spanning stages including data collection, training, hyperparameter tuning, offline evaluation and model optimization for on-device resource efficiency before deployment. In this work, we considered an existing water leakage acoustic dataset for polyvinyl chloride pipelines. To prepare the acoustic data for analysis, we performed preprocessing to transform it into scalograms. We devised a water leak detection method by applying transfer learning to five distinct Convolutional Neural Network (CNN) variants, which are namely EfficientNet, ResNet, AlexNet, MobileNet V1, and MobileNet V2. The CNN models were found to be able to detect leakages where a maximum testing accuracy, recall, precision, and F1 score of 97.45%, 98.57%, 96.70%, and 97.63%, respectively, were observed using the EfficientNet model. To enable seamless deployment on the Arduino Nano 33 BLE edge device, the EfficientNet model is compressed using quantization resulting in a low inference time of 1932 ms, a peak RAM usage of 255.3 kilobytes, and a flash usage requirement of merely 48.7 kilobytes.

An End-to-end Energy-efficient Approach for Intake Detection With Low Inference Time Using Wrist-worn Sensor.

Automated detection of intake gestures with wearable sensors has been a critical area of research for advancing our understanding and ability to intervene in people's eating behavior. Numerous algorithms have been developed and evaluated in terms of accuracy. However, ensuring the system is not only accurate in making predictions but also efficient in doing so is critical for real-world deployment. Despite the growing research on accurate detection of intake gestures using wearables, many of these algorithms are often energy inefficient, impeding on-device deployment for continuous and real-time monitoring of diet. This paper presents a template-based optimized multicenter classifier that enables accurate intake gesture detection while maintaining low-inference time and energy consumption using a wrist-worn accelerometer and gyroscope. We designed an Intake Gesture Counter smartphone application (CountING) and validated the practicality of our algorithm against seven state-of-the-art approaches on three public datasets (In-lab FIC, Clemson, and OREBA). Compared with other methods, we achieved optimal accuracy (81.60% F1 score) and very low inference time (15.97 msec per 2.20-sec data sample) on the Clemson dataset, and among the top performing algorithms, we achieve comparable accuracy (83.0% F1 score compared with 85.6% in the top performing algorithm) but superior inference time (13.8x faster, 33.14 msec per 2.20-sec data sample) on the In-lab FIC dataset and comparable accuracy (83.40% F1 score compared with 88.10% in the top-performing algorithm) but superior inference time (33.9x faster, 16.71 msec inference time per 2.20-sec data sample) on the OREBA dataset. On average, our approach achieved a 25-hour battery lifetime (44% to 52% improvement over state-of-the-art approaches) when tested on a commercial smartwatch for continuous real-time detection. Our approach demonstrates an effective and efficient method, enabling real-time intake gesture detection using wrist-worn devices in longitudinal studies.

LIDN: A novel light invariant image dehazing network

With the increasing demand for efficient image dehazing solutions in computer vision algorithms, particularly for autonomous systems, a research focus on low-inference time yet high-performance dehazing techniques has emerged. Existing dehazing methods predominantly rely on either daytime or nighttime haze models, limiting their effectiveness in handling haziness under varying lighting conditions. To address this limitation, this research paper introduces the Light Invariant Dehazing Network (LIDN), an end-to-end image dehazing network consisting of four sub-modules: Feature Extractor, Deep Global Atmospheric Light Estimator, Medium Transmission Extractor, and Encoder-Decoder. The proposed model, trained using Quadruplet loss, effectively reduces artifacts and produces sharper dehazed images. Extensive experiments conducted under diverse lighting conditions demonstrate the superior performance of the proposed LIDN model compared to state-of-the-art daytime and nighttime dehazing approaches. Remarkably, the proposed model achieves these exceptional results with a runtime of only 0.24s per image, making it highly efficient than existing dehazing algorithms.

MPF6D: masked pyramid fusion 6D pose estimation

Object pose estimation has multiple important applications, such as robotic grasping and augmented reality. We present a new method to estimate the 6D pose of objects that improves upon the accuracy of current proposals and can still be used in real-time. Our method uses RGB-D data as input to segment objects and estimate their pose. It uses a neural network with multiple heads to identify the objects in the scene, generate the appropriate masks and estimate the values of the translation vectors and the quaternion that represents the objects’ rotation. These heads leverage a pyramid architecture used during feature extraction and feature fusion. We conduct an empirical evaluation using the two most common datasets in the area, and compare against state-of-the-art approaches, illustrating the capabilities of MPF6D. Our method can be used in real-time with its low inference time and high accuracy.

AAEE‐Net: Attention‐guided aggregation and error‐aware enhancement network for accurate and efficient stereo matching

AbstractStereo matching is a fundamental and long‐standing task in computer vision. Although learning‐based stereo matching algorithms have made remarkable progress, two major challenges still persist. Firstly, existing cost aggregation methods that use stacked three‐dimensional convolutions are complex, leading to heavy computation and memory costs. Secondly these methods continue to struggle with establishing reliable matches in weakly matchable such as that edges and thin structures. To overcome these limitations, we propose an accurate and efficient network called Attention‐guided Aggregation and Error‐aware Enhancement Network (AAEE‐Net). Our approach involves designing an Attention‐guided Aggregation Mechanism (AAM) based on simple image features. This mechanism uses attention weights generated from image features to guide cost aggregation with a more efficient and effective strategy. Additionally, we propose an Error‐aware Enhancement Module (EEM) that refines the raw disparity by combining high‐frequency information from the original image and warp error between the left and right views. EEM enables the network to learn error correction capabilities that produce excellent subtle details and sharp edges. The experimental results on the SceneFlow and KITTI benchmark datasets demonstrate that AAEE‐Net achieves state‐of‐the‐art performance with low inference time. The qualitative results show that AAEE‐Net significantly improves predictions, especially for thin structures.

Extreme Gradient Boosting to Predict Atomic Layer Deposition for Platinum Nano-Film Coating.

Extreme gradient boosting (XGBoost) is an artificial intelligence algorithm capable of high accuracy and low inference time. The current study applies this XGBoost to the production of platinum nano-film coating through atomic layer deposition (ALD). In order to generate a database for model development, platinum is coated on α-Al2O3 using a rotary-type ALD equipment. The process is controlled by four parameters: process temperature, stop valve time, precursor pulse time, and reactant pulse time. A total of 625 samples according to different process conditions are obtained. The ALD coating index is used as the Al/Pt component ratio through ICP-AES analysis during postprocessing. The four process parameters serve as the input data and produces the Al/Pt component ratio as the output data. The postprocessed data set is randomly divided into 500 training samples and 125 test samples. XGBoost demonstrates 99.9% accuracy and a coefficient of determination of 0.99. The inference time is lower than that of random forest regression, in addition to a higher prediction safety than that of the light gradient boosting machine.

Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training.

Image-text retrieval is a central problem for understanding the semantic relationship between vision and language, and serves as the basis for various visual and language tasks. Most previous works either simply learn coarse-grained representations of the overall image and text, or elaborately establish the correspondence between image regions or pixels and text words. However, the close relations between coarse- and fine-grained representations for each modality are important for image-text retrieval but almost neglected. As a result, such previous works inevitably suffer from low retrieval accuracy or heavy computational cost. In this work, we address image-text retrieval from a novel perspective by combining coarse- and fine-grained representation learning into a unified framework. This framework is consistent with human cognition, as humans simultaneously pay attention to the entire sample and regional elements to understand the semantic content. To this end, a Token-Guided Dual Transformer (TGDT) architecture which consists of two homogeneous branches for image and text modalities, respectively, is proposed for image-text retrieval. The TGDT incorporates both coarse- and fine-grained retrievals into a unified framework and beneficially leverages the advantages of both retrieval approaches. A novel training objective called Consistent Multimodal Contrastive (CMC) loss is proposed accordingly to ensure the intra- and inter-modal semantic consistencies between images and texts in the common embedding space. Equipped with a two-stage inference method based on the mixed global and local cross-modal similarity, the proposed method achieves state-of-the-art retrieval performances with extremely low inference time when compared with representative recent approaches. Code is publicly available: github.com/LCFractal/TGDT.

Plant Root Phenotyping Using Deep Conditional GANs and Binary Semantic Segmentation

This paper develops an approach to perform binary semantic segmentation on Arabidopsis thaliana root images for plant root phenotyping using a conditional generative adversarial network (cGAN) to address pixel-wise class imbalance. Specifically, we use Pix2PixHD, an image-to-image translation cGAN, to generate realistic and high resolution images of plant roots and annotations similar to the original dataset. Furthermore, we use our trained cGAN to triple the size of our original root dataset to reduce pixel-wise class imbalance. We then feed both the original and generated datasets into SegNet to semantically segment the root pixels from the background. Furthermore, we postprocess our segmentation results to close small, apparent gaps along the main and lateral roots. Lastly, we present a comparison of our binary semantic segmentation approach with the state-of-the-art in root segmentation. Our efforts demonstrate that cGAN can produce realistic and high resolution root images, reduce pixel-wise class imbalance, and our segmentation model yields high testing accuracy (of over 99%), low cross entropy error (of less than 2%), high Dice Score (of near 0.80), and low inference time for near real-time processing.

Intelligent Video Ingestion for Real-time Traffic Monitoring

As an indispensable part of modern critical infrastructures, cameras deployed at strategic places and prime junctions in an intelligent transportation system can help operators in observing traffic flow, identifying any emergency situation, or making decisions regarding road congestion without arriving on the scene. However, these cameras are usually equipped with heterogeneous and turbulent networks, making the real-time smooth playback of traffic monitoring videos with high quality a grand challenge. In this article, we propose a lightweight Deep Reinforcement Learning-based approach, namely, sRC-C (smart bitRate Control with a Continuous action space) , to enhance the quality of real-time traffic monitoring by adjusting the video bitrate adaptively. Distinguished from the existing bitrate adjusting approaches, sRC-C can overcome the bias incurred by deterministic discretization of candidate bitrates by adjusting the video bitrate with more fine-grained control from a continuous action space, thus significantly improving the Quality-of-Service (QoS). With carefully designed state space and neural network model, sRC-C can be implemented on cameras with scarce resources to support real-time live video streaming with low inference time. Extensive experiments show that sRC-C can reduce the frame loss counts and hold time by 24% and 15.5%, respectively, even with comparable bandwidth utilization. Meanwhile, compared to the-state-of-art approaches, sRC-C can improve the QoS by 30.4%.

Prosody-TTS: An End-to-End Speech Synthesis System with Prosody Control

End-to-end text-to-speech synthesis systems achieved immense success in recent times, with improved naturalness and intelligibility. However, the end-to-end models, which primarily depend on the attention-based alignment, do not offer an explicit provision to modify/incorporate the desired prosody while synthesizing the speech. Moreover, the state-of-the-art end-to-end systems use autoregressive models for synthesis, making the prediction sequential. Hence, the inference time and the computational complexity are quite high. This paper proposes Prosody-TTS, a data-efficient end-to-end speech synthesis model that combines the advantages of statistical parametric models and end-to-end neural network models. It also has a provision to modify or incorporate the desired prosody at the finer level by controlling the fundamental frequency (\(f_0\)) and the phone duration. Generating speech utterances with appropriate prosody and rhythm helps in improving the naturalness of the synthesized speech. We explicitly model the duration of the phoneme and the \(f_0\) to have a finer level control over them during the synthesis. The model is trained in an end-to-end fashion to directly generate the speech waveform from the input text, which in turn depends on the auxiliary subtasks of predicting the phoneme duration, \(f_0\), and Mel spectrogram. Experiments on the Telugu language data of the IndicTTS database show that the proposed Prosody-TTS model achieves state-of-the-art performance with a mean opinion score of 4.08, with a very low inference time using just 4 hours of training data.

An anomalous sound detection methodology for predictive maintenance

In the last decade, Anomalous Sound Detection (ASD) is becoming an increasingly challenging task for a plethora of applications due to the widespread diffusion of Deep Neural Networks. Nevertheless, the arise of recent cyber–physical attacks (i.e. Triton or Stuxnet), that deceive monitoring platforms, pose novel and challenging issues. For this reason, advanced predictive maintenance techniques are starting to exploit sounds generated by particular industrial equipment, whose analysis can unveil symptom of possible failures. For this kind of context, it is very easy to collect data related to normal and abnormal behavior of a given machinery, thus several kinds of deep neural architectures can be effectively trained to predict eventual downtime situations. In this paper, we propose a novel deep learning-based methodology for anomalous sound detection task, having flexibility, modularity and efficiency characteristics. The proposed methodology analyzes audio clips based on the mel-spectogram and ID equipment information, while a one-hot encoding method extracts features that are, successively, used to train an ID Conditioned Network. In particular, the main novelty of the proposed methodology concerns the conditioning of an autoencoder by jointly analyzing the relationships between mel-spectogram and the related machine identifier through an encoder–decoder architecture for computing an anomaly score related to the input sequence. Several experiments have been made for investigating the efficiency and effectiveness of the proposed methodology on multiple instances of different industrial machines (pumps, valves, slide rails and fans), achieving low inference time and memory requirements w.r.t. the other approaches in the literature.

Automatic Face Mask Detection System in Public Transportation in Smart Cities Using IoT and Deep Learning

The World Health Organization (WHO) has stated that the spread of the coronavirus (COVID-19) is on a global scale and that wearing a face mask at work is the only effective way to avoid becoming infected with the virus. The pandemic made governments worldwide stay under lock-downs to prevent virus transmissions. Reports show that wearing face masks would reduce the risk of transmission. With the rise in population in cities, there is a greater need for efficient city management in today’s world for reducing the impact of COVID-19 disease. For smart cities to prosper, significant improvements to occur in public transportation, roads, businesses, houses, city streets, and other facets of city life will have to be developed. The current public bus transportation system, such as it is, should be expanded with artificial intelligence. The autonomous mask detection and alert system are needed to find whether the person is wearing a face mask or not. This article presents a novel IoT-based face mask detection system in public transportation, especially buses. This system would collect real-time data via facial recognition. The main objective of the paper is to detect the presence of face masks in real-time video stream by utilizing deep learning, machine learning, and image processing techniques. To achieve this objective, a hybrid deep and machine learning model was designed and implemented. The model was evaluated using a new dataset in addition to public datasets. The results showed that the transformation of Convolution Neural Network (CNN) classifier has better performance over the Deep Neural Network (DNN) classifier; it has almost complete face-identification capabilities with respect to people’s presence in the case where they are wearing masks, with an error rate of only 1.1%. Overall, compared with the standard models, AlexNet, Mobinet, and You Only Look Once (YOLO), the proposed model showed a better performance. Moreover, the experiments showed that the proposed model can detect faces and masks accurately with low inference time and memory, thus meeting the IoT limited resources.

A 2.5D semantic segmentation of the pancreas using attention guided dual context embedded U-Net

Automatic segmentation of the pancreas from medical images is important for clinical assessment of pancreas-related diseases. However, pancreatic segmentation based on computer tomography (CT) images is time-consuming and prone to errors because of the variances in shape and texture. Since various studies based on 2D/3D convolution neural networks (CNNs) have achieved encouraging performance for medical image segmentation, the 2D methods enjoy low inference time but suffer from a lack of 3D information. 3D methods are superior in performance for difficult targets requiring contextual information, but encounter the issue of high computational cost. Thus, we proposed a 2.5D segmentation method for pancreatic segmentation to balance utilizing contextual information and the high computational cost. This represents the 3D structural relationship among contiguous slices in a special representation. In the preprocessing stage, light-weight 3D voxels and the corresponding label mapping method were designed to explicitly express the differences in the target structure in contiguous slices. This would enable the network to learn spatial relationships directly. A 2D CNN embedded multi-attention mechanism and dual-context feature fusion method were designed to describe 3D information through 2D operations. In the post-processing stage, a fusion method was used to refine the segmentation results. The proposed method was evaluated on an abdominal contrast-enhanced CT dataset. Results showed Dice was 87.19%. Compared to the corresponding 2D and 3D methods, the proposed 2.5D method improved Dice by 1.14% and 2.80%, and was 60 times faster than the 3D method by using 0.1 times of the trainable parameters. Moreover, evaluations were performed on the NIH Pancreas-CT dataset, and the proposed 2.5D method achieved better segmentation performance than state-of-the-art methods.

Automated Detection of Subsurface Defects Using Active Thermography and Deep Learning Object Detectors

The need for improved quality control in industry makes object detection crucial. This work addresses the challenging problem of subsurface defect detections using a combination of active thermography and deep learning. The novel contribution of this work is to pose the problem as one of object detection rather than semantic segmentation or classification. The images used as input for the deep learning algorithms are three-channel color images obtained using principal component thermography (PCT). The use of these images improves the signal-to-noise ratio (SNR). A framework to label ground truths automatically is also created. The most widely used deep learning detector algorithms were evaluated, and the fifth version of you only look once (YOLO), i.e., YOLOv5, was selected because of its excellent average precision (AP) and its low inference time. The resulting combination of this algorithm and active thermography is effective and accurate in detecting subsurface defects.

Spatio-temporal deep learning model for distortion classification in laparoscopic video

Background: Laparoscopy is a surgery performed in the abdomen without making large incisions in the skin and with the aid of a video camera, resulting in laparoscopic videos. The laparoscopic video is prone to various distortions such as noise, smoke, uneven illumination, defocus blur, and motion blur. One of the main components in the feedback loop of video enhancement systems is distortion identification, which automatically classifies the distortions affecting the videos and selects the video enhancement algorithm accordingly. This paper aims to address the laparoscopic video distortion identification problem by developing fast and accurate multi-label distortion classification using a deep learning model. Current deep learning solutions based on convolutional neural networks (CNNs) can address laparoscopic video distortion classification, but they learn only spatial information. Methods: In this paper, utilization of both spatial and temporal features in a CNN-long short-term memory (CNN-LSTM) model is proposed as a novel solution to enhance the classification. First, pre-trained ResNet50 CNN was used to extract spatial features from each video frame by transferring representation from large-scale natural images to laparoscopic images. Next, LSTM was utilized to consider the temporal relation between the features extracted from the laparoscopic video frames to produce multi-label categories. A novel laparoscopic video dataset proposed in the ICIP2020 challenge was used for training and evaluation of the proposed method. Results: The experiments conducted show that the proposed CNN-LSTM outperforms the existing solutions in terms of accuracy (85%), and F1-score (94.2%). Additionally, the proposed distortion identification model is able to run in real-time with low inference time (0.15 sec). Conclusions: The proposed CNN-LSTM model is a feasible solution to be utilized in laparoscopic videos for distortion identification.