Deep Learning-based Solutions Research Articles

DeepMarkerNet: Leveraging supervision from the Duchenne Marker for spontaneous smile recognition

Distinguishing between spontaneous and posed smiles from videos poses a significant challenge in pattern classification literature. Researchers have developed feature-based and deep learning-based solutions for this problem. To this end, deep learning outperforms feature-based methods. However, certain aspects of feature-based methods could improve deep learning methods. For example, previous research has shown that Duchenne Marker (or D-Marker) features from the face play a vital role in spontaneous smiles, which can be useful to improve deep learning performances. In this study, we propose a deep learning solution that leverages D-Marker features to improve performance further. Our multi-task learning framework, named DeepMarkerNet, integrates a transformer network with the utilization of facial D-Markers for accurate smile classification. Unlike past methods, our approach simultaneously predicts the class of the smile and associated facial D-Markers using two different feed-forward neural networks, thus creating a symbiotic relationship that enriches the learning process. The novelty of our approach lies in incorporating supervisory signals from the pre-calculated D-Markers (instead of as input in previous works), harmonizing the loss functions through a weighted average. In this way, our training utilizes the benefits of D-Markers, but the inference does not require computing the D-Marker. We validate our model’s effectiveness on four well-known smile datasets: UvA-NEMO, BBC, MMI facial expression, and SPOS datasets, and achieve state-of-the-art results.

A survey for table recognition based on deep learning

Tables, as an important information interaction media containing much structured information, are popularly adopted in our life. There are a lot of table recognition methods for understanding and applying these information. Among them, table recognition based on deep learning is the mainstream. Therefore, we conducted this survey to summarize the latest research on table recognition based on deep learning in recent years. Table recognition can be divided into three sub-tasks, namely table detection, table structure recognition and table content recognition, so we firstly introduce the importance and significance of table recognition research and analyze the current challenges faced by each sub-task. Then, we list the existing public table recognition datasets and introduce commonly used evaluation metrics. Furthermore, we review deep learning-based solutions proposed in recent years for three different sub-tasks and reclassify them. Especially for table structure recognition, we reclassify them according to their annotation schemes. Finally, we report the performance of some excellent methods, such as CascadeTabNet, TSRFormer, SEM, etc, on public datasets, and describe the future research directions, including table recognition in natural scenes, question and answer about tables, visual information extraction, domain adaptation etc. In summary, this survey reviews the data, evaluations, methods, and development directions of table recognition. Table understanding and domain adaptation need as well to be focused on. The former can predict the relationships between two table elements, which helps us to understand the table. The latter can enhance the generalization ability of table recognition methods, thereby obtaining more robust recognition performance. Both are important development directions for table recognition in the future.

Pixel-wise segmentation of cells in digitized Pap smear images

A simple and cheap way to recognize cervical cancer is using light microscopic analysis of Pap smear images. Training artificial intelligence-based systems becomes possible in this domain, e.g., to follow the European recommendation to screen negative smears to reduce false negative cases. The first step for such a process is segmenting the cells. A large and manually segmented dataset is required for this task, which can be used to train deep learning-based solutions. We describe a corresponding dataset with accurate manual segmentations for the enclosed cells. Altogether, the APACS23 (Annotated PAp smear images for Cell Segmentation 2023) dataset contains about 37 000 manually segmented cells and is separated into dedicated training and test parts, which could be used for an official benchmark of scientific investigations or a grand challenge.

Adaptive bridge model for compressed domain point cloud classification

The recent adoption of deep learning-based models for the processing and coding of multimedia signals has brought noticeable gains in performance, which have established deep learning-based solutions as the uncontested state-of-the-art both for computer vision tasks, targeting machine consumption, as well as, more recently, coding applications, targeting human visualization. Traditionally, applications requiring both coding and computer vision processing require first decoding the bitstream and then applying the computer vision methods to the decompressed multimedia signals. However, the adoption of deep learning-based solutions enables the use of compressed domain computer vision processing, with gains in performance and computational complexity over the decompressed domain approach. For point clouds (PCs), these gains have been demonstrated in the single available compressed domain computer vision processing solution, named Compressed Domain PC Classifier, which processes JPEG Pleno PC coding (PCC) compressed streams using a PC classifier largely compatible with the state-of-the-art spatial domain PointGrid classifier. However, the available Compressed Domain PC Classifier presents strong limitations by imposing a single, specific input size which is associated to specific JPEG Pleno PCC configurations; this limits the compression performance as these configurations are not ideal for all PCs due to their different characteristics, notably density. To overcome these limitations, this paper proposes the first Adaptive Compressed Domain PC Classifier solution which includes a novel adaptive bridge model that allows to process the JPEG Pleno PCC encoded bit streams using different coding configurations, now maximizing the compression efficiency. Experimental results show that the novel Adaptive Compressed Domain PC Classifier allows JPEG PCC to achieve better compression performance by not imposing a single, specific coding configuration for all PCs, regardless of its different characteristics. Moreover, the added adaptability power can achieve slightly better PC classification performance than the previous Compressed Domain PC Classifier and largely better PC classification performance (and lower number of weights) than the PointGrid PC classifier working in the decompressed domain.

Attention-based Imputation of Missing Values in Electronic Health Records Tabular Data.

The imputation of missing values (IMV) in electronic health records tabular data is crucial to enable machine learning for patient-specific predictive modeling. While IMV methods are developed in biostatistics and recently in machine learning, deep learning-based solutions have shown limited success in learning tabular data. This paper proposes a novel attention-based missing value imputation framework that learns to reconstruct data with missing values leveraging between-feature (self-attention) or between-sample attentions. We adopt data manipulation methods used in contrastive learning to improve the generalization of the trained imputation model. The proposed self-attention imputation method outperforms state-of-the-art statistical and machine learning-based (decision-tree) imputation methods, reducing the normalized root mean squared error by 18.4% to 74.7% on five tabular data sets and 52.6% to 82.6% on two electronic health records data sets. The proposed attention-based missing value imputation method shows superior performance across a wide range of missingness (10% to 50%) when the values are missing completely at random.

Enhancing the Safety of Autonomous Vehicles in Adverse Weather by Deep Learning-Based Object Detection

Recognizing and categorizing items in weather-adverse environments poses significant challenges for autonomous vehicles. To improve the robustness of object-detection systems, this paper introduces an innovative approach for detecting objects at different levels by leveraging sensors and deep learning-based solutions within a traffic circle. The suggested approach improves the effectiveness of single-stage object detectors, aiming to advance the performance in perceiving autonomous racing environments and minimizing instances of false detection and low recognition rates. The improved framework is based on the one-stage object-detection model, incorporating multiple lightweight backbones. Additionally, attention mechanisms are integrated to refine the object-detection process further. Our proposed model demonstrates superior performance compared to the state-of-the-art method on the DAWN dataset, achieving a mean average precision (mAP) of 99.1%, surpassing the previous result of 84.7%.

How to detect occluded crosswalks in overview images? Comparing three methods in a heavily occluded area

Crosswalk presence data is crucial for pedestrian safety and urban planning. However, obtaining such data at a large scale is often challenging due to the high cost associated with traditional collection methods. While automated methods based on computer vision have been explored to detect crosswalks from aerial images, a major obstacle to their application is the handling of candidate crosswalks occluded by objects or shadows in the aerial imagery. To address this challenge, this study explores different deep learning-based solutions, including the aerial-view and street-view methods, which are commonly used, and a combination of the two − dual-perspective method. Deep learning models based on Convolutional Neural Networks with VGG16 architecture were trained using 16,815 images to automatically detect crosswalks from both aerial and street view images. To compare the performance of these methods in handling occlusions, 1,378 images from a heavily occluded area were processed separately by the three methods. The results showed that the aerial-view method suffered the most when dealing with images from a heavily occluded area, resulting in the lowest accuracy, precision, recall, and F1 score among the three methods. On the other hand, the street-view method outperformed the aerial-view method significantly. The dual-perspective method demonstrated the highest accuracy and precision values, indicating its superiority in accurately predicting the location of a crosswalk. However, the street-view method exhibited the highest recall value, highlighting its superior ability to recover an occluded crosswalk among all methods.

Structure-aware deep model for MHC-II peptide binding affinity prediction

The prediction of major histocompatibility complex (MHC)-peptide binding affinity is an important branch in immune bioinformatics, especially helpful in accelerating the design of disease vaccines and immunity therapy. Although deep learning-based solutions have yielded promising results on MHC-II molecules in recent years, these methods ignored structure knowledge from each peptide when employing the deep neural network models. Each peptide sequence has its specific combination order, so it is worth considering adding the structural information of the peptide sequence to the deep model training. In this work, we use positional encoding to represent the structural information of peptide sequences and validly combine the positional encoding with existing models by different strategies. Experiments on three datasets show that the introduction of position-coding information can further improve the performance built upon the existing model. The idea of introducing positional encoding to this field can provide important reference significance for the optimization of the deep network structure in the future.

CC4S: Encouraging Certainty and Consistency in Scribble-Supervised Semantic Segmentation.

Deep learning-based solutions have achieved impressive performance in semantic segmentation but often require large amounts of training data with fine-grained annotations. To alleviate such requisition, a variety of weakly supervised annotation strategies have been proposed, among which scribble supervision is emerging as a popular one due to its user-friendly annotation way. However, the sparsity and diversity of scribble annotations make it nontrivial to train a network to produce deterministic and consistent predictions directly. To address these issues, in this paper we propose holistic solutions involving the design of network structure, loss and training procedure, named CC4S to improve Certainty and Consistency for Scribble-Supervised Semantic Segmentation. Specifically, to reduce uncertainty, CC4S embeds a random walk module into the network structure to make neural representations uniformly distributed within similar semantic regions, which works together with a soft entropy loss function to force the network to produce deterministic predictions. To encourage consistency, CC4S adopts self-supervision training and imposes the consistency loss on the eigenspace of the probability transition matrix in the random walk module (we named neural eigenspace). Such self-supervision inherits the category-level discriminability from the neural eigenspace and meanwhile helps the network focus on producing consistent predictions for the salient parts and neglect semantically heterogeneous backgrounds. Finally, to further improve the performance, CC4S uses the network predictions as pseudo-labels and retrains the network with an extra color constraint regularizer on pseudo-labels to boost semantic consistency in color space. Rich experiments demonstrate the excellent performance of CC4S. In particular, under scribble supervision, CC4S achieves comparable performance to those from fully supervised methods. Comprehensive ablation experiments verify the effectiveness of the design choices in CC4S and its robustness under extreme supervision cases, i.e., when scribbles are shrunk proportionally or dropped randomly. The code for this work has been open-sourced at https://github.com/panzhiyi/CC4S.

Improving the Accuracy of Facial Micro-Expression Recognition: Spatio-Temporal Deep Learning with Enhanced Data Augmentation and Class Balancing

Aim/Purpose: This study presents a novel deep learning-based framework designed to enhance spontaneous micro-expression recognition by effectively increasing the amount and variety of data and balancing the class distribution to improve recognition accuracy. Background: Micro-expression recognition using deep learning requires large amounts of data. Micro-expression datasets are relatively small, and their class distribution is not balanced. Methodology: This study developed a framework using a deep learning-based model to recognize spontaneous micro-expressions on a person’s face. The framework also includes several technical stages, including image and data preprocessing. In data preprocessing, data augmentation is carried out to increase the amount and variety of data and class balancing to balance the distribution of sample classes in the dataset. Contribution: This study’s essential contribution lies in enhancing the accuracy of micro-expression recognition and overcoming the limited amount of data and imbalanced class distribution that typically leads to overfitting. Findings: The results indicate that the proposed framework, with its data preprocessing stages and deep learning model, significantly increases the accuracy of micro-expression recognition by overcoming dataset limitations and producing a balanced class distribution. This leads to improved micro-expression recognition accuracy using deep learning techniques. Recommendations for Practitioners: Practitioners can utilize the model produced by the proposed framework, which was developed to recognize spontaneous micro-expressions on a person’s face, by implementing it as an emotional analysis application based on facial micro-expressions. Recommendation for Researchers: Researchers involved in the development of a spontaneous micro-expression recognition framework for analyzing hidden emotions from a person’s face are playing an essential role in advancing this field and continue to search for more innovative deep learning-based solutions that continue to explore techniques to increase the amount and variety of data and find solutions to balancing the number of sample classes in various micro-expression datasets. They can further improvise to develop deep learning model architectures that are more suitable and relevant according to the needs of recognition tasks and the various characteristics of different datasets. Impact on Society: The proposed framework could significantly impact society by providing a reliable model for recognizing spontaneous micro-expressions in real-world applications, ranging from security systems and criminal investigations to healthcare and emotional analysis. Future Research: Developing a spontaneous micro-expression recognition framework based on spatial and temporal flow requires the learning model to classify optimal features. Our future work will focus more on exploring micro-expression features by developing various alternative learning models and increasing the weights of spatial and temporal features.

Biotic Stress Management in Soil-Less Agriculture Systems: A Deep Learning Approach for Identification of Leaf Miner Pest Infestation

Leaf miner pests pose a serious threat to the productivity, profitability, and sustainability of soil-less tomato cultivation systems. Early and accurate identification of leaf miner infestation is crucial for timely pest control measures. This study presents an efficient approach using attention-based convolutional neural networks for timely identification of this pest infestation. The proposed approach uses both spatial and channel attention modules to enhance the feature extraction capability of the convolutional neural network. The custom model developed was trained using an image dataset collected from tomatoes grown in a hydroponic environment. The different hyper parameters were tuned to get the optimal model performance. The experimental results show that the proposed attention-based CNN model achieved an overall accuracy of 97.87%, 97.10% precision, 98.53% recall, and 97.81% F1-score. Additionally, the model performance was compared with other pre-trained models viz., AlexNet, VGG16, and VGG19, and was found to outperform these state-of-the-art CNN models due to its improved feature extraction capability. The efficiency of the model underlines its potential to be deployed as part of automated pest monitoring systems in hydroponic environments. This work contributes to the development of computer vision and deep learning-based solutions for precision agriculture applications.

A Smart Contract Vulnerability Detection Method Based on Multimodal Feature Fusion and Deep Learning

With the proliferation of blockchain technology in decentralized applications like decentralized finance and supply chain and identity management, smart contracts operating on a blockchain frequently encounter security issues such as reentrancy vulnerabilities, timestamp dependency vulnerabilities, tx.origin vulnerabilities, and integer overflow vulnerabilities. These security concerns pose a significant risk of causing substantial losses to user accounts. Consequently, the detection of vulnerabilities in smart contracts has become a prominent area of research. Existing research exhibits limitations, including low detection accuracy in traditional smart contract vulnerability detection approaches and the tendency of deep learning-based solutions to focus on a single type of vulnerability. To address these constraints, this paper introduces a smart contract vulnerability detection method founded on multimodal feature fusion. This method adopts a multimodal perspective to extract three modal features from the lifecycle of smart contracts, leveraging both static and dynamic features comprehensively. Through deep learning models like Graph Convolutional Networks (GCNs) and bidirectional Long Short-Term Memory networks (bi-LSTMs), effective detection of vulnerabilities in smart contracts is achieved. Experimental results demonstrate that the proposed method attains detection accuracies of 85.73% for reentrancy vulnerabilities, 85.41% for timestamp dependency vulnerabilities, 83.58% for tx.origin vulnerabilities, and 90.96% for integer Overflow vulnerabilities. Furthermore, ablation experiments confirm the efficacy of the newly introduced modal features, highlighting the significance of fusing dynamic and static features in enhancing detection accuracy.

A systematic review of image-level camouflaged object detection with deep learning

Camouflaged object detection (COD) aims to search and identify disguised objects that are hidden in their surrounding environment, thereby deceiving the human visual system. As an interesting and challenging task, COD has received increasing attention from the community in the past few years, especially for image-level camouflaged object segmentation task. So far, some advanced image-level COD models have been proposed, mainly dominated by deep learning-based solutions. To have an in-depth understanding of existing image-level COD methods in the deep learning era, in this paper, we give a comprehensive review on model structure and paradigm classification, public benchmark datasets, evaluation metrics, model performance benchmark, and potential future development directions. Specifically, we first review 96 existing deep COD algorithms. Subsequently, we summarize and analyze the existing five widely used COD datasets and evaluation metrics. Furthermore, we benchmark a set of representative models and provide a detailed analysis of the comparison results from both quantitative and qualitative perspectives. Moreover, we further discuss the challenges of COD and the corresponding solutions. Finally, based on the understanding of this field, future development trends and potential research directions are prospected. In conclusion, the purpose of this paper is to provide researchers with a review of the latest COD methods, increase their understanding of COD research, and gain some enlightenment.

An improved multi-scale gradient generative adversarial network for enhancing classification of colorectal cancer histological images.

Deep learning-based solutions for histological image classification have gained attention in recent years due to their potential for objective evaluation of histological images. However, these methods often require a large number of expert annotations, which are both time-consuming and labor-intensive to obtain. Several scholars have proposed generative models to augment labeled data, but these often result in label uncertainty due to incomplete learning of the data distribution. To alleviate these issues, a method called InceptionV3-SMSG-GAN has been proposed to enhance classification performance by generating high-quality images. Specifically, images synthesized by Multi-Scale Gradients Generative Adversarial Network (MSG-GAN) are selectively added to the training set through a selection mechanism utilizing a trained model to choose generated images with higher class probabilities. The selection mechanism filters the synthetic images that contain ambiguous category information, thus alleviating label uncertainty. Experimental results show that compared with the baseline method which uses InceptionV3, the proposed method can significantly improve the performance of pathological image classification from 86.87% to 89.54% for overall accuracy. Additionally, the quality of generated images is evaluated quantitatively using various commonly used evaluation metrics. The proposed InceptionV3-SMSG-GAN method exhibited good classification ability, where histological image could be divided into nine categories. Future work could focus on further refining the image generation and selection processes to optimize classification performance.

Deep Learning for Logo Detection: A Survey

Logo detection has gradually become a research hotspot in the field of computer vision and multimedia for its various applications, such as social media monitoring, intelligent transportation, and video advertising recommendation. Recent advances in this area are dominated by deep learning-based solutions, where many datasets, learning strategies, network architectures, and loss functions have been employed. This article reviews the advance in applying deep learning techniques to logo detection. First, we discuss a comprehensive account of public datasets designed to facilitate performance evaluation of logo detection algorithms, which tend to be more diverse, more challenging, and more reflective of real life. Next, we perform an in-depth analysis of the existing logo detection strategies and their strengths and weaknesses of each learning strategy. Subsequently, we summarize the applications of logo detection in various fields, from intelligent transportation and brand monitoring to copyright and trademark compliance. Finally, we analyze the potential challenges and present the future directions for the development of logo detection. This study aims better to inform readers about the current state of logo detection and encourage more researchers to get involved in logo detection.

A Primer on Seq2Seq Models for Generative Chatbots

The recent spread of Deep Learning-based solutions for Artificial Intelligence and the development of Large Language Models has pushed forwards significantly the Natural Language Processing area. The approach has quickly evolved in the last ten years, deeply affecting NLP, from low-level text pre-processing tasks –such as tokenisation or POS tagging– to high-level, complex NLP applications like machine translation and chatbots. This article examines recent trends in the development of open-domain data-driven generative chatbots, focusing on the Seq2Seq architectures. Such architectures are compatible with multiple learning approaches, ranging from supervised to reinforcement and, in the last years, allowed to realise very engaging open-domain chatbots. Not only do these architectures allow to directly output the next turn in a conversation but, to some extent, they also allow to control the style or content of the response. To offer a complete view on the subject, we examine possible architecture implementations as well as training and evaluation approaches. Additionally, we provide information about the openly available corpora to train and evaluate such models and about the current and past chatbot competitions. Finally, we present some insights on possible future directions, given the current research status.

A review of the machine learning datasets in mammography, their adherence to the FAIR principles and the outlook for the future

The increasing rates of breast cancer, particularly in emerging economies, have led to interest in scalable deep learning-based solutions that improve the accuracy and cost-effectiveness of mammographic screening. However, such tools require large volumes of high-quality training data, which can be challenging to obtain. This paper combines the experience of an AI startup with an analysis of the FAIR principles of the eight available datasets. It demonstrates that the datasets vary considerably, particularly in their interoperability, as each dataset is skewed towards a particular clinical use-case. Additionally, the mix of digital captures and scanned film compounds the problem of variability, along with differences in licensing terms, ease of access, labelling reliability, and file formats. Improving interoperability through adherence to standards such as the BIRADS criteria for labelling and annotation, and a consistent file format, could markedly improve access and use of larger amounts of standardized data. This, in turn, could be increased further by GAN-based synthetic data generation, paving the way towards better health outcomes for breast cancer.

Deep Learning-based Human Pose Estimation: A Survey

Human pose estimation aims to locate the human body parts and build human body representation (e.g., body skeleton) from input data such as images and videos. It has drawn increasing attention during the past decade and has been utilized in a wide range of applications including human-computer interaction, motion analysis, augmented reality, and virtual reality. Although the recently developed deep learning-based solutions have achieved high performance in human pose estimation, there still remain challenges due to insufficient training data, depth ambiguities, and occlusion. The goal of this survey article is to provide a comprehensive review of recent deep learning-based solutions for both 2D and 3D pose estimation via a systematic analysis and comparison of these solutions based on their input data and inference procedures. More than 260 research papers since 2014 are covered in this survey. Furthermore, 2D and 3D human pose estimation datasets and evaluation metrics are included. Quantitative performance comparisons of the reviewed methods on popular datasets are summarized and discussed. Finally, the challenges involved, applications, and future research directions are concluded. A regularly updated project page is provided: https://github.com/zczcwh/DL-HPE .

Deep learning-based scene understanding for autonomous robots: a survey

Autonomous robots are a hot research subject within the fields of science and technology, which has a big impact on social-economic development. The ability of the autonomous robot to perceive and understand its working environment is the basis for solving more complicated issues. In recent years, an increasing number of artificial intelligence-based methods have been proposed in the field of scene understanding for autonomous robots, and deep learning is one of the current key areas in this field. Outstanding gains have been attained in the field of scene understanding for autonomous robots based on deep learning. Thus, this paper presents a review of recent research on the deep learning-based scene understanding for autonomous robots. This survey provides a detailed overview of the evolution of robotic scene understanding and summarizes the applications of deep learning methods in scene understanding for autonomous robots. In addition, the key issues in autonomous robot scene understanding are analyzed, such as pose estimation, saliency prediction, semantic segmentation, and object detection. Then, some representative deep learning-based solutions for these issues are summarized. Finally, future challenges in the field of the scene understanding for autonomous robots are discussed.

Simplified review on cyber security threats detection in IoT environment using deep learning approach

Wearable technology, sensor networks, and home utilities are just a few of the businesses where the Internet of Things (IoT) is spreading quickly. With the development of the IoT, billions of gadgets are now connected to the internet and exchanging data. The proliferation of IoT devices has increased the number of IoT-based cyberattacks. In 2016 a massive denial of service (DDOS) cyber-attack was lunched utilizing infected internet of things devices a major website including Netflix and CNN was shutdown. Therefore, new ways for recognizing threats posed by hacked IoT nodes must be developed to overcome this concern. In that same context, ML and DL approaches are the best appropriate investigative control solution against IoT device-based intrusions. The point of the study is to offer a complete grasp of the IoT system-relevant technologies, standards, architecture, and the increasing dangers from corrupted IoT gadgets and an introduction to intrusion detection systems. Additionally, this research focuses on deep learning-based solutions for identifying IoT devices susceptible to cyber-attacks. The detection rate provided by deep learning algorithms shows promising results which reached 99% detection accuracy in some cases.