Computer Vision Research Research Articles

Explainable Deep Learning approach for Shoulder Abnormality Detection in X-Rays Dataset

Computer vision researchers and decision-makers have struggled to understand how deep neural networks (DNNs) accomplish image classification tasks and interpret their results. Due to a lack of understanding of their internal functioning, these models are commonly referred to as "black boxes." As a part of the development process, the DNNs can be easily explained. In this research work, we introduce an explainable technique for shoulder abnormality detection. The motivation behind this study is to enhance patients' and medical professionals' trust in DNNs technology. DNNs are implemented frequently in the medical domain. The suggested abnormality detector, which is based on IGrad-CAM++, is capable of detecting Shoulder X-rays abnormality. Grad-CAM is a common approach that combines the activation maps received from the model to create such a visualization. The average gradient-based terms used in this technique, on the other hand, understate the contribution of the model's identified representations to its predictions. In order to address this issue, we offer a technique that uses Grad-CAM++ to compute the route integral of the gradient-based terms. By assessing different techniques, it is discovered that the recommended procedure can perform very effectively and efficiently in X-ray images provides more visual explanation than existing techniques.

Real-Time Face Recognition Civil Servant Presence System Using DNN Algorithm

Facial recognition has become a growing topic among Computer Vision researchers because it can solve real-life problems, including during the COVID-19 pandemic. The pandemic is why the Indonesian government has imposed social restrictions and physical contact in public places. Before the pandemic, most touch-based attendance systems used fingerprints or Radio Frequency Identification (RFID) cards. The solution proposed in this study is to identify real-time facial recognition of the Civil Service presence system using a Deep Neural Network. The goal is to minimize physical contact. The research stages include data collection, augmentation and preprocessing, CNN modeling and training, model evaluation, converting to OpenCV DNN, implementation of transfer learning, and identification of test data. This research contributes to testing variations in distance and position so it can recognize a person's face even when wearing a mask and glasses. This DNN model produces a validation accuracy value of 99.48% and a validation loss of 0.0273 with a data training process of 10 times. Tests for variations in distance, position, use of masks, and glasses on MTCNN detection provide an average accuracy for each trial of 100%, 96%, and 100%, respectively. Therefore, the average accuracy of the Haar Cascades detection test is 100%, 85%, and 100%.

A Deep Learning Framework for Accurate Vehicle Yaw Angle Estimation from a Monocular Camera Based on Part Arrangement.

An accurate object pose is essential to assess its state and predict its movements. In recent years, scholars have often predicted object poses by matching an image with a virtual 3D model or by regressing the six-degree-of-freedom pose of the target directly from the pixel data via deep learning methods. However, these approaches may ignore a fact that was proposed in the early days of computer vision research, i.e., that object parts are strongly represented in the object pose. In this study, we propose a novel and lightweight deep learning framework, YAEN (yaw angle estimation network), for accurate object yaw angle prediction from a monocular camera based on the arrangement of parts. YAEN uses an encoding–decoding structure for vehicle yaw angle prediction. The vehicle part arrangement information is extracted by the part-encoding network, and the yaw angle is extracted from vehicle part arrangement information by the yaw angle decoding network. Because vehicle part information is refined by the encoder, the decoding network structure is lightweight; the YAEN model has low hardware requirements and can reach a detection speed of on a graphics cards. To improve the performance of our model, we used asymmetric convolution and SSE (sum of squared errors) loss functions of adding the sign. To verify the effectiveness of this model, we constructed an accurate yaw angle dataset under real-world conditions with two vehicles equipped with high-precision positioning devices. Experimental results prove that our method can achieve satisfactory prediction performance in scenarios in which vehicles do not obscure each other, with an average prediction error of less than 3.1° and an accuracy of 96.45% for prediction errors of less than 10° in real driving scenarios.

Boosting Archimedes optimization algorithm using trigonometric operators based on feature selection for facial analysis.

Due to technical advancements and the proliferation of mobile applications, facial analysis (FA) of humans has recently become an important area for computer vision research. FA investigates a variety of difficulties, including gender recognition, facial expression recognition, age and race recognition, with the goal of automatically comprehending social interactions. Due to the dimensional challenge posed by pre-trained CNN networks, the scientific community has developed numerous techniques inspired by biology, swarm intelligence theory, physics, and mathematical rules. This article presents a gender recognition system based on scAOA, that is a modified version of the Archimedes optimization algorithm (AOA). The latest variant (scAOA) enhances the exploitation stage by using trigonometric operators inspired by the sine cosine algorithm (SCA) in order to prevent local optima and to accelerate the convergence. The main purpose of this paper is to apply scAOA to select the relevant deep features provided by two pretrained models of CNN (AlexNet & ResNet) to recognize the gender of a human person categorized into two classes (men and women). Two datasets are used to evaluate the proposed approach (scAOA): the Brazilian FEI dataset and the Georgia Tech Face dataset (GT). In terms of accuracy, Fscore and statistical test, the comparison analysis demonstrates that scAOA outperforms other modern and competitive optimizers such as AOA, SCA, Ant lion optimizer (ALO), Salp swarm algorithm (SSA), Grey wolf optimizer (GWO), Simple genetic algorithm (SGA), Grasshopper optimization algorithm (GOA) and Particle swarm optimizer (PSO).

The Core of Smart Cities: Knowledge Representation and Descriptive Framework Construction in Knowledge-Based Visual Question Answering

Visual question answering (VQA), which is an important presentation form of AI-complete task and visual Turing tests, coupled with its potential application value, attracted widespread attention from both researchers in computer vision and natural language processing. However, there are no relevant research regarding the expression and participation methods of knowledge in VQA. Considering the importance of knowledge for answering questions correctly, this paper analyzes and researches the stratification, expression and participation process of knowledge in VQA and proposes a knowledge description framework (KDF) to guide the research of knowledge-based VQA (Kb-VQA). The KDF consists of a basic theory, implementation methods and specific applications. This paper focuses on describing mathematical models at basic theoretical levels, as well as the knowledge hierarchy theories and key implementation behaviors established on this basis. In our experiment, using the statistics of VQA’s accuracy in the relevant literature, we propose a good corroboration of the research results from knowledge stratification, participation methods and expression forms in this paper.

Automated detection of pain levels using deep feature extraction from shutter blinds-based dynamic-sized horizontal patches with facial images

Pain intensity classification using facial images is a challenging problem in computer vision research. This work proposed a patch and transfer learning-based model to classify various pain intensities using facial images. The input facial images were segmented into dynamic-sized horizontal patches or “shutter blinds”. A lightweight deep network DarkNet19 pre-trained on ImageNet1K was used to generate deep features from the shutter blinds and the undivided resized segmented input facial image. The most discriminative features were selected from these deep features using iterative neighborhood component analysis, which were then fed to a standard shallow fine k-nearest neighbor classifier for classification using tenfold cross-validation. The proposed shutter blinds-based model was trained and tested on datasets derived from two public databases—University of Northern British Columbia-McMaster Shoulder Pain Expression Archive Database and Denver Intensity of Spontaneous Facial Action Database—which both comprised four pain intensity classes that had been labeled by human experts using validated facial action coding system methodology. Our shutter blinds-based classification model attained more than 95% overall accuracy rates on both datasets. The excellent performance suggests that the automated pain intensity classification model can be deployed to assist doctors in the non-verbal detection of pain using facial images in various situations (e.g., non-communicative patients or during surgery). This system can facilitate timely detection and management of pain.

Online Multiplayer Tracking by Extracting Temporal Contexts with Transformer

Sports competition is one of the most popular programs for many audiences. Tracking the players in sports game videos from broadcasts is a nontrivial challenge for computer vision researchers. In sports videos, the direction of an athlete’s movement changes quickly and unpredictably. Mutual occlusion between athletes is also more frequent in team competitions. However, the rich temporal contexts among the adjacent frames have been excluded from consideration. To address this dilemma, we propose an online transformer-based learnable framework in an end-to-end fashion. We use a transformer architecture to extract the temporal contexts between the successive frames and add them to the network training, which is robust to occlusion and complex direction changes in multiplayer tracking. We demonstrate the effectiveness of our method on three sports video datasets by comparing them with recently advanced multiplayer trackers.

Attention Guided Feature Encoding for Scene Text Recognition.

The real-life scene images exhibit a range of variations in text appearances, including complex shapes, variations in sizes, and fancy font properties. Consequently, text recognition from scene images remains a challenging problem in computer vision research. We present a scene text recognition methodology by designing a novel feature-enhanced convolutional recurrent neural network architecture. Our work addresses scene text recognition as well as sequence-to-sequence modeling, where a novel deep encoder–decoder network is proposed. The encoder in the proposed network is designed around a hierarchy of convolutional blocks enabled with spatial attention blocks, followed by bidirectional long short-term memory layers. In contrast to existing methods for scene text recognition, which incorporate temporal attention on the decoder side of the entire architecture, our convolutional architecture incorporates novel spatial attention design to guide feature extraction onto textual details in scene text images. The experiments and analysis demonstrate that our approach learns robust text-specific feature sequences for input images, as the convolution architecture designed for feature extraction is tuned to capture a broader spatial text context. With extensive experiments on ICDAR2013, ICDAR2015, IIIT5K and SVT datasets, the paper demonstrates an improvement over many important state-of-the-art methods.

Classification of Brain Tumor from Magnetic Resonance Imaging Using Vision Transformers Ensembling.

The automated classification of brain tumors plays an important role in supporting radiologists in decision making. Recently, vision transformer (ViT)-based deep neural network architectures have gained attention in the computer vision research domain owing to the tremendous success of transformer models in natural language processing. Hence, in this study, the ability of an ensemble of standard ViT models for the diagnosis of brain tumors from T1-weighted (T1w) magnetic resonance imaging (MRI) is investigated. Pretrained and finetuned ViT models (B/16, B/32, L/16, and L/32) on ImageNet were adopted for the classification task. A brain tumor dataset from figshare, consisting of 3064 T1w contrast-enhanced (CE) MRI slices with meningiomas, gliomas, and pituitary tumors, was used for the cross-validation and testing of the ensemble ViT model’s ability to perform a three-class classification task. The best individual model was L/32, with an overall test accuracy of 98.2% at 384 × 384 resolution. The ensemble of all four ViT models demonstrated an overall testing accuracy of 98.7% at the same resolution, outperforming individual model’s ability at both resolutions and their ensembling at 224 × 224 resolution. In conclusion, an ensemble of ViT models could be deployed for the computer-aided diagnosis of brain tumors based on T1w CE MRI, leading to radiologist relief.

Detection and Classification of Human-Carrying Baggage Using DenseNet-161 and Fit One Cycle

In recent decades, the crime rate has significantly increased. As a result, the automatic video monitoring system has become increasingly important for researchers in computer vision. A person’s baggage classification is essential in knowing who has abandoned baggage. This paper proposes a model for classifying humans carrying baggage. Two approaches are used for comparison using a deep learning technique. The first approach is based on categorizing human-containing image regions as either with or without baggage. The second approach classifies human-containing image regions based on the human position direction attribute. The proposed model is based on the pretrained DenseNet-161 architecture. It uses a "fit-one-cycle policy" strategy to reduce the training time and achieve better accuracy. The Fastai framework is used for implementation due to its super computational ability, simple workflow, and unique data cleansing functionalities. Our proposed model was experimentally validated, and the results show that the process is sufficiently precise, faster, and outperforms the existing methods. We achieved an accuracy of between 96% and 98.75% for the binary classification and 96.67% and 98.33% for the multi-class classification. For multi-class classification, the datasets, such as PETA, INRIA, ILIDS, and MSMT17, are re-annotated with one’s direction information about one’s stance to test the suggested approach’s efficacy.

Complementary characteristics fusion network for weakly supervised salient object detection

Salient object detection (SOD) is a challenging and fundamental research in computer vision and image processing. Since the cost of pixel-level annotations is high, scribble annotations are usually used as weak supervisions. However, scribble annotations are too sparse and always located inside the objects with lacking annotations close to the semantic boundaries, which can't make confident predictions. To alleviate these issues, we propose a novel and effective scribble-based weakly supervised approach for SOD, named complementary characteristics fusion network (CCFNet). To be more specific, we design an edge fusion module (EFM) by taking account of local and high-level semantic information to equip our model, which would be beneficial to enhance the power of aggregating edge information. Then to achieve the complementary role of different features, a series of feature correlation modules (FCMs) are employed to strengthen the localization information and details learning. This is based on low-level, high-level global and edge information, which will complement each other to obtain relatively complete salient regions. Alternatively, to encourage the network to learn structural information and further improve the results of saliency maps in foreground and background, we propose a self-supervised salient detection (SSD) loss. Extensive experiments using five benchmark datasets demonstrate that our proposed approach performs favorably against the state-of-the-art weakly supervised algorithms, and even surpasses the performance of those fully supervised.

Dual Expression Fusion: A Universal Microexpression Recognition Framework

Microexpression recognition from short video sequences is a challenging problem in computer vision and multimedia research. This article proposes a dual expression fusion (DEF) microexpression recognition framework, which performs better on more general videos. This microexpression recognition framework uses deep learning models to extract the facial features from a single frame while directly predicting the action units (AUs) states of that frame. Then, a long short-term memory network (LSTM) predicts the microexpression category on the sequence features. To perform better across different datasets, DEF uses a capsule network to learn more subtle structural information from faces. Since the effective facial features are extracted by a deep convolutional network and the capsule network, the framework performs well through feature fusion. It has relatively low requirements for the content of video data. Based on the proposed framework, we won first place among 12 qualified teams in the ICIP2020 microexpression Recognition Challenge with an average F1 score of 0.6297.

Action recognition based on RGB and skeleton data sets: A survey

Action recognition is a major branch of computer vision research. As a widely used technology, action recognition has been applied to human–computer interaction, intelligent pension, and intelligent transportation system. Because of the explosive growth of action recognition related methods, the performance of action recognition on many difficult data sets has improved significantly. In terms of the different data sets used for action recognition, action recognition can mainly be divided into RGB-based action recognition method and skeleton-based action recognition method. The former method can take advantage of the prior knowledge of image recognition. However, it has high requirements for computing power and storage ability, and it is difficult to avoid the influence of irrelevant background and illumination. In contrast, the latter method’s calculation amount and required storage space are reduced significantly. However, it lacks context information that is useful for action recognition. This review provides a comprehensive description of these two methods, covering the milestone algorithms, the state-of-the-art algorithms, the commonly used data sets, evaluation metrics, challenges, and promising future directions. So far as we know, this work is the first survey covering traditional methods of action recognition, RGB-based end-to-end action recognition method, pose estimation, and skeleton-based action recognition in one review. This survey aims to help scholars who study action recognition technology to systematically learn action recognition technology, select data sets, understand current challenges, and choose promising future research directions.

Monocular Visual Traffic Surveillance: A Review

To facilitate the monitoring and management of modern transportation systems, monocular visual traffic surveillance systems have been widely adopted for speed measurement, accident detection, and accident prediction. Thanks to the recent innovations in computer vision and deep learning research, the performance of visual traffic surveillance systems has been significantly improved. However, despite this success, there is a lack of survey papers that systematically review these new methods. Therefore, we conduct a systematic review of relevant studies to fill this gap and provide guidance to future studies. This paper is structured along the visual information processing pipeline that includes object detection, object tracking, and camera calibration. Moreover, we also include important applications of visual traffic surveillance systems, such as speed measurement, behavior learning, accident detection and prediction. Finally, future research directions of visual traffic surveillance systems are outlined.

Deep Learning for Downscaling Remote Sensing Images: Fusion and super-resolution

The past few years have seen an accelerating integration of deep learning (DL) techniques into various remote sensing (RS) applications, highlighting their power to adapt and achieving unprecedented advancements. In the present review, we provide an exhaustive exploration of the DL approaches proposed specifically for the spatial downscaling of RS imagery. A key contribution of our work is the presentation of the major architectural components and models, metrics, and data sets available for this task as well as the construction of a compact taxonomy for navigating through the various methods. Furthermore, we analyze the limitations of the current modeling approaches and provide a brief discussion on promising directions for image enhancement, following the paradigm of general computer vision (CV) practitioners and researchers as a source of inspiration and constructive insight.

Face Recognition Attendance System Method Based on Fusion of LBP and HOG

As one of the hot topics in the field of computer vision research, face recognition technology has received significant attention due to its potentiality for a wide range of applications in government as well as commercial purposes. In practical applications, although several existing face recognition methods have achieved good performances in specific scenes, they easily suffer from a sharp decline in recognition rate if affected by different conditions of light, expression, posture and occlusion. Among many factors, influences of complex illuminations on face recognition are particularly significant. To further improve the performance of the existing local binary pattern (LBP) operator, neighbourhood weighted average LBP (NWALBP) is first proposed for fully considering the strong correlations between pixel pairs in the neighbourhood, which extends the traditional LBP uni-layer neighbourhood template window to the bi-layer neighbourhood template window and calculates the weighted average of bi-layer neighbourhood pixels in each direction. Then, inspired by centre symmetric LBP (CS-LBP), centre symmetric NWALBP (CS-NWALBP) is further proposed, which can effectively reduce computation complexity by only comparing the weighted average values of the neighbourhood pixels that are symmetric about the centre pixel. Finally, by combining the merit of histogram of oriented gradient (HOG), a feature fusion algorithm named CS-NWALBP+HOG is suggested. Several experiments have eventually demonstrated that our proposed algorithms have more robust performance under complex illumination conditions if compared with many other latest algorithms.

Identifying and Monitoring Students’ Classroom Learning Behavior Based on Multisource Information

Understanding human activity and behavior, particularly real-time understanding in video feeds, is one of the most active areas of research in Computer Vision (CV) and Artificial Intelligence (AI) nowadays. To advance the topic of integrating learning engagement research with university teaching practice, accurate and efficient assessment, and analysis of students’ classroom learning behavior engagement is very important. The recently proposed classroom behavior recognition algorithms have some limitations, such as the inability to quickly and accurately identify students’ classroom behaviors because they do not consider the motion information of students between consecutive frames. In recent years, action recognition algorithms based on Convolutional Neural Networks (CNN) have improved significantly. To address the limitations of existing algorithms, in this study, a 3D-CNN is selected as a network model for classroom student behavior recognition, which increases information multisourcing and classroom student localization with high accuracy and robustness. For better analysis of human behavior in videos, the 3D convolution extends the 2D convolution to the spatial–temporal domain. In the proposed system, first of all, a real-time picture stream of each student is obtained by combining real-time target detection and tracking. Then, a deep spatiotemporal residual CNN is used to learn the spatiotemporal features of each student’s behavior, so, as to achieve real-time recognition of classroom behaviors for multistudent targets in classroom teaching scenarios. To verify the effectiveness of the proposed model, different experiments are conducted using the labeled classroom behavior dataset. The experimental results demonstrate that the proposed model exhibits better performance in classroom behavior recognition. The accurate recognition of classroom behaviors can assist the teachers and students to understand the classroom learning situation and help to promote the development of smart classroom.

Using ISEF Edge Detection, Text Extraction from Natural Images of Different Languages

In this research, we presented a method for word extraction from various linguistic imagery. In computer vision research, text in images is crucial. Here, we employ edge-based text extraction with ISEF (infinite symmetrical edge filter). The best edge detector, known as ISEF, provides reliable findings for text in images. Detection, localization, tracking, and improvement are all involved in text extraction. For text extraction, numerous techniques have been proposed. Our goal is to propose a reliable method for text extraction from photos in many languages.

Capsule networks for image classification: A review

Over the past few years, the computer vision domain has evolved and made a revolutionary transition from human-engineered features to automated features to address challenging tasks. Computer vision is an ever-evolving domain with its roots deeply correlated with neuroscience; any new findings that trigger a more intuitive understanding and working of the human visual system generally impact the design strategy of computer vision algorithms. The convolutional neural network is one such algorithm that is currently the de facto standard for most computer vision tasks such as image classification, object detection, image segmentation, etc. As convolutional neural networks are associated with inherent constraints such as the requirement for an immense amount of labeled data and an inefficient data routing policy, capsule networks could be a viable alternative. Upheld by the backpropagation and the dynamic routing algorithm, the capsule network has set the new paradigm for developing reliable computer vision algorithms. Despite the phenomenal theoretical backing from neuroscience and the groundbreaking performance on benchmark datasets, the lack of information concerning the conception and working of capsule networks becomes the major impediment to adopting them for computer vision algorithms. This paper presents a concise overview of capsule network-based classification architectures, routing algorithms, performance analysis, limitations, and future scope, helping the research community to adopt capsule networks at the forefront of modern computer vision research.

Self-Supervised Learning for Scene Classification in Remote Sensing: Current State of the Art and Perspectives

Deep learning methods have become an integral part of computer vision and machine learning research by providing significant improvement performed in many tasks such as classification, regression, and detection. These gains have been also observed in the field of remote sensing for Earth observation where most of the state-of-the-art results are now achieved by deep neural networks. However, one downside of these methods is the need for large amounts of annotated data, requiring lots of labor-intensive and expensive human efforts, in particular for specific domains that require expert knowledge such as medical imaging or remote sensing. In order to limit the requirement on data annotations, several self-supervised representation learning methods have been proposed to learn unsupervised image representations that can consequently serve for downstream tasks such as image classification, object detection or semantic segmentation. As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.