Masked Regions Research Articles

Iterative facial image inpainting based on an encoder-generator architecture

Facial image inpainting is a challenging problem as it requires generating new pixels that include semantic information for masked key components in a face, e.g., eyes and nose. Recently, remarkable methods have been proposed in this field. Most of these approaches use encoder–decoder architectures and have different limitations such as allowing unique results for a given image and a particular mask. Alternatively, some optimization-based approaches generate promising results using different masks with generator networks. However, these approaches are computationally more expensive. In this paper, we propose an efficient solution to the facial image painting problem using the Cyclic Reverse Generator (CRG) architecture, which provides an encoder-generator model. We use the encoder to embed a given image to the generator space and incrementally inpaint the masked regions until a plausible image is generated; we trained a discriminator model to assess the quality of the generated images during the iterations and determine the convergence. After the generation process, for the post-processing, we utilize a Unet model that we trained specifically for this task to remedy the artifacts close to the mask boundaries. We empirically observed that even in the absence of important facial features, the encoder model is capable of embedding images in semantically rich regions in the latent space, utilizing the surrounding context in the images. Cultivating the feedback loop between the encoder and generator gradually improves the missing content in the images in an iterative fashion, and only a few iterations are sufficient to generate realistic content. Since the models are not trained for particular mask types, our method allows applying sketch-based inpaintings, using a variety of mask types, and producing multiple and diverse results. We compared our method with the state-of-the-art models both quantitatively and qualitatively, and observed that our method can compete with the other models in all mask types; it is particularly better in images where larger masks are utilized. Our code, dataset and models are available at: https://github.com/yahyadogan72/iterative_facial_image_inpainting.

AFD-StackGAN: Automatic Mask Generation Network for Face De-Occlusion Using StackGAN.

To address the problem of automatically detecting and removing the mask without user interaction, we present a GAN-based automatic approach for face de-occlusion, called Automatic Mask Generation Network for Face De-occlusion Using Stacked Generative Adversarial Networks (AFD-StackGAN). In this approach, we decompose the problem into two primary stages (i.e., Stage-I Network and Stage-II Network) and employ a separate GAN in both stages. Stage-I Network (Binary Mask Generation Network) automatically creates a binary mask for the masked region in the input images (occluded images). Then, Stage-II Network (Face De-occlusion Network) removes the mask object and synthesizes the damaged region with fine details while retaining the restored face’s appearance and structural consistency. Furthermore, we create a paired synthetic face-occluded dataset using the publicly available CelebA face images to train the proposed model. AFD-StackGAN is evaluated using real-world test images gathered from the Internet. Our extensive experimental results confirm the robustness and efficiency of the proposed model in removing complex mask objects from facial images compared to the previous image manipulation approaches. Additionally, we provide ablation studies for performance comparison between the user-defined mask and auto-defined mask and demonstrate the benefits of refiner networks in the generation process.

Pre-training Model Based on Parallel Cross-Modality Fusion Layer.

Visual Question Answering (VQA) is a learning task that combines computer vision with natural language processing. In VQA, it is important to understand the alignment between visual concepts and linguistic semantics. In this paper, we proposed a Pre-training Model Based on Parallel Cross-Modality Fusion Layer (P-PCFL) to learn the fine-grained relationship between vision and language. The P-PCFL model is composed of three Encoders: Object Encoder, Language Encoder, and Parallel Cross-Modality Fusion Encoder, with Transformer as the core. We use four different Pre-training missions, namely, Cross-Modality Mask Language Modeling, Cross-Modality Mask Region Modeling, Image-Text Matching, and Image-Text Q&A, to pre-train the P-PCFL model and improve its reasoning and universality, which help to learn the relationship between Intra-modality and Inter-modality. Experimental results on the platform of Visual Question Answering dataset VQA v2.0 show that the Pre-trained P-PCFL model has a good effect after fine-tuning the parameters. In addition, we also conduct ablation experiments and provide some results of Attention visualization to verify the effectiveness of P-PCFL model.

Unsupervised person re-identification via simultaneous clustering and mask prediction

Extracting meaningful representation is a key challenge for person re-identification (re-ID) task, especially in the absence of ground truth labels. However, existing unsupervised approaches simply utilize pseudo labels generated from clustering to supervise re-ID model and thus have not yet fully explored the semantic information existing in data itself. This also limits the representation capabilities of learned models. To address the above problem, we propose mask prediction (MaskPre) as a pretext task for unsupervised re-ID, such that the clustering network can capture more semantic information and separate the images into semantic clusters automatically. Specifically, MaskPre masks region-level features with dynamic dropblock layer to generate differently masked views of a single image. To predict the masked regions and bridge the domain gap across views, we design mask prediction head and moving-average model to learn visual consistency from still image and temporal consistency during training process. Meanwhile, we optimize the model by grouping the two masked views into the same cluster, thus enhancing the consistency across views. Experimental results on three public benchmark datasets show that our proposed method outperforms the existing state-of-the-art approaches.

An Intelligent Deep Learning Enabled Marine Fish Species Detection and Classification Model

In recent times, marine fish species recognition becomes an important research area to protect the ocean environment. It is a tough and time-consuming operation to manually detect marine fish species on the ocean floor. Depending on the situation, extensive sample efforts may be required. These efforts might be harmful to the marine ecosystem. Automated classification methods are capable of properly classifying these fish on a consistent basis. An increasing number of people are becoming interested in utilizing electronic monitoring and reporting with artificial intelligence for the aim of fish identification and enhancing present techniques. It is becoming more usual to use video and pictures of fish (either underwater or on ships) in fishing operations. These techniques are operational, transportable, and non-invasive, and they provide high-quality pictures at a lower cost than traditional approaches. Automated image processing techniques such as Deep Learning (DL) and Machine Learning (ML) are now available, and they may be customized to perform efficient fish species identification and segmentation. In this aspect, this paper presents an Intelligent DL based Marine Fish Species Classification (IDL-MFSC) technique. The proposed IDL-MFSC technique involves three major processes such as pre-processing, fish detection and fish classification. Primarily, Weiner filtering-based noise removal process takes place as a pre-processing step. In addition, Mask R-CNN (Mask Region Based Convolutional Neural Networks) with Residual Network as a backbone network is used for fish detection. Moreover, Optimal Deep Kernel Extreme Learning Machine (ODKELM) based classification method is employed for determining the class labels of the marine fish species in which the parameter tuning of the DKELM model takes place using Water Wave Optimization (WWO) technique. The performance of the proposed method is tested using an openly accessible Fish4Knowledge dataset. The experimental result highlights the supremacy of the IDL-MFSC technique over the recent techniques with respect to various measures.

Deep Learning-Based Chest CT Image Features in Diagnosis of Lung Cancer.

This study was to evaluate the diagnostic value of deep learning-optimized chest CT in the patients with lung cancer. 90 patients who were diagnosed with lung cancer by surgery or puncture in hospital were selected as the research subjects. The Mask Region Convolutional Neural Network (Mask-RCNN) model was a typical end-to-end image segmentation model, and Dual Path Network (DPN) was used in nodule detection. The results showed that the accuracy of DPN algorithm model in detecting lung lesions in lung cancer patients was 88.74%, the accuracy of CT diagnosis of lung cancer was 88.37%, the sensitivity was 82.91%, and the specificity was 87.43%. Deep learning-based CT examination combined with serum tumor detection, factoring into Neurospecific enolase (N S E), cytokeratin 19 fragment (CYFRA21), Carcinoembryonic antigen (CEA), and squamous cell carcinoma (SCC) antigen, improved the accuracy to 97.94%, the sensitivity to 98.12%, and the specificity to 100%, all showing significant differences (P < 0.05). In conclusion, this study provides a scientific basis for improving the diagnostic efficiency of CT imaging in lung cancer and theoretical support for subsequent lung cancer diagnosis and treatment.

Underwater Fish Detection and Counting Using Mask Regional Convolutional Neural Network

Fish production has become a roadblock to the development of fish farming, and one of the issues encountered throughout the hatching process is the counting procedure. Previous research has mainly depended on the use of non-machine learning-based and machine learning-based counting methods and so was unable to provide precise results. In this work, we used a robotic eye camera to capture shrimp photos on a shrimp farm to train the model. The image data were classified into three categories based on the density of shrimps: low density, medium density, and high density. We used the parameter calibration strategy to discover the appropriate parameters and provided an improved Mask Regional Convolutional Neural Network (Mask R-CNN) model. As a result, the enhanced Mask R-CNN model can reach an accuracy rate of up to 97.48%.

E2F-GAN: Eyes-to-Face Inpainting via Edge-Aware Coarse-to-Fine GANs

Face inpainting is a challenging task aiming to fill the damaged or masked regions in face images with plausibly synthesized contents. Based on the given information, the reconstructed regions should look realistic and more importantly preserve the demographic and biometric properties of the individual. The aim of this paper is to reconstruct the face based on the periocular region (eyes-to-face). To do this, we proposed a novel GAN-based deep learning model called Eyes-to-Face GAN (E2F-GAN) which includes two main modules: a coarse module and a refinement module. The coarse module along with an edge predictor module attempts to extract all required features from a periocular region and to generate a coarse output which will be refined by a refinement module. Additionally, a dataset of eyes-to-face synthesis has been generated based on the public face dataset called CelebA-HQ for training and testing. Thus, we perform both qualitative and quantitative evaluations on the generated dataset. Experimental results demonstrate that our method outperforms previous learning-based face inpainting methods and generates realistic and semantically plausible images. We also provide the implementation of the proposed approach to support reproducible research via (<uri>https://github.com/amiretefaghi/E2F-GAN</uri>).

Deep Reinforcement Learning Enabled Smart City Recycling Waste Object Classification

The Smart City concept revolves around gathering real time data from citizen, personal vehicle, public transports, building, and other urban infrastructures like power grid and waste disposal system. The understandings obtained from the data can assist municipal authorities handle assets and services effectually. At the same time, the massive increase in environmental pollution and degradation leads to ecological imbalance is a hot research topic. Besides, the progressive development of smart cities over the globe requires the design of intelligent waste management systems to properly categorize the waste depending upon the nature of biodegradability. Few of the commonly available wastes are paper, paper boxes, food, glass, etc. In order to classify the waste objects, computer vision based solutions are cost effective to separate out the waste from the huge dump of garbage and trash. Due to the recent developments of deep learning (DL) and deep reinforcement learning (DRL), waste object classification becomes possible by the identification and detection of wastes. In this aspect, this paper designs an intelligence DRL based recycling waste object detection and classification (IDRL-RWODC) model for smart cities. The goal of the IDRL-RWODC technique is to detect and classify waste objects using the DL and DRL techniques. The IDRL-RWODC technique encompasses a two-stage process namely Mask Regional Convolutional Neural Network (Mask RCNN) based object detection and DRL based object classification. In addition, DenseNet model is applied as a baseline model for the Mask RCNN model, and a deep Q-learning network (DQLN) is employed as a classifier.Moreover, a dragonfly algorithm (DFA) based hyperparameter optimizer is derived for improving the efficiency of the DenseNet model. In order to ensure the enhanced waste classification performance of the IDRL-RWODC technique, a series of simulations take place on benchmark dataset and the experimental results pointed out the better performance over the recent techniques with maximal accuracy of 0.993.

CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editing

The text-based speech editor allows the editing of speech through intuitive cutting, copying, and pasting operations to speed up the process of editing speech. However, the major drawback of current systems is that edited speech often sounds unnatural due to cut-copy-paste operation. In addition, it is not obvious how to synthesize records according to a new word not appearing in the transcript, which often needs the help of text-to-speech (TTS) and voice conversion (VC) technology at the same time. This paper first proposes a novel end-to-end text-based speech editing method called context-aware mask prediction network (CampNet). The model can simulate the text-based speech editing process by randomly masking part of speech and then predicting the masked region by sensing the speech context. It can solve unnatural prosody in the edited region and synthesize the speech corresponding to the unseen words in the transcript. Secondly, for the possible operation of text-based speech editing, we design three text-based operations based on CampNet: deletion, insertion, and replacement. These operations can cover various situations of speech editing. Thirdly, to synthesize the speech corresponding to long text in insertion and replacement operations, a word-level autoregressive generation method is proposed, which can synthesize the speech of arbitrary length text. Fourthly, we propose a speaker adaptation method using only one sentence for CampNet and explore the ability of few-shot learning based on CampNet, which provides a new idea for speech forgery tasks. The subjective and objective experiments<xref ref-type="fn" rid="fn1">¹</xref><fn id="fn1"><label>¹</label> Examples of generated speech can be found at <uri>https://hairuo55.github.io/CampNet</uri>. </fn> on VCTK and LibriTTS datasets show that the speech editing results based on CampNet are better than TTS technology, manual editing, and VoCo method (the combination of TTS and VC). We also conduct detailed ablation experiments to explore the effect of the CampNet structure on its performance. Finally, the experiment shows that speaker adaptation with only one sentence can further improve the naturalness of speech editing for one-shot learning.

Deep Learning Based Hand Wrist Segmentation using Mask R-CNN

Deep learning is one of the trending technologies in computer vision to identify and classify objects. Deep learning is a subset of Machine Learning and Artificial Intelligence. Detecting and classifying the object was a challenging task in traditional computer vision techniques, and now there are numerous deep learning Techniques scaled up to achieve this. The primary purpose of the research is to detect and segment the human hand wrist region using deep learning methods. This research is widespread to deep learning enthusiasts who needs to segment custom objects using instance segmentation. We demonstrated a segmented hand wrist using the Mask Regional Convolutional Neural Network (R-CNN) technique with an average accuracy of 99.73%. This work also compares the performance evaluation of baseline and custom Hand Wrist Mask R-CNN. The achieved validation class loss is 0.00866 training and 0.02736 validation; both the values are comparatively deficient compared with baseline Mask R-CNN.

Intelligent Deep Learning Based Automated Fish Detection Model for UWSN

An exponential growth in advanced technologies has resulted in the exploration of Ocean spaces. It has paved the way for new opportunities that can address questions relevant to diversity, uniqueness, and difficulty of marine life. Underwater Wireless Sensor Networks (UWSNs) are widely used to leverage such opportunities while these networks include a set of vehicles and sensors to monitor the environmental conditions. In this scenario, it is fascinating to design an automated fish detection technique with the help of underwater videos and computer vision techniques so as to estimate and monitor fish biomass in water bodies. Several models have been developed earlier for fish detection. However, they lack robustness to accommodate considerable differences in scenes owing to poor luminosity, fish orientation, structure of seabed, aquatic plant movement in the background and distinctive shapes and texture of fishes from different genus. With this motivation, the current research article introduces an Intelligent Deep Learning based Automated Fish Detection model for UWSN, named IDLAFD-UWSN model. The presented IDLAFD-UWSN model aims at automatic detection of fishes from underwater videos, particularly in blurred and crowded environments. IDLAFD-UWSN model makes use of Mask Region Convolutional Neural Network (Mask RCNN) with Capsule Network as a baseline model for fish detection. Besides, in order to train Mask RCNN, background subtraction process using Gaussian Mixture Model (GMM) model is applied. This model makes use of motion details of fishes in video which consequently integrates the outcome with actual image for the generation of fish-dependent candidate regions. Finally, Wavelet Kernel Extreme Learning Machine (WKELM) model is utilized as a classifier model. The performance of the proposed IDLAFD-UWSN model was tested against benchmark underwater video dataset and the experimental results achieved by IDLAFD-UWSN model were promising in comparison with other state-of-the-art methods under different aspects with the maximum accuracy of 98% and 97% on the applied blurred and crowded datasets respectively.

A Naive and Semantic Approach for Detecting Face Mask Region Based Convolutional Neural Networks (R-CNN)

Due to Covid-19, most of the health care organizations and governments has ordered their citizens to wear face mask to protect themself. This novel research presents a tactical methodology for rapid detection whether a person is wearing a face mask or not. It is entirely different from the existing system; the singular aim of the proposed system is to train the deep learning model with a minimum number of image samples and to operate face mask instance segmentation and along with object box detection. In this work, we proposed a novel and semantic pixel-to-pixel region based deep network which can detect no of face mask instances in different categories pixel wise to organize the segment bounding box and the confidence of the various categories for each pixel. This system experimental output demonstrate that this approach can effectively and precisely detect the face mask with multi-feature combination. It is also reported that our application performance outperforms the existing system.

Breed recognition and estimation of live weight of cattle based on methods of machine learning and computer vision

A method of measuring cattle parameters using neural network methods of image processing was proposed. To this end, several neural network models were used: a convolutional artificial neural network and a multilayer perceptron. The first is used to recognize a cow in a photograph and identify its breed followed by determining its body dimensions using the stereopsis method. The perceptron was used to estimate the cow's weight based on its breed and size information. Mask RCNN (Mask Regions with CNNs) convolutional network was chosen as an artificial neural network. To clarify information on the physical parameters of animals, a 3D camera (Intel RealSense D435i) was used. Images of cows taken from different angles were used to determine the parameters of their bodies using the photogrammetric method. The cow body dimensions were determined by analyzing animal images taken with synchronized cameras from different angles. First, a cow was identified in the photograph and its breed was determined using the Mask RCNN convolutional neural network. Next, the animal parameters were determined using the stereopsis method. The resulting breed and size data were fed to a predictive model to determine the estimated weight of the animal. When modeling, Ayrshire, Holstein, Jersey, Krasnaya Stepnaya breeds were considered as cow breeds to be recognized. The use of a pre-trained network with its subsequent training applying the SGD algorithm and Nvidia GeForce 2080 video card has made it possible to significantly speed up the learning process compared to training in a CPU. The results obtained confirm the effectiveness of the proposed method in solving practical problems.

Implementation and Practice of Deep Learning-Based Instance Segmentation Algorithm for Quantification of Hepatic Fibrosis at Whole Slide Level in Sprague-Dawley Rats.

Exponential development in artificial intelligence or deep learning technology has resulted in more trials to systematically determine the pathological diagnoses using whole slide images (WSIs) in clinical and nonclinical studies. In this study, we applied Mask Regions with Convolution Neural Network (Mask R-CNN), a deep learning model that uses instance segmentation, to detect hepatic fibrosis induced by N-nitrosodimethylamine (NDMA) in Sprague-Dawley rats. From 51 WSIs, we collected 2011 cropped images with hepatic fibrosis annotations. Training and detection of hepatic fibrosis via artificial intelligence methods was performed using Tensorflow 2.1.0, powered by an NVIDIA 2080 Ti GPU. From the test process using tile images, 95% of model accuracy was verified. In addition, we validated the model to determine whether the predictions by the trained model can reflect the scoring system by the pathologists at the WSI level. The validation was conducted by comparing the model predictions in 18 WSIs at 20× and 10× magnifications with ground truth annotations and board-certified pathologists. Predictions at 20× showed a high correlation with ground truth (R2 = 0.9660) and a good correlation with the average fibrosis rank by pathologists (R2 = 0.8887). Therefore, the Mask R-CNN algorithm is a useful tool for detecting and quantifying pathological findings in nonclinical studies.

Automatic vectorization of point symbols on archive maps using deep convolutional neural network

Abstract. Archive topographical maps are a key source of geographical information from past ages, which can be valuable for several science fields. Since manual digitization is usually slow and takes much human resource, automatic methods are preferred, such as deep learning algorithms. Although automatic vectorization is a common problem, there have been few approaches regarding point symbols. In this paper, a point symbol vectorization method is proposed, which was tested on Third Military Survey map sheets using a Mask Regional Convolutional Neural Network (MRCNN). The MRCNN implementation uses the ResNet101 network improved with the Feature Pyramid Network architecture and is developed in a Google Colab environment. The pretrained network was trained on four point symbol categories simultaneously. Results show 90% accuracy, while 94% of symbols detected for some categories on the complete test sheet.

Analyzing fibrous tissue pattern in fibrous dysplasia bone images using deep R-CNN networks for segmentation

Predictive health monitoring systems help to detect human health threats in the early stage. Evolving deep learning techniques in medical image analysis results in efficient feedback in quick time. Fibrous dysplasia (FD) is a genetic disorder, triggered by the mutation in Guanine Nucleotide binding protein with alpha stimulatory activities in the human bone genesis. It slowly occupies the bone marrow and converts the bone cell into fibrous tissues. It weakens the bone structure and leads to permanent disability. This paper proposes the study of FD bone image analyzing techniques with deep networks. Also, the linear regression model is annotated for predicting the bone abnormality levels with observed coefficients. Modern image processing begins with various image filters. It describes the edges, shades, texture values of the receptive field. Different types of segmentation and edge detection mechanisms are applied to locate the tumor, lesion, and fibrous tissues in the bone image. Extract the fibrous region in the bone image using the region-based convolutional neural network algorithm. The segmented results are compared with their accuracy metrics. The segmentation loss is reduced by each iteration. The overall loss is 0.24% and the accuracy is 99%, segmenting the masked region produces 98% of accuracy, and building the bounding boxes is 99% of accuracy.

Efficient masked face recognition method during the COVID-19 pandemic.

The coronavirus disease (COVID-19) is an unparalleled crisis leading to a huge number of casualties and security problems. In order to reduce the spread of coronavirus, people often wear masks to protect themselves. This makes face recognition a very difficult task since certain parts of the face are hidden. A primary focus of researchers during the ongoing coronavirus pandemic is to come up with suggestions to handle this problem through rapid and efficient solutions. In this paper, we propose a reliable method based on occlusion removal and deep learning-based features in order to address the problem of the masked face recognition process. The first step is to remove the masked face region. Next, we apply three pre-trained deep Convolutional Neural Networks (CNN), namely VGG-16, AlexNet, and ResNet-50, and use them to extract deep features from the obtained regions (mostly eyes and forehead regions). The Bag-of-features paradigm is then applied to the feature maps of the last convolutional layer in order to quantize them and to get a slight representation comparing to the fully connected layer of classical CNN. Finally, Multilayer Perceptron (MLP) is applied for the classification process. Experimental results on Real-World-Masked-Face-Dataset show high recognition performance compared to other state-of-the-art methods.

Segmentation of Overlapping Cervical Cells with Mask Region Convolutional Neural Network.

The task of segmenting cytoplasm in cytology images is one of the most challenging tasks in cervix cytological analysis due to the presence of fuzzy and highly overlapping cells. Deep learning-based diagnostic technology has proven to be effective in segmenting complex medical images. We present a two-stage framework based on Mask RCNN to automatically segment overlapping cells. In stage one, candidate cytoplasm bounding boxes are proposed. In stage two, pixel-to-pixel alignment is used to refine the boundary and category classification is also presented. The performance of the proposed method is evaluated on publicly available datasets from ISBI 2014 and 2015. The experimental results demonstrate that our method outperforms other state-of-the-art approaches with DSC 0.92 and FPRp 0.0008 at the DSC threshold of 0.8. Those results indicate that our Mask RCNN-based segmentation method could be effective in cytological analysis.

Deep leaf: Mask R-CNN based leaf detection and segmentation from digitized herbarium specimen images

• A modified mask RCNN named Deep Leaf is developed to identify the leaves from the digitized herbarium specimens. • Deep Leaf measures automatically the morphological traits of the extracted leaves. • Deep features are extracted through an improved ResNet50/101, which is chosen as the backbone network of the feature extraction. • We achieved better performance compared with the original mask RCNN algorithm for leaves detection. The generation of morphological traits of plants such as the leaf length, width, perimeter, area, and petiole length are fundamental features of herbarium specimens, thus providing high-quality data to investigate plant responses to ongoing climatic change and plant history evolution. However, the existing measurement methods are primarily associated with manual analysis, which is labor-intensive and inefficient. This paper proposes a deep learning-based approach, called Deep Leaf, for detecting and pixel-wise segmentation of leaves based on the improved state-of-the-art instance segmentation approach, Mask Region Convolutional Neural Network (Mask R-CNN). Deep Leaf can accurately detect each leaf in the herbarium specimen and measure the associated morphological traits. The experimental results indicate that our automated approach can segment the leaves of different families. Compared to manual measurement done by ecologist and botanist experts, the average relative error of leaf length is 4.6%, while the average relative error of leaf width is 5.7%.