Perform Data Augmentation Research Articles

The Role of the Input in Natural Language Video Description

Natural language video description (NLVD) has recently received strong interest in the computer vision, natural language processing (NLP), multimedia, and autonomous robotics communities. The state-of-the-art (SotA) approaches obtained remarkable results when tested on the benchmark datasets. However, those approaches poorly generalize to new datasets. In addition, none of the existing works focus on the processing of the input to the NLVD systems, which is both visual and textual. In this paper, an extensive study is presented to deal with the role of the visual input, evaluated with respect to the overall NLP performance. This is achieved by performing data augmentation of the visual component, applying common transformations to model camera distortions, noise, lighting, and camera positioning that are typical in real-world operative scenarios. A t-SNE-based analysis is proposed to evaluate the effects of the considered transformations on the overall visual data distribution. For this study, the English subset of the Microsoft Research Video Description (MSVD) dataset is considered, which is used commonly for NLVD. It was observed that this dataset contains a relevant amount of syntactic and semantic errors. These errors have been amended manually, and the new version of the dataset (called MSVD-v2) is used in the experimentation. The MSVD-v2 dataset is released to help to gain insight into the NLVD problem.

Weight Rotation as a Regularization Strategy in Convolutional Neural Networks.

Convolutional Neural Networks (CNN) have become the gold standard in many visual recognition tasks including medical applications. Due to their high variance, however, these models are prone to over-fit the data they are trained on. To mitigate this problem, one of the most common strategies, is to perform data augmentation. Rotation, scaling and translation are common operations. In this work we propose an alternative method to rotation-based data augmentation where the rotation transformation is performed inside the CNN architecture. In each training batch the weights of all convolutional layers are rotated by the same random angle. We validate our proposed method empirically showing its usefulness under different scenarios.

A dynamical system as the source of augmentation in a deep learning problem

In this work we build a convolutional neural network capable of identifying individual birds by their songs. Since the actual data available from each individual is very limited, we use a dynamical system capable of synthesizing realistic songs, to generate surrogate-training data. The different synthetic songs are the result of integrating the dynamical system with slightly varied parameters. We show that a data set built in this way allows us to train the network to successfully identify the different individuals in our study. In this way, we present a novel way to perform data augmentation using dynamical systems.

Data augmentation of random grid-hiding for video object segmentation

Video object segmentation is an important field in computer vision. However, the challenges in video object segmentation such as background clutter, occlusion and edge ambiguity cannot be avoided. In addition, existing labeled video object segmentation datasets are limited in size, which prevents CNN models from reaching their full generalization capabilities. In this paper, we propose a novel approach, called random grid-hiding (RGH), to perform data augmentation. We divide the training image into several rectangular regions and hide some regions randomly during model training. Thus, the convolutional neural network automatically focuses on the discriminative parts of the image. When the most discriminative part of the image is hidden, it compels the network focus on the other related parts of the image. Further, occlusion images are randomly generated in various levels. More features can be obtained by random grid-hiding, which can effectively reduce the risk of overfitting. Our approach is an effective extension of the data augmentation (such as random cropping and random flipping), and leads to improved accuracy in the task of the video object segmentation method on DAVIS dataset. Our experimental results show that the proposed method is a stable and effective method for data augmentation.

Enhancing convolutional neural network scheme for rheumatoid arthritis grading with limited clinical data**Project supported by the National Key Research and Development Program of China (Grant No. 2017YFC0111402) and the Natural Science Funds of Jiangsu Province of China (Grant No. BK20181256).

The gray-scale ultrasound (US) imaging method is usually used to assess synovitis in rheumatoid arthritis (RA) in clinical practice. This four-grade scoring system depends highly on the sonographer’s experience and has relatively lower validity compared with quantitative indexes. However, the training of a qualified sonographer is expensive and time-consuming while few studies focused on automatic RA grading methods. The purpose of this study is to propose an automatic RA grading method using deep convolutional neural networks (DCNN) to assist clinical assessment. Gray-scale ultrasound images of finger joints are taken as inputs while the output is the corresponding RA grading results. Firstly, we performed the auto-localization of synovium in the RA image and obtained a high precision in localization. In order to make up for the lack of a large annotated training dataset, we performed data augmentation to increase the number of training samples. Motivated by the approach of transfer learning, we pre-trained the GoogLeNet on ImageNet as a feature extractor and then fine-tuned it on our own dataset. The detection results showed an average precision exceeding 90%. In the experiment of grading RA severity, the four-grade classification accuracy exceeded 90% while the binary classification accuracies exceeded 95%. The results demonstrate that our proposed method achieves performances comparable to RA experts in multi-class classification. The promising results of our proposed DCNN-based RA grading method can have the ability to provide an objective and accurate reference to assist RA diagnosis and the training of sonographers.

Conditional Generative Adversarial Network- Based Data Augmentation for Enhancement of Iris Recognition Accuracy

Presently, lots of previous studies on biometrics employ convolutional neural networks (CNN) which requires a large amount of labeled training data. However, biometric data are considered as important personal information, and it is difficult to obtain large amounts of data due to individual privacy issues. Training with a small amount of data is a major cause of overfitting and low testing accuracy. To resolve this problem, previous studies have performed data augmentation that are based on geometric transforms and the adjustment of image brightness. Nevertheless, the data created by these methods have high correlation with the original data, and they cannot adequately reflect individual diversities. To resolve this problem, this study proposes iris image augmentation based on a conditional generative adversarial network (cGAN), as well as a method for improving recognition performance that uses this augmentation method. In our method, normalized iris images that are generated through arbitrary changes in the iris and pupil coordinates are used as input in the cGAN-based model to generate iris images. Due to the limitations of the cGAN model, data augmentation, which uses the periocular region, was found to fail with regard to the improvement of performance. Based on this information, only the iris region was used as input for the cGAN model. The augmentation method proposed in this paper was tested using NICE.II training dataset (selected from UBIRS.v2), MICHE database, and CASIA-Iris-Distance database. The results showed that the recognition performance was improved compared to existing studies.

Reducing the Model Variance of a Rectal Cancer Segmentation Network

In preoperative imaging, the demarcation of rectal cancer with magnetic resonance images provides an important basis for cancer staging and treatment planning. Recently, deep learning has greatly improved the state-of-the-art method in automatic segmentation. However, limitations in data availability in the medical field can cause large variance and consequent overfitting to medical image segmentation networks. In this study, we propose methods to reduce the model variance of a rectal cancer segmentation network by adding a rectum segmentation task and performing data augmentation; the geometric correlation between the rectum and rectal cancer motivated the former approach. Moreover, we propose a method to perform a bias-variance analysis within an arbitrary region-of-interest (ROI) of a segmentation network, which we applied to assess the efficacy of our approaches in reducing model variance. As a result, adding a rectum segmentation task reduced the model variance of the rectal cancer segmentation network within tumor regions by a factor of 0.90; data augmentation further reduced the variance by a factor of 0.89. These approaches also reduced the training duration by a factor of 0.96 and a further factor of 0.78, respectively. Our approaches will improve the quality of rectal cancer staging by increasing the accuracy of its automatic demarcation and by providing rectum boundary information since rectal cancer staging requires the demarcation of both rectum and rectal cancer. Besides such clinical benefits, our method also enables segmentation networks to be assessed with bias-variance analysis within an arbitrary ROI, such as a cancerous region.

Pixel-Based Image Encryption Without Key Management for Privacy-Preserving Deep Neural Networks

We present a novel privacy-preserving scheme for deep neural networks (DNNs) that enables us not to only apply images without visual information to DNNs but to also consider the use of independent encryption keys for both training and testing images for the first time. In this paper, a novel pixel-based image encryption method that maintains important features of original images is proposed for privacy-preserving DNNs. For training, a DNN model is trained with images encrypted by using the proposed method with independent encryption keys. For testing, the model enables us to apply both encrypted images and plain images for image classification. Therefore, there is no need to manage keys. In addition, the proposed method allows us to perform data augmentation in the encrypted domain. In an experiment, the proposed method is applied to well-known networks, that is, deep residual networks and densely connected convolutional networks, for image classification. The experimental results demonstrate that the proposed method, under the use of independent encryption keys, can maintain a high classification performance, and it is robust against ciphertext-only attacks (COAs). Moreover, the results confirm that the proposed scheme is able to classify plain images as well as encrypted images, even when data augmentation is carried out in the encrypted domain.

Exploiting Repetitions for Image‐Based Rendering of Facades

AbstractStreet‐level imagery is now abundant but does not have sufficient capture density to be usable for Image‐Based Rendering (IBR) of facades. We present a method that exploits repetitive elements in facades ‐ such as windows ‐ to perform data augmentation, in turn improving camera calibration, reconstructed geometry and overall rendering quality for IBR. The main intuition behind our approach is that a few views of several instances of an element provide similar information to many views of a single instance of that element. We first select similar instances of an element from 3–4 views of a facade and transform them into a common coordinate system, creating a “platonic” element. We use this common space to refine the camera calibration of each view of each instance and to reconstruct a 3D mesh of the element with multi‐view stereo, that we regularize to obtain a piecewise‐planar mesh aligned with dominant image contours. Observing the same element under multiple views also allows us to identify reflective areas ‐ such as glass panels ‐ which we use at rendering time to generate plausible reflections using an environment map. Our detailed 3D mesh, augmented set of views, and reflection mask enable image‐based rendering of much higher quality than results obtained using the input images directly.

A Methodological Proposal for Collecting and Creating Macroscopic Photograph Collections of Tropical Woods with Potential for Use in Deep Learning

Costa Rica is one of the countries with highest species biodiversity density in the world. More than 2,000 tree species have already been identified, many of which are used in the building, furniture, and packaging industries (Grayum et al. 2003). This rich diversity makes the correct identification of tree species very difficult. As a result, it is common to see in the national market that species are commercialized with mistaken identifications, which makes quality control particularly challenging. In addition, because 90 timber tree species have been classified as “threatened” in Costa Rica, correct identifications are indispensable for law-enforcement. The traditional system for tree species identification is based on macro and microscopic evaluations of the anatomy of the wood. It entails assesing anatomical features such as patterns of vessels, parenchymas, and fibers. Typically, 7.7 x 10 cm pieces of wood cuts are used to identify the tree species (Pan and Kudo 2011, Yusof et al. 2013). However, assessing these features is extremely difficult for taxonomists because properties of the wood can vary considerably due to environmental conditions and intra-specific genetic variability. Deep learning techniques have recently been used to identify plant species (Carranza-Rojas et al. 2017a, Carranza-Rojas et al. 2017b) and are potentially useful to detect subtle differences in patterns of vessels, parenchyma, and other anatomical features of wood. However, it is necessary to have a large collection of macroscopic photographs of individuals from various parts of the country (Pan and Kudo 2011). As a first step in the application of deep learning techniques, we have defined a formal, standard protocol for collecting wood samples, physically processing them, taking pictures, performing data augmentation, and using metadata to provide the primary data necessary for deep learning applications. Unlike traditional xylotheque sampling methods that destroy trees or use wood from fallen trees, we propose a method that extracts small size samples with sufficient quality for anatomical characterization but does not affect the growth and survival of the individual. This study has been developed in three forest permanent plots in Costa Rica, all of which are sites with historical growth data over the last 20 years. We have so far evaluated 40 species (10 individuals per species) with diameters greater than 20 cm. From each individual, a cylindrical sample of 12 mm diameter and 7.5 cm in length was extracted with a cordless drill. Each sample is then cut into five of 8 x 8 x 8 mm cubes and further processed to result in curated xylotheque samples, a dataset with all relevant metadata and original images, and a dataset with images obtained by performing data augmentation on the original images.

Data augmentation for unbalanced face recognition training sets

Face recognition remains a challenging problem. While one-to-one face verification has been largely tackled, verification-based classification problem still demands effort. To further enhance the verification models, one solution is to fully utilize the unbalanced training sets, where, while abundant samples are provided for some subjects, there are often so few samples available for the rest. These subjects with too few samples can contribute little to the model learning. Therefore, before training a model, algorithms usually perform data augmentation on the whole dataset, especially on subjects with insufficient samples. In this paper, a new augmentation method is proposed, targeting on data augmentation for face classification algorithms. Instead of directly manipulating the input image, we perform virtual sample generating on feature level. The distribution of feature maps is first estimated, then random noise consistent to the distribution is applied to the feature vectors of training samples. Our method is based on Joint Bayesian Face Analysis, and we also develop an algorithm to boost the whole procedure. We conduct experiments based on high dimensional LBP features and features extracted by a shallow Convolutional Neural Network, and succeed to verify the effectiveness of this method, using image data from benchmark dataset LFW.