Limited-data transfer learning for semantic segmentation and pre-labeling of 3D shell construction LiDAR scans

Training data is an essential ingredient within supervised learning, but time consuming, expensive and for some applications impossible to acquire. A possible solution is to use synthetic training data. However, the domain shift of synthetic data makes it challenging to obtain good results when used as training data for deep learning models. It is therefore of interest to refine synthetic data, e.g. using image-to-image translation, to improve results. The aim of this work is to compare different methods to do image-to-image translation of synthetic training data of thermal IR-images using generative adversarial networks (GANs). Translation is done both using synthetic thermal IR-images alone, as well as including pixelwise depth and/or semantic information. To evaluate, we propose a new measure based on the Frechet Inception Distance, adapted to work for thermal IR-images. We show that by adapting a GAN model to also include corresponding pixelwise depth data to each synthetic IR-image, the performance is improved compared to using only IR-images.

Inadequate training data is a significant challenge for deep learning techniques, particularly in applications where data is difficult to get, and publicly available datasets are uncommon owing to ethical and privacy concerns. Various approaches, such as data augmentation and transfer learning, are employed to address this problem, which help to some extent in removing this limitation. However, after a certain amount of data augmentation, the quality of the generated data stalls, and transfer learning suffers from the issue of negative transfer. This paper proposes a novel generative adversarial network-based synthetic data training (GAN-ST) model to generate synthetic data for training a lightweight convolutional neural network (CNN). An enhanced generator is proposed to quickly saturate and cover the colour space of the training distribution. The GAN-ST model is based on Deep Convolutional Generative Adversarial Network(s) (DCGAN) and Conditional Generative Adversarial Network(s) (CGAN) models, which consist of an enhanced generator. The study evaluates the accuracy of a CNN model on the MNIST and CIFAR 10 datasets using both original and synthetic data. The results revealed an impressive classifier accuracy on the MNIST dataset, achieving an accuracy of 99.38% on GAN-ST-generated synthetic training data, which is only 0.05% lower than the performance on original data-based training. The classifier performance on the CIFAR dataset is also remarkable, achieving an accuracy of 90.23%. The performance of CNN trained using GAN-ST-based synthetic data is notable, with the most considerable improvement of 0.66% and 7.06%, over a single GAN-based synthetic data training for the MNIST and CIFAR datasets, respectively. By training two GANs independently, the GAN-ST model covers different parts of the original data distribution, resulting in a more diverse and realistic training data set for the classifier. This diverse set of synthetic data, when used to train a CNN, shows better generalization to new data, leading to improved classification accuracy.

Separation of diffracted from reflected events in seismic data is still challenging due to the relatively low amplitude of the diffracted wavefield compared to the reflected wavefield as well as the overlap in the kinematics of reflection and diffraction events. A workflow based on deep learning can be a simple and fast alternative, but using training data made by physics‐based modelling is expensive and lacks diversity in terms of noise, amplitude, frequency content and wavelet. This results in poor generalization beyond the training data without retraining and transfer learning. In this paper, we demonstrate successful applications of reflection–diffraction separation using a conventional U‐net architecture. The novelty of our approach is that we do not use synthetic data created from physics‐based modelling, but instead use only synthetic data built from basic geometric shapes. Our domain of application is the pre‐migration common‐offset domain where reflected events resemble local geology and the diffracted wavefield consists of downward convex hyperbolic diffraction patterns. Both patterns were randomly perturbed in many ways while maintaining their intrinsic features. This approach is inspired by the common practice of data augmentation in deep learning for machine vision applications. Since many of the standard data augmentation techniques lack a geophysical motivation, we have instead perturbed our synthetic training data in ways to make more sense from a signal processing perspective or given our ‘domain knowledge’ of the problem at hand. We did not have to retrain the network to show good results on the field data set. The large variety and diversity in examples enabled to trained neural networks to show encouraging results on synthetic and field data sets that were not used in training.

Semantic segmentation - labelling each pixel in an image to a specific class- models require large amounts of manually labeled and collected data to train. These huge datasets often are lacking in many fields such as specific weather events and climates. They also lack variety as most datasets are of city and street scenes as opposed to more rural scenes. Synthetic-to-real domain adaptation can be used to fill those gaps. Although there has been prior work on how to generate virtual data, in this paper we present insight on how features of the training data can affect the model and ultimately, the outcome. They can also provide insight on the dataset selection process. We hypothesized that different synthetic training data on real test data will affect the outcome of semantic segmentation. By testing real-world data on two iterations of a model trained by two separate virtual datasets using Python and deep learning models, we compared the results and analyzed the differences and similarities the datasets yielded for semantic segmentation. We unearthed critical insights that shed light on the dataset selection process, enabling researchers and practitioners to make more informed decisions when choosing the appropriate dataset type for semantic segmentation tasks. We contributed valuable findings, unveiling the limitations of substituting real datasets with virtual counterparts and offering guidance for dataset selection.

The rise of artificial intelligence (AI) and in particular modern machine learning (ML) algorithms during the last decade has been met with great interest in the agricultural industry. While undisputedly powerful, their main drawback remains the need for sufficient and diverse training data. The collection of real datasets and their annotation are the main cost drivers of ML developments, and while promising results on synthetically generated training data have been shown, their generation is not without difficulties on their own. In this paper, we present a development model for the iterative, cost-efficient generation of synthetic training data. Its application is demonstrated by developing a low-cost early disease detector for tomato plants (Solanum lycopersicum) using synthetic training data. A neural classifier is trained by exclusively using synthetic images, whose generation process is iteratively refined to obtain optimal performance. In contrast to other approaches that rely on a human assessment of similarity between real and synthetic data, we instead introduce a structured, quantitative approach. Our evaluation shows superior generalization results when compared to using non-task-specific real training data and a higher cost efficiency of development compared to traditional synthetic training data. We believe that our approach will help to reduce the cost of synthetic data generation in future applications.

Robust low frequency seismic bandwidth extension with a U-net and synthetic training data

Deep learning shows promise in automated brain tumour segmentation, but it depends on costly expert annotations. Recent advances in unsupervised learning offer an alternative by using synthetic data for training. However, the discrepancy between real and synthetic data limits the accuracy of the unsupervised approaches. In this paper, we propose an approach for unsupervised brain tumour segmentation on magnetic resonance (MR) images via a two-stage image synthesis strategy. This approach accounts for the domain gap between real and synthetic data and aims to generate realistic synthetic data for model training. In the first stage, we train a junior segmentation model using synthetic brain tumour images generated by hand-crafted tumour shape and intensity models, and employs a validation set with distribution shift for model selection. The trained junior model is applied to segment unlabelled real tumour images, generating pseudo labels that capture realistic tumour shape, intensity, and texture. In the second stage, realistic synthetic tumour images are generated by mixing brain images with tumour pseudo labels, closing the domain gap between real and synthetic images. The generated synthetic data is then used to train a senior model for final segmentation. In experiments on five brain imaging datasets, the proposed approach, named as SynthTumour, surpasses existing unsupervised methods and demonstrates high performance for both brain tumour segmentation and ischemic stroke lesion segmentation tasks.

Attenuation of seismic migration smile artifacts with deep learning

Semantic image segmentation is of crucial importance to many applications, such as autonomous driving, robot vision, and scene understanding. However, the border of a segmented image tends to be rough, and the labeling process is tedious and labor-intensive. Therefore, this study is the first proposing to use a deep generative adversarial network (GAN) with double-layered upsampling based on max-pooling indexed deconvolution. Our proposed upsampling method replaces the bilinear interpolation upsampling method; i.e., we fuse the deep deconvolution method by saving the indices of relative locations of the max weights computed during pooling. Combined with the deep GAN, our upsampling method can improve the extraction of low-resolution features, and compensate for the loss of the image size. To further reduce the whole network's dependence on labeled datasets, a weakly supervised feedback method is proposed. The unlabeled data can improve the generalization ability of the model. Considering the generalization to unseen image domains, we introduce transfer learning based on a deep GAN and a weakly supervised method. The segmentation model using the trained data in the source domain can obtain good segmentation in the target domain using transfer learning. Extensive experiments in various domains demonstrate the advantages of the proposed method compared to the generalization ability of semantic segmentation. This method also significantly decreases the dependence on labeled data and ensures the network accuracy.

Synthetic training data promise considerable performance improvements in machine learning (ML) surveillance tasks, including such applications as crowd counting, pedestrian tracking, and face recognition. In this context, synthetic training data constitute techno-fixes primarily by virtue of acting as “edge cases”—data that are hard to come by in the “real world” yet straightforward to produce synthetically—which are used to enhance ML systems’ resilience. In this dialogue paper, I mobilize Haggerty and Ericson’s (2000) concept of the surveillant assemblage to argue that synthetic training data raise well-known, entrenched surveillance issues. Specifically, I contend that conceptualizing synthetic data as but one component of larger surveillant assemblages is analytically meaningful because it challenges techno-deterministic imaginaries that posit synthetic data as fixes to deep-rooted surveillance issues. To exemplify this stance, I draw from several examples of how synthetic training data are already used, illustrating how they may both intensify the disappearance of disappearance and contribute to the leveling of hierarchies of surveillance depending upon the surveillant assemblage that they reconfigure. Overall, this intervention urges surveillance studies scholarship to attend to how synthetic data reconfigure specific surveillant assemblages, with both problematic and emancipatory implications.

Semantic segmentation is a pixel-wise classification task, which is to predict class label to every pixel within an image. However, one of the obstacles limiting the development of semantic segmentation is that the pixel-wise segmentation annotations for the training images are quite difficult and expensive to obtain. Thus, it is difficult to apply the segmentation methods to new datasets or semantic classes since it is labor-intensive to obtain new annotated data. The target of this thesis is to reduce the annotation work load for semantic segmentation tasks on images. From the view of data domain, the approaches could be roughly classified into intra-domain approaches and inter-domain approaches. The category of intra-domain approaches is to utilize weaker level of supervisions within the datasets of same data domain. The supervision format of image-level labels for weakly supervised semantic segmentation is investigated in this thesis. To recover the pixel-wise annotations from image-level labels, region-mining models are trained to approximate the target object regions. The goal is to train the region-mining models which could highlight the integral object regions instead of only the most discriminative regions. In this thesis, I investigate regularizing the region-mining model both in the forward pass and the backward pass of the training process. The category of inter-domain approaches is to transfer the pixel-wise knowledge from another domain, whose data and pixel-wise annotations are easier to generate, to the target data domain. This thesis investigates the case of transferring from the synthetic source data with pixel-wise annotations to the real-world unlabeled target data. Adversarial learning approaches are applied to narrow the domain gap between the synthetic data and real-world data. Adversarial learning approaches usually suffer from the problem of content misalignment. To alleviate the content misalignment problem, two approaches are proposed in this thesis to regularize adversarial learning methods: the first is to embed the global structure knowledge into the feature-level adversarial learning step. The second is to back-propagate the final task loss into the pixel-wise adversarial learning step. This thesis presents the methods of both the categories to alleviate the need of pixelwise annotations for semantic image segmentation. The experiments show that the proposed methods could achieve promising segmentation performance without utilizing the pixel-wise annotations.

Summary One of the major challenges related to the use of machine learning for seismic processing and seismic interpretation is the notion of acquiring appropriate training data where we do not have the “ground truth”. Here, we present a novel approach to create survey adapted synthetic training data with cycleGANs. CycleGANs can be trained to translate between domains without paired input-output examples, and it has been applied to a wide range of applications, including collection style transfer, object transfiguration, season transfer and photo enhancement. We train a cycleGAN to translate from simple synthetic reflectivity examples (i.e. noise free and not convolved with a wavelet) to “real” or survey adapted examples, and the idea is that the survey adaption will automatically generate suitable synthetic training data for a given survey without any manual preconditioning or engineering. The approach can be used to generate training data for many different problems within seismic interpretation and seismic processing. We demonstrate the approach with a multiple attenuation case example on seismic pre-stack data.

Page segmentation and table detection play an important role in understanding the structure of documents. We present a page segmentation algorithm that incorporates state-of-the-art deep learning methods for segmenting three types of document elements: text blocks, tables, and figures. We propose a multi-scale, multi-task fully convolutional neural network (FCN) for the tasks of semantic page segmentation and element contour detection. The semantic segmentation network accurately predicts the probability at each pixel of the three element classes. The contour detection network accurately predicts instance level "edges" around each element occurrence. We propose a conditional random field (CRF) that uses features output from the semantic segmentation and contour networks to improve upon the semantic segmentation network output. Given the semantic segmentation output, we also extract individual table instances from the page using some heuristic rules and a verification network to remove false positives. We show that although we only consider a page image as input, we produce comparable results with other methods that relies on PDF file information and heuristics and hand crafted features tailored to specific types of documents. Our approach learns the representative features for page segmentation from real and synthetic training data. %, and produces good results on real documents. The learning-based property makes it a more general method than existing methods in terms of document types and element appearances. For example, our method reliably detects sparsely lined tables which are hard for rule-based or heuristic methods.

Visual recognition, encompassing tasks such as image classification, object detection, and semantic segmentation, has been a fundamental challenge in computer vision research. It also serves as the foundation for numerous applications, including autonomous driving, remote sensing, and robotics. The emergence of deep learning has significantly advanced this field, enabling remarkable achievements through the use of end-to-end trainable deep neural networks (DNNs). However, this remarkable progress comes at the cost of requiring vast amounts of annotated training data, the collection of which is often extremely labor-intensive and time-consuming. This constraint presents a significant barrier to the scalability and adaptability of various deep learning approaches. Transfer learning, which aims to transfer the off-the-shelf learned knowledge from one or multiple related source domains to target domains, is a promising way to mitigate this constraint. In this thesis, we explore label-efficient visual recognition with various transfer learning techniques, where the objective is to learn effective visual representations for target domains without any annotations by transferring the off-the-shelf knowledge learned in different but related source domains. Specifically, we investigate three representative types of transfer learning problems, including unsupervised domain adaptation, black-box unsupervised domain adaptation and test-time prompt tuning. As one of the most typical transfer learning problems, unsupervised domain adaptation aims to adapt a visual recognition model that is trained on labeled source-domain data to perform well on a related but unlabeled target-domain data. The primary challenge is to mitigate the distribution differences between the source and target domains, which are also referred to as domain gap or domain discrepancy, such that the model can generalize well on the target domain. In this thesis, we propose three innovative unsupervised domain adaptation techniques designed to efficiently transfer knowledge from labeled source domains to unlabeled target domains across a range of visual recognition tasks. Specifically, we explore diverse approaches to align domains from multiple perspectives (e.g., data or feature space alignment), thereby reducing the domain gap and enabling to perform well on unlabeled target domain. Black-box unsupervised domain adaptation aims to learn visual recognition model for target domain with the initial predictions of target data obtained by black-box source-domain model. In another word, during the adaptation process, only the model predictions of target data are available while either source data or source models are inaccessible. Compared to conventional unsupervised domain adaptation, black-box unsupervised domain adaptation offers notable advantages in terms of data privacy and flexibility, allowing adaptation to different target models regardless of the architecture or details of the source-trained black-box model. In this thesis, we analyze the challenges in black-box domain adaptation and design a novel approach for effectively exploiting the initial predictions of target data, leading to better adapted model on unlabeled target data. We further explore label-efficient visual recognition in the recent era of vision-language models. Despite the impressive zero-shot generalization capabilities of vision-language models, they always face unneglectable gaps while applied to target datasets, i.e., a target dataset typically features unique image styles and text formats tailored to its specific tasks. To circumvent this limitation, we explore an effective transfer learning approach for vision-language models, i.e., test-time prompt tuning, where the prompts are optimized as learnable vectors during the test stage without requiring any annotations for target data. Specifically, we examine the limitation of existing test-time prompt tuning methods, and propose a novel approach that could effectively exploit the learned knowledge in vision-language models and learn prompts for target data on-the-fly. Extensive experiments conducted over various visual recognition benchmarks indicate that our proposed methods achieve superior performance over target datasets, enabling to largely mitigate the dependence of labeled data across various visual recognition tasks.

In recent years, there has been a growing interest in synthetic data for several computer vision applications, such as automotive, detection and tracking, surveillance, medical image analysis and robotics. Early use of synthetic data was aimed at performing controlled experiments under the analysis by synthesis approach. Currently, synthetic data are mainly used for training computer vision models, especially deep learning ones, to address well-known issues of real data, such as manual annotation effort, data imbalance and bias, and privacy-related restrictions. In this work, we survey the use of synthetic training data focusing on applications related to video surveillance, whose relevance has rapidly increased in the past few years due to their connection to security: crowd counting, object and pedestrian detection and tracking, behaviour analysis, person re-identification and face recognition. Synthetic training data are even more interesting in this kind of application, to address further, specific issues arising, e.g., from typically unconstrained image or video acquisition conditions and cross-scene application scenarios. We categorise and discuss the existing methods for creating synthetic data, analyse the synthetic data sets proposed in the literature for each of the considered applications, and provide an overview of their effectiveness as training data. We finally discuss whether and to what extent the existing synthetic data sets mitigate the issues of real data, highlight existing open issues, and suggest future research directions in this field.