Large-scale Training Data Research Articles

Efficient Photoacoustic Image Synthesis with Deep Learning.

Photoacoustic imaging potentially allows for the real-time visualization of functional human tissue parameters such as oxygenation but is subject to a challenging underlying quantification problem. While in silico studies have revealed the great potential of deep learning (DL) methodology in solving this problem, the inherent lack of an efficient gold standard method for model training and validation remains a grand challenge. This work investigates whether DL can be leveraged to accurately and efficiently simulate photon propagation in biological tissue, enabling photoacoustic image synthesis. Our approach is based on estimating the initial pressure distribution of the photoacoustic waves from the underlying optical properties using a back-propagatable neural network trained on synthetic data. In proof-of-concept studies, we validated the performance of two complementary neural network architectures, namely a conventional U-Net-like model and a Fourier Neural Operator (FNO) network. Our in silico validation on multispectral human forearm images shows that DL methods can speed up image generation by a factor of 100 when compared to Monte Carlo simulations with 5×108 photons. While the FNO is slightly more accurate than the U-Net, when compared to Monte Carlo simulations performed with a reduced number of photons (5×106), both neural network architectures achieve equivalent accuracy. In contrast to Monte Carlo simulations, the proposed DL models can be used as inherently differentiable surrogate models in the photoacoustic image synthesis pipeline, allowing for back-propagation of the synthesis error and gradient-based optimization over the entire pipeline. Due to their efficiency, they have the potential to enable large-scale training data generation that can expedite the clinical application of photoacoustic imaging.

Map of Land Cover Agreement: Ensambling Existing Datasets for Large-Scale Training Data Provision

Land cover information plays a critical role in supporting sustainable development and informed decision-making. Recent advancements in satellite data accessibility, computing power, and satellite technologies have boosted large-extent high-resolution land cover mapping. However, retrieving a sufficient amount of reliable training data for the production of such land cover maps is typically a demanding task, especially using modern deep learning classification techniques that require larger training sample sizes compared to traditional machine learning methods. In view of the above, this study developed a new benchmark dataset called the Map of Land Cover Agreement (MOLCA). MOLCA was created by integrating multiple existing high-resolution land cover datasets through a consensus-based approach. Covering Sub-Saharan Africa, the Amazon, and Siberia, this dataset encompasses approximately 117 billion 10m pixels across three macro-regions. The MOLCA legend aligns with most of the global high-resolution datasets and consists of nine distinct land cover classes. Noteworthy advantages of MOLCA include a higher number of pixels as well as coverage for typically underrepresented regions in terms of training data availability. With an estimated overall accuracy of 96%, MOLCA holds great potential as a valuable resource for the production of future high-resolution land cover maps.

Physics-added neural networks: An image-based deep learning for material printing system

Drop-on-demand material printing condition optimization with empirical knowledge based on the trial-and-error method and implementation of computational fluid dynamics (CFD) requires a significant amount of time and costs. In addition, current state-of-the-art neural network algorithms can reduce such time and cost by predicting droplet formation over a temporal domain; however, insufficient data and physical inconsistencies across output results limit their extrapolative prediction performance. In this study, we performed multi-GPU-implemented CFD simulations to collect large-scale training data, and we propose a novel algorithm for embedding physics knowledge in neural networks: physics-added neural networks. Integrating an image-to-image deep learning algorithm with our novel physics-embedding module, we obtain more consistent physical characteristics of the material printing process, such as mass and energy. Furthermore, by guiding the neural network to adhere to mass and energy conservation laws, the possible solution space of neural network training is restricted, which enhances convergence stability and decreases training time.

Exploring the potential of Siamese network for RGBT object tracking

Siamese tracking is one of the most promising object tracking methods today due to its balance of performance and speed. However, it still performs poorly when faced with some challenges such as low light or extreme weather. This is caused by the inherent limitations of visible images, and a common way to cope with it is to introduce infrared data as an aid to improve the robustness of tracking. However, most of the existing RGBT trackers are variants of MDNet (Hyeonseob Nam and Bohyung Han, Learning multi-domain convolutional neural networks for visual tracking, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4293–4302.), which have significant limitations in terms of operational efficiency. On the contrary, the potential of Siamese tracking in the field of RGBT tracking has not been effectively exploited due to the reliance on large-scale training data. To solve this dilemma, in this paper, we propose an end-to-end Siamese RGBT tracking framework that is based on cross-modal feature enhancement and self-attention (SiamFEA). We draw on the idea of migration learning and employ local fine-tuning to reduce the dependence on large-scale RGBT data and verify the feasibility of this approach, and then we propose a reliable fusion approach to efficiently fuse the features of different modalities. Specifically, we first propose a cross-modal feature enhancement module to exploit the complementary properties of dual-modality, followed by capturing non-local attention in channel and spatial dimensions for adaptive weighted fusion, respectively. Our network was trained end-to-end on the LasHeR (Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking, CoRR abs/2104.13202, 2021) training set and reached new SOTAs on GTOT (C. Li, H. Cheng, S. Hu, X. Liu, J. Tang, L. Lin, Learning collaborative sparse representation for grayscale-thermal tracking, IEEE Trans. Image Process, 25 (12) (2016) 5743–5756.), RGBT234 (C. Li, X. Liang, Y. Lu, N. Zhao, and J. Tang, “Rgb-t object tracking: Benchmark and baseline,” Pattern Recognition, vol. 96, p. 106977, 2019.), and LasHeR (Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking, CoRR abs/2104.13202, 2021) while running in real-time.

Fast CO2 saturation simulations on large-scale geomodels with artificial intelligence-based Wavelet Neural Operators

Carbon capture and storage (CCS) projects rely on numerical reservoir simulations for determining where and how much CO2 can safely be stored underground. Conventional simulations are computationally expensive and time consuming, thereby increasing the turn-around time of decision-making processes and projects. Recent research on artificial intelligence (AI) based approaches for solving the underlying two-phase flow equations has shown enormous promise, as the computational burden can be shifted to the training time, while simulations can be performed in fractions of a second at test time. Applications of AI methods for simulating CO2 flow and other differential equations have so far been limited to 2D or small 3D problems with relatively simple geometries. We introduce a cloud-native framework for parallel simulations of large-scale training data sets using the Open Porous Media (OPM) simulator and we propose a new AI architecture based on Fourier Neural Operators (FNOs) with 3D wavelet transforms. This architecture performs better than FNOs on data with shock fronts, such as simulating CO2 saturation. We also provide a new training dataset for CO2 flow based on the Sleipner CO2 storage geomodel from the Norwegian continental shelf and show that both FNOs and Wavelet Neural Operators (WNOs) can be trained on models with over 200,000 cells and lead to significant speed-ups of up to 50,000× (FNOs) and 100× (Wavelet NOs) over numerical simulations of CO2 saturation with OPM.

EzDPS: An Efficient and Zero-Knowledge Machine Learning Inference Pipeline

Machine Learning as a service (MLaaS) permits resource-limited clients to access powerful data analytics services ubiquitously. Despite its merits, MLaaS poses significant concerns regarding the integrity of delegated computation and the privacy of the server’s model parameters. To address this issue, Zhang et al. (CCS'20) initiated the study of zero-knowledge Machine Learning (zkML). Few zkML schemes have been proposed afterward; however, they focus on sole ML classification algorithms that may not offer satisfactory accuracy or require large-scale training data and model parameters, which may not be desirable for some applications. We propose ezDPS, a new efficient and zero-knowledge ML inference scheme. Unlike prior works, ezDPS is a zkML pipeline in which the data is processed in multiple stages for high accuracy. Each stage of ezDPS is harnessed with an established ML algorithm that is shown to be effective in various applications, including Discrete Wavelet Transformation, Principal Components Analysis, and Support Vector Machine. We design new gadgets to prove ML operations effectively. We fully implemented ezDPS and assessed its performance on real datasets. Experimental results showed that ezDPS achieves one-to-three orders of magnitude more efficient than the generic circuit-based approach in all metrics while maintaining more desirable accuracy than single ML classification approaches.

ISTD-PDS7: A Benchmark Dataset for Multi-Type Pavement Distress Segmentation from CCD Images in Complex Scenarios

The lack of large-scale, multi-scene, and multi-type pavement distress training data reduces the generalization ability of deep learning models in complex scenes, and limits the development of pavement distress extraction algorithms. Thus, we built the first large-scale dichotomous image segmentation (DIS) dataset for multi-type pavement distress segmentation, called ISTD-PDS7, aimed to segment highly accurate pavement distress types from natural charge-coupled device (CCD) images. The new dataset covers seven types of pavement distress in nine types of scenarios, along with negative samples with texture similarity noise. The final dataset contains 18,527 images, which is many more than the previously released benchmarks. All the images are annotated with fine-grained labels. In addition, we conducted a large benchmark test, evaluating seven state-of-the-art segmentation models, providing a detailed discussion of the factors that influence segmentation performance, and making cross-dataset evaluations for the best-performing model. Finally, we investigated the effectiveness of negative samples in reducing false positive prediction in complex scenes and developed two potential data augmentation methods for improving the segmentation accuracy. We hope that these efforts will create promising developments for both academics and the industry.

A Study on the Effectiveness of Deep Learning-Based Anomaly Detection Methods for Breast Ultrasonography.

In the medical field, it is delicate to anticipate good performance in using deep learning due to the lack of large-scale training data and class imbalance. In particular, ultrasound, which is a key breast cancer diagnosis method, is delicate to diagnose accurately as the quality and interpretation of images can vary depending on the operator's experience and proficiency. Therefore, computer-aided diagnosis technology can facilitate diagnosis by visualizing abnormal information such as tumors and masses in ultrasound images. In this study, we implemented deep learning-based anomaly detection methods for breast ultrasound images and validated their effectiveness in detecting abnormal regions. Herein, we specifically compared the sliced-Wasserstein autoencoder with two representative unsupervised learning models autoencoder and variational autoencoder. The anomalous region detection performance is estimated with the normal region labels. Our experimental results showed that the sliced-Wasserstein autoencoder model outperformed the anomaly detection performance of others. However, anomaly detection using the reconstruction-based approach may not be effective because of the occurrence of numerous false-positive values. In the following studies, reducing these false positives becomes an important challenge.

Siamese PointNet: 3D Head Pose Estimation with Local Feature Descriptor

Head pose estimation is an important part of the field of face analysis technology. It can be applied to driver attention monitoring, passenger monitoring, effective information screening, etc. However, illumination changes and partial occlusion interfere with the task, and due to the non-stationary characteristic of the head pose change process, normal regression networks are unable to achieve very accurate results on large-scale synthetic training data. To address the above problems, a Siamese network based on 3D point clouds was proposed, which adopts a share weight network with similar pose samples to constrain the regression process of the pose’s angles; meanwhile, a local feature descriptor was introduced to describe the local geometric features of the objects. In order to verify the performance of our method, we conducted experiments on two public datasets: the Biwi Kinect Head Pose dataset and Pandora. The results show that compared with the latest methods, our standard deviation was reduced by 0.4, and the mean error was reduced by 0.1; meanwhile, our network also maintained a good real-time performance.

Tropical Cyclone Wind Speed Estimation: A Large Scale Training Data Set and Community Benchmarking

AbstractTropical cyclones (TCs) cause significant disruptions to infrastructure and livelihood. The scale of loss due to TCs may be mitigated by prompt and accurate advisories about TC wind speed. Current advisories are consensus based and have a time delay of about 6 hr between each new update. As part of efforts to increase the frequency of wind speed estimations without compromising on the accuracy, we curate a high‐quality hurricane image and wind speed data set and develop a machine learning (ML) model for frequent wind speed estimation. The image and wind speed pairings are consolidated from different sources. The wind speeds are interpolated hourly to curate the TC wind speed estimation data set. The data set is supplemented with metadata that eases adoption of the data set by the ML community. We have also designed a competition to garner interest and encourage community involvement to build state‐of‐the‐art ML models to estimate wind speeds from satellite images of TCs. The competition winners were able to improve the wind speed estimation by almost 50% over the benchmark model.

Spatio-temporal deep forest for emotion recognition based on facial electromyography signals

Emotion recognition is a key component of human–computer interaction technology, for which facial electromyogram (fEMG) is an important physiological modality. Recently, deep-learning-based emotion recognition using fEMG signals has drawn increased attention. However, the ability of effective feature extraction and the demand of large-scale training data are two dominant factors that restrict the performance of emotion recognition. In this paper, a novel spatio-temporal deep forest (STDF) model is proposed to classify three categories of discrete emotions (neutral, sadness, and fear) using multi-channel fEMG signals.The feature extraction module fully extracts effective spatio-temporal features of fEMG signals using a combination of 2D frame sequences and multi-grained scanning. Meanwhile, a cascade forest-based classifier is designed to provide optimal structures for different scales of training data via automatically adjusting the number of cascade layers.The proposed model and five comparison methods were evaluated on our in-house fEMG dataset that included three discrete emotions and three channels of fEMG electrodes with a total of twenty-seven subjects. Experimental results demonstrate that the proposed STDF model achieves the best recognition performance with an average accuracy of 97.41%. Besides, our proposed STDF model can reduced the scale of training data to 50% while the average accuracy of emotion recognition is only reduced by about 5%. Our proposed model offers an effective solution for practical applications of fEMG-based emotion recognition.

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Vision transformers have shown great potential in various computer vision tasks owing to their strong capability to model long-range dependency using the self-attention mechanism. Nevertheless, they treat an image as a 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance, which is instead learned implicitly from large-scale training data with longer training schedules. In this paper, we leverage the two IBs and propose the ViTAE transformer, which utilizes a reduction cell for multi-scale feature and a normal cell for locality. The two kinds of cells are stacked in both isotropic and multi-stage manners to formulate two families of ViTAE models, i.e., the vanilla ViTAE and ViTAEv2. Experiments on the ImageNet dataset as well as downstream tasks on the MS COCO, ADE20K, and AP10K datasets validate the superiority of our models over the baseline and representative models. Besides, we scale up our ViTAE model to 644 M parameters and obtain the state-of-the-art classification performance, i.e., 88.5% Top-1 classification accuracy on ImageNet validation set and the best 91.2% Top-1 classification accuracy on ImageNet Real validation set, without using extra private data. It demonstrates that the introduced inductive bias still helps when the model size becomes large. The source code and pretrained models are publicly available atcode.

Toward 3D scene reconstruction from locally scale-aligned monocular video depth

Monocular depth estimation methods have achieved excellent robustness on diverse scenes, usually by predicting affine-invariant depth, up to an unknown scale and shift, rather than metric depth in that it is much easier to collect large-scale affine-invariant depth training data. However, in some video-based scenarios such as video depth estimation and 3D scene reconstruction, the unknown scale and shift residing in per-frame prediction may cause the predicted depth to be inconsistent. To tackle this problem, we propose a locally weighted linear regression method to recover the scale and shift map with very sparse anchor points, which ensures the consistency along consecutive frames. Extensive experiments show that our method can drop the Rel error (relative error) of existing state-of-the-art approaches significantly over several zero-shot benchmarks. Besides, we merge 6.3 million RGBD images to train robust depth models. By locally recovering scale and shift, our produced ResNet50-backbone model even outperforms the state-of-the-art DPT ViT-Large model. Combined with geometry-based reconstruction methods, we formulate a new dense 3D scene reconstruction pipeline, which benefits from both the scale consistency of sparse points and the robustness of monocular methods. By performing simple per-frame prediction over a video, the accurate 3D scene geometry can be recovered.

SiamON: Siamese Occlusion-Aware Network for Visual Tracking

Occlusion has been proven to be one of the most challenging factors faced by most visual trackers. There are mainly two difficulties, the first one is that the number of occlusion samples are very limited even though collecting a large-scale training data set, and another one is how to correctly learn the features of the target when comes to occlusion situations. In this paper, we tried to solve these two problems together in our proposed model. To this end, we propose a novel Siamese Occlusion-aware Network (SiamON) for high-performance visual tracking. In particular, we predefine some soft-masks to solve the problem of fewer occlusion samples, which perceive patterns of occlusion contents at different locations and take these masks as the conditions to guide occlusion-aware feature learning. Meanwhile, we propose a target-aware attention mechanism allows the model to pay more attention to the target and further weaken the impact of occlusion. Extensive experiments on several popular benchmarks show that our tracking method exceeds many state-of-the-art trackers especially in the presence of occlusion and meets the requirements of real-time.

Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition.

In this paper, we present Vision Permutator, a conceptually simple and data efficient MLP-like architecture for visual recognition. By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections. This allows Vision Permutator to capture long-range dependencies and meanwhile avoid the attention building process in transformers. The outputs are then aggregated in a mutually complementing manner to form expressive representations. We show that our Vision Permutators are formidable competitors to convolutional neural networks (CNNs) and vision transformers. Without the dependence on spatial convolutions or attention mechanisms, Vision Permutator achieves 81.5% top-1 accuracy on ImageNet without extra large-scale training data (e.g., ImageNet-22k) using only 25M learnable parameters, which is much better than most CNNs and vision transformers under the same model size constraint. When scaling up to 88M, it attains 83.2% top-1 accuracy, greatly improving the performance of recent state-of-the-art MLP-like networks for visual recognition. We hope this work could encourage research on rethinking the way of encoding spatial information and facilitate the development of MLP-like models. PyTorch/MindSpore/Jittor code is available at https://github.com/Andrew-Qibin/VisionPermutator.

STAR-FC: Structure-Aware Face Clustering on Ultra-Large-Scale Graphs.

Face clustering is a promising method for annotating unlabeled face images. Recent supervised approaches have boosted the face clustering accuracy greatly, however their performance is still far from satisfactory. These methods can be roughly divided into global-based and local-based ones. Global-based methods suffer from the limitation of training data scale, while local-based ones are inefficient for inference due to the use of numerous overlapped subgraphs. Previous approaches fail to tackle these two challenges simultaneously. To address the dilemma of large-scale training and efficient inference, we propose the STructure-AwaRe Face Clustering (STAR-FC) method. Specifically, we design a structure-preserving subgraph sampling strategy to explore the power of large-scale training data, which can increase the training data scale from 105 to 107. On this basis, a novel hierarchical GCN training paradigm is further proposed for better capturing the dynamic local structure. During inference, the STAR-FC performs efficient full-graph clustering with two steps: graph parsing and graph refinement. And the concept of node intimacy is introduced in the second step to mine the local structural information, where a calibration module is further proposed for fairer edge scores. The STAR-FC gets 93.21 pairwise F-score on standard partial MS1M within 312 seconds, which far surpasses the state-of-the-arts while maintaining high inference efficiency. Furthermore, we are the first to train on an ultra-large-scale graph with 20M nodes, and achieve superior inference results on 12M testing data. Overall, as a simple and effective method, the proposed STAR-FC provides a strong baseline for large-scale face clustering. Code is available in https://github.com/sstzal/STAR-FC.

Faster Domain Adaptation Networks

It is widely acknowledged that the success of deep learning is built upon large-scale training data and tremendous computing power. However, the data and computing power are not always available for many real-world applications. In this paper, we address the machine learning problem where it lacks training data and limits computing power. Specifically, we investigate domain adaptation which is able to transfer knowledge from one labeled source domain to an unlabeled target domain, so that we do not need much training data from the target domain. At the same time, we consider the situation that the running environment is confined, e.g., in edge computing the end device has very limited running resources. Technically, we present the Faster Domain Adaptation (FDA) protocol and further report two paradigms of FDA: early stopping and amid skipping. The former accelerates domain adaptation by multiple early exit points. The latter speeds up the adaptation by wisely skip several amid neural network blocks. Extensive experiments on standard benchmarks verify that our method is able to achieve the comparable and even better accuracy but employ much less computing resources. To the best of our knowledge, there are very few works which investigated accelerating knowledge adaptation in the community. This work is expected to inspire the topic for more discussion.

LSCIDMR: Large-Scale Satellite Cloud Image Database for Meteorological Research.

People can infer the weather from clouds. Various weather phenomena are linked inextricably to clouds, which can be observed by meteorological satellites. Thus, cloud images obtained by meteorological satellites can be used to identify different weather phenomena to provide meteorological status and future projections. How to classify and recognize cloud images automatically, especially with deep learning, is an interesting topic. Generally speaking, large-scale training data are essential for deep learning. However, there is no such cloud images database to date. Thus, we propose a large-scale cloud image database for meteorological research (LSCIDMR). To the best of our knowledge, it is the first publicly available satellite cloud image benchmark database for meteorological research, in which weather systems are linked directly with the cloud images. LSCIDMR contains 104 390 high-resolution images, covering 11 classes with two different annotation methods: 1) single-label annotation and 2) multiple-label annotation, called LSCIDMR-S and LSCIDMR-M, respectively. The labels are annotated manually, and we obtain a total of 414 221 multiple labels and 40 625 single labels. Several representative deep learning methods are evaluated on the proposed LSCIDMR, and the results can serve as useful baselines for future research. Furthermore, experimental results demonstrate that it is possible to learn effective deep learning models from a sufficiently large image database for the cloud image classification.

Comparing models of learning and relearning in large-scale cognitive training data sets

Practice in real-world settings exhibits many idiosyncracies of scheduling and duration that can only be roughly approximated by laboratory research. Here we investigate 39,157 individuals’ performance on two cognitive games on the Lumosity platform over a span of 5 years. The large-scale nature of the data allows us to observe highly varied lengths of uncontrolled interruptions to practice and offers a unique view of learning in naturalistic settings. We enlist a suite of models that grow in the complexity of the mechanisms they postulate and conclude that long-term naturalistic learning is best described with a combination of long-term skill and task-set preparedness. We focus additionally on the nature and speed of relearning after breaks in practice and conclude that those components must operate interactively to produce the rapid relearning that is evident even at exceptionally long delays (over 2 years). Naturalistic learning over long time spans provides a strong test for the robustness of theoretical accounts of learning, and should be more broadly used in the learning sciences.

Anomaly segmentation in retinal images with poisson-blending data augmentation

Diabetic retinopathy (DR) is one of the most important complications of diabetes. Accurate segmentation of DR lesions is of great importance for the early diagnosis of DR. However, simultaneous segmentation of multi-type DR lesions is technically challenging because of 1) the lack of pixel-level annotations and 2) the large diversity between different types of DR lesions. In this study, first, we propose a novel Poisson-blending data augmentation (PBDA) algorithm to generate synthetic images, which can be easily utilized to expand the existing training data for lesion segmentation. We perform extensive experiments to recognize the important attributes in the PBDA algorithm. We show that position constraints are of great importance and that the synthesis density of one type of lesion has a joint influence on the segmentation of other types of lesions. Second, we propose a convolutional neural network architecture, named DSR-U-Net++ (i.e., DC-SC residual U-Net++), for the simultaneous segmentation of multi-type DR lesions. Ablation studies showed that the mean area under precision recall curve (AUPR) for all four types of lesions increased by >5% with PBDA. The proposed DSR-U-Net++ with PBDA outperformed the state-of-the-art methods by 1.7%-9.9% on the Indian Diabetic Retinopathy Image Dataset (IDRiD) and 67.3% on the e-ophtha dataset with respect to mean AUPR. The developed method would be an efficient tool to generate large-scale task-specific training data for other medical anomaly segmentation tasks.