Large-scale Synthetic Dataset Research Articles

Review on synergizing the Metaverse and AI-driven synthetic data: enhancing virtual realms and activity recognition in computer vision

The Metaverse’s emergence is redefining digital interaction, enabling seamless engagement in immersive virtual realms. This trend’s integration with AI and virtual reality (VR) is gaining momentum, albeit with challenges in acquiring extensive human action datasets. Real-world activities involve complex intricate behaviors, making accurate capture and annotation difficult. VR compounds this difficulty by requiring meticulous simulation of natural movements and interactions. As the Metaverse bridges the physical and digital realms, the demand for diverse human action data escalates, requiring innovative solutions to enrich AI and VR capabilities. This need is underscored by state-of-the-art models that excel but are hampered by limited real-world data. The overshadowing of synthetic data benefits further complicates the issue. This paper systematically examines both real-world and synthetic datasets for activity detection and recognition in computer vision. Introducing Metaverse-enabled advancements, we unveil SynDa’s novel streamlined pipeline using photorealistic rendering and AI pose estimation. By fusing real-life video datasets, large-scale synthetic datasets are generated to augment training and mitigate real data scarcity and costs. Our preliminary experiments reveal promising results in terms of mean average precision (mAP), where combining real data and synthetic video data generated using this pipeline to train models presents an improvement in mAP (32.35%), compared to the mAP of the same model when trained on real data (29.95%). This demonstrates the transformative synergy between Metaverse and AI-driven synthetic data augmentation.

Enhancing Oil Recovery Predictions by Leveraging Polymer Flooding Simulations and Machine Learning Models on a Large-Scale Synthetic Dataset

Enhancing Oil Recovery Predictions by Leveraging Polymer Flooding Simulations and Machine Learning Models on a Large-Scale Synthetic Dataset

Development of large-scale synthetic 3D point cloud datasets for vision-based bridge structural condition assessment

Visual recognition of 3D point cloud data of bridge inspection scenes is a key step in automating the visual inspection process, which is currently largely manual and inefficient. To alleviate the lack of large-scale annotated point cloud datasets for training such 3D visual recognition algorithms, this research investigates an approach for developing large-scale synthetic point cloud datasets. The proposed approach proceeds in four steps: (1) random generation of different types of bridges in computer graphics environments; (2) sampling of camera trajectories that represent data collection scenarios during bridge inspection; (3) 3D reconstruction using Structure from Motion (SfM) applied to rendered synthetic images; (4) automated annotation of the reconstructed point cloud using ground truth masks obtained with synthetic images. Besides, this research proposes to store point uncertainty information defined by the error between the ground truth depth and the depth calculated from the SfM results. Prior to training, thresholds can be applied to this uncertainty information to control the levels of outliers in the dataset. This research demonstrates the proposed approach by generating point cloud datasets for two data collection scenarios. The effectiveness of the generated datasets is investigated by training 3D semantic segmentation algorithms and evaluating the performance on real and synthetic point cloud data. The proposed approach for point cloud dataset generation will facilitate the development of generalizable and high level-of-detail 3D recognition algorithms toward autonomous bridge inspection.

ModelNet-O: A large-scale synthetic dataset for occlusion-aware point cloud classification

Recently, 3D point cloud classification has made significant progress with the help of many datasets. However, these datasets do not reflect the incomplete nature of real-world point clouds caused by occlusion, which limits the practical application of current methods. To bridge this gap, we propose ModelNet-O, a large-scale synthetic dataset of 123,041 samples that emulates real-world point clouds with self-occlusion caused by scanning from monocular cameras. ModelNet-O is 10 times larger than existing datasets and offers more challenging cases to evaluate the robustness of existing methods. Our observation on ModelNet-O reveals that well-designed sparse structures can preserve structural information of point clouds under occlusion, motivating us to propose a robust point cloud processing method that leverages a critical point sampling (CPS) strategy in a multi-level manner. We term our method PointMLS. Through extensive experiments, we demonstrate that our PointMLS achieves state-of-the-art results on ModelNet-O and competitive results on regular datasets such as ModelNet40 and ScanObjectNN, and we also demonstrate its robustness and effectiveness. Code available: https://github.com/fanglaosi/ModelNet-O_PointMLS.

BrepMFR: Enhancing machining feature recognition in B-rep models through deep learning and domain adaptation

Feature Recognition (FR) plays a crucial role in modern digital manufacturing, serving as a key technology for integrating Computer-Aided Design (CAD), Computer-Aided Process Planning (CAPP) and Computer-Aided Manufacturing (CAM) systems. The emergence of deep learning methods in recent years offers a new approach to address challenges in recognizing highly intersecting features with complex geometric shapes. However, due to the high cost of labeling real CAD models, neural networks are usually trained on computer-synthesized datasets, resulting in noticeable performance degradation when applied to real-world CAD models. Therefore, we propose a novel deep learning network, BrepMFR, designed for Machining Feature Recognition (MFR) from Boundary Representation (B-rep) models. We transform the original B-rep model into a graph representation as network-friendly input, incorporating local geometric shape and global topological relationships. Leveraging a graph neural network based on Transformer architecture and graph attention mechanism, we extract the feature representation of high-level semantic information to achieve machining feature recognition. Additionally, employing a two-step training strategy under a transfer learning framework, we enhance BrepMFR's generalization ability by adapting synthetic training data to real CAD data. Furthermore, we establish a large-scale synthetic CAD model dataset inclusive of 24 typical machining features, showcasing diversity in geometry that closely mirrors real-world mechanical engineering scenarios. Extensive experiments across various datasets demonstrate that BrepMFR achieves state-of-the-art machining feature recognition accuracy and performs effectively on CAD models of real-world mechanical parts.

Transformer-Based No-Reference Image Quality Assessment via Supervised Contrastive Learning

Image Quality Assessment (IQA) has long been a research hotspot in the field of image processing, especially No-Reference Image Quality Assessment (NR-IQA). Due to the powerful feature extraction ability, existing Convolution Neural Network (CNN) and Transformers based NR-IQA methods have achieved considerable progress. However, they still exhibit limited capability when facing unknown authentic distortion datasets. To further improve NR-IQA performance, in this paper, a novel supervised contrastive learning (SCL) and Transformer-based NR-IQA model SaTQA is proposed. We first train a model on a large-scale synthetic dataset by SCL (no image subjective score is required) to extract degradation features of images with various distortion types and levels. To further extract distortion information from images, we propose a backbone network incorporating the Multi-Stream Block (MSB) by combining the CNN inductive bias and Transformer long-term dependence modeling capability. Finally, we propose the Patch Attention Block (PAB) to obtain the final distorted image quality score by fusing the degradation features learned from contrastive learning with the perceptual distortion information extracted by the backbone network. Experimental results on six standard IQA datasets show that SaTQA outperforms the state-of-the-art methods for both synthetic and authentic datasets. Code is available at https://github.com/I2-Multimedia-Lab/SaTQA.

Joint Segmentation and Identification Feature Learning for Occlusion Face Recognition.

The existing occlusion face recognition algorithms almost tend to pay more attention to the visible facial components. However, these models are limited because they heavily rely on existing face segmentation approaches to locate occlusions, which is extremely sensitive to the performance of mask learning. To tackle this issue, we propose a joint segmentation and identification feature learning framework for end-to-end occlusion face recognition. More particularly, unlike employing an external face segmentation model to locate the occlusion, we design an occlusion prediction module supervised by known mask labels to be aware of the mask. It shares underlying convolutional feature maps with the identification network and can be collaboratively optimized with each other. Furthermore, we propose a novel channel refinement network to cast the predicted single-channel occlusion mask into a multi-channel mask matrix with each channel owing a distinct mask map. Occlusion-free feature maps are then generated by projecting multi-channel mask probability maps onto original feature maps. Thus, it can suppress the representation of occlusion elements in both the spatial and channel dimensions under the guidance of the mask matrix. Moreover, in order to avoid misleading aggressively predicted mask maps and meanwhile actively exploit usable occlusion-robust features, we aggregate the original and occlusion-free feature maps to distill the final candidate embeddings by our proposed feature purification module. Lastly, to alleviate the scarcity of real-world occlusion face recognition datasets, we build large-scale synthetic occlusion face datasets, totaling up to 980193 face images of 10574 subjects for the training dataset and 36721 face images of 6817 subjects for the testing dataset, respectively. Extensive experimental results on the synthetic and real-world occlusion face datasets show that our approach significantly outperforms the state-of-the-art in both 1:1 face verification and 1:N face identification.

Layout-aware Single-image Document Flattening

Single image rectification of document deformation is a challenging task. Although some recent deep learning-based methods have attempted to solve this problem, they cannot achieve satisfactory results when dealing with document images with complex deformations. In this article, we propose a new efficient framework for document flattening. Our main insight is that most layout primitives in a document have rectangular outline shapes, making unwarping local layout primitives essentially homogeneous with unwarping the entire document. The former task is clearly more straightforward to solve than the latter due to the more consistent texture and relatively smooth deformation. On this basis, we propose a layout-aware deep model working in a divide-and-conquer manner. First, we employ a transformer-based segmentation module to obtain the layout information of the input document. Then a new regression module is applied to predict the global and local UV maps. Finally, we design an effective merging algorithm to correct the global prediction with local details. Both quantitative and qualitative experimental results demonstrate that our framework achieves favorable performance against state-of-the-art methods. In addition, the current publicly available document flattening datasets have limited 3D paper shapes without layout annotation and also lack a general geometric correction metric. Therefore, we build a new large-scale synthetic dataset by utilizing a fully automatic rendering method to generate deformed documents with diverse shapes and exact layout segmentation labels. We also propose a new geometric correction metric based on our paired document UV maps. Code and dataset will be released at https://github.com/BunnySoCrazy/LA-DocFlatten .

Deep learning based multi-view stereo matching and 3D scene reconstruction from oblique aerial images

In this paper, we propose a practical three-dimensional (3D) real-scene reconstruction framework named Deep3D, which is paired with a deep learning based multi-view stereo (MVS) matching model named the adaptive multi-view aggregation matching (Ada-MVS) model, to obtain a 3D textured mesh model from multi-view oblique aerial images. Deep3D is the first deep learning based framework for 3D scene reconstruction, in which aerial triangulation and view selection are first performed on the input images, and the depth map of each image is then inferred using the pretrained Ada-MVS model. All the inferred depth maps are then fused into a dense point cloud after filtering the outliers. Finally, the 3D textured mesh is extracted from the dense 3D points as the final product. In the Ada-MVS model, a novel adaptive inter-view aggregation module is specially proposed to address the inconsistent information among oblique views and to fuse the multi-view costs into a robust cost volume. A lightweight recurrent regularization module is also designed for high-efficiency processing of high-capacity aerial images with large depth variations. Moreover, as oblique aerial image datasets are currently lacking, we built a large-scale synthetic multi-view oblique aerial image dataset (WHU-OMVS dataset) for deep learning based model training and methodology evaluation for the task of 3D scene reconstruction. The experimental results show that, firstly, the proposed Ada-MVS model has obvious advantages when used with high-capacity oblique aerial images, compared with several relevant learning-based MVS methods. Secondly, through a comprehensive comparison with popular commercial software packages and open-source solutions, it is shown that the proposed Deep3D framework outperforms all the other solutions in terms of reconstruction quality, and outperforms all the open-source solutions and some of the software packages in terms of efficiency on the WHU-OMVS dataset. Thirdly, the Deep3D framework shows a stable generalization ability and excellent performance when applied to other oblique or nadir aerial images, without any further fine-tuning. The dataset and code will be available at http://gpcv.whu.edu.cn/data.

Frame-Recurrent Video Crowd Counting

Since video data contains temporal information, video crowd counting demonstrates more potential than single-frame crowd counting for scenarios requiring high accuracy. However, learning robust relationships among frames efficiently and cheaply is very challenging. Existing methods for video crowd counting lack explicit temporal correlation modeling and robustness, and they are complex. In this paper, we propose the Frame-Recurrent Video Crowd Counting (FRVCC) framework to solve these issues. Specifically, we design a frame-recurrent manner to recursively relate the density maps in the temporal dimension, which efficiently explores long-term inter-frame knowledge and ensures the continuity of feature map responses. FRVCC consists of three plug-in modules: an optical flow estimation module, a single-frame counting module, and a density map fusion module. For the fusion module, we propose the ResTrans network to robustly learn complementary features between visual-based and correlation-based feature maps through residual strategy and vision transformer. To constrain the output distribution to be consistent with the ground truth distribution, we introduce an adversarial loss to rectify the training process. Additionally, we release a large-scale synthetic video crowd-counting dataset, CrowdXV, to evaluate the proposed method and further improve its performance. We have conducted extensive experiments on several video-counting datasets. The results demonstrate that FRVCC achieves state-of-the-art performance and, concurrently, high generalization, high flexibility, and less complexity.

A Large-scale Virtual Dataset and Egocentric Localization for Disaster Responses.

With the increasing social demands of disaster response, methods of visual observation for rescue and safety have become increasingly important. However, because of the shortage of datasets for disaster scenarios, there has been little progress in computer vision and robotics in this field. With this in mind, we present the first large-scale synthetic dataset of egocentric viewpoints for disaster scenarios. We simulate pre- and post-disaster cases with drastic changes in appearance, such as buildings on fire and earthquakes. The dataset consists of more than 300K high-resolution stereo image pairs, all annotated with ground-truth data for the semantic label, depth in metric scale, optical flow with sub-pixel precision, and surface normal as well as their corresponding camera poses. To create realistic disaster scenes, we manually augment the effects with 3D models using physically-based graphics tools. We train various state-of-the-art methods to perform computer vision tasks using our dataset, evaluate how well these methods recognize the disaster situations, and produce reliable results of virtual scenes as well as real-world images. We also present a convolutional neural network-based egocentric localization method that is robust to drastic appearance changes, such as the texture changes in a fire, and layout changes from a collapse. To address these key challenges, we propose a new model that learns a shape-based representation by training on stylized images, and incorporate the dominant planes of query images as approximate scene coordinates. We evaluate the proposed method using various scenes including a simulated disaster dataset to demonstrate the effectiveness of our method when confronted with significant changes in scene layout. Experimental results show that our method provides reliable camera pose predictions despite vastly changed conditions.

TGF-Net: Sim2Real Transparent Object 6D Pose Estimation Based on Geometric Fusion

Transparent objects are a common part of daily life, but their unique optical properties make estimating their 6D pose a challenging task. In this letter, we propose TGF-Net, a monocular instance-level 6D pose estimation method for transparent objects based on geometric fusion. TGF-Net learns the edge features and surface fragments of transparent objects as intermediate features and reduces the influence of appearance changes on the 6D pose estimation of transparent objects by fusing rich geometric features in the network. Moreover, we propose an approach for generating high-fidelity large-scale synthetic datasets of transparent objects using Blender and use this approach to generate a synthetic dataset Trans6D-32K. Trans6D-32K contains rendered RGB images and poses information about transparent objects in a variety of different backgrounds, perspectives, and lighting conditions. To evaluate the performance of TGF-Net on 6D pose estimation of transparent objects, we compare with multiple related works on the dataset Trans6D-32K. TGF-Net can be trained entirely on synthetic datasets without fine-tuning and applied directly to real-world scenarios. Multiple challenging real-scene experiments demonstrate the good performance of TGF-Net, while grasping experiments demonstrate the application value of TGF-Net in transparent object manipulation.

A Large-scale Synthetic Pathological Dataset for Deep Learning-enabled Segmentation of Breast Cancer

The success of training computer-vision models heavily relies on the support of large-scale, real-world images with annotations. Yet such an annotation-ready dataset is difficult to curate in pathology due to the privacy protection and excessive annotation burden. To aid in computational pathology, synthetic data generation, curation, and annotation present a cost-effective means to quickly enable data diversity that is required to boost model performance at different stages. In this study, we introduce a large-scale synthetic pathological image dataset paired with the annotation for nuclei semantic segmentation, termed as Synthetic Nuclei and annOtation Wizard (SNOW). The proposed SNOW is developed via a standardized workflow by applying the off-the-shelf image generator and nuclei annotator. The dataset contains overall 20k image tiles and 1,448,522 annotated nuclei with the CC-BY license. We show that SNOW can be used in both supervised and semi-supervised training scenarios. Extensive results suggest that synthetic-data-trained models are competitive under a variety of model training settings, expanding the scope of better using synthetic images for enhancing downstream data-driven clinical tasks.

Enhancing vehicle re-identification via synthetic training datasets and re-ranking based on video-clips information

Vehicle re-identification (ReID) aims to find a specific vehicle identity across multiple non-overlapping cameras. The main challenge of this task is the large intra-class and small inter-class variability of vehicles appearance, sometimes related with large viewpoint variations, illumination changes or different camera resolutions. To tackle these problems, we proposed a vehicle ReID system based on ensembling deep learning features and adding different post-processing techniques. In this paper, we improve that proposal by: incorporating large-scale synthetic datasets in the training step; performing an exhaustive ablation study showing and analyzing the influence of synthetic content in ReID datasets, in particular CityFlow-ReID and VeRi-776; and extending post-processing by including different approaches to the use of gallery video-clips of the target vehicles in the re-ranking step. Additionally, we present an evaluation framework in order to evaluate CityFlow-ReID: as this dataset has not public ground truth annotations, AI City Challenge provided an on-line evaluation service which is no more available; our evaluation framework allows researchers to keep on evaluating the performance of their systems in the CityFlow-ReID dataset.

Methods and Applications of Clusterwise Linear Regression: A Survey and Comparison

Clusterwise linear regression (CLR) is a well-known technique for approximating a data using more than one linear function. It is based on the combination of clustering and multiple linear regression methods. This article provides a comprehensive survey and comparative assessments of CLR including model formulations, description of algorithms, and their performance on small to large-scale synthetic and real-world datasets. Some applications of the CLR algorithms and possible future research directions are also discussed.

A refinement network embedded with attention mechanism for computer vision based post-earthquake inspections of railway viaduct

Post-earthquake inspection of structures based on computer vision is developing rapidly due to the advantages of high efficiency and without manual feature extraction. However, it is still necessary to investigate how to accurately recognize structural components and damage from the perspective of pixels. Fortunately, refinement network which named RefineNet has been developed for semantic segmentation of images, which helps to combine low-level features and high-level semantics to generate high resolution segmented images for efficient end-to-end learning. Therefore, RefineNet is used in this study as a network architecture for semantic segmentation tasks of recognizing railway viaducts components and damages. Moreover, it is proposed to embed the convolutional block attention mechanism in the down-sampling process of the RefineNet to extract image features, which helps the network to assign different weights to image regions of different importance and effectively improve the extraction effect of intermediate features. With the provided large-scale synthetic railway viaduct image dataset, which named Tokaido Dataset, the proposed RefineNet with Attention Mechanism (RefineNet-AM) is used for structural condition assessment of railway viaduct, including semantic segmentation tasks of components and damages of railway viaduct. Based on the test dataset, it is shown that proposed RefineNet-AM can inspect the structural components and damage of railway viaduct with satisfactory accuracy.

SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses.

Deep learning has demonstrated its power in image rectification by leveraging the representation capacity of deep neural networks via supervised training based on a large-scale synthetic dataset. However, the model may overfit the synthetic images and generalize not well on real-world fisheye images due to the limited universality of a specific distortion model and the lack of explicitly modeling the distortion and rectification process. In this paper, we propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of a same scene from different lenses should be the same. Specifically, we devise a new network architecture with a shared encoder and several prediction heads, each of which predicts the distortion parameter of a specific distortion model. We further leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters and exploit the intra- and inter-model consistency between them during training, thereby leading to a self-supervised learning scheme without the need for ground-truth distortion parameters or normal images. Experiments on synthetic dataset and real-world fisheye images demonstrate that our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art (SOTA) methods. The proposed self-supervised method also provides a possible way to improve the universality of distortion models while keeping their self-consistency. Code and datasets will be available at https://github.com/loong8888/SIR.

Scene point cloud understanding and reconstruction technologies in 3D space

Scene point cloud understanding and reconstruction technologies in 3D space

SDNSandbox — Enabling learning-based innovation in provider networks

Provider networks are looking to follow the footsteps of cloud-based networks/data centers and incorporate Software-Defined Networking (SDN) technology. This move is problematic for various reasons, such as the networks’ size and the providers’ inability to control users’ activity. Additionally, research into these networks is handicapped by the lack of information stemming from the confidentiality of these complex networks. To that end, we have created SDNSandbox — an SDN-based provider network simulator prototype. SDNSandbox is an open-source, easy-to-use, provider-network in-a-laptop simulator. It aims to facilitate the creation of reproducible experiments and large-scale synthetic datasets. In its current prototype form, it uses a basic traffic generator module alongside real-world provider topologies. SDNSandbox allows users to simulate provider networks, enabling them to conduct research in the field and examine practical applications. To demonstrate SDNSandbox, we use the prototype to simulate basic traffic conditions over several topologies. We then feed the generated datasets to DCRNN, a Convolutional Neural Network (CNN) traffic patterns prediction module. We adapt DCRNN to accept SDNSandbox output and show that it can predict traffic conditions at various points within the network tens of seconds into the future. We further compare its performance with other baseline algorithms. Our results demonstrate that SDNSandbox can also be used as a testbed for a digital twin, creating datasets that are hard to replicate in production networks. It also serves as a demonstration of the framework’s power and versatility as a modular research tool.

Learning large-scale fuzzy cognitive maps under limited resources

Research on the problem of learning large-scale fuzzy cognitive maps (FCMs) with a limited computational budget is outstanding. To learn large-scale FCMs from time series, in most work, this problem is decomposed into learning local connections of each concept, respectively, and then one optimizer is employed to optimize each such sub-problem. Each sub-problem may have different requirements for the computational resource, but the existing methods ignore this issue and allocate the same amounts of computational resources for each sub-problem. In this paper, we propose two strategies to address this problem. We first develop a dynamic resource allocation strategy to maximize the performance of the decomposition-based optimizer under a limited computational budget. Second, we propose a half-thresholding memetic algorithm to improve the performance of the traditional evolutionary algorithm. We term our proposal as a half-thresholding memetic algorithm with a dynamic resource allocation strategy (HTMA-DRA). Finally, the experiments on large-scale synthetic data and DREAM datasets compared with the existing state-of-the-art methods demonstrate the effectiveness of the proposed HTMA-DRA.