Generation Pipeline Research Articles

AugmentA: Patient-specific augmented atrial model generation tool.

Digital twins of patients' hearts are a promising tool to assess arrhythmia vulnerability and to personalize therapy. However, the process of building personalized computational models can be challenging and requires a high level of human interaction. We propose a patient-specific Augmented Atria generation pipeline (AugmentA) as a highly automated framework which, starting from clinical geometrical data, provides ready-to-use atrial personalized computational models. AugmentA identifies and labels atrial orifices using only one reference point per atrium. If the user chooses to fit a statistical shape model to the input geometry, it is first rigidly aligned with the given mean shape before a non-rigid fitting procedure is applied. AugmentA automatically generates the fiber orientation and finds local conduction velocities by minimizing the error between the simulated and clinical local activation time (LAT) map. The pipeline was tested on a cohort of 29 patients on both segmented magnetic resonance images (MRI) and electroanatomical maps of the left atrium. Moreover, the pipeline was applied to a bi-atrial volumetric mesh derived from MRI. The pipeline robustly integrated fiber orientation and anatomical region annotations in 38.4±5.7s. In conclusion, AugmentA offers an automated and comprehensive pipeline delivering atrial digital twins from clinical data in procedural time.

Single Pixel Spectral Color Constancy

Color constancy is still one of the biggest challenges in camera color processing. Convolutional neural networks have been able to improve the situation but there are still problems in many conditions, especially in scenes where a single color is dominating. In this work, we approach the problem from a slightly different setting. What if we could have some other information than the raw RGB image data. What kind of information would help to bring significant improvements while still be feasible in a mobile device. These questions sparked an idea for a novel approach for computational color constancy. Instead of raw RGB images used by the existing algorithms to estimate the scene white points, our approach is based on the scene’s average color spectra-single pixel spectral measurement. We show that as few as 10–14 spectral channels are sufficient. Notably, the sensor output has five orders of magnitude less data than in raw RGB images of a 10MPix camera. The spectral sensor captures the “spectral fingerprints” of different light sources and the illuminant white point can be accurately estimated by a standard regressor. The regressor can be trained with generated measurements using the existing RGB color constancy datasets. For this purpose, we propose a spectral data generation pipeline that can be used if the dataset camera model is known and thus its spectral characterization can be obtained. To verify the results with real data, we collected a real spectral dataset with a commercial spectrometer. On all datasets the proposed Single Pixel Spectral Color Constancy obtains the highest accuracy in the both single and cross-dataset experiments. The method is particularly effective for the difficult scenes for which the average improvements are 40–70% compared to state-of-the-arts. The approach can be extended to multi-illuminant case for which the experimental results also provide promising results.

REALITY-ENHANCED SYNTHETIC IMAGE TRAINING DATASET FOR COMPUTER VISION CONSTRUCTION MONITORING

Recent growth in the availability of aerial and ground-level images from construction sites has created a surge in computer vision-based techniques for construction monitoring applications. To address the need for large volumes of visual data with ground truth for training the underlying machine learning models, the application of synthetic data, particularly from 3D/4D BIM, has gained traction in the research community. However, the existing domain gap between real and synthetic images, such as rendering noise from repeated material texture patterns, randomized positional camera parameters that collect the ground truth of BIM elements, and the poorly visibility of BIM elements to the camera viewpoints within these environments, has negatively impacted the potential use of these synthetic datasets as scale. To address these limitations, this paper presents a new synthetic image generation pipeline that optimizes and integrates camera extrinsic parameters with a novel synthetic image appearance enhancement technique to generate high volume of quality synthetic data with respective ground-truth. The use of synthetic datasets for training progress monitoring models is validated through several real-data segmentation cases that incorporate: 1) the automatic collection of synthetic images and ground-truth annotations from high-LoD BIM model disciplines, 2) optimization of positional camera parameters using element visibility metrics, and 3) the enhancement of realism of synthetic images using a patch-based generative approach. The benefits and the current limitations to automated progress monitoring, as well as robotic path automation and optimization for progress motoring real data collection are discussed.

HeartDIS: A Generalizable End-to-End Energy Disaggregation Pipeline

The need for a more energy-efficient future is now more evident than ever. Energy disagreggation (NILM) methodologies have been proposed as an effective solution for the reduction in energy consumption. However, there is a wide range of challenges that NILM faces that still have not been addressed. Herein, we propose HeartDIS, a generalizable energy disaggregation pipeline backed by an extensive set of experiments, whose aim is to tackle the performance and efficiency of NILM models with respect to the available data. Our research (i) shows that personalized machine learning models can outperform more generic models; (ii) evaluates the generalization capabilities of these models through a wide range of experiments, highlighting the fact that the combination of synthetic data, the decreased volume of real data, and fine-tuning can provide comparable results; (iii) introduces a more realistic synthetic data generation pipeline based on other state-of-the-art methods; and, finally, (iv) facilitates further research in the field by publicly sharing synthetic and real data for the energy consumption of two households and their appliances.

One‐to‐Many Spectral Upsampling of Reflectances and Transmittances

AbstractSpectral rendering is essential for the production of physically‐plausible synthetic images, but requires to introduce several changes in the content generation pipeline. In particular, the authoring of spectral material properties (e.g., albedo maps, indices of refraction, transmittance coefficients) raises new problems. While a large panel of computer graphics methods exists to upsample a RGB color to a spectrum, they all provide a one‐to‐one mapping. This limits the ability to control interesting color changes such as the Usambara effect or metameric spectra. In this work, we introduce a one‐to‐many mapping in which we show how we can explore the set of all spectra reproducing a given input color. We apply this method to different colour changing effects such as vathochromism – the change of color with depth, and metamerism.

Improving traffic time‐series predictability by imputing continuous non‐random missing data

AbstractContinuous non‐random data missing can be a challenging task for model prediction in intelligent transport system (ITS). In ITS, many methods have been proposed to solve this problem. However, the imputation accuracy of them is far from accurate. Thus, the authors propose a novel cross‐modality generative adversarial network, named as cross‐modality GAN, to impute continuous non‐random missing data from the cross‐modality perspective. This model uses the cross‐modality data fusion technique to fuse spatial and temporal modal data into vectorized features, and then imputes the target unseen missing data by a data generation pipeline. Different from the other existing models, this model overcomes the problem of zero observation data, and realizes long‐term missing time series imputation. Many comparative experiments are conducted. The results verify that the cross‐modality GAN achieves better imputation performances on Performance Measurement System (PeMS) dataset, a real public traffic dataset, compared to other baseline models. Furthermore, the results verify that the imputed data of cross‐modality GAN can provide more traffic time‐series predictability information, and improve prediction accuracy of prediction models effectively.

Early Outcomes of the Pipeline Vantage Flow Diverter : AMulticentre Study.

The recently introduced Pipeline Vantage Embolization Device with Shield Technology is the fourth generation of Pipeline flow diverter devices. Due to the relatively high rate of intraprocedural technical complications, modifications were subsequently made to the device after alimited release of the device in 2020. This study aimed to evaluate the safety and efficacy of the modified version of this device. This was amulticentre retrospective series. The primary efficacy endpoint was aneurysm occlusion in the absence of retreatment. The primary safety endpoint was any neurological morbidity or death. Ruptured and unruptured aneurysms were included in the study. A total of 52procedures were performed for 60target aneurysms. Treatment was performed on 5patients with ruptured aneurysms. The technical success rate was 98%. The mean clinical follow-up time was 5.5months. In patients presenting with unruptured aneurysms there were no deaths, 3 (6.4%) major complications and 7 (13%) minor complications. In the five patients presenting with subarachnoid haemorrhage there were 2 (40%) major complications with 1 (20%) of these resulting in death, and 1 (20%) minor complication. Of the patients 29(56%) had undergone 6‑monthly postprocedural angiographic imaging with amean time of 6.6months demonstrating that 83% of patients had achieved adequate occlusion (RROC1/2) of the aneurysm. In this non-industry-sponsored study, the occlusion rates and safety outcomes were similar to those seen in previously published studies with flow diverter devices and earlier generation Pipeline devices. Modifications to the device appear to have improved ease of deployment.

Detecting and Grounding Important Characters in Visual Stories

Characters are essential to the plot of any story. Establishing the characters before writing a story can improve the clarity of the plot and the overall flow of the narrative. However, previous work on visual storytelling tends to focus on detecting objects in images and discovering relationships between them. In this approach, characters are not distinguished from other objects when they are fed into the generation pipeline. The result is a coherent sequence of events rather than a character-centric story. In order to address this limitation, we introduce the VIST-Character dataset, which provides rich character-centric annotations, including visual and textual co-reference chains and importance ratings for characters. Based on this dataset, we propose two new tasks: important character detection and character grounding in visual stories. For both tasks, we develop simple, unsupervised models based on distributional similarity and pre-trained vision-and-language models. Our new dataset, together with these models, can serve as the foundation for subsequent work on analysing and generating stories from a character-centric perspective.

Generative Pipeline for Data Augmentation of Unconstrained Document Images with Structural and Textural Degradation (Student Abstract)

Generative Pipeline for Data Augmentation of Unconstrained Document Images with Structural and Textural Degradation (Student Abstract)

Non-rigid point cloud registration for middle ear diagnostics with endoscopic optical coherence tomography

PurposeMiddle ear infection is the most prevalent inflammatory disease, especially among the pediatric population. Current diagnostic methods are subjective and depend on visual cues from an otoscope, which is limited for otologists to identify pathology. To address this shortcoming, endoscopic optical coherence tomography (OCT) provides both morphological and functional in vivo measurements of the middle ear. However, due to the shadow of prior structures, interpretation of OCT images is challenging and time-consuming. To facilitate fast diagnosis and measurement, improvement in the readability of OCT data is achieved by merging morphological knowledge from ex vivo middle ear models with OCT volumetric data, so that OCT applications can be further promoted in daily clinical settings.MethodsWe propose C2P-Net: a two-staged non-rigid registration pipeline for complete to partial point clouds, which are sampled from ex vivo and in vivo OCT models, respectively. To overcome the lack of labeled training data, a fast and effective generation pipeline in Blender3D is designed to simulate middle ear shapes and extract in vivo noisy and partial point clouds.ResultsWe evaluate the performance of C2P-Net through experiments on both synthetic and real OCT datasets. The results demonstrate that C2P-Net is generalized to unseen middle ear point clouds and capable of handling realistic noise and incompleteness in synthetic and real OCT data.ConclusionsIn this work, we aim to enable diagnosis of middle ear structures with the assistance of OCT images. We propose C2P-Net: a two-staged non-rigid registration pipeline for point clouds to support the interpretation of in vivo noisy and partial OCT images for the first time. Code is available at: https://gitlab.com/nct_tso_public/c2p-net.

Left Atrial Ligation in the Avian Embryo as a Model for Altered Hemodynamic Loading During Early Vascular Development.

Due to its four-chambered mature ventricular configuration, ease of culture, imaging access, and efficiency, the avian embryo is a preferred vertebrate animal model for studying cardiovascular development. Studies aiming to understand the normal development and congenital heart defect prognosis widely adopt this model. Microscopic surgical techniques are introduced to alter the normal mechanical loading patterns at a specific embryonic time point and track the downstream molecular and genetic cascade. The most common mechanical interventions are left vitelline vein ligation, conotruncal banding, and left atrial ligation (LAL), modulating the intramural vascular pressure and wall shear stress due to blood flow. LAL, particularly if performed in ovo, is the most challenging intervention, with very small sample yields due to the extremely fine sequential microsurgical operations. Despite its high risk, in ovo LAL is very valuable scientifically as it mimics hypoplastic left heart syndrome (HLHS) pathogenesis. HLHS is a clinically relevant, complex congenital heart disease observed in human newborns. A detailed protocol for in ovo LAL is documented in this paper. Briefly, fertilized avian embryos were incubated at 37.5 °C and 60% constant humidity typically until they reached Hamburger-Hamilton (HH) stages 20 to 21. The egg shells were cracked open, and the outer and inner membranes were removed. The embryo was gently rotated to expose the left atrial bulb of the common atrium. Pre-assembled micro-knots from 10-0 nylon sutures were gently positioned and tied around the left atrial bud. Finally, the embryo was returned to its original position, and LAL was completed. Normal and LAL-instrumented ventricles demonstrated statistically significant differences in tissue compaction. An efficient LAL model generation pipeline would contribute to studies focusing on synchronized mechanical and genetic manipulation during the embryonic development of cardiovascular components. Likewise, this model will provide a perturbed cell source for tissue culture research and vascular biology.

Detecting Road Damages in Mobile Mapping Point Clouds using Competitive Reconstruction Networks

Abstract. LiDAR scanning technology is an established method for capturing landscapes, buildings, or roads in order to create a so-called spatial digital twin of the reality, stored as a large collection of 3D coordinates called 3D point cloud. This spatial data offers high density and precision at the cost of hard to extract shape or object information. One popular application of LiDAR 3D point clouds is road condition quality exams. This task is challenging due to a lack of dedicated algorithms to extract and evaluate road point cloud features and due to the large variety of road damages. Deep learning approaches are very promising, but require extensive training data. The data and damage characteristics make data labeling a very difficult and tedious task that often results in mislabeled data, even when performed by trained human operators.We propose a semi supervised generative adversarial network (GAN) based approach for labeling 2D images rendered from LiDAR point cloud data captured by mobile mapping vehicles, named Competitive Reconstruction Networks (CRN). Our solution trains multiple networks with the same architecture in an ”all vs all” fashion. Our method achieves reliable and robust results on two road image datasets as well as the MVTecAD dataset, and surpass comparable anomaly detection approaches in anomaly detection performance. We also implemented a data generation pipeline to render training images from 3D point cloud of roads and remap anomaly scores back to those 3D point clouds to use the full potential of the 3D data for further analysis.

Adaptive search query generation and refinement in systematic literature review

Systematic literature reviews (SLRs) are a central part of evidence-based research, which involves collecting and integrating empirical evidence on specific research questions. A key step in this process is building Boolean search queries, which are at the core of information retrieval systems that support literature search. This involves turning general research aims into specific search terms that can be combined into complex Boolean expressions. Researchers must build and refine search queries to ensure they have sufficient coverage and properly represent the literature. In this paper, we propose an adaptive query generation and refinement pipeline for SLR search that uses reinforcement learning to learn the optimal modifications to a query based on feedback from researchers about its performance. Empirical evaluations with 10 SLR datasets showed our approach achieves comparable performance to queries manually composed by SLR authors. We also investigate the impact of design decisions on the performance of the query generation and refinement pipeline. Specifically, we study the effects of the type of input seed, the use of general versus domain-specific word embedding models, the sampling strategy for relevance feedback, and number of iterations in the refinement process. Our results provide insights into the effects of these choices on the pipeline’s performance.

IngridKG: A FAIR Knowledge Graph of Graffiti

Graffiti is an urban phenomenon that is increasingly attracting the interest of the sciences. To the best of our knowledge, no suitable data corpora are available for systematic research until now. The Information System Graffiti in Germany project (Ingrid) closes this gap by dealing with graffiti image collections that have been made available to the project for public use. Within Ingrid, the graffiti images are collected, digitized and annotated. With this work, we aim to support the rapid access to a comprehensive data source on Ingrid targeted especially by researchers. In particular, we present IngridKG, an RDF knowledge graph of annotated graffiti, abides by the Linked Data and FAIR principles. We weekly update IngridKG by augmenting the new annotated graffiti to our knowledge graph. Our generation pipeline applies RDF data conversion, link discovery and data fusion approaches to the original data. The current version of IngridKG contains 460,640,154 triples and is linked to 3 other knowledge graphs by over 200,000 links. In our use case studies, we demonstrate the usefulness of our knowledge graph for different applications.

Automatic item generation: foundations and machine learning-based approaches for assessments

This mini review summarizes the current state of knowledge about automatic item generation in the context of educational assessment and discusses key points in the item generation pipeline. Assessment is critical in all learning systems and digitalized assessments have shown significant growth over the last decade. This leads to an urgent need to generate more items in a fast and efficient manner. Continuous improvements in computational power and advancements in methodological approaches, specifically in the field of natural language processing, provide new opportunities as well as new challenges in automatic generation of items for educational assessment. This mini review asserts the need for more work across a wide variety of areas for the scaled implementation of AIG.

Generative AI and Algorithmic Art: Disrupting the Framing of Meaning and Rethinking the Subject-Object Dilemma

In the revision of treatments of contemporary art in the 21st century, art historians are recognizing 2022 as the dawn of the age of creative artificial intelligence (AI). The emergence of generative AI tools like ChatGPT and Stable Diffusion in late 2022 immediately disrupted the established practices of the art world, leading to debates about the validity of "AI Art" and the emergence of a new market for NFTs. However, fears regarding the "death of the artist" are unwarranted when considering the historical adoption of new technologies by artists, such as photography. The role of the artist will undoubtedly transform, and the definition of "art" will be redefined once again. To better understand how AI generative art will impact traditional art-making practices, this study will present an AI generative art development pipeline and provide recommendations for future technical and theoretical considerations of the subject-object dilemma in art through a post structuralist reading of reception theory.

A high throughput bispecific antibody discovery pipeline

Bispecific antibodies (BsAbs) represent an emerging class of immunotherapy, but inefficiency in the current discovery has limited their broad clinical availability. Here we report a high throughput, agnostic, single-cell-based functional screening pipeline, comprising molecular and cell engineering for efficient generation of BsAb library cells, followed by functional interrogation at the single-cell level to identify and sort positive clones and downstream sequence identification and functionality characterization. Using a CD19xCD3 bispecific T cell engager (BiTE) as a model, we demonstrate that our single-cell platform possesses a high throughput screening efficiency of up to one and a half million variant library cells per run and can isolate rare functional clones at a low abundance of 0.008%. Using a complex CD19xCD3 BiTE-expressing cell library with approximately 22,300 unique variants comprising combinatorially varied scFvs, connecting linkers and VL/VH orientations, we have identified 98 unique clones, including extremely rare ones (~ 0.001% abundance). We also discovered BiTEs that exhibit novel properties and insights to design variable preferences for functionality. We expect our single-cell platform to not only increase the discovery efficiency of new immunotherapeutics, but also enable identifying generalizable design principles based on an in-depth understanding of the inter-relationships between sequence, structure, and function.

A Stack-Propagation Framework for Low-Resource Personalized Dialogue Generation

With the resurgent interest in building open-domain dialogue systems, the dialogue generation task has attracted increasing attention over the past few years. This task is usually formulated as a conditional generation problem, which aims to generate a natural and meaningful response given dialogue contexts and specific constraints, such as persona. And maintaining a consistent persona is essential for the dialogue systems to gain trust from the users. Although tremendous advancements have been brought, traditional persona-based dialogue models are typically trained by leveraging a large number of persona-dense dialogue examples. Yet, such persona-dense training data are expensive to obtain, leading to a limited scale. This work presents a novel approach to learning from limited training examples by regarding consistency understanding as a regularization of response generation. To this end, we propose a novel stack-propagation framework for learning a generation and understanding pipeline. Specifically, the framework stacks a Transformer encoder and two Transformer decoders, where the first decoder models response generation and the second serves as a regularizer and jointly models response generation and consistency understanding. The proposed framework can benefit from the stacked encoder and decoders to learn from much smaller personalized dialogue data while maintaining competitive performance. Under different low-resource settings, subjective and objective evaluations prove that the stack-propagation framework outperforms strong baselines in response quality and persona consistency and largely overcomes the shortcomings of traditional models that rely heavily on the persona-dense dialogue data.

Hybrid synthetic data generation pipeline that outperforms real data

Fine-tuning a pretrained model with real data for a machine learning task requires many hours of manual work, especially for computer vision tasks, where collection and annotation of data can be very time-consuming. We present a framework and methodology for synthetic data collection that is not only efficient in terms of time taken to collect and annotate data, making use of free- and open-source software tools and 3D assets but also beats the state-of-the-art against real data, which is the ultimate test for any similar-to-real approach. We test our approach on a set of image classes from ObjectNet, which is a challenging image classification benchmark test dataset that is designed to be similar in many respects to ImageNet but with a wider variety of viewpoints, rotations, and backgrounds, which can make it more difficult for transfer learning problems. The novelty of our approach stems from the way we create complex backgrounds for 3D models using 2D images laid out as decals in a 3D game engine, where synthetic images are captured programmatically with a large number of systematic variations. We demonstrate that our approach is highly effective, resulting in a deep learning model with a top-1 accuracy of 72% on the ObjectNet data, which is a new state-of-the-art result. In addition, we present an efficient strategy for learning rate tuning that is an order of magnitude faster than regular grid search.

Automatic Image Generation Pipeline for Instance Segmentation of Deformable Linear Objects.

Robust detection of deformable linear objects (DLOs) is a crucial challenge for the automation of handling and assembly of cables and hoses. The lack of training data is a limiting factor for deep-learning-based detection of DLOs. In this context, we propose an automatic image generation pipeline for instance segmentation of DLOs. In this pipeline, a user can set boundary conditions to generate training data for industrial applications automatically. A comparison of different replication types of DLOs shows that modeling DLOs as rigid bodies with versatile deformations is most effective. Further, reference scenarios for the arrangement of DLOs are defined to generate scenes in a simulation automatically. This allows the pipelines to be quickly transferred to new applications. The validation of models trained with synthetic images and tested on real-world images shows the feasibility of the proposed data generation approach for segmentation of DLOs. Finally, we show that the pipeline yields results comparable to the state of the art but has advantages in reduced manual effort and transferability to new use cases.