MVAESynth: a unified framework for multimodal data generation, modality restoration, and controlled generation

Abstract

Synthetic data generation is used nowadays in a number of applications with privacy issues, such as training and testing of systems for analyzing the behavior of social network users or bank customers. Very often, personal data is complex and describes different aspects of a person, some of which may be missing for some records, which makes it very hard to deal with. In this paper, we present MVAESynth, a novel framework for the data-driven generation of multimodal synthetic data. It contains our implementation of a multimodal variational auto-encoder (MVAE), which is capable of generating user multimodal personal profiles (for example, social media profiles data and transactional data) and training even with missing modalities. Extensive experimental studies of MVAESynth performance were conducted demonstrating its effectiveness compared with the available solutions for the following tasks 1) training on data with missing modalities; 2) generating realistic social network profiles; 3) restoring missing profile modalities; 4) generating profiles with the specified characteristics.