Prediction of human behavior with synthetic data

Abstract

The truly relevant dataset creation tasks are aimed at assessing human action. The approach to detect and recognize a person falling allows to promptly warn about dangerous incidents for further analysis of the consequences that can help in the tasks of healthcare industry. The problem can be solved using modeled data based on digital human simulation. The result of a proposed modular pipeline for synthetic data generation of digital human interaction with the 3D environment was demonstrated in this paper. The research includes the following contributions: the synthetic dataset based on procedural generation of realistic movements and fall which taking into account physics model of a digital human; registering basic rgb and segmentation rendering maps while simulating a digital human fall; in segmentation maps, we present hitting coordinate masks with the interaction of the human model and 3D scene. The pipeline is implemented using Unreal Engine that provides automatic “playback” of various scenarios for simulation. We used the generated synthetic data to train the Mask R-CNN framework. It is shown that a fallen person can be recognized with an accuracy of 97.6% and the type of person’s impact can be classified, including hitting the head when falling with training the model on simulation data. The proposed method also allows covering a variety of scenarios that can have a positive effect at a CNN training stage in the tasks of data creation for human action estimation.