ESS MS-G3D: extension and supplement shift MS-G3D network for the assessment of severe mental retardation

Abstract

Automated mental retardation (MR) assessment is potential for improving the diagnostic efficiency and objectivity in clinical practice. Based on the researches on abnormal behavior characteristics of patients with MR, we propose an extension and supplement shift multi-scale G3D (ESS MS-G3D) network for video-based assessment of MR. Specifically, all videos are collected from clinical diagnostic scenarios and the skeleton sequence of human body is extracted from videos through an advanced pose estimation model. To solve the shortcomings of existing behavior characteristic learning methods, we present: (1) three G3D styles, enable the network to have different input forms; (2) two G3D graphs and two extension graphs, redefine and extend the graph structure of spatial–temporal nodes; (3) two learnable parameters, realize adaptive adjustment of graph structure; (4) a shift layer, enable the network to learn global features. Finally, we construct a three-branch model ESS MS-STGC, which can capture the discriminative spatial–temporal features and explore the co-occurrence relationship between spatial and temporal domains. Experiments in clinical video data set show that our proposed model has good performance in MR assessment and is superior to the existing vision-based methods. In two-classification task, our model with joint stream achieves the highest accuracy of 94.63%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$94.63\\%$$\\end{document} in validation set and 89.13%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$89.13\\%$$\\end{document} in test set. The results are further improved to 96.52%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$96.52\\%$$\\end{document} and 93.22%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$93.22\\%$$\\end{document}, respectively, by utilizing multi-stream fusion strategy. In four-classification task, our model obtains Top1 accuracy of 78.84%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$78.84\\%$$\\end{document} and Top2 accuracy of 91.34%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$91.34\\%$$\\end{document} in test set. The proposed method provides a new idea for clinical mental retardation assessment.