DeepConnection: classifying momentary relationship state from images of romantic couples

Abstract

Detecting momentary relationship state and quality in romantic couples is an important endeavor for relationship research, couple therapy, and of course couples themselves. Yet current methods to achieve this are intrusive, asynchronous, plagued by ceiling effects, and only assess subjective responses to questionnaires while trying to capture the objective state of a relationship. According to social appraisal theory, human beings rely on emotional responses to assess interpersonal situations, a key element for relationship functioning in couples. Using couples is particularly advantageous as strong emotional reactions are triggered in romantic relationships. Here, we employ deep learning methods to assess the momentary relationship state of romantic couples from predominantly stock images via facial and bodily emotion expression and other features. Our new model, DeepConnection, comprises pre-trained residual neural networks, spatial pyramid pooling layers, and power mean transformations to extract relevant features from images for binary classification. With this, we achieved an average accuracy of nearly 97% on a separate validation dataset. We also engaged in model interpretation using Gradient-weighted Class Activation Mapping (Grad-CAM) to identify which features allow DeepConnection to detect binarized momentary relationship state. To demonstrate generalizability and robustness, we used DeepConnection to analyze videos of couples exhibiting a range of different postures and facial expressions. Here, we achieved an average accuracy of about 85% with a trained DeepConnection model. The work presented here could inform couples, advance relationship research, and find application in couple therapy to assist the therapist.