Against the background of weakening face-to-face social interaction, the mental health of college students deserves attention. There are few existing studies on the impact of audiovisual interaction on interactive behavior, especially emotional perception in specific spaces. This study aims to indicate whether the perception of one’s music environment has influence on college students’ emotion during communication in different indoor conditions including spatial function, visual and sound atmospheres, and interior furnishings. The three-dimensional pleasure–arousal–dominance (PAD) emotional model was used to evaluate the changes of emotions before and after communication. An acoustic environmental measurement was performed and the evaluations of emotion during communication was investigated by a questionnaire survey with 331 participants at six experimental sites [including a classroom (CR), a learning corridor (LC), a coffee shop (CS), a fast food restaurant (FFR), a dormitory (DT), and a living room(LR)], the following results were found: Firstly, the results in different functional spaces showed no significant effect of music on communication or emotional states during communication. Secondly, the average score of the musical evaluation was 1.09 higher in the warm-toned space compared to the cold-toned space. Thirdly, the differences in the effects of music on emotion during communication in different sound environments were significant and pleasure, arousal, and dominance could be efficiently enhanced by music in the quiet space. Fourthly, dominance was 0.63 higher in the minimally furnished space. Finally, we also investigated influence of social characteristics on the effect of music on communication in different indoor spaces, in terms of the intimacy level, the gender combination, and the group size. For instance, when there are more than two communicators in the dining space, pleasure and arousal can be efficiently enhanced by music. This study shows that combining the sound environment with spatial factors (for example, the visual and sound atmosphere) and the interior furnishings can be an effective design strategy for promoting social interaction in indoor spaces.