Music videos, as a prevalent form of multimedia entertainment, deliver engaging audio-visual experiences to audiences and have gained immense popularity among singers and fans. Creators can express their interpretations of music naturally through visual elements. However, the creation process of music video demands proficiency in script design, video shooting, and music-video synchronization, posing significant challenges for non-professionals. Previous work has designed automated music video generation frameworks. However, they suffer from complexity in input and poor output quality. In response, we present MV-Crafter, a system capable of producing high-quality music videos with synchronized music-video rhythm and style. Our approach involves three technical modules that simulate the human creation process: the script generation module, video generation module, and music-video synchronization module. MV-Crafter leverages a large language model to generate scripts considering the musical semantics. To address the challenge of synchronizing short video clips with music of varying lengths, we propose a dynamic beat matching algorithm and visual envelope-induced warping method to ensure precise, monotonic music-video synchronization. Besides, we design a user-friendly interface to simplify the creation process with intuitive editing features. Extensive experiments have demonstrated that MV-Crafter provides an effective solution for improving the quality of generated music videos.

Personality primarily refers to the unique and stable way of a person’s thinking and behavior. A few studies have recently been conducted on personality recognition using physiological signals, most of which have used 2D emotional stimulus materials. Virtual reality (VR) has been utilized in many fields, and its superiority over 2D in emotion recognition has been proven. However, relevant research on VR scenes is lacking in the field of personality recognition. In this study, based on the psychological principle that emotional arousal can expose an individual’s personality, we attempt to explore the feasibility and effect of using electrocardiogram (ECG) signals in response to VR emotional stimuli for personality identification. For this purpose, a VR-2D emotion-induction experiment was conducted in which ECG signals were collected, and physiological datasets of emotional personalities were constructed through preprocessing and feature extraction. Statistical analysis of the emotion scale scores and ECG features of the participants showed that the VR group had a higher number of significantly correlated features. Meanwhile, VR- and 2D-based personality recognition models were constructed using machine learning algorithms. The results showed that the VR-based personality recognition model achieved better results for the four personality dimensions, with a maximum accuracy of 79.76%. These findings indicate that VR not only enhances the physiological correlation between emotion and personality but also improves the classification accuracy of personality recognition.

Conversational agents (CAs) are one of the most promising technologies for helping older adults maintain independence longer by augmenting their support and social networks. Voice-based technology in particular is especially powerful in this regard due to its accessibility and ease of use. There is also a growing body of evidence supporting the potential use of such technology in mitigating common issues such as loneliness and isolation, particularly for independent older adults aging in place. One of the key challenges for smart technologies deployed in this context is the development and maintenance of long-term user engagement and adoption, which is often addressed by attempting to closely mimic human social interactions. However, the more human-like the system, the more glaring fault conditions become, and the more jarring they are for users. In this study we explore the effectiveness of an alternative conversational strategy meant to encourage users to engage in positive reflection and introspection. We detail the iterative design and implementation of a prototype CA developed to engage in social conversation with older adults on selected topics of interest. We then use this system as part of a multi-method approach to investigate the effect of deliberate positivity as a conversational strategy, including its impact on user impressions and willingness to continue using the CA. Our results from different approaches, including methods such as psycholinguistic analysis, user self-report, and researcher-based coding, paint a promising picture of this conversational design. We show that the deliberate encouragement by a CA of positive conversation and reflection in users has a measurable positive impact on both user enjoyment and desire to continue engaging with a system. We further demonstrate how some user characteristics may amplify this effect, and discuss the implications of these results for the design and testing of future conversational systems for older adults.

Diversifying movie recommendations is an effective way to address choice overload, a phenomenon where recommenders generate lists with highly similar recommendations that are difficult to choose from. However, existing diversification algorithms often rely on latent features, which limits their interpretability and makes it less clear why a particular set of movies is recommended. Given that movies are designed to elicit emotional responses, researchers have suggested leveraging these responses to enhance recommender system performance. This study introduces a novel “emotion diversification” approach, which diversifies movie recommendations based on emotional signals extracted from audience reviews. We evaluate this method against latent and non-diversified baselines in a controlled user study (N = 115), finding that it significantly improves perceived taste coverage and system satisfaction without compromising recommendation quality. Going beyond the traditional rating- and/or interaction data used by traditional recommender systems, our work demonstrates the user experience benefits of extracting emotional data from rich, qualitative user feedback and using it to give users a more emotionally diverse set of recommendations.

The increasing adoption of machine learning (ML) raises ethical concerns, particularly regarding bias. This study explores how ML practitioners with limited experience in bias understand and apply bias definitions, detection measures, and mitigation methods. Through a take-home task, exercises, and interviews with 22 participants, we identified five key themes: sources of bias, selecting bias metrics, detecting bias, mitigating bias, and ethical considerations. Participants faced unresolved conflicts, such as applying fairness definitions in practice, selecting context-dependent bias metrics, addressing real-world biases, balancing model performance with bias mitigation, and relying on personal perspectives over data-driven metrics. While bias mitigation techniques helped identify biases in two datasets, participants could not fully eliminate bias, citing the oversimplification of complex processes into models with limited variables. We propose designing bias detection tools that encourage practitioners to focus on the underlying assumptions and integrating bias concepts into ML practices, such as using a harmonic mean-based approach, akin to the F1 score, to balance bias and accuracy.

Adversarial machine learning (AML) studies attacks that can fool machine learning algorithms into generating incorrect outcomes as well as the defenses against worst-case attacks to strengthen model robustness. Specifically for image classification, it is challenging to understand adversarial attacks due to their use of subtle perturbations that are not human-interpretable, as well as the variability of attack impacts influenced by diverse methodologies, instance differences, and model architectures. Through a design study with AML learners, and teachers, we introduce AdvEx , a multi-level interactive visualization system that comprehensively presents the properties and impacts of evasion attacks on different image classifiers for novice AML learners. We quantitatively and qualitatively assessed AdvEx in a two-part evaluation including user studies and expert interviews. Our results show that AdvEx is not only highly effective as a visualization tool for understanding AML mechanisms but also provides an engaging and enjoyable learning experience, thus demonstrating its overall benefits for AML learners.

We present HAIFAI—a novel two-stage system where humans and AI interact to tackle the challenging task of reconstructing a visual representation of a face that exists only in a person’s mind. In the first stage, users iteratively rank images our reconstruction system presents based on their resemblance to a mental image. These rankings, in turn, allow the system to extract relevant image features, fuse them into a unified feature vector and use a generative model to produce an initial reconstruction of the mental image. The second stage leverages an existing face editing method, allowing users to manually refine and further improve this reconstruction using an easy-to-use slider interface for face shape manipulation. To avoid the need for tedious human data collection for training the reconstruction system, we introduce a computational user model of human ranking behaviour. For this, we collected a small face ranking dataset through an online crowd-sourcing study containing data from 275 participants. We evaluate HAIFAI and an ablated version in a 12-participant user study and demonstrate that our approach outperforms the previous state of the art regarding reconstruction quality, usability, perceived workload and reconstruction speed. We further validate the reconstructions in a subsequent face ranking study with 18 participants and show that HAIFAI achieves a new state-of-the-art identification rate of 60.6%. These findings represent a significant advancement towards developing new interactive intelligent systems capable of reliably and effortlessly reconstructing a user’s mental image.

State-of-the-art agent-modelling tools support the creation of powerful Socially Intelligent Agents (SIAs) capable of engaging in social interactions with participants in various roles and environments. However, their deployment demands a labourious authoring task as it is necessary to manually define behaviour rules and create content for different interaction scenarios. While Socially Intelligent Agents (SIAs) research has centred on the user experience, we shift focus to the authors. To understand the challenges faced by authors who create these agents, we performed an innovative analysis of the authoring experience in modern agent modelling tools. One key finding is that, while SIA concepts are generally understandable, emotional-based concepts are not as easily comprehended or used by authors. We propose a hybrid solution approach that culminated in the development of Authoring-Assisted FAtiMA-Toolkit. The augmented agent modelling tool incorporates a data-driven Authoring Assistant to boost author productivity while promoting transparency and authorial control. To evaluate the impact of this framework on the authoring experience, we conducted a user study. Results showed that authors using the Authoring-Assisted FAtiMA-Toolkit were on average able to create more SIA-related content in less time. Our findings suggest that data-augmented, theory-grounded agent modelling tools can support the development of affective social agents by reducing the authoring burden without sacrificing the framework’s clarity or the authors’ control over the content.

Users can adapt contemporary natural language interfaces (NLIs) by teaching the NLIs how to handle new natural language (NL) inputs. One promising approach is interactive task learning (ITL), which enables users to teach new NL inputs for multi-modal systems. While recent advances enable users to teach the syntactic and semantic level of the NL inputs through ITL, NLIs are still not able to learn how to consider the context, such as the current state of the graphical user interface (GUI). To address this challenge, we designed MALACHITE through three formative studies. MALACHITE enables users to successfully teach NL inputs on a semantic and syntactic level leveraging the GUI screen of a data visualization tool. With two evaluative studies, we provide evidence that with MALACHITE ’s suggestions, users significantly improve their accuracy by a factor of 2.3 in teaching GUI-dependent NL inputs in contrast to those without MALACHITE ’s suggestions.