Interaction Logs Research Articles

EBook Technology Facilitating University Education During COVID-19: Japanese Experience

UNESCO reported that 90% of students are affected in some way by COVID-19 pandemic. Like many countries, Japan too imposed emergency remote teaching and learning at both school and university level. In this study, we focus on a national university in Japan, and investigate how teaching and learning were facilitated during this pandemic period using an ebook platform, BookRoll, which was linked as an external tool to the university’s learning management system. Such an endeavor also reinforced the Japanese national thrust regarding explorations of e-book-based technologies and using Artificial Intelligence in education. Teachers could upload reading materials for instance their course notes and associate an audio of their lecture. While students who registered in their course accessed the learning materials, the system collected their interaction logs in a learning record store. Across the spring semesters from April - July 2020, BookRoll system collected nearly 1.5 million reading interaction logs from more than 6300 students across 243 courses in 6 domains. The analysis highlighted that during emergency remote teaching and learning BookRoll maintained a weekly average traffic above 1,900 learners creating more than 78,000 reading logs and teachers perceived it as useful for orchestrating their course.

Is the Reign of Interactive Search Eternal? Findings from the Video Browser Showdown 2020

Comprehensive and fair performance evaluation of information retrieval systems represents an essential task for the current information age. Whereas Cranfield-based evaluations with benchmark datasets support development of retrieval models, significant evaluation efforts are required also for user-oriented systems that try to boost performance with an interactive search approach. This article presents findings from the 9th Video Browser Showdown, a competition that focuses on a legitimate comparison of interactive search systems designed for challenging known-item search tasks over a large video collection. During previous installments of the competition, the interactive nature of participating systems was a key feature to satisfy known-item search needs, and this article continues to support this hypothesis. Despite the fact that top-performing systems integrate the most recent deep learning models into their retrieval process, interactive searching remains a necessary component of successful strategies for known-item search tasks. Alongside the description of competition settings, evaluated tasks, participating teams, and overall results, this article presents a detailed analysis of query logs collected by the top three performing systems, SOMHunter, VIRET, and vitrivr. The analysis provides a quantitative insight to the observed performance of the systems and constitutes a new baseline methodology for future events. The results reveal that the top two systems mostly relied on temporal queries before a correct frame was identified. An interaction log analysis complements the result log findings and points to the importance of result set and video browsing approaches. Finally, various outlooks are discussed in order to improve the Video Browser Showdown challenge in the future.

Task estimation for software company employees based on computer interaction logs

Task estimation for software company employees based on computer interaction logs

Direct Answer Threshold Optimization in Dialogue Systems

The presence of dialogue systems is rising in a wide array of industries. While complex, human-like conversational flow and turn-taking have been the focus of recent research advances, we make the point that direct answers are equally important in human-bot interactions. This is especially true in an information seeking task where prompt, correct answers with minimal back-and-forth are desirable. We define a direct answer as a response given to a user query without requiring further clarifications from the user. To determine whether a direct answer is to be given or not, a threshold is applied to the to the confidence level of the predicted intent; in the case where the confidence is higher than the threshold, the user receives a direct answer. This threshold is often set intuitively or on the basis of a few observations, usually between 50% - 75%. In this paper, we propose a method to estimate this threshold based on the intent classification confidence level combined with several intent volumetrics. The goal of our method is to maximize the number of correct direct responses for as many intents as possible in order to minimize user frustration from unnecessary requests for clarifications. Moreover, our method is applicable in the earlier stages of a dialogue system when real interaction logs are scarce. We show that our method improves the accuracy of directly answered queries by 3 to 14% while maximizing the number of accurately answered intents on two dialogue system datasets of 32 and 152 intents.

Lessons Learned from the Usability Evaluation of a Simulated Patient Dialogue System

Simulated consultations through virtual patients allow medical students to practice history-taking skills. Ideally, applications should provide interactions in natural language and be multi-case, multi-specialty. Nevertheless, few systems handle or are tested on a large variety of cases. We present a virtual patient dialogue system in which a medical trainer types new cases and these are processed without human intervention. To develop it, we designed a patient record model, a knowledge model for the history-taking task, and a termino-ontological model for term variation and out-of-vocabulary words. We evaluated whether this system provided quality dialogue across medical specialities (n = 18), and with unseen cases (n = 29) compared to the cases used for development (n = 6). Medical evaluators (students, residents, practitioners, and researchers) conducted simulated history-taking with the system and assessed its performance through Likert-scale questionnaires. We analysed interaction logs and evaluated system correctness. The mean user evaluation score for the 29 unseen cases was 4.06 out of 5 (very good). The evaluation of correctness determined that, on average, 74.3% (sd = 9.5) of replies were correct, 14.9% (sd = 6.3) incorrect, and in 10.7% the system behaved cautiously by deferring a reply. In the user evaluation, all aspects scored higher in the 29 unseen cases than in the 6 seen cases. Although such a multi-case system has its limits, the evaluation showed that creating it is feasible; that it performs adequately; and that it is judged usable. We discuss some lessons learned and pivotal design choices affecting its performance and the end-users, who are primarily medical students.

How do students develop computational thinking? Assessing early programmers in a maze-based online game

ABSTRACT Background and Context: Despite many initiatives to develop Computational Thinking (CT), not much is known about how early programmers develop CT and how we can assess their learning. Objective: Determine if the analysis of students’ interactions with an online platform allows understanding the development of CT, how we can convert data collected into valuable insights, and the aspects that should be considered in platforms design. Method: We developed an online platform with a fine-grained log–recording system. We analysed the data collected from 1004 students (ages 8-14) to understand the difficulties they face. We explain our platform and the tools to process and filter the interaction logs. We calculate additional indicators that provide useful information about student’s behaviour. Findings: Age and gender have shown to influence on CT learning. Generating additional indicators from basic interaction data provide valuable insights. We provide a list of recommendations for developing more effective programming learning platforms.

Seeing like an infrastructure: avidity and difference in algorithmic recommendation

ABSTRACT As the influence of algorithmic systems has grown, critics have come to appreciate that algorithms are not autonomous technical forces, but rather heterogeneous sociotechnical systems. The people who build and maintain these infrastructures play integral roles in their functioning: in the tight and continuous cycles of contemporary software development, the thinking of developers shapes how data drives ‘data-driven’ organizations. This article contributes to contemporary debates on infrastructural politics by describing how the vernacular social theorizing of one group of developers tangles with their technical work. Drawing on ethnographic fieldwork with developers of music recommender systems in the US, I examine how they understand the variability of music listeners. I find that the dominant frame for making sense of listener variation is avidity: a level of enthusiasm for music, which manifests as a willingness to expend effort in finding listening material. For people working in this industry, avidity displaces other ways of understanding human variety – particularly demography. While the technical communities behind these systems were predominantly white and male, they understood the difference that set them apart from most users to be their enthusiasm for music. Centreing avidity provided a way to claim elite cultural status and to avoid talking about demographic diversity. It also reflects the infrastructures through which recommender system developers know and intervene upon their users: avidity is what users look like when seen through interaction logs. Less avid users leave fewer traces, and the goal of the recommender system is to encourage them to leave more. As a result, the figure of the less avid listener serves to justify increasingly rapacious data collection practices.

HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks.

To mitigate the pain of manually tuning hyperparameters of deep neural networks, automated machine learning (AutoML) methods have been developed to search for an optimal set of hyperparameters in large combinatorial search spaces. However, the search results of AutoML methods significantly depend on initial configurations, making it a non-trivial task to find a proper configuration. Therefore, human intervention via a visual analytic approach bears huge potential in this task. In response, we propose HyperTendril, a web-based visual analytics system that supports user-driven hyperparameter tuning processes in a model-agnostic environment. HyperTendril takes a novel approach to effectively steering hyperparameter optimization through an iterative, interactive tuning procedure that allows users to refine the search spaces and the configuration of the AutoML method based on their own insights from given results. Using HyperTendril, users can obtain insights into the complex behaviors of various hyperparameter search algorithms and diagnose their configurations. In addition, HyperTendril supports variable importance analysis to help the users refine their search spaces based on the analysis of relative importance of different hyperparameters and their interaction effects. We present the evaluation demonstrating how HyperTendril helps users steer their tuning processes via a longitudinal user study based on the analysis of interaction logs and in-depth interviews while we deploy our system in a professional industrial environment.

Characterizing Stage-aware Writing Assistance for Collaborative Document Authoring

Writing is a complex non-linear process that begins with a mental model of intent, and progresses through an outline of ideas, to words on paper (and their subsequent refinement). Despite past research in understanding writing, Web-scale consumer and enterprise collaborative digital writing environments are yet to greatly benefit from intelligent systems that understand the stages of document evolution, providing opportune assistance based on authors' situated actions and context. In this paper, we present three studies that explore temporal stages of document authoring. We first survey information workers at a large technology company about their writing habits and preferences, concluding that writers do in fact conceptually progress through several distinct phases while authoring documents. We also explore, qualitatively, how writing stages are linked to document lifespan. We supplement these qualitative findings with an analysis of the longitudinal user interaction logs of a popular digital writing platform over several million documents. Finally, as a first step towards facilitating an intelligent digital writing assistant, we conduct a preliminary investigation into the utility of user interaction log data for predicting the temporal stage of a document. Our results support the benefit of tools tailored to writing stages, identify primary tasks associated with these stages, and show that it is possible to predict stages from anonymous interaction logs. Together, these results argue for the benefit and feasibility of more tailored digital writing assistance.

Kajian Penggunaan Data Log Mahasiswa untuk Berbagai Permasalahan Analisis Pembelajaran

Pada saat ini, penyelenggaraan sistem pembelajaran daring menjadi hal yang penting di tengah pandemi untuk menekan persebaran virus COVID-19. Namun, sistem ini sangat sulit menjaga motivasi dan tingkat keterlibatan mahasiswa karena tidak ada interaksi langsung antara pengajar dengan mahasiswa. Makalah ini meninjau penggunaan data log mahasiswa untuk kebutuhan analisis pembelajaran guna memprediksi kinerja atau kecenderungan drop-out mahasiswa dari suatu mata kuliah dengan melihat pada data log interaksi mahasiswa dengan sistem dan data demografis mahasiswa menggunakan suatu data terbuka, yaitu Open University Learning Analytics Dataset (OULAD). Dari tinjauan beberapa artikel penelitian yang merujuk pada dataset tersebut, ada beberapa hal yang perlu ditinjau: 1) permasalahan yang sering diangkat, yaitu prediksi kecenderungan gagal dari mata kuliah tertentu, prediksi kinerja, dan prediksi keterlibatan mahasiswa; 2) fitur yang digunakan pada saat pemodelan, yaitu fitur demografis dan interaksi, baik yang diringkas secara harian atau mingguan dengan berbagai representasi fitur; 3) metode analisis pembelajaran yang secara khusus menggunakan metode pembelajaran mesin yang sering digunakan, yaitu Decision Tree (DT), Support Vector Machine (SVM), Multi-Layer Perceptron (MLP), dan Long Short-Term Memory (LSTM). Makalah ini juga mendiskusikan proses mitigasi dari mahasiswa yang berisiko, perancangan sistem data yang mendukung analisis pembelajaran, dan permasalahan yang sering ditemui pada saat proses pemodelan.

205 - Building and Using an Apps Library for People Living with Dementia and their Carers

205 - Building and Using an Apps Library for People Living with Dementia and their Carers

Understanding Digital Checklist Use Through Team Communication.

Introducing technology support in a complex, team-based work setting requires a study of teamwork effects on technology use. In this paper, we present our initial analysis of team communications in a trauma resuscitation setting, where we deployed a digital checklist to support trauma team leaders in guiding patient care. By analyzing speech transcripts, checklist interaction logs, and videos of 15 resuscitations, we identified several tensions that arose from the use of a checklist in a team-based process with multi-step tasks. The tensions included incorrect markings of in-progress tasks as completed, failure to mark completed tasks due to missed communications, failure to record planned tasks, and difficulties in recording dynamic values. From these findings, we discuss design implications for checklist design for dynamic, team-based activities.

Robotic Process Mining: Vision and Challenges

Robotic process automation (RPA) is an emerging technology that allows organizations automating repetitive clerical tasks by executing scripts that encode sequences of fine-grained interactions with Web and desktop applications. Examples of clerical tasks include opening a file, selecting a field in a Web form or a cell in a spreadsheet, and copy-pasting data across fields or cells. Given that RPA can automate a wide range of routines, this raises the question of which routines should be automated in the first place. This paper presents a vision towards a family of techniques, termed robotic process mining (RPM), aimed at filling this gap. The core idea of RPM is that repetitive routines amenable for automation can be discovered from logs of interactions between workers and Web and desktop applications, also known as user interactions (UI) logs. The paper defines a set of basic concepts underpinning RPM and presents a pipeline of processing steps that would allow an RPM tool to generate RPA scripts from UI logs. The paper also discusses research challenges to realize the envisioned pipeline.

Predicting school performance and early risk of failure from an intelligent tutoring system

In many rural Indian schools, English is a second language for teachers and students. Intelligent tutoring systems have good potential because they enable students to learn at their own pace, in an exploratory manner. This paper describes a 3-year longitudinal study of 2123 Indian students who used the intelligent tutoring system, AmritaITS. The aim of the study was to use the students’ interaction logs with AmritaITS to: (1) predict student performance, in English and Mathematics subjects, via summative and formative assessments, (2) predict students who may be at risk of failing the final examination and (3) screen students who may have reading difficulties. The prediction models for summative assessments were significantly improved by formative assessments scores, along with AmritaITS logs. The receiver operating characteristic (ROC) curve showed that students at risk of failing a class could be identified early, with high sensitivity and specificity. The models also provide recommendations for the amount of time required for students to use the system, and reach the appropriate grade level. Finally, the models demonstrated promise in identifying students who might be at risk of suffering from reading difficulties.

Characterizing the Quality of Insight by Interactions: A Case Study.

Understanding the quality of insight has become increasingly important with the trend of allowing users to post comments during visual exploration, yet approaches for qualifying insight are rare. This article presents a case study to investigate the possibility of characterizing the quality of insight via the interactions performed. To do this, we devised the interaction of a visualization tool-MediSyn-for insight generation. MediSyn supports five types of interactions: selecting, connecting, elaborating, exploring, and sharing. We evaluated MediSyn with 14 participants by allowing them to freely explore the data and generate insights. We then extracted seven interaction patterns from their interaction logs and correlated the patterns to four aspects of insight quality. The results show the possibility of qualifying insights via interactions. Among other findings, exploration actions can lead to unexpected insights; the drill-down pattern tends to increase the domain values of insights. A qualitative analysis shows that using domain knowledge to guide exploration can positively affect the domain value of derived insights. We discuss the study's implications, lessons learned, and future research opportunities.

Breaking the Eyes: How Do Users Get Started with a Coordinated and Multiple View Geovisualization Tool?

ABSTRACT Maps are frequently combined with data displays in the form of coordinated and multiple views (CMV). Although CMV are valuable geovisualization tools, novice users may find them complex and thus require explanation. However, no tutorial guidelines have been developed that indicate what is helpful in understanding CMV geovisualization tools. We therefore conducted a study on the learnability of a CMV tool, informed with eye-tracking data, talk-aloud and interaction logs. We have investigated how untrained users work with a CMV geovisualization tool. The study revealed that: (1) despite their initial confusion, users found the tested tool pleasant to play with while getting to grips with how dynamic brushing works, (2) when examining the tool’s interface, participants mainly looked freely at explanatory elements, such as labels and the legend, but they explored interactive techniques only to a limited degree. We conclude with tips about tutorial design and layout design for CMV tools.

Prediction of students’ early dropout based on their interaction logs in online learning environment

ABSTRACTOnline learning has become more popular in higher education since it adds convenience and flexibility to students’ schedule. But, it has faced difficulties in the retention of the continuity of students and ensure continual growth in course. Dropout is a concerning factor in online course continuity. Therefore, it has sparked great interest among educators and researchers to offer many models and strategies to reduce online course students’ dropout by analyzing students’ behavior and their individual and academic information. However, online education platforms still face challenges of high dropout rate and the difficulty of accurate prediction to reduce it. The key aim of the present study is to construct a predictive model to early predict students who are at-risk of dropout. This model is useful to the course instructors to make effective and timely interventions. We have noticed that dropout prediction is basically a sequence labeling or time series prediction problem. For these reasons, we proposed two models; the Logistic Regression by adding a regularization term and the Input-Output Hidden Markov Model (IOHMM). Results showed that the proposed models achieved an accuracy of 84% compared to the baseline of Machine Learning Models for prediction of the students at-risk of dropping out.

GUI information-based interaction logging and visualization for asynchronous usability testing

Asynchronous usability testing is efficient and promising usability testing methodology. However, evaluators are required to dedicate a considerable amount of effort for identifying usability problems in asynchronous testing. Evaluators cannot observe user interactions directly; only the collected data is available to them for identifying the usability problems. To reduce the amount of effort required, this paper presents a framework and tool for interaction visualization. This framework tracing user interactions based on meaningful GUI changes. The collected interaction log is visualized in a way that is similar to the visualization of traversed path used in web domains. Using our approach, the evaluator shall be able to understand user behavior more easily. In addition, we designed a black-box-based interaction logging for minimizing the implementation costs. Our tool especially helpful when testing has a large number of interaction data to analyze. To evaluate the proposed framework and tool, user interaction data were collected, and the results were evaluated by mobile application experts. The evaluation results demonstrate that the user interaction data traced by the proposed framework accurately reflect the user interactions and that the visualization results are valid.

JEPPY: An Interactive Pedagogical Agent to Aid Novice Programmers in Correcting Syntax Errors

Programming is a complicated task and correcting syntax error is just one among the many tasks that makes it difficult. Error messages produced by the compiler allow novice learners to know their errors. However, these messages are puzzling, and most of the times misleading due to cascading of errors, which can be detrimental to running a syntax-error free program. In most laboratory setting, it is the role of the teachers to assist their students while doing activities. However, in our experienced, considering the large number of students in a class, it may seem difficult for teachers to assist their students one-by- one given the time constraints. In this paper, the design and implementation of an interactive pedagogical agent named JEPPY is presented. It is intended to assist novice learners learning to program using C++ as a programming language. In order to see on how students struggle or progress in dealing with errors, the proponents implemented the Error Quotient (EQ) developed by Jadud. The principles of the cognitive requirements of an agent- based learning environment were followed. The agent was put into test by novice learners in a laboratory setting. Logs of interaction between the embodied agent and the participants were recorded, aside from the compile errors and edit actions. These mechanisms show us some insight on the interaction behavior of learner to the agent.

Human Feedback as Action Assignment in Interactive Reinforcement Learning

Teaching by demonstrations and teaching by assigning rewards are two popular methods of knowledge transfer in humans. However, showing the right behaviour (by demonstration) may appear more natural to a human teacher than assessing the learner’s performance and assigning a reward or punishment to it. In the context of robot learning, the preference between these two approaches has not been studied extensively. In this article, we propose a method that replaces the traditional method of reward assignment with action assignment (which is similar to providing a demonstration) in interactive reinforcement learning. The main purpose of the suggested action is to compute a reward by seeing if the suggested action was followed by the self-acting agent or not. We compared action assignment with reward assignment via a user study conducted over the web using a two-dimensional maze game. The logs of interactions showed that action assignment significantly improved users’ ability to teach the right behaviour. The survey results showed that both action and reward assignment seemed highly natural and usable, reward assignment required more mental effort, repeatedly assigning rewards and seeing the agent disobey commands caused frustration in users, and many users desired to control the agent’s behaviour directly.