User Evaluation Study Research Articles

Those were the days: learning to rank social media posts for reminiscence

Social media posts are a great source for life summaries aggregating activities, events, interactions and thoughts of the last months or years. They can be used for personal reminiscence as well as for keeping track with developments in the lives of not-so-close friends. One of the core challenges of automatically creating such summaries is to decide which posts are memorable, i.e., should be considered for retention and which ones to forget. To address this challenge, we design and conduct user evaluation studies and construct a corpus that captures human expectations towards content retention. We analyze this corpus to identify a small set of seed features that are most likely to characterize memorable posts. Next, we compile a broader set of features that are leveraged to build general and personalized machine-learning models to rank posts for retention. By applying feature selection, we identify a compact yet effective subset of these features. The models trained with the presented feature sets outperform the baseline models exploiting an intuitive set of temporal and social features.

High-Performance Virtual Reality Volume Rendering of Original Optical Coherence Tomography Point-Cloud Data Enhanced With Real-Time Ray Casting.

PurposeFeasibility testing of a novel volume renders technology to display optical coherence tomography data (OCT) in a virtual reality (VR) environment.MethodsA VR program was written in C++/OpenGL to import and display volumetric OCT data in real time with 180 frames per second using a high-end computer and a tethered head-mounted display. Following exposure, participants completed a Simulator Sickness Questionnaire (SSQ) to assess for nausea, disorientation, and oculomotor disturbances. A user evaluation study of this software was conducted to explore the potential utility of this application.ResultsFifty-seven subjects completed the user testing (34 males and 23 females). Mean age was 48.5 years (range, 21–77 years). Mean acquired work experience of the 35 ophthalmologists (61.40%) included in the group was 15.46 years (range, 1–37 years). Twenty-nine participants were VR-naïve. The SSQ showed a mean total score of 5.8 (SD = 9.44) indicating that the system was well tolerated and produced minimal side effects. No difference was reported between VR-naïve participants and experienced users. Overall, immersed subjects reported an enjoyable VR-OCT presence effect.ConclusionsA usable and satisfying VR imaging technique was developed to display and interact with original OCT data.Translational RelevanceAn advanced high-end VR image display method was successfully developed to provide new views and interactions in an ultra high-speed projected digital scenery using point-cloud OCT data. This represents the next generation of OCT image display technology and a new tool for patient engagement, medical education, professional training, and telecommunications.

Can I have more of these please?

PurposeDuring the literature review phase, the task of finding similar research papers can be a difficult proposition for researchers due to the procedural complexity of the task. Current systems and approaches help in finding similar papers for a given paper, even though researchers tend to additionally search using a set of papers. This paper aims to focus on conceptualizing and developing recommendation techniques for key literature review and manuscript preparatory tasks that are interconnected. In this paper, the user evaluation results of the task where seed basket-based discovery of papers is performed are presented.Design/methodology/approachA user evaluation study was conducted on a corpus of papers extracted from the ACM Digital Library. Participants in the study included 121 researchers who had experience in authoring research papers. Participants, split into students and staff groups, had to select one of the provided 43 topics and run the tasks offered by the developed assistive system. A questionnaire was provided at the end of each task for evaluating the task performance.FindingsThe results show that the student group evaluated the task more favourably than the staff group, even though the difference was statistically significant for only 5 of the 16 measures. The measures topical relevance, interdisciplinarity, familiarity and usefulness were found to be significant predictors for user satisfaction in this task. A majority of the participants, who explicitly stated the need for assistance in finding similar papers, were satisfied with the recommended papers in the study.Originality/valueThe current research helps in bridging the gap between novices and experts in terms of literature review skills. The hybrid recommendation technique evaluated in this study highlights the effectiveness of combining the results of different approaches in finding similar papers.

A New Dactylology and Interactive System Development for Blind–Computer Interaction

Although a lot of work has been performed in the gesture-based human–computer interface, blind users still feel it is difficult to interact with computers. One of the major stumbling blocks is the lack of knowledge about their preferences toward hand gestures. A user evaluation study is conducted with 25 blind users to understand this fact and an optimal gesture set is devised. These gestures are selected based on the performance and preference measure analysis. Performance measure includes rating of gestures on four subjective criteria: easiness , naturalness , learning , and reproducibility . In preference measure, a new parameter called as preference index is proposed in this paper. The optimal gestures are further categorized into two groups: tier-1 and tier-2. On the basis of these gestures, a dactylology is proposed for blind users so that they can interact with a computer easily. A prototype model of the proposed interactive system has been developed and encouraging experimental results are obtained.

Wearable Inertial Sensor Systems for Lower Limb Exercise Detection and Evaluation: A Systematic Review.

Analysis of lower limb exercises is traditionally completed with four distinct methods: (1) 3D motion capture; (2) depth-camera-based systems; (3) visual analysis from a qualified exercise professional; and (4) self-assessment. Each method is associated with a number of limitations. The aim of this systematic review is to synthesise and evaluate studies which have investigated the capacity for inertial measurement unit (IMU) technologies to assess movement quality in lower limb exercises. A systematic review of studies identified through the databases of PubMed, ScienceDirect and Scopus was conducted. Articles written in English and published in the last 10years which investigated an IMU system for the analysis of repetition-based targeted lower limb exercises were included. The quality of included studies was measured using an adapted version of the STROBE assessment criteria for cross-sectional studies. The studies were categorised into three groupings: exercise detection, movement classification or measurement validation. Each study was then qualitatively summarised. From the 2452 articles that were identified with the search strategies, 47 papers are included in this review. Twenty-six of the 47 included studies were deemed as being of high quality. Wearable inertial sensor systems for analysing lower limb exercises is a rapidly growing field of research. Research over the past 10years has predominantly focused on validating measurements that the systems produce and classifying users' exercise quality. There have been very few user evaluation studies and no clinical trials in this field to date.

Drawing OWL 2 ontologies with Eddy the editor

In this paper we introduce Eddy, a new open-source tool for the graphical editing of OWL 2 ontologies. Eddy is specifically designed for creating ontologies in Graphol, a completely visual ontology language that is equivalent to OWL 2. Thus, in Eddy ontologies are easily drawn as diagrams, rather t han written as sets of formulas, as commonly happens in popular ontology design and engineering environments. This makes Eddy particularly suited for usage by people who are more familiar with diagramatic languages for conceptual modeling rather than with typical ontology formalisms, as is often required in non-academic and industrial contexts. Eddy provides intuitive functionalities for specifying Graphol diagrams, guarantees their syntactic correctness, and allows for exporting them in standard OWL 2 syntax. A user evaluation study we conducted shows that Eddy is perceived as an easy and intuitive tool for ontology specification.

Community‐driven user evaluation of the Inuvialuit cultural heritage digital library

ABSTRACTPreservation of and access to aboriginal and indigenous cultural heritage is emerging as a key area of information science research and development. In this study, we report on a community‐driven user interface evaluation within a cultural heritage digital library that was developed for the Inuit communities in the Inuvialuit Settlement Region in Canada's north. The study adopted a culturally‐aware, multi‐method and multidisciplinary user evaluation framework to examine the usability and usefulness of the Digital Library North user interface. The three‐phase user evaluation study consisted of such methods and approaches as environmental scanning, surveys, interviews, information audits, information tables, open houses as well as various community workshops. The iterative nature of the digital library interface development and usability evaluation provided rich and varied feedback and suggestions for the improvement of the user interface and its searching, browsing and navigation functionalities. The three thematized categories that emerged from the diverse data collected over the course of the three phases include user engagement, search and browse, and interface features. The study found that the development of cultural heritage digital libraries and their user interfaces for northern communities requires diverse, multidisciplinary and longitudinal methodologies that ensure their success, and more importantly, their sustainability and acceptance.

Systematic Literature Review on Usability of Firewall Configuration

Firewalls are network security components that handle incoming and outgoing network traffic based on a set of rules. The process of correctly configuring a firewall is complicated and prone to error, and it worsens as the network complexity grows. A poorly configured firewall may result in major security threats; in the case of a network firewall, an organization’s security could be endangered, and in the case of a personal firewall, an individual computer’s security is threatened. A major reason for poorly configured firewalls, as pointed out in the literature, is usability issues. Our aim is to identify existing solutions that help professional and non-professional users to create and manage firewall configuration files, and to analyze the proposals in respect of usability. A systematic literature review with a focus on the usability of firewall configuration is presented in the article. Its main goal is to explore what has already been done in this field. In the primary selection procedure, 1,202 articles were retrieved and then screened. The secondary selection led us to 35 articles carefully chosen for further investigation, of which 14 articles were selected and summarized. As main contributions, we propose a taxonomy of existing solutions as well as a synthesis and in-depth discussion about the state of the art in firewall usability. Among the main findings, we perceived that there is a lack (or even an absence) of usability evaluation or user studies to validate the proposed models. Although all articles are related to the topic of usability, none of them clearly defines it, and only a few actually employ usability design principles and/or guidelines.

Quantifying Collaboration with a Co-Creative Drawing Agent

This article describes a new technique for quantifying creative collaboration and applies it to the user study evaluation of a co-creative drawing agent. We present a cognitive framework called creative sense-making that provides a new method to visualize and quantify the interaction dynamics of creative collaboration, for example, the rhythm of interaction, style of turn taking, and the manner in which participants are mutually making sense of a situation. The creative sense-making framework includes a qualitative coding technique, interaction coding software, an analysis method, and the cognitive theory behind these applications. This framework and analysis method are applied to empirical studies of the Drawing Apprentice collaborative sketching system to compare human collaboration with a co-creative AI agent vs. a Wizard of Oz setup. The analysis demonstrates how the proposed technique can be used to analyze interaction data using continuous functions (e.g., integrations and moving averages) to measure and evaluate how collaborations unfold through time.

A new pen device for injection of recombinant human growth hormone: a convenience, functionality and usability evaluation study.

PurposeAdherence to recombinant human growth hormone (r-hGH) is critical to growth and other outcomes in patients with growth disorders, but the requirement for daily injections means that ease of use is an important factor. This study assessed the perceived ease of use and functionality of the prototype of a reusable pen injector (pen device) for r-hGH that incorporates several advanced features.Participants and methodsSemi-structured 60-minute qualitative interviews were conducted in 5 countries with 57 health care professionals (HCPs) and 30 patients with GH deficiency/caregivers administering r-hGH to patients, including children. HCPs had to be responsible for training in the use of r-hGH pen devices and to see ≥4 r-hGH patients/caregivers per month. Patients/caregivers had to have experience with r-hGH administration for at least 6 months.ResultsThirty-seven (65%) of HCPs described the pen device as “simple” or “easy” to use. The aluminum body was generally perceived as attractive, high quality and comfortable to hold and operate. The ease of preparation and use made it suitable for both children and adults. The ability to dial back the r-hGH dose, if entered incorrectly, was mentioned as a major benefit, because other devices need several user steps to reset. Patients/caregivers felt the pen device was easy to use and the injection-feedback features reassured them that the full dose had been given. Overall, 40 (70%) HCPs and 16 (52%) patients/caregivers were likely to recommend or request the pen device. Moreover, patients/caregivers rated the pen device higher than their current reusable pens and almost equal to the leading disposable device for ease of learning, preparation, administration and ease of use.ConclusionThe prototype pen device successfully met its design objectives and was very well accepted by HCPs and patients/caregivers for its ease of use, appearance and functionality.

User evaluation of a task for shortlisting papers from researcher’s reading list for citing in manuscripts

PurposeAlthough many interventional approaches have been proposed to address the apparent gap between novices and experts for literature review (LR) search tasks, there have been very few approaches proposed for manuscript preparation (MP) related tasks. The purpose of this paper is to describe a task and an incumbent technique for shortlisting important and unique papers from the reading list (RL) of researchers, meant for citation in a manuscript.Design/methodology/approachA user evaluation study was conducted on the prototype system which was built for supporting the shortlisting papers (SP) task along with two other LR search tasks. A total of 119 researchers who had experience in authoring research papers participated in this study. An online questionnaire was provided to the participants for evaluating the task. Both quantitative and qualitative analyses were performed on the collected evaluation data.FindingsGraduate research students prefer this task more than research and academic staff. The evaluation measures relevance, usefulness and certainty were identified as predictors for the output quality measure “good list”. The shortlisting feature and information cues were the preferred aspects while limited data set and rote steps in the study were ascertained as critical aspects from the qualitative feedback of the participants.Originality/valueFindings point out that researchers are clearly interested in this novel task of SP from the final RL prepared during LR. This has implications for digital library, academic databases and reference management software where this task can be included to benefit researchers at the manuscript preparatory stage of the research lifecycle.

AuditingThe Completeness and Legibility ofComputerizedRadiological Request Forms.

CertainSaudi healthcare organizations transferoutpatientsto medical imaging departments for radiological examinations in a manual process that relies on the use of paper-based forms. With the increased implementation of electronic medical records in Saudi Hospitals, littleis known about the completeness and legibility of information captured in electronic-based medical imaging forms. The purpose of this study is to audit the completeness and legibilityof medical imaging paper-based forms in comparison with electronic-based medical imagingforms. As a secondary objective, we also examined the number of errors found on the forms.An observational retrospective cross-sectional study was utilizedtoaudit the completeness and legibilityof both paper and electronic forms collected betweenMarch 1 and May 15, 2015.The study measuredthe association among categorical variables using Chi-Square analysis. The results of this investigationshow a significant association between formcompletion and type of record(i.e., paper vs. electronic) where electronic-based systems were found to be more complete than paper-based records. Electrnoic based records were also found toimprove form legibility, promote useradherence to complete the forms and minimize entry errors.In conclusion, electronic-based medical imaging forms are more complete and legible than paper based forms. Future studies should evaluate other hospitals and compare both legibility and completeness of electronic-based medical imaging forms and conduct usability evaluation studies with users to explore the impacts of system design on both completeness and legibility of electronic forms, in general, but more specifically, electronic-based medical imaging forms.

Using Eye Tracking as a Tool to Teach Informatics Students the Importance of User Centered Design

In this article the authors describe how they incorporate eye tracking in a human-computer interaction (HCI) course that forms part of a postgraduate Informatics degree. The focus is on an eye tracking assignment that involves student groups performing usability evaluation studies for real world clients. Over the past three years the authors have observed how this experience positively affected students' attitude towards usability and user experience (UX) evaluation. They therefore believe that eye tracking is a powerful tool to convince students of the importance of user centered design. To investigate the soundness of their informal observations, the authors conducted a survey amongst 2016 HCI students and analysed student course evaluation results from 2014 to 2016. The findings confirm that students regard the eye tracking assignment as a mind altering experience and that it is potentially an effective tool for convincing future IT professionals of the importance of usability, UX and user centered design.

Usability evaluation of assistive technologies through qualitative research focusing on people with mild dementia

The assistive technologies (ATs) are commonly used for the wellbeing of people with dementia (PWD). Research shows that current ATs are not performing to their best and high rate of AT abandonment still exist. Although empirical evaluations greatly impact AT success, yet only few studies investigated AT usability for PWD. To the best of authors knowledge there is no AT usability evaluation study conducted in the South Asia. Therefore a qualitative study is carried out to identify which AT factors encourage and discourage the PWD through semi-structured interviews. Thematic analysis is used to generate themes and sub-themes. The (happy users, non-happy users and technology and human care) emerged as three main themes while, (communication, monitoring and reminders) as popular AT types. Most PWD use ATs for socialization and health monitoring. Overall the PWD appreciated the role of ATs in their lives, but showed concerns about interface efficacy, function simplicity and elderly requirements adaptation. The already met and unmet needs of the PWD are also investigated. The AT producers should make user interface simpler and tailor future ATs to the specific requirements of the PWD. The user centered techniques should be adopted for the development of new ATs.

Temporal Factor-Aware Video Affective Analysis and Recommendation for Cyber-Based Social Media

As an important cyber-enabled application, online video recommendation is seeing significant interest from both industry and academia. To effectively recommend video content becomes a popular research topic. However, it has been found that existing recommendation methods based on video affective analysis ignore the temporal factor, leading to poor performance especially when the order of emotion components does affect the recommendation quality. This motivates us to study the feature of emotion fluctuation, which we call Temporal Factor of Emotion (TFE). In this paper, a novel recommendation method based on the Grey Relational Analysis (GRA) is proposed to tackle this problem. GRA preserves the temporal factor of objects during analysis and is suitable for analyzing systems with unknown correlation (a set of independent videos). In our work, first, specific video features are extracted and mapped to the well-known Lovheim emotion-space, through the SVMs (Support Vector Machine). Then, GRA is applied to compute the quantitative relation among videos by using extracted emotions as factors. Finally, a pick-filter pattern and GRA-based recommendation method under the Fisher model are proposed. To evaluate the performance of our method, an online video recommendation system is developed. Experimental results of both user study and parameter evaluation demonstrate that the GRA-based method can improve accuracy of video affective analysis and performance of video recommendation.

Evaluating a threefold intervention framework for assisting researchers in literature review and manuscript preparatory tasks

PurposeSystems to support literature review (LR) and manuscript preparation tend to focus on only one or two of the tasks involved. The purpose of this paper is to describe an intervention framework that redesigns a particular set of tasks, allowing for interconnectivity between the tasks and providing appropriate user interface display features for each task in a prototype system.Design/methodology/approachA user evaluation study was conducted on the prototype system. The system supports the three tasks: building a reading list (RL) of research papers, finding similar papers based on a set of papers and shortlisting papers from the final RL for inclusion in manuscript based on article type. A total of 119 researchers who had experience in authoring research papers, participated in the evaluation study. They had to select one of the provided 43 topics and execute the tasks offered by the system. Three questionnaires were provided for evaluating the tasks and system. Both quantitative and qualitative analyses were performed on the collected evaluation data.FindingsTask redesign aspects had a positive impact in user evaluation for the second task of finding similar papers while improvement was found to be required for the first and third tasks. The tasks interconnectivity features seed basket and RL were helpful for the participants in conveniently searching for papers within the system. Two of the four proposed informational display features, namely, information cue labels and shared co-relations were the most preferred features of the system. Student user group found the task recommendations and the overall system to be more useful and effective than the staff group.Originality/valueThis study validates the importance of interconnected task design and novel informational display features in accentuating task-based recommendations for LR and manuscript preparatory tasks. The potential for improvement in recommendations was shown through the task redesign exercise where new requirements for the tasks were identified. The resultant prototype system helps in bridging the gap between novices and experts in terms of LR skills.

Creating a reference data set for the summarization of discussion forum threads

In this paper we address extractive summarization of long threads in online discussion fora. We present an elaborate user evaluation study to determine human preferences in forum summarization and to create a reference data set. We showed long threads to ten different raters and asked them to create a summary by selecting the posts that they considered to be the most important for the thread. We study the agreement between human raters on the summarization task, and we show how multiple reference summaries can be combined to develop a successful model for automatic summarization. We found that although the inter-rater agreement for the summarization task was slight to fair, the automatic summarizer obtained reasonable results in terms of precision, recall, and ROUGE. Moreover, when human raters were asked to choose between the summary created by another human and the summary created by our model in a blind side-by-side comparison, they judged the model’s summary equal to or better than the human summary in over half of the cases. This shows that even for a summarization task with low inter-rater agreement, a model can be trained that generates sensible summaries. In addition, we investigated the potential for personalized summarization. However, the results for the three raters involved in this experiment were inconclusive. We release the reference summaries as a publicly available dataset.

A system for extracting and comparing memes in online forums

From their origins in the sociological field, memes have recently become of interest in the context of `viral' transmission of basic information units (memes) in online social networks. However, much work still needs to be done in terms of metrics and practical data processing issues. In this paper we define a theoretical basis and processing system for extracting and matching memes from free format text. The system facilitates the work of a text analyst in extracting this type of data structures from online text corpuses and n performing empirical experiments in a controlled manner. The general aspects related to the solution are the automatic processing of unstructured text without need for preprocessing (such as labelling and tagging), identification of co-occurences of concepts and corresponding relations, construction of semantic networks and selecting the top memes. The system integrates these processes which are generally separate in other state of the art systems. The proposed system is important because unstructured online text content is growing at a greater rate than other content (e.g. semi-structured, structured) and integrated and automated systems for knowledge extraction from this content will be increasingly important in the future. To illustrate the method and metrics we process several real online discussion forums, extracting the principal concepts and relations, building the memes and then identifying the key memes for each document corpus using a sophisticated matching process. The results show that our method can automatically extract coherent key knowledge from free text, which is corroborated by benchmarking with a set of other text analysis approaches, as well as a user study evaluation.

Using author-specified keywords in building an initial reading list of research papers in scientific paper retrieval and recommender systems

An initial reading list is prepared by researchers at the start of literature review for getting an overview of the research performed in a particular area. Prior studies have taken the approach of merely recommending seminal or popular papers to aid researchers in such a task. In this paper, we present an alternative technique called the AKR (Author-specified Keywords based Retrieval) technique for providing popular, recent, survey and a diverse set of papers as a part of the initial reading list. The AKR technique is based on a novel coverage value that has its calculation centered on author-specified keywords. We performed an offline evaluation experiment with four variants of the AKR technique along with three state-of-the-art approaches involving collaborative filtering and graph ranking algorithms. Findings show that the Hyperlink-Induced Topic Search (HITS) enhanced variant of the AKR technique performs better than other techniques, satisfying most requirements for a reading list. A user evaluation study was conducted with 132 researchers to gauge user interest on the proposed technique using 14 evaluation measures. Results show that (i) students group are more satisfied with the recommended papers than staff group, (ii) popularity measure is strongly correlated with the output quality measures and (iii) the measures familiarity, usefulness and ‘agreeability on a good list’ were found to be strong predictors for user satisfaction. The AKR technique provides scope for extension in future information retrieval (IR) and content-based recommender systems (RS) studies.

Phonological Disorders in Children? Design and user experience evaluation of a mobile serious game approach

This paper presents the concept and the user evaluation study of the first Super-Fon’s prototype, which is a mobile health app with a serious game approach for Android Tablets. It was developed with the main goal of working as a complement to the therapeutic intervention in phonological disorders in children between 3 and 8 years old. The app comprises a range of activities, grouped into levels, presenting a therapeutic intervention that follows the Metaphon methodology. The serious game dimension was added to better engage children in its use. The paper presents a user study conducted with two groups of children, without and with the phonological disorders. The first one worked as a control group to better help finding out if the prototype is well designed and provides a positive user experience to be well received by the children, engaging them, despite their condition. The results of the study were very positive and promising.