User Performance Evaluation Research Articles

Artificial intelligence and machine learning for additive manufacturing composites toward enriching Metaverse technology

<p class="Abstract">As a result of the growing significance and application of technology across a wide range of fields, digital environments such as Metaverse started to take shape over the span of the previous decade. This study aims to discover an area of engineering that could benefit from this new technology by developing an artificial intelligence (AI)—based approach to analyzing and predicting the mechanical properties of carbon fiber reinforced syntactic thermoset composites that are made through additive manufacturing (AM). These composites are intended to be utilized as a tool for metaverse technology in a variety of domains—as the presence of the limitations in the currently experimental methods. The metaverse allows for the generation of simulations through the application of artificial intelligence (AI) and machine learning (ML). Consequently, this paves the way for individuals to investigate various design possibilities and view the virtual manifestation of those possibilities. This is made possible by the use of machine learning algorithms, which allow for the monitoring and evaluation of user performance, as well as the provision of individualized feedback and suggestions for improvement. As a consequence of this, it is feasible that professionals will be able to get education and training that are both more efficient and effective. Consequently, this work aims to introduce an Adaptive Neuro-Fuzzy Inference System (ANFIS)—based model, which is able to effectively anticipate the behavior of mechanical systems in a variety of settings without the need for significant measurements. The validity of the ANFIS model was determined through the utilization of flexure and compression testing. The approach that was used to improve the technical assessment of the manufactured composites—is verified by the model’s near-realistic predictions. Moreover, this method is superb for lowering weight, enhancing mechanical qualities, and minimizing product complexity.</p>

ASSESSMENT AND TECHNICAL CONSTRAINT IDENTIFICATION OF SMALLHOLDER IRRIGATION PUMP IN ARSI AND EAST SHEWA ZONE, OROMIA

Efficient irrigation begins with properly installed and maintained pumps, motors, and engines. For this, the study was conducted on potential irrigation pump users of the Arsi and East Shewa zones with the objective of identifying the technical constraints of smallholder irrigation pump users. Doddota from Arsi and Dugda and Bora woreda from the East Shewa zone were selected. 211 samples size from 2321 pump users were used for data collection. The constraints listed by respondents, like pump damage, pump cost, long priming, and fuel, were the most common bottlenecks in pump irrigation schemes. Pump damage was positively correlation with use of un-recommended suction and delivery head, age of pump, long priming time and continues operation time but negatively correlation with pump maintenance, experience in irrigation farming and educational level of the household. Long priming and fuel consumption was positively (+Ve) correlation with use of un-recommended suction and delivery head, age of pump, operation time and pump size. From the existing pump type respondent responses, KAMA pumps were highly available on the market, vulnerable to damage, and had sufficient spare parts with values of 60.66, 64.45 and 44.08%, respectively. The lowest were Cushion their values were 1.42%, 3.32% and 3.78%, respectively. The respondents also reply, due to absence of governmental pump maintaining organization, cost of maintenance at local and private garage were very expensive (68.25%) and expensive (26.07%). This makes another problem on small holder irrigation pump. Therefore, it is recommended to government to add pump maintaining structure at engineering center which serves' closely to scheme users and providing regular training on maintenance checklist. It is also recommended for engineering and socio-economic researchers to conduct research collaboratively to assess the gap frequently for other kebele pump user and technical performance evaluation for pump that have problems.

A Question Answering Chatbot for Gastric Cancer Patients After Curative Gastrectomy: Development and Evaluation of User Experience and Performance.

Postoperative gastric cancer patients have many questions about managing their daily lives with various symptoms and discomfort. This study aimed to develop a knowledge-based question answering (QA) chatbot for their self-management and to evaluate the user experience and performance of the chatbot. To support the chatbot's natural language processing, we analyzed QA texts from an online self-help group, clinical guidelines, and refined frequently asked questions related to gastric cancer. We developed a named entity classification with seven superconcepts, 4544 subconcepts, and 1415 synonyms. We also developed a knowledge base by linking the users' classified question intents with the experts' answers and knowledge resources, including 677 question intents and scripts with standard QA pairs and similar question phrases. A chatbot called "GastricFAQ" was built, reflecting the question topics of the named entity classification and QA pairs of the knowledge base. User experience evaluation (N = 56) revealed the highest mean score for usefulness (4.41/5.00), with all other items rated 4.00 or higher, except desirability (3.85/5.00). The chatbot's accuracy, precision, recall, and F score ratings were 85.2%, 87.6%, 96.8%, and 92.0%, respectively, with immediate answers. GastricFAQ could be provided as one option to obtain immediate information with relatively high accuracy for postoperative gastric cancer patients.

A Study on Innovative Design Approaches for Implementing an Intelligent ICT-Based Smart Highway

This research, conducted over a three-year period starting from 2020, was a collaborative project between practitioners in the field of VE design and the Korea Expressway Corporation, aimed at innovating the design VE project to construct an intelligent ICT-based smart highway and enhancing highway driving performance and customer satisfaction.
 The research included R&D support for payment and management systems integrated with IOT technology, the application of the developed prototype at highway sites, and analysis of user performance evaluation. Based on the Innovation Work Roadmap of the Korea Expressway Corporation, the research process involved collaboration between a public participation group and an expert group, considering the technical feasibility of data-based ICT technology through the National Idea VE Contest.
 The final VE design results were implemented at the Seongnam tollgate site with budget support from relevant departments, followed by a satisfaction survey. It is expected that the continuous contributions to achieving an ESG society, such as the increase in the number of customers using the tolling zone equipped with smart upgrades, along with their satisfaction rate, and the reduction in carbon emissions and total settlement time of highway users, will be sustained.

Building Higher-Order Abstractions from the Components of Recommender Systems

We present a modular recommender system framework that tightly integrates yet maintains the independence of individual components, thus satisfying two of the most critical aspects of industrial applications, generality and specificity. On the one hand, we ensure that each component remains self-contained and is ready to serve in other applications beyond recommender systems. On the other hand, when these components are combined, a unified theme emerges for recommender systems. We present the details of each component in the context of recommender systems and other applications. We release each component as an open-source library, and most importantly, we release their integration under MAB2REC, an industry-strength open-source software for building bandit-based recommender systems. By bringing standalone components together, Mab2Rec realizes a powerful and scalable toolchain to build and deploy business-relevant personalization applications. Finally, we share our experience and best practices for user training, adoption, performance evaluation, deployment, and model governance within the enterprise and the broader community.

Assessment and Technical Constraint Identification of Smallholder Irrigation Pump in Arsi and East Shewa Zone, Oromia

Efficient irrigation begins with properly installed and maintained pumps, motors, and engines. For this, the study was conducted on potential irrigation pump users of the Arsi and East Shewa zones with the objective of identifying the technical constraints of smallholder irrigation pump users. Doddota from Arsi and Dugda and Bora woreda from the East Shewa zone were selected. 211 samples size from 2321 pump users were used for data collection. The constraints listed by respondents, like pump damage, pump cost, long priming, and fuel, were the most common bottlenecks in pump irrigation schemes. Pump damage was positively correlation with use of un-recommended suction and delivery head, age of pump, long priming time and continues operation time but negatively correlation with pump maintenance, experience in irrigation farming and educational level of the household. Long priming and fuel consumption was positively (+Ve) correlation with use of un-recommended suction and delivery head, age of pump, operation time and pump size. From the existing pump type respondent responses, KAMA pumps were highly available on the market, vulnerable to damage, and had sufficient spare parts with values of 60.66, 64.45 and 44.08%, respectively. The lowest were Cushion their values were 1.42%, 3.32% and 3.78%, respectively. The respondents also reply, due to absence of governmental pump maintaining organization, cost of maintenance at local and private garage were very expensive (68.25%) and expensive (26.07%). This makes another problem on small holder irrigation pump. Therefore it is recommended to government to add pump maintaining structure at engineering center which serves' closely to scheme users and providing regular training on maintenance checklist. It is also recommended for engineering and socio-economic researchers to conduct research collaboratively to assess the gap frequently for other kebele pump user and technical performance evaluation for pump that have problems.

Methods for Evaluating Learning Analytics and Learning Analytics Dashboards in Adaptive Learning Platforms: A Systematic Review

This research paper highlights and addresses the lack of a systematic review of the methods used to evaluate Learning Analytics (LA) and Learning Analytics Dashboards (LAD) of Adaptive Learning Platforms (ALPs) in the current literature. Addressing this gap, the authors built upon the work of Tretow-Fish and Khalid (2022) and analyzed 32 papers, which were grouped into six categories (C1-6) based on their themes. The categories include C1) the evaluation of LA and LAD design and framework, C2) the evaluation of user performance with LA and LAD, C3) the evaluation of adaptivity, C4) the evaluation of ALPs through perceived value, C5) the evaluation of Multimodal methods, and C6) the evaluation of the pedagogical implementation of ALP’s LA and LAD. The results include a tabular summary of the papers including the categories, evaluation unit(s), methods, variables and purpose. While there are numerous studies in categories C1-4 that focus on the design, development, and impact assessment of ALP's LA and LAD, there are only a few studies in categories C5 and C6. For the category of C5), very few studies applied any evaluation methods assessing the multimodal features of LA and LADs on ALPs. Especially for C6), evaluating the pedagogical implementation of ALP's LA and LAD, the three dimensions of signature pedagogy are used to assess the level of pedagogy evaluation. Findings showed that no studies focus on evaluating the deep or implicit structure of ALP's LA. All studies examine the structural surface dimension of learning activities and interactions between students, teachers, and ALP's LA and LAD, as examined in categories C2-C5. No studies were exclusively categorized as a C6 category, indicating that all studies evaluate ALP's LA and LAD on the surface structure dimension of signature pedagogy. This review highlights the lack of pedagogical methodology and theory in ALP's LA and LAD, which are recommended to be emphasized in future research and ALP development and implementation.

A review on smart robotic wheelchairs with advancing mobility and independence for individuals with disabilities

This research paper presents a comprehensive review of smart robotic wheelchairs and their impact on enhancing mobility and independence for individuals with disabilities. Traditional wheelchairs often impose limitations on users, resulting in reduced freedom of movement and limited accessibility. The emergence of smart robotic wheelchairs offers a promising solution to address these challenges. This paper provides an overview of wheelchair technology, identifies the specific challenges faced by individuals with disabilities, and explores the advantages and limitations of smart robotic wheelchairs through a review of previous research studies. The features and functionalities of smart robotic wheelchairs, including navigation and obstacle avoidance capabilities, autonomous and semi-autonomous modes, and customizable control options, are discussed. User experience and performance evaluation, along with the impact on mobility and independence, are examined. The paper concludes with future directions and recommendations to guide further research and development in this important field, aiming to empower individuals with disabilities and improve their quality of life.

A Human-Adaptive Model for User Performance and Fatigue Evaluation during Gaze-Tracking Tasks

Eye gaze interfaces are an emerging technology that allows users to control graphical user interfaces (GUIs) simply by looking at them. However, using gaze-controlled GUIs can be a demanding task, resulting in high cognitive and physical load and fatigue. To address these challenges, we propose the concept and model of an adaptive human-assistive human–computer interface (HA-HCI) based on biofeedback. This model enables effective and sustainable use of computer GUIs controlled by physiological signals such as gaze data. The proposed model allows for analytical human performance monitoring and evaluation during human–computer interaction processes based on the damped harmonic oscillator (DHO) model. To test the validity of this model, the authors acquired gaze-tracking data from 12 healthy volunteers playing a gaze-controlled computer game and analyzed it using odd–even statistical analysis. The experimental findings show that the proposed model effectively describes and explains gaze-tracking performance dynamics, including subject variability in performance of GUI control tasks, long-term fatigue, and training effects, as well as short-term recovery of user performance during gaze-tracking-based control tasks. We also analyze the existing HCI and human performance models and develop an extension to the existing physiological models that allows for the development of adaptive user-performance-aware interfaces. The proposed HA-HCI model describes the interaction between a human and a physiological computing system (PCS) from the user performance perspective, incorporating a performance evaluation procedure that interacts with the standard UI components of the PCS and describes how the system should react to loss of productivity (performance). We further demonstrate the applicability of the HA-HCI model by designing an eye-controlled game. We also develop an analytical user performance model based on damped harmonic oscillation that is suitable for describing variability in performance of a PC game based on gaze tracking. The model’s validity is tested using odd–even analysis, which demonstrates strong positive correlation. Individual characteristics of users established by the damped oscillation model can be used for categorization of players under their playing skills and abilities. The experimental findings suggest that players can be categorized as learners, whose damping factor is negative, and fatiguers, whose damping factor is positive. We find a strong positive correlation between amplitude and damping factor, indicating that good starters usually have higher fatigue rates, but slow starters have less fatigue and may even improve their performance during play. The proposed HA-HCI model and analytical user performance models provide a framework for developing an adaptive human-oriented HCI that enables monitoring, analysis, and increased performance of users working with physiological-computing-based user interfaces. The proposed models have potential applications in improving the usability of future human-assistive gaze-controlled interface systems.

A portfolio recommendation system based on machine learning and big data analytics

<abstract><p>This research paper introduces a portfolio recommendation system that utilizes machine learning and big data analytics to offer a profitable stock portfolio and stock analytics via a web application. The system's effectiveness was evaluated through backtesting and user evaluation studies, which consisted of two parts: user evaluation and performance evaluation. The findings indicate that the development of a machine learning-based portfolio recommendation system and big data analytics can effectively meet the expectations of the majority of users and enhance users' financial knowledge. This study contributes to the growing body of research on utilizing advanced technologies for portfolio recommendation and highlights the potential of machine learning and big data analytics in the financial industry.</p></abstract>

Artificial Intelligence-Driven Serious Games in Health Care: Scoping Review.

Artificial intelligence (AI)-driven serious games have been used in health care to offer a customizable and immersive experience. Summarizing the features of the current AI-driven serious games is very important to explore how they have been developed and used and their current state to plan on how to leverage them in the current and future health care needs. This study aimed to explore the features of AI-driven serious games in health care as reported by previous research. We conducted a scoping review to achieve the abovementioned objective. The most popular databases in the information technology and health fields (ie, MEDLINE, PsycInfo, Embase, CINAHL, IEEE Xplore, ACM Digital Library, and Google Scholar) were searched using keywords related to serious games and AI. Two reviewers independently performed the study selection process. Three reviewers independently extracted data from the included studies. A narrative approach was used for data synthesis. The search process returned 1470 records. Of these 1470 records, 46 (31.29%) met all eligibility criteria. A total of 64 different serious games were found in the included studies. Motor impairment was the most common health condition targeted by these serious games. Serious games were used for rehabilitation in most of the studies. The most common genres of serious games were role-playing games, puzzle games, and platform games. Unity was the most prominent game engine used to develop serious games. PCs were the most common platform used to play serious games. The most common algorithm used in the included studies was support vector machine. The most common purposes of AI were the detection of disease and the evaluation of user performance. The size of the data set ranged from 36 to 795,600. The most common validation techniques used in the included studies were k-fold cross-validation and training-test split validation. Accuracy was the most commonly used metric for evaluating the performance of AI models. The last decade witnessed an increase in the development of AI-driven serious games for health care purposes, targeting various health conditions, and leveraging multiple AI algorithms; this rising trend is expected to continue for years to come. Although the evidence uncovered in this study shows promising applications of AI-driven serious games, larger and more rigorous, diverse, and robust studies may be needed to examine the efficacy and effectiveness of AI-driven serious games in different populations with different health conditions.

User Performance Evaluation and System Accuracy Assessment of Four Blood Glucose Monitoring Systems With Color Coding of Measurement Results.

Blood glucose monitoring systems (BGMSs) are a cornerstone in diabetes management. They have to provide sufficiently accurate results in the hands of lay users, particularly in insulin-treated patients. The aim of this study was user performance evaluation and system accuracy assessment of four BGMSs with color coding of results. Study procedures were based on ISO 15197:2013. User performance evaluation included data from 100 participants, each of whom used every system with one reagent lot. Study personnel observed user techniques. For the system accuracy assessment, 100 capillary samples were obtained for measurement in duplicate with each of three reagent system lots per system, resulting in 600 results per system. All assessed BGMSs exhibited a sufficient level of accuracy, with small differences between them. In the user performance evaluation, study personnel observed the smallest total number of user errors with Contour Next (Ascensia), followed by Accu-Chek Instant (Roche), Medisafe Fit Smile (Terumo), and OneTouch Ultra Plus Reflect (LifeScan). Approximately 90% of participants stated that a consistent color scheme, eg, for low blood glucose (BG) values, should be used across all BGMSs. There was no clear preference among the four tested BGMSs regarding the specific way of displaying color coding. The four BGMSs assessed in this study showed only small differences in an overall sufficient level of accuracy. User handling errors, as observed by study personnel, differed between the systems.

IoT Sensing for Reality-Enhanced Serious Games, a Fuel-Efficient Drive Use Case.

Internet of Things technologies are spurring new types of instructional games, namely reality-enhanced serious games (RESGs), that support training directly in the field. This paper investigates a key feature of RESGs, i.e., user performance evaluation using real data, and studies an application of RESGs for promoting fuel-efficient driving, using fuel consumption as an indicator of driver performance. In particular, we propose a reference model for supporting a novel smart sensing dataflow involving the combination of two modules, based on machine learning, to be employed in RESGs in parallel and in real-time. The first module concerns quantitative performance assessment, while the second one targets verbal recommendation. For the assessment module, we compared the performance of three well-established machine learning algorithms: support vector regression, random forest and artificial neural networks. The experiments show that random forest achieves a slightly better performance assessment correlation than the others but requires a higher inference time. The instant recommendation module, implemented using fuzzy logic, triggers advice when inefficient driving patterns are detected. The dataflow has been tested with data from the enviroCar public dataset, exploiting on board diagnostic II (OBD II) standard vehicular interface information. The data covers various driving environments and vehicle models, which makes the system robust for real-world conditions. The results show the feasibility and effectiveness of the proposed approach, attaining a high estimation correlation (R2 = 0.99, with random forest) and punctual verbal feedback to the driver. An important word of caution concerns users’ privacy, as the modules rely on sensitive personal data, and provide information that by no means should be misused.

Human performance evaluation of user interfaces for spatially-constrained text portrayal and access

To maximize visualization effectiveness, graphical data are commonly augmented with text to provide detailed information and define specific values. This text is often displayed in a pop-up dialog box pertaining to an object, permitting simultaneous display of the object and associated alphanumeric information. However, a human operator performance cost may be incurred when the resulting portrayal occludes critical information within the visualization. To address this issue, we developed and evaluated three alternative, spatially constrained, text portrayal techniques. These techniques and their associated access interface were designed to reduce occlusion while providing rapid access to desired alphanumeric data. Each technique was evaluated against the pop-up dialog using a dual-task human performance paradigm. Performance measures included accuracy, response time, display occlusion, and subjective feedback. The basis for the spatially-constrained text access technique design, their implementation affordances, and limitations are discussed. The alternative techniques and their user interface concepts resulted in mixed accuracy and response time performance compared to the pop-up dialog. Specific design features reduced data access time to one third of the time required to access the spatially-constrained text access techniques. Overall, equivalent performance was obtained among the variants while the potential for occlusion was reduced during use of the novel designs.

Development and prospective validation of a scoring system for the Basic Endoscopic Skills Training (BEST) box

The increased use of endoscopy as a minimally invasive therapeutic technique has created a great demand for endoscopic training. The Basic Endoscopic Skills Training (BEST) box provides a low-cost solution by adapting the Fundamentals of Laparoscopic Surgery (FLS) box for flexible endoscopic simulation. The BEST box consists of six endoscopic tasks with a 5-min time limit per task. This study aims to develop a scoring system for objective evaluation of user performance. A total of 165 participants were tested on the BEST box. Participants were divided into two groups: retrospective analysis (n = 100) and prospective analysis (n = 65). From the retrospective group, 55 individuals were also scored on the Global Assessment of Gastrointestinal Endoscopic Skills-Upper Endoscopy (GAGES-UE). Linear regression between user performance on BEST box and GAGES-UE was performed to develop the scoring system. Receiver Operating Characteristic curve was used to determine a threshold score to help users appreciate their endoscopic expertise. Prospective scoring of 65 individuals was then performed using the formula developed (20 experts and 45 trainees). The minimum and maximum possible scores are 30 and 110, respectively. Retrospective analysis showed that the scoring system was able to distinguish between experts and trainees (p < 0.001), correlated with GAGES-UE (p < 0.001), and had a reliability constant of r = 0.765 (p < 0.001). On prospective testing using the scoring system the expert group received a final average score of 92, whereas the average score for the trainee group was 61 (p < 0.001). The developed BEST box scoring system correlates with the experience level of the test taker as well as with the GAGES-UE scoring system. The results of this study add further evidence to the validity of the BEST box as an effective, low-cost endoscopic simulator with the scores used by trainees to track their performance level overtime.

Accuracy and user performance evaluation of a blood glucose monitoring system which wirelessly transmits results to compatible insulin pumps

Objective: To perform evaluations of the CONTOUR PLUS LINK 2.4 blood glucose monitoring system (BGMS) assessed according to ISO 15197:2013 criteria.Methods: Clinical trial registered at ClinicalTrials.gov (NCT01824355). In a laboratory study (Study 1), capillary fingertip blood samples from 100 subjects were evaluated in duplicate, using three test strip lots. In a clinical trial (Study 2), 113 adults with diabetes were enrolled, and BGMS results and Yellow Springs Instruments (YSI) analyzer reference measurements were compared for subject- and trial staff-obtained fingertip blood and subject-obtained palm blood. Subjects completed a questionnaire to evaluate BGMS ease of use.Results: In Study 1, 100% of combined results (all test strip lots) fulfilled ISO 15197:2013 section 6.3 criteria. In Study 2, 97.7% of subject-obtained fingertip results and 100% of trial staff-obtained fingertip results met ISO 15197:2013 section 8 criteria. Additionally, 93.8% of palm results were within ± 15 mg/dL of mean YSI measurements for glucose concentrations <100 mg/dL or ± 15% for glucose concentrations ≥100 mg/dL. Most subjects found the BGMS easy to use. There were three non-serious, non-device related adverse events.Conclusion: The BGMS exceeded minimum ISO 15197:2013-specified accuracy criteria in the laboratory and in the hands of lay users with diabetes.

User performance evaluation of the systems for self-monitoring of blood glucose GL50 evo and GL 44 following DIN EN ISO 15197:2015 – A comparison of accuracy in glucose concentration rages &lt;100 mg/dl and ≥100 mg/dl

User performance evaluation of the systems for self-monitoring of blood glucose GL50 evo and GL 44 following DIN EN ISO 15197:2015 – A comparison of accuracy in glucose concentration rages &lt;100 mg/dl and ≥100 mg/dl

Performance Evaluation of Alarm Forms under Different Visual Coding on the Digital Interface

To investigate user performance evaluation for different alarm forms on the digital interface under single dimensional information coding, such as color, shape, flash frequency, size and number, this study designed 5 visual alarm modes by following the design specification of visual alarm signal and carried out behavioral experiments for a certain radar interface. Based on the statistical analysis of accuracy rate and reaction time and comprehensive experimental interviews, the results reveal that, visual coding alarm form in the best identification performance is number and the worst is flash frequency. The research conclusions can be applied to the actual alarm interface design. It has a certain reference value for reducing the cognitive friction between the user and the digital interface, establishing a good man-machine cognitive matching relationship and perfecting the usability evaluation system of information coding.

Evaluation of User Performance in Interactive and Static 3D Maps

Interactive 3D visualizations of geospatial data are currently available and popular through various applications such as Google EarthTM and others. Several studies have focused on user performance with 3D maps, but static 3D maps were mostly used as stimuli. The main objective of this paper was to identify differences between interactive and static 3D maps. We also explored the role of different tasks and inter-individual differences of map users. In the experimental study, we analyzed effectiveness, efficiency, and subjective preferences, when working with static and interactive 3D maps. The study included 76 participants and used a within-subjects design. Experimental testing was performed using our own testing tool 3DmoveR 2.0, which was based on a user logging method and open web technologies. We demonstrated statistically significant differences between interactive and static 3D maps in effectiveness, efficiency, and subjective preferences. Interactivity influenced the results mainly in ‘spatial understanding’ and ‘combined’ tasks. From the identified differences, we concluded that the results of the user studies with static 3D maps as stimuli could not be transferred to interactive 3D visualizations or virtual reality.

Su1811 - Home Based Fecal Calprotectin Testing: A Canadian User Performance Evaluation Study of IBDoc®

Su1811 - Home Based Fecal Calprotectin Testing: A Canadian User Performance Evaluation Study of IBDoc®