Towards reducing teacher burden in Performance-Based assessments using aivaluate: an emotionally intelligent LLM-Augmented pedagogical AI conversational agent

Background Magnetic resonance imaging (MRI) safety screening is a crucial procedure for patient preparation before entering into MRI room. Many hospitals in Malaysia are still using the MRI safety checklist printed form. Besides, clinicians will not get a definite conclusion about whether the patient is contraindicated for MRI or not. Hence, we have created one mobile application named MagnetoSafe to overcome this issue. The application will provide an instant decision on whether the patient has no contraindication, relative contraindication, or absolute contraindicated for MRI. We need to check for acceptability and user experience for any newly created mobile application. Objective This study was designed to check the validity of the adapted Technology Acceptance Model (TAM) and System Usability Scale (SUS) Questionnaire. Method The validity and reliability of the questionnaire were investigated. Subsequently, 52 fully completed responses were collected. Results Face and content validity of the questionnaires are considered acceptable with only minor changes to Item 10 of SUS. The Cronbach's alpha for the SUS questionnaire (10 questions) is -0.49, which is not acceptable. The Cronbach's alpha for TAM questionnaire (3 domains; 14 questions) is acceptable, which is 0.910 for perceived usefulness, 0.843 for perceived ease of use, and 0.915 for intention to use. Conclusion Face validity of the adapted SUS and modified TAM questionnaires is acceptable with only minor changes to Item 10 in SUS. Content validity with experts is good. However, the reliability of the SUS questionnaire is not acceptable and therefore adapted SUS will not be used for assessing user experience. The reliability of the modified TAM questionnaire with the original three-factor structure is considered acceptable and can be used to evaluate the user's acceptability of MagnetoSafe.

Introduction: There are many information systems in Sri Lanka. However, there is no widely accepted tool to measure their usability. System Usability Scale (SUS) is a standardised metric or a quantitative measuring scale for measuring the usability of a website, information system or interactive system. SUS was released to the world by John Brooke in 1986. Currently, SUS has become an industry standard. In SUS, a user responds to 10 short questions on a 5-point Likert scale about the level of agreement. A special methodology is used to prevent unintentional rating responses. A SUS score above 68 would be considered above average. This questionnaire is originally in English language. However, even a native English speaker has implications on reliability and validity of the SUS due to the misunderstanding and misinterpretation of terminology or conflicting language. Thus, translating to native local languages is considered to increase the reliability and validity of SUS. The aim was to develop a Sinhala/Tamil System Usability Scale (SUS) to be used in Sri Lanka for assessing the information system usability. Methods: Translation of SUS questions in English to Sinhala and Tamil Languages after brainstorming sessions. Questions were validated by multi-disciplinary specialists. Translations were aimed at keeping the content simple, clear and concise, achieving a conceptual translation rather than a literal translation and avoid technology jargon. The tool was pretested by using a group of users in Electronic Reproductive Health Information Management System (eRHMIS) of the Family Health Bureau. Results: Sinhala/Tamil questions were finalized, and final SUS questionnaire was developed including a 5-point Likert scale. This tool was selected for measuring the usability of the eRHMIS of Family Health Bureau. Conclusions: There was immediate acceptance as well as plans for future use of translated SUS. This SUS questionnaire can be used to measure the usability of existing information systems in the country.

Over the past few decades, medicine and healthcare have significantly improved and decentralized, offering high-quality yet accessible care to individuals. Despite mHealth's substantial role in postpartum and maternity care, the adoption rate in rural and remote areas remains limited in Indonesia. mHealth apps for tracking and monitoring during and after pregnancy can potentially improve both the mother and child’s quality of life. Primagravida is a web-based application introduced for maternal and childcare in a municipality at Kulon Progo Regency, Yogyakarta, Indonesia. A collaborative approach with two primary health care and the local government facilitates expert teams to reach out to pregnant women and parents at risk for health complications. Field supervision sessions were organized to coach targeted populations and voluntary health workers. At the end of the sessions, a System Usability Scale (SUS) questionnaire was distributed to capture users’ experiences. Generally, a SUS score over 68 is considered good [3]. Primagravida application was acceptable and easy to learn among the users. However, the SUS questionnaire was intended as a quick and dirty assessment tool for shallow evaluation. The questions are aimed at identifying possible inconsistencies in the system rapidly. The future study must address more detailed identification with rigorous and comprehensive methods. The study's overall finding is that Primagravida was valued as satisfactory by the targeted audience of users. However, the app's usability is expected to improve through an iterative approach continuously. The findings of this study highlight the connection between efficient health apps and how their layout may foster patient engagement in care delivery. Therefore, developing health apps is essential to introduce an interdisciplinary strategy with early target group participation.

Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context

The aim of this paper is to evaluate user satisfaction based on System Usability Scale (SUS) questionnaire on Augmented Reality (AR) application for productive vocabulary using speech recognition. There is still lack of research focus on user satisfaction towards the use of AR-based app using speech recognition for vocabulary learning in early education. The first objective of this paper is to develop an AR application for children using speech recognition to enhance productive vocabulary learning that integrates visual script (orthography) and audio (phonology). The second objective is to evaluate and identify the user satisfaction in enhancing productive vocabulary methods by combining AR technology and speech recognition to the parents and teachers by doing a mixed method testing such as questionnaire and interview. To achieve this, an interview session was conducted with the experts and SUS questionnaire was given to the teachers and parents of the student to evaluate user satisfaction. The results show that the research hypotheses of this study were successfully achieved. It revealed that the teachers and students were satisfied with the application based on SUS score (SUS score > 68). Since the total SUS score is 80.3, which is above average, it shows that they were satisfied with the overall application. Apart from that, the mean of the usability study based on questionnaire also shows that the users have found that this application is usable in terms of learnability study and that this application is easy to use.

Background: Intelligent powered wheelchairs remain a popular research topic that can improve users’ quality of life. Although our multidisciplinary research team has put a lot of effort into adding features based on end-users needs and impairments since 2006, there are still open issues regarding the usability and functionalities of an intelligent powered wheelchair (IPW). Methods: For this reason, this research presents an experience with our IPW followed by a study in two parts: a quantitative one based on the System Usability Scale (SUS) questionnaire and a qualitative one through open questions regarding IPW functionalities with novice users, e.g., IPW non-users. These users never used an IPW before, but are users and aware of the impacts of the technology used in our IPW, being undergraduate to postdoctoral students and staff (faculty, lecturers, research engineers) at the Faculty of Engineering of Université de Sherbrooke. Results: The qualitative analyses identified different behaviours among the novice users. The quantitative analysis via SUS questionnaire done with novice users reports an “okay” rating (equivalent with a C grade or 68 SUS Score) for our IPW’s usability. Moreover, advantages and disadvantages opinions were gathered on the IPW as well as comments which can be used to improve the system. Conclusions: The results reported in these studies show that the system, e.g., IPW, was judged to be sufficiently usable and robust by novice users, with and without experience with the software used in developing the IPW.

The rapid advancement in technology necessitates robust financial information systems (FIS) in educational institutions to manage the increasing complexity and volume of financial transactions. This study evaluates the usability of developed web-based Financial Information System for schools and the Foundation, designed to automate and streamline the management of student tuition fee records. The evaluation employs two prominent usability metrics: the System Usability Scale (SUS) and the Usability Metric for User Experience-Lite (UMUX-Lite). The SUS and UMUX-Lite are tools designed to quantitatively assess a system's usability and user satisfaction. The SUS provides a quick, reliable measure of a product's usability, while UMUX-Lite offers a concise evaluation of user satisfaction with less respondent time without compromising insight quality. Our research methodology included distributing questionnaires to 41 schools, employing SUS and UMUX-Lite questions to gather comprehensive feedback on the FIS's usability. Analysis of the data revealed a SUS score of 77.5, placing the system's usability in the "B+" category, indicative of good user satisfaction. The UMUX-Lite score was slightly higher at 81.69, reflecting excellent usability and aligning with user needs and ease of use. These results affirm the effectiveness of the developed FIS, highlighting its potential to significantly improve financial management processes within schools. The close correlation between SUS and UMUX-Lite scores underscores the system's robustness and intuitive design, suggesting that minor enhancements could further elevate its usability and overall user experience. This study contributes to the field by demonstrating the practical application of SUS and UMUX-Lite in evaluating the usability of financial information systems in educational settings, offering insights for future system enhancements and user-centered design practices.

The integration of Augmented Reality (AR) in chemistry education, particularly in acid-base titration, offers an interactive approach that can improve students' understanding. This study aimed to assess the effectiveness of AR in enhancing learning outcomes for tenth-grade students at SMA Negeri 4 Denpasar, while also evaluating the user experience through the System Usability Scale (SUS) and User Experience Questionnaire (UEQ). A total of 60 students participated in AR-based learning, and the evaluation focused on changes in learning outcomes before and after using AR. The SUS was used to measure system usability, while the UEQ assessed user experience in terms of attractiveness, Perspicuity, efficiency, stimulation, and novelty. Results showed a significant improvement in student understanding, with an average test score increase of 25%. The SUS yielded a high usability score of 85, indicating that the AR system was easy to use. Additionally, the UEQ highlighted a positive user experience, particularly in stimulation and attractiveness, both rated "excellent." These findings suggest that AR not only improves learning outcomes but also provides an engaging and motivating learning experience.

The Pharmaceutical Services Program (PSP) has not yet evaluated the usability of the web application system. The System Usability Scale (SUS) and Net Promoter Score (NPS) assessment tools give you a quick overview of your system's usability and help you improve it further. The goal of this study was to assess the usability of the PRiSMA and My.Pharma-C systems using SUS and NPS. From April 2019 to July 2020, a survey was conducted using a validated self-administered questionnaire that was prompted after each successful log-out from the PRiSMA and My.Pharma-C systems. The questionnaire was divided into three sections: demographic, device type, and SUS and NPS questionnaire. The mean SUS and NPS scores were computed. The survey was completed by 3,959 PRiSMA users and 811 My.Pharma-C users. The SUS score was interpreted as a grade, adjective, and range of acceptability. The PRiSMA mean SUS score was 70.2 (SD 15.44), indicating a 'Grade C' and 'Good' system. The mean SUS score for My.Pharma-C was 58.2 (SD 15.13), indicating a 'Grade D' and 'OK' system. Users accepted both systems on the basis of their usability. Calculated NPS value for PRiSMA was p=42%, n=49% and d=9%, NPS value=+33, whereas My.Pharma-C p=18%, n=46% and d=36%, NPS value=-18. A positive NPS value for PRiSMA represents the user's readiness to recommend the system to others compared to My.Pharma-C. The findings also established that SUS alone is insufficient and needs to be complemented by another method, such as qualitative instruments. It could help to drill down in depth to understand how to address potentially problematic areas for web application systems and assist in the improvement phases. However, both of the tools are applicable to be used to evaluate the usability of the web application systems that managed by the Pharmaceutical Services Program.

BackgroundAfter a mobile health (mHealth) app is created, an important step is to evaluate the usability of the app before it is released to the public. There are multiple ways of conducting a usability study, one of which is collecting target users’ feedback with a usability questionnaire. Different groups have used different questionnaires for mHealth app usability evaluation: The commonly used questionnaires are the System Usability Scale (SUS) and Post-Study System Usability Questionnaire (PSSUQ). However, the SUS and PSSUQ were not designed to evaluate the usability of mHealth apps. Self-written questionnaires are also commonly used for evaluation of mHealth app usability but they have not been validated.ObjectiveThe goal of this project was to develop and validate a new mHealth app usability questionnaire.MethodsAn mHealth app usability questionnaire (MAUQ) was designed by the research team based on a number of existing questionnaires used in previous mobile app usability studies, especially the well-validated questionnaires. MAUQ, SUS, and PSSUQ were then used to evaluate the usability of two mHealth apps: an interactive mHealth app and a standalone mHealth app. The reliability and validity of the new questionnaire were evaluated. The correlation coefficients among MAUQ, SUS, and PSSUQ were calculated.ResultsIn this study, 128 study participants provided responses to the questionnaire statements. Psychometric analysis indicated that the MAUQ has three subscales and their internal consistency reliability is high. The relevant subscales correlated well with the subscales of the PSSUQ. The overall scale also strongly correlated with the PSSUQ and SUS. Four versions of the MAUQ were created in relation to the type of app (interactive or standalone) and target user of the app (patient or provider). A website has been created to make it convenient for mHealth app developers to use this new questionnaire in order to assess the usability of their mHealth apps.ConclusionsThe newly created mHealth app usability questionnaire—MAUQ—has the reliability and validity required to assess mHealth app usability.

This paper presents the usability evaluation of a gimmick tracking application designed by students in the Human-Computer Interaction class to solve the problem in one of the biggest food companies in Indonesia. Gimmick tracking is important for the company to make sure the gimmick can be transferred to the customers with minimum-to-none problems. The gimmick tracking application should be user-friendly for the users to make sure the application meets the purpose of its development. Quantitative and qualitative usability evaluation using the system usability scale (SUS) questionnaire and think-aloud method were used to evaluate how effective is the real-time gimmick tracking application. A total of 12 subjects with an age range from 18 to 27 from the grocery stores in Malang were recruited to participate in this study. The SUS questionnaire was used to measure the hypotheses related to learnability, efficiency, memorability, errors, user satisfaction, and usability. The qualitative data such as recorded videos and notes were transcribed into the text for further analysis. The SUS score of the design of the gimmick tracking application was 70.83, this score is higher than 55 percent of other applications. Thus, it can be concluded that our design generally has fulfilled all usability requirements.

BackgroundA mobile app is a programmed system designed to be used by a target user on a mobile device. The usability of such a system refers not only to the extent to which product can be used to achieve the task that it was designed for, but also its effectiveness and efficiency, as well as user satisfaction. The System Usability Scale is one of the most commonly used questionnaires used to assess the usability of a system. The original 10-item version of System Usability Scale was developed in English and thus needs to be adapted into local languages to assess the usability of a mobile apps developed in other languages.ObjectiveThe aim of this study is to translate and validate (with cross-cultural adaptation) the English System Usability Scale questionnaire into Malay, the main language spoken in Malaysia. The development of a translated version will allow the usability of mobile apps to be assessed in Malay.MethodsForward and backward translation of the questionnaire was conducted by groups of Malay native speakers who spoke English as their second language. The final version was obtained after reconciliation and cross-cultural adaptation. The content of the Malay System Usability Scale questionnaire for mobile apps was validated by 10 experts in mobile app development. The efficacy of the questionnaire was further probed by testing the face validity on 10 mobile phone users, followed by reliability testing involving 54 mobile phone users.ResultsThe content validity index was determined to be 0.91, indicating good relevancy of the 10 items used to assess the usability of a mobile app. Calculation of the face validity index resulted in a value of 0.94, therefore indicating that the questionnaire was easily understood by the users. Reliability testing showed a Cronbach alpha value of .85 (95% CI 0.79-0.91) indicating that the translated System Usability Scale questionnaire is a reliable tool for the assessment of usability of a mobile app.ConclusionsThe Malay System Usability Scale questionnaire is a valid and reliable tool to assess the usability of mobile app in Malaysia.

<div class="section abstract"><div class="htmlview paragraph">The System Usability Scale (SUS) is used across industries, to evaluate a product’s ease of use. As the automotive industry increases its digital footprint, the SUS has found its application as a simple and reliable assessment of various in-vehicle human machine interfaces. These evaluations cover a broad scope and it is important to design studies with participant fatigue, study time, and study cost in mind. Reducing the number of items in the SUS questionnaire could save researchers time and resources. The SUS is a ten-item questionnaire that can measure usability and learnability, depending on the system. These ten questions are highly correlated to each other suggesting the SUS score can be determined with fewer items. Thus, the focus of this paper is two-fold: using principal component analysis (PCA) to determine the dimensionality of SUS and using this finding to reduce variables and build a regression equation for SUS scores for in-vehicle human machine interfaces. Data from 42 systems were used for this evaluation. Building a prediction model using items 3 (I thought the system was easy to use), 9 (I felt very confident using the system) and 10 (I needed to learn a lot of things before I could get going with the system) was determined to be the best way to reduce the questionnaire while preserving the learnability dimension that occurs in measuring in-vehicle systems. Further studies should be done on how a reduced questionnaire impacts the assumptions built into SUS protocol.</div></div>

Given that Indonesia has four times the number of districts as towns, regency creation needs serious attention. However, few studies look at the life of regencies from the standpoint of ICT utility governance. This study aims to explore and test the dimensions of usability in Smart Regency Mobile-Apps application services. The application being tested is Sumekar Online, a Smart Regency Mobile-Apps service application used in Sumenep Regency. The testing process was carried out by 10 participants, including testing using the System Usability Scale (SUS) and User Experience Questionnaire (UEQ), adapted to the Indonesian context. The test results with SUS show that mobile-based applications get a score of 77.75, which means that the Acceptability Range category is "Acceptable," the Grade Scale category is "C,"; and the Adjective Rating category is "Good". Meanwhile, the evaluation results with UEQ get a value of 1.35 with details of attractiveness is 0.96, clarity is 0.72, novelty is 0.3, stimulation is 0.62, accuracy is 0.75, and efficiency is 0.85. This study contributes by providing suggestions and recommendations so developers and policymakers pay better attention to influencing the usability dimensions of Smart Regency Mobile-Apps services.

Promotion in dental health services in Indonesia must adhere to the dental ethics code. One form of promotion is introducing the various services and facilities of a dental clinic. A website can serve as a medium for introducing a dental clinic. This study aims to design and develop a dental clinic website to improve dental health services and patient satisfaction. The research method used is research and development with an action research approach. The study employed the waterfall System Development Life Cycle (SDLC) method. The website development process was based on a needs analysis, which included interviews with the dental clinic owner, website creation, testing, and evaluation using the System Usability Scale (SUS) questionnaire containing 10 Likert-scale questions. The study results showed that the average evaluation score using the SUS questionnaire was 84.3. Consequently, the adjective ratings fell into the "Excellent" category, the acceptability ranges fell into the "Acceptable" category, and the letter grade fell into category A. The conclusion drawn was that developing a dental clinic website prototype provided beneficial results for both the clinic owner and patients, particularly in obtaining information about dental health services.