Aloud Protocols Research Articles

Understanding decision processes in guessing games: a protocol analysis approach

Protocol analysis, in the form of concurrent verbal ‘thinking aloud’ reports, is a method of collecting and analyzing data about cognitive processes. This approach can help economists in evaluating competing theories of behavior and in categorizing heterogeneity of thinking patterns. As a proof of concept, I tested this method in the context of a guessing game. I found that concurrent think aloud protocols can inform us about individual’s thought processes without affecting decisions. The method allowed me to identify game theoretic thinking and heterogeneous approaches to unravelling the guessing game. The think aloud protocol is inexpensive and scalable, and it is a useful tool for identifying empirical regularities regarding decision processes.

The influence of multimedia development knowledge and workplace pressures on the design decisions of the instructional designer

This study explored the interaction of multimedia production competencies of expert and novice instructional designers on the design decisions made during the instructional design process/workflow. This multiple measures study used qualitative survey instruments to access and measure the production competencies of participants, then a design aloud protocol to capture and measure the instructional design decision-making process for those same participants. A follow-on interview after the initial design aloud session was conducted in order to triangulate and confirm any trends or findings uncovered during the earlier design aloud session. Ultimately, the objective of this study was to provide some evidence that suggests whether certain production skills are influencing instructional design decision-making. Employer influence on the instructional designer’s decision-making was also explored. Results indicated that a substantial number of instructional designers (n = 30) who participated in this study were selecting media as a preliminary step in their workflow process, and were often then using analysis as a measure to confirm the early media selection. Expert instructional designers appeared to be less susceptible to the early media selection behavior, though not immune. Results indicate that one reason the expert instructional designers were less likely to adopt media as a preliminary instructional design step was that the experts conducted a more diverse set of analysis activities. Additionally, results indicated that instructional designers were often experiencing pressure to adopt media based on employer demands, and project constraints such as budget and time.

A Mobile Phone App for the Self-Management of Pediatric Concussion: Development and Usability Testing.

BackgroundConcussion is a common injury among Canadian children and adolescents that leads to a range of neurobehavioral deficits. However, noticeable gaps continue to exist in the management of pediatric concussion, with poor health outcomes associated with the inadequate application of best practice guidelines.ObjectiveThe aim of this study was to describe the development and assess the usability of a mobile phone app to aid youth in the self-management of concussion. A secondary objective was to assess the usefulness of the app.MethodsAn agile user-centered design approach was used to develop the technology, followed by a formative lab-based usability study for assessment and improvement proposals. Youths aged 10 to 18 years with a history of concussion and health care professionals involved in concussion management were recruited. This study included participants performing 12 tasks with the mobile phone app while using the think aloud protocol and the administration of the System Usability Scale (SUS), posttest questionnaire, and a semistructured interview.ResultsA mobile phone app prototype called NeuroCare, an easily accessible pediatric concussion management intervention that provides easy access to expert-informed concussion management strategies and helps guide youth in self-managing and tracking their concussion recovery, was developed. A total of 7 youths aged between 10 and 18 years with a history of concussion and 7 health care professionals were recruited. The mean SUS score was 81.9, mean task success rates were greater than 90% for 92% (11/12) of the tasks, 92% (11/12) of tasks had a total error frequency of less than 11 errors, and mean task completion times were less than 2 min for 100% of the tasks.ConclusionsResults suggest that participants rated this app as highly usable, acceptable to users, and that it may be useful in helping youth self-manage concussion.

Emerging patterns and problems of higher-order thinking skills (HOTS) mathematical problem-solving in the Form-three assessment (PT3)

Higher-order thinking skills (HOTS) mathematical problem-solving was first introduced in the Form-3 assessment (PT3) in 2014. However, to date, there have been no studies of students’ ability to solve the mathematical problems in this assessment. Therefore, this study investigated the emerging patterns and problems of HOTS mathematical problem-solving in the PT3. This investigation was a case study and classified under a qualitative research approach. Oral reporting (i.e. thinking aloud protocol) was used to obtain the data. The participants were 10 Form-3 students who were candidates for PT3 in 2015. They were students in a secondary school in a district in Johor Bahru. The results show that students who successfully solved the HOTS mathematical problems produced the same process starting with understanding, followed by phases of planning, implementation, and ending with the final answer. The students who failed to answer the HOTS mathematics questions produced a solution pattern starting with understanding followed by planning and implementation. Based on the patterns, this study also identified the problems that emerged in every step of the HOTS mathematical problems-solving processes and discusses how they could be overcome and improved. Keywords: Form-three assessment (PT3); higher-order thinking skills (HOTS); mathematical problem-solving; Model of Polya

Empirical Insights of Individual Website Adjustments for People with Dyslexia.

The existing body of knowledge reveals that customisable websites may lead to an increase in accessibility and usability for people with disabilities. In this way, the main goal of this research was to investigate how people with dyslexia respond to a customised version of a website in terms of its effectiveness, efficiency, satisfaction and suitability when compared to the default version of the website. The customisation of the investigated website was enabled with the aid of integrated assistive technology that offers people with dyslexia the opportunity to adjust a website themselves in accordance with their individual needs, demands and preferences. They can do this by changing the parameters, such as font size, font type and contrast between the background and text. The answers to the research questions were obtained with complementary research methods and techniques, including formal usability testing, thinking aloud protocol, log analyses, questionnaires and interviews. The empirical results show that participants experienced more issues when interacting with the default website, and they enjoyed more benefits when using the customised website. Too much information on the screen, not enough graphic elements, issues with visual appearance and inappropriately presented information were identified as the most common issues when interacting with the default website. When using the customised website, all participants agreed on a better user experience and, as the majority of them reported, this was due to appropriate contrast and font size. Additionally, the majority of participants also expressed desire to use the individual website adjustments regularly in the future. The conclusions of this investigation are that the individual website adjustments used in this research can not only help to minimise issues, but also eliminate challenges that people with dyslexia have when interacting with a website. Therefore, the primary contributions of this research are the empirical insights of interaction with both the default and customised version of the website for people with dyslexia. Furthermore, this research also has three secondary contributions: (1) detailed presentation and application the general usability evaluation procedure to a specific target group (people with dyslexia); (2) recommendations to adapt the usability evaluation methods for people with dyslexia; and (3) the usage of quantitative measurement instruments for the evaluation of a website’s usability and suitability for people with dyslexia.

Assessing usability of eHealth technology: A comparison of usability benchmarking instruments

BackgroundIt is generally assumed that usability benchmarking instruments are technology agnostic. The same methods for usability evaluations are used for digital commercial, educational, governmental and healthcare systems. However, eHealth technologies have unique characteristics. They need to support patients’ health, provide treatment or monitor progress. Little research is done on the effectiveness of different benchmarks (qualitative and quantitative) within the eHealth context. ObjectivesIn this study, we compared three usability benchmarking instruments (logging task performance, think aloud and the SUS, the System Usability Scale) to assess which metric is most indicative of usability in an eHealth technology. Also, we analyzed how these outcome variables (task completion, system usability score, serious and critical usability issues) interacted with the acceptance factors Perceived benefits, Usefulness and Intention to use. MethodsA usability evaluation protocol was set up that incorporated all three benchmarking methods. This protocol was deployed among 36 Dutch participants and across three different eHealth technologies: a gamified application for older adults (N = 19), an online tele-rehabilitation portal for healthcare professionals (N = 9), and a mobile health app for adolescents (N = 8). ResultsThe main finding was that task completion, compared to the SUS, had stronger correlations with usability benchmarks. Also, serious and critical issues were stronger correlated to task metrics than the SUS. With regard to acceptance factors, there were no significant differences between the three usability benchmarking instruments. ConclusionsWith this study, we took a first step in examining how to improve usability evaluations for eHealth. The results show that listing usability issues from think aloud protocols remains one of the most effective tools to explain the usability for eHealth. Using the SUS as a stand-alone usability metric for eHealth is not recommended. Preferably, the SUS should be combined with task metrics, especially task completion. We recommend to develop a usability benchmarking instrument specifically for eHealth.

Psikologi Peluang Kewirausahaan: Proses Kognitif Pengusaha Startup Digital dalam Opportunity Recognition

Developed countries have at least 2% entrepreneurs from the total population of the country. However, Indonesia have a very low number of entrepreneurs as well as an increasing unemployment rate over the years. Recently, the development of entrepreneurship in Indonesia dominated by the growing number of digital startup. Moreover, Indonesia is a country with the largest number of digital startup in Southeast Asia. Initial psychological studies on entrepreneurship are focused on the personal characteristics of the entrepreneurs. However, conclusions from more than 30 years of research indicate that there are no special personality characteristics of entrepreneurs and non-entrepreneurs. In essence, the heart of entrepreneurship lies in the ability to recognize an opportunity. Opportunity recognition is a mechanism that happens in an individual’s cognitive process. Therefore, this study intended to reveal the cognitive processes that take place when entrepreneurs, in particular the founders of digital startups, in the identification of entrepreneurial opportunities. This research use think aloud protocol method with protocol analysis. Based on this research, we found that in the process of opportunity recognition, entrepreneurs focus their cognitive efforts on the market (demand) and technology (supply) as well as build relationships and meaningful patterns in these two aspects through structural relationship processing. This study provides an in-depth description of the cognitive processes that occur when entrepreneurs recognize entrepreneurial opportunities through structural alignment processes.

Language and environmental justice: articulating intersectionality within energy policy deliberations

ABSTRACT The widespread practice of shale gas extraction via high volume hydraulic fracturing (HVHF) has created challenges for communities who attempt to understand HVHF risk. This study uses a critical discourse lens to understand how intersectionality, and environmental justice language frames might be more clearly established in tools that stakeholders must use to engage in energy policy deliberation. I examine how stakeholders in Ohio, Pennsylvania, and New York navigate public comment tools in an attempt to understand environmental risk. Using both interview and think aloud protocol, I find that possibilities to voice concern about HVHF risk vary by locality and state. HVHF policy deliberation that does not often include ‘intersectional’ and ‘environmental justice’ frameworks results in an uneven distribution of HVHF risk across and within states. By understanding various discourse frames within policy tools in different states, a more just system of risk identification might emerge.

Assessing information interaction to improve library ebook collections and services

That university students encounter issues and experience frustration when using ebooks is well documented, but more research into the specific issues they encounter is needed. This study focused on how successfully undergraduate and graduate students use ebooks and whether there are measurable differences between the two groups. Researchers employed user testing methodologies, including task analysis and talk aloud protocol, to research student interaction with the content and structure of ebooks on two platforms: ProQuest Ebook Central and EBSCO eBooks. Findings include suggested improvements for ebook platforms, considerations for collection development librarians, and areas of focus for library instructors.

Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: A case study and beginning method validation.

Self-management of chronic diseases using mobile health (mHealth) systems and applications is becoming common. Current evaluation methods such as formal usability testing can be very costly and time-consuming; others may be more efficient but lack a user focus. We propose an enhanced cognitive walkthrough (CW) method, the user-centered CW (UC-CW), to address identified deficiencies in the original technique and perform a beginning validation with think aloud protocol (TA) to assess its effectiveness, efficiency and user acceptance in a case study with diabetes patient users on a mHealth self-management application. A total of 12 diabetes patients at University of Utah Health, USA, were divided into UC-CW and think aloud (TA) groups. The UC-CW method included: making the user the main evaluator for detecting usability problems, having a dual domain facilitator, and using three other improved processes: validated task development, higher level tasks and a streamlined evaluation process. Users interacted with the same mHealth application for both methods. Post-evaluation assessments included the NASA RTLX instrument and a set of brief interview questions. Participants had similar demographic characteristics. A total of 26 usability problems were identified with the UC-CW and 20 with TA. Both methods produced similar ratings: severity across all views (UC-CW = 2.7 and TA = 2.6), numbers of problems in the same views (Main View [UC-CW = 11, TA = 10], Carbohydrate Entry View [UC-CW = 4, TA = 3] and List View [UC-CW = 3, TA = 3]) with similar heuristic violations (Match Between the System and Real World [UC-CW = 19, TA = 16], Consistency and Standards [UC-CW = 17, TA = 15], and Recognition Rather than Recall [UC-CW = 13, TA = 10]). Both methods converged on eight usability problems, but the UC-CW group detected five critical issues while the TA group identified two. The UC-CW group identified needed personalized features for patients' disease needs not identified with TA. UC-CW was more efficient on average time per identified usability problem and on the total evaluation process with patients. NASA RTLX scores indicated that participants experienced the UC-CW half as cognitively demanding. Common themes from interviews indicated the UC-CW as enjoyable and easy to perform while TA was considered somewhat awkward and more cognitively challenging. UC-CW was effective for finding severe, recurring usability problems and it highlighted the need for personalized user features. The method was also efficient and had high user acceptance. These results indicate UC-CW's utility and user acceptance in evaluating a mHealth self-management application. It provides an additional usability evaluation technique for researchers.

The relation of student’s diagram comprehension, knowledge and cognitive activities while studying mosses metagenesis diagram

The students’ success in understanding the visual representation such as diagram, can be determined by cognitive activities that occurs in working memory. One of the diagrams that are abstract and difficult to understand while studying Biological material is plant metagenesis diagram. The aim of the study is to reveal the types of cognitive activities that relate to knowledge and diagram comprehension while students read mosses metagenesis diagram. Twenty five Biology education students completed mosses metagenesis knowledge test and diagram comprehension test (m = 48,9). Verbal data were collected by Think Aloud Protocol to find several types of cognitive activities while students read mosses metagenesis diagram. Four types of cognitive activities that found and correlated strongly with diagram comprehension, were 1) identifying of image detail (r = 0.917; p <0.01), 2) symbols interpretation (r = 0.862; p <0.01), 3 ) activating prior knowledge (r = 0.703; p <0.01), and 4) inference (r = 0.773; p <0.01). These results indicate the high frequency of identifying image detail, symbols interpretation, activation prior knowledge, and inference contributing to the high understanding of mosses metagenesis diagram. This study also reveals that students’ knowledge has contributed highly to students’ diagrams comprehension.

Mobile health for older adult patients: Using an aging barriers framework to classify usability problems

BackgroundWith populations aging, digital health tools and mobile health applications (mHealth) are becoming more common to assist older people in independent living and self-management of (chronic) illnesses. These mHealth services can be beneficial to older patients, provided that they are adjusted to their needs and characteristics, as the current mHealth landscape lacks user-friendly services for this target group. Understanding of intrinsic aging barriers, which cause and impact usability problems older patients encounter, is needed to achieve this. ObjectivesThis study set out to assess usability problems older patients encounter in two mHealth apps and aims to show the value of MOLD-US, a recent aging barriers framework, as a classification tool to identify the intrinsic cause of these problems. MethodA case-study design, with in-depth analysis of usability issues older adult patients’ experience. Data on usability issues were collected using the Think Aloud Protocol for two mHealth apps. The MOLD-US framework and Nielsen’s severity rating were used to classify identified issues and their potential impact. ResultsIn total 28 high severe usability issues of the mHealth apps were identified. Core natures of most issues were related to motivational and cognitive barriers of older adults. Participants had difficulties in understanding the navigation structure of the apps. Important text, buttons and icon elements were overseen. ConclusionCurrent knowledge on creating interfaces for older target groups is not well applied within the assessed mHealth designs. Specifically, design guidelines should address older adults’ diminishing cognition skills, physical ability and motivational barriers. By classifying usability problems with MOLD-US, insights on these barriers can be enhanced to adequately address these issues in new designs. In addition, we propose that future research focuses on investigating suitable usability evaluation methods adapted to older patients’ characteristics to ultimately be able to gain unbiased sight on usability issues older patients may experience while interacting with technology.

Low-fidelity Prototype Design for Serious Game for Slow-reading Students

Serious game is an alternative teaching aid that is getting a place of use by teachers and parents. Its widespread use has basically changed the way of life and learning of children and has a positive impact on achievement and increased the motivation of children in learning. However, not all serious game designs are suitable for slow-reading students. They are slightly different from other students in terms of cognitive potential and are struggling to meet academic demands in the class. Therefore, the main objective of this study is to produce low-fidelity prototypes involving target users as early as the design process. This study focuses on the production of storyboard contents suitable for slow-reading students to save time and cost of game model development. This study uses a child-centered design (CCD) method that involves paper prototypes, chauffeured prototype, think aloud protocols and observations. The results of this study are low-fidelity prototypes in the form of computerized storyboards that have been verified and will be used for heuristic assessments. These low-fidelity prototypes are expected to give an early look and help researchers in developing high-fidelity prototypes.

Cognitive Analysis of Constructing Algebraic Proof Processes: A Mixed Method Research

This study aimed in order to investigate cognitive aspects of the processes of constructing algebraic proofs of pre-service and service secondary math teachers. We used explanatory sequential design from mixed research methods in the study. Secondary math teachers working in the province east of Turkey and pre-service secondary math teacher in this province participated to the study. Quantitative data of the study were collected through a “Proving Diagnostic Test” and qualitative data were gathered from the participants through a think aloud protocol. We used two different activity cards that each of them included one algebraic proof question in the think aloud protocol. Quantitative data were analyzed using descriptive and inferential statistics. Content analysis was applied to qualitative data. The study show that the cognitive skills performed by pre-service and service secondary math teachers emerge five categories: read the proposition of the proof, evaluating the correctness, determining strategies, carry out plans and heuristic shortcuts thinking strategy. Eight categories were identified in the theme (context of) of meta-cognitive skills. These categories are: facilitating the operations, questioning, awareness, planning, strategy determination, controlling, correlating, and analogical reasoning.

English

From the author&rsquo;s teaching experience, a greater number of Nigerian university students either stick monotonously to the active English sentences or use the expletive &ldquo;It&rdquo; structure. This paper investigated the difficulties Igbo bilinguals encounter when learning the English passive and the grammar learning strategies they adopt to overcome the difficulties. For the study, 30 Igbo speaking 100 level students of the Department of English and Communication Arts, Ignatius Ajuru University of Education (IAUE) Port Harcourt were selected. At the end of teaching the English passive voice, two written achievement tests were given to the class from which the scripts of the sampled students were selected. Data from three academic sessions were used for the purpose of ensuring correctness and objectivity of results. For data elicitation on the difficulties encountered and learning strategies used, the students were asked to write diaries of their problems and how they coped with them. In &ndash;class observation and think aloud protocols were also used. The findings revealed that the most difficult aspect was the tense-aspect changes of the verb phrase and the most used strategy was affective strategy. Based on the findings, this paper discussed some pedagogical implications. &nbsp; Key words: English passive voice, difficulties, learning strategies, Igbo learners, pedagogical implications.

Influence of L1 Metacognitive Reading Strategies on L2 Academic Reading of Chinese College Students

Abstract With the fast increasing number of second language (L2) learners in universities, the ability to read L2 academic texts has become one of the most important skills that L2 college students need to acquire. However, L2 learners still face various problems and difficulties in their L2 reading and past literature suggests that reading strategies have proven to be effective in enhancing L2 reading. Therefore, it is of great significance to examine the factors influencing L2 reading strategies use. Among factors influencing L2 reading strategy use, L1 reading strategies have been identified as a significant influence. However, few studies in this regard have looked into college students in mainland China which has a large number of L2 learners. This study aimed to seek the connection between first language (L1) and L2 metacognitive reading strategies by comparing and contrasting mainland China college students’ metacognitive strategy patterns in their L1 and L2 academic reading. The study also explored factors influencing Chinese college students’ L2 reading strategy use. The study conducted a survey on reading strategies (SORS), think aloud protocol (TAP) and stimulated recall interviews (SRI) to explore the research issues. The study found that Chinese college students employed strategies on a more frequent basis in their L2 academic reading as compared to their L1 academic reading. Two factors, namely, limited L2 proficiency (LLP) and test-oriented reading approach (TORA) were identified as the major factors influencing the L2 metacognitive strategy patterns of Chinese college students. The present study adds to the existing knowledge on the relationship between L1 and L2 reading strategy patterns for Chinese college students and examines factors shaping their L2 reading strategy use. This study assists English language teachers to identify factors influencing Chinese college students’ reading strategies patterns while taking into consideration of the factors influencing their L2 strategy use.

How Signaling Principle Affects Learning: An Eye Tracking Study

Studies examining the effects of signaling principle on learning mostly used data sources such as interviews, achievement tests and think aloud protocols. There is, however, limited research on the use of quantitative measures to promote findings from interviews and think aloud procedures. This study aimed to investigate the effects of signaling principle on learning by supporting the results with the eye movement data. Participants included 34 Information and Communication Technology (ICT) pre-service teachers divided into experimental and control groups randomly. Participants were presented with two types of MS Power Point presentations with and without highlighted clues as multimedia material. Participants’ pre-test, post-test, and retention test scores were measured. Participants’ gaze behavior (fixation duration, fixation counts and time to first fixation) during the presentation of instructional material was also recorded. The results revealed that while the experimental group had higher scores than control group in the post- and retention tests, there was no significant difference between the two groups. The analysis of eye movement data showed that signaling directed the attention of the students on relevant information and decreased students’ cognitive effort. Results revealed that both eye movement data and achievement scores were consistent with each other.

Investigating stress and coping during practice and competition in tennis using think aloud

Aim: The purpose of this study was to examine stress and coping in both competition and practice in tennis and to further investigate gender difference using Think Aloud protocol (TA) in real-time. Method: 16 (8 males and 8 females) competitive tennis players took part. A within groups design was implemented, and participants verbalised their thoughts between points of a championship tie-break during a practice and a competition condition. Data were transcribed verbatim, analysed for stressors (confidence, performance, external, physical) and coping responses (problem, emotion, avoidance) using deductive analysis. A CSAI-2R questionnaire was used to assess anxiety levels prior to practice and competition. Results: CSAI-2R results showed cognitive anxiety significantly increased from practice to competition. Performance-focused coping (e.g. planning, technical) was verbalised most frequently in both conditions. Performance stressors (e.g. outcome, tactics) were verbalised most frequently in both conditions. Males verbalised significantly more performance stress in competition and physical stress in practice. Females verbalised external stress and utilise problem-focused responses more in competition than practice. Problem-focused coping was utilised most for males and females in both conditions. Conclusion: Through the use of a novel data collection method (TA) this study provides context-specific findings within tennis, which support previous research in stress and coping where gender differences occur only for the type of stressor appraised. TA has also been found to be a viable method to assess stress and coping data in tennis. Findings can inform coaches, players, and psychologists about stressors and coping responses utilised during practice and competition.

Conceptualizing the information seeking of college students on the autism spectrum through participant viewpoint ethnography

Due to increased numbers of diagnoses, targeted programs and initiatives, more students with autism spectrum disorder (ASD) are attending college (Heflin & Isbell, 2012; Zager & Alpern, 2007) but academic librarians and their staffs have not been trained to optimally serve this growing population. Utilizing wayfinding, think aloud protocol (TAP), retrospective think aloud protocol (RTAP), and a wearable camera, the actions, thoughts and feelings of a student with ASD and a neurotypical peer are evidenced as they navigate their campus library in search of materials. The library website, virtual maps to resource locations, and library workers served equally as enablers and barriers to both students in their information seeking. This study demonstrates that participant viewpoint ethnography is a viable research methodology for both neurotypical college students and those with autism.

Observing response processes with eye tracking in international large-scale assessments: evidence from the OECD PIAAC assessment

This paper reports on a pilot study that used eye tracking techniques to make detailed observations of item response processes in the OECD Programme for the International Assessment of Adult Competencies (PIAAC). The lab-based study also recorded physiological responses using measures of pupil diameter and electrodermal activity. The study tested 14 adult respondents as they individually completed the PIAAC computer-based assessment. The eye tracking observations help to fill an ‘explanatory gap’ by providing data on variation in item response processes that are not captured by other sources of process data such as think aloud protocols or computer-generated log files. The data on fixations and saccades provided detailed information on test item response strategies, enabling profiling of respondent engagement and response processes associated with successful performance. Much of that activity does not include the use of the keyboard and mouse, and involves ‘off-screen’ use of pen and paper (and calculator) that are not captured by assessment log-files. In conclusion, this paper points toward an important application of eye tracking in large-scale assessments. This includes insights into response processes in new domains such as adaptive problem-solving that aim to identify individuals’ ability to select and combine resources from the digital and physical environment.