Paper-based Assessments Research Articles

Designing usability/user experience heuristics to evaluate e-assessments administered to children

The application of electronic assessments (e-assessments) has increased, particularly among elementary-school-aged children. Paper-based assessments are frequently converted into digital formats for efficiency gains, with little thought given to their user experience (UX) and usability. Individual differences, particularly among young children, can inhibit test-takers from completing the assessment tasks that are not designed to match their needs and abilities. Consequently, studies have raised concerns about the generalizability and fairness of e-assessments. Whereas heuristic evaluation is a standard method for evaluating and enhancing the efficacy of a product with respect to a set of guidelines, more information is needed about its added value when designing e-assessments for children. This paper synthesizes heuristics on the basis of the literature and expert judgments to accommodate children's abilities for interacting with e-assessment platforms. We present a final set of 10 heuristics, validated and refined by applying a heuristic evaluation workshop and collecting 24 expert surveys. The results indicate that the derived heuristics can help evaluate the UX and usability-related aspects of e-assessments with 6- to 12-year-old children. Moreover, the present paper proposes recommendations for a framework for developing usability/UX heuristics that can be used to help researchers develop domain-specific heuristics in the future.

Examining The Integration Of ICT Tools In English Language Teaching: Comparative Analysis Of High School Teachers' Practices In India And Vietnam

This study explores the integration of Information and Communication Technology (ICT) tools in English language teaching practices of high school teachers in India and Vietnam. A survey with chart-based responses was distributed to 60 teachers(30 from each country) from nearby districts. The research investigates howteachers utilize ICT tools, their assessment practices, and perceived effects on student engagement and learning. The findings reveal that both countries employ ICT tools, with a focus on social media platforms, language learning applications, and online dictionaries. Vietnamese teachers demonstrated a slightly higher adoption rate, particularly for virtual reality (VR) and augmented reality (AR) technologies. While paper-based assessments remain prevalent, online tools like Google Forms and online assessment platforms are gaining traction. Teachers reported positive perceptions of ICT tools, believing they enhance student engagement and learning outcomes. However, challenges were identified, including limited access to technology resources and a lack of training on effective ICT integration. This study highlights the need for further research to directly measure the impact of ICT tools on student achievement. Additionally, it emphasizes the importance of addressing teacher training needs to unlock the full potential of ICT tools and create more engaging and effective English language learning experiences for students.

Improving symptom screening rates for patients with head and neck cancer.

e23246 Background: Effective symptom management in cancer care relies on regular and accurate reporting using validated patient-reported outcome measures (PROMs), such as the Edmonton Symptom Assessment System (ESAS). The ESAS measures nine common cancer symptoms, rating them on a scale from 0 to 10 to improve monitoring and communication regarding patient symptoms and prevent adverse outcomes such as emergency department visits. There is a need to enhance completion rates to ensure that head and neck cancer (HNC) symptoms are effectively addressed. High completion rates provide valuable data for cancer centers and clinicians to intervene early and support patients through their cancer journey. This makes it an important target for quality improvement (QI) with wide-ranging implications for patient outcomes and care quality. Our objectives were to: Investigate underlying reasons for poor ESAS completion rates. Lay the groundwork for strategic interventions aimed at bolstering symptom screening among HNC patients. Methods: This was a QI project targeting the HNC patient population at a regional cancer centre (RCC). Our approach included stakeholder engagement meetings alongside the expertise of QI specialists. Comprehensive data collection was conducted to analyze symptom screening rates and identify potential barriers to ESAS completion. We tracked ESAS completion on weekly HNC clinic days from 2022-2023 via manual and automated chart abstraction. Patient, staff, and volunteer interviews, along with direct clinic observations, provided data for Ishikawa diagrams, facilitating a root cause analysis. Results: We identified a marked decline in ESAS completion rates, steadily decreasing from an average of 29.4% during the 2022-2023 fiscal year to an average of 7.6% between April-November 2023, with rates dropping as low as 3.3% in September 2023. Root cause analysis pinpointed several barriers to ESAS completion, including the transition from paper-based assessments used during the pandemic to electronic formats, low patient awareness of the purpose of symptom screening, perceived lack of value in screening by patients and staff, and the absence of direct guidance from staff in helping patients complete their ESAS. Variable communication practices and inconsistencies in registration staff directing patients to kiosks was also observed. Improving infrastructure and support for volunteer services was identified as a potential solution, which may optimize their effectiveness in symptom screening roles. Conclusions: The project has laid the foundation for targeted QI interventions aimed at improving ESAS screening adherence. By addressing the identified barriers, this work endeavors to re-engage patients in active symptom reporting and to integrate this critical aspect of care into the daily routine of the outpatient hospital setting, thereby enhancing the overall management of HNC symptoms.

SALUS-A Study on Self-Tonometry for Glaucoma Patients: Design and Implementation of the Electronic Case File.

In times of omnipresent digitization and big data, telemedicine and electronic case files (ECFs) are gaining ground for networking between players in the health care sector. In the context of the SALUS study, this approach is applied in practice in the form of electronic platforms to display and process disease-relevant data of glaucoma patients. The SALUS ECF is designed and implemented to support data acquisition and presentation, monitoring, and outcome control for patients suffering from glaucoma in a clinical setting. Its main aim is to provide a means for out- and inpatient exchange of information between various stakeholders with an intuitive user interface in ophthalmologic care. Instrument data, anamnestic data, and diagnostic assessments need to be accessible and historic data stored for patient monitoring. Quality control of the data is ensured by a reading center. Based on an intensive requirement analysis, we implemented the ECF as a web-based application in React with a Datomic back-end exposing REST and GraphQL APIs for data access and import. A flexible role management was developed, which addresses the various tasks of multiple stakeholders in the SALUS study. Data security is ensured by a comprehensive encryption concept. We evaluated the usability and efficiency of the ECF by measuring the durations medical doctors need to enter and work with the data. The evaluation showed that the ECF is time-saving in comparison to paper-based assessments and offers supportive monitoring and outcome control for numerical and imaging-related data. By allowing patients and physicians to access the digital ECF, data connectivity as well as patient autonomy were enhanced. ECFs have a great potential to efficiently support all patients and stakeholders involved in the care of glaucoma patients. They benefit from the efficient management and view of the data tailored to their specific role.

Comparison of animal welfare assessment tools and methodologies: need for an effective approach for captive elephants in Asia.

Welfare is a fundamental aspect of animal management and conservation. In light of growing public awareness and welfare concerns about captive elephants, there is an urgent need for comprehensive, globally coordinated efforts for Asian elephants (Elephas maximus) that participate in religious, logging, or tourist activities in range countries where the majority reside, and where welfare issues have been identified but not addressed. This review provides a comparative analysis of available animal assessment tools. Each offers distinct features for assessment that allow institutions to select criteria for specific needs and available resources. Most are applied to general animal welfare assessments, although some are tailored to particular species, including elephants. The tools span diverse formats, from digital to primarily paper-based assessments. Assessments operate at individual and institutional levels and across multiple welfare domains. Methodologies rely on keeper ratings or expert evaluations, incorporate numerical scoring and Likert scales for welfare grading, and encompass inputs including behaviors, health, and physiological indicators. For tourist camp elephants, one challenge is that the tools were developed in zoos, which may or may not have application to non-zoological settings. Digital tools and assessment methodologies such as keeper ratings face logistical challenges when applied across tourist venues. As with any tool, reliability, validity, and repeatability are essential and must address the unique welfare challenges of diverse captive settings. We propose that a holistic, context-specific, evidence-based, and practical tool be developed to ensure high elephant welfare standards in non-zoological facilities throughout Asia.

Improving Administrative Data at Scale: Experimental Evidence on Digital Testing in Indian Schools

Abstract Large-scale student assessments are a cornerstone of proposed educational reforms to improve student achievement from very low levels in low- and middle-income countries. Yet, this promise relies on their presumed reliability. I use direct audit evidence from a large Indian state (Andhra Pradesh) to show that, as currently administered, official learning assessments substantially overstate proficiency and understate the ‘learning crisis’ of low student achievement. In an experiment covering over 2,400 schools, I evaluate whether digital tablet-based testing could reduce distortion. Although paper-based assessments proctored by teachers severely exaggerate achievement, tablet-based assessments closely match independent test data and are much less likely to be flagged for cheating. Furthermore, I use the direct audit-based retest to directly validate existing (indirect) statistical procedures for detecting cheating at scale and establish that it would be feasible to monitor data integrity cheaply and at scale with such methods. Overall, these results suggest that well-designed technology-aided interventions may improve data integrity at scale, without which these learning assessments are unlikely to serve as a catalyst for policy action.

Computer-based assessments in medical education and strategies to curb malpractice in online assessments: Narrative review

ABSTRACT Assessments are an integral part of the medical curriculum delivery process and give a clear indication of whether students have acquired the set competencies. The purpose of the current review is to explore the scope of computer-based assessments (CBA), identify the ways for malpractice in online assessments, and the strategies to curb the incidence of such unfair practices. An extensive search of all materials related to the topic was carried out on the PubMed search engine and a total of 25 articles were selected based on their suitability with the current review objectives. Keywords used in the search include CBA and online assessments in the title alone only. CBA has found extensive scope in medical education and it is predominantly due to the fact that they enable teachers to increase the periodicity of assessments. CBAs have been linked with potential challenges, such as the mandatory presence of the Internet, technical support, and investment to establish the assessment system, training of the users, and cheating. In conclusion, many medical institutions are gradually shifting from traditional paper-based assessments to CBA. This move is expected to have some concerns, with the most important being the scope for students to adopt any malpractice during online assessments. However, the employment of standard measures and implementation of a strict protocol is bound to curb the incidence of any such malpractices and thus enables the organizers to conduct fair and transparent examinations.

Validation of Dietary Intake Estimated by Web-Based Dietary Assessment Methods and Usability Using Dietary Records or 24-h Dietary Recalls: A Scoping Review.

The goal was to summarize studies comparing the accuracy of web-based dietary assessments with those of conventional face-to-face or paper-based assessments using 24-h dietary recall or dietary record methods in the general population. Using two databases, mean differences and correlation coefficients (CCs) for intakes of energy, macronutrients, sodium, vegetables, and fruits were extracted from each study independently by the authors. We also collected information regarding usability from articles reporting this. From 17 articles included in this review, the mean dietary intake differences in the web-based dietary assessment compared to conventional methods, were -11.5-16.1% for energy, -12.1-14.9% for protein, -16.7-17.6% for fat, -10.8-8.0% for carbohydrates, -11.2-9.6% for sodium, -27.4-3.9% for vegetables, and -5.1-47.6% for fruits. The CC was 0.17-0.88 for energy, protein, fat, carbohydrates, and sodium, and 0.23-0.85 for vegetables and fruits. In three out of four studies reporting usability, more than half of the participants preferred the web-based dietary assessment. In conclusion, % difference and CC of dietary intake were acceptable in both web-based dietary records and 24-h dietary recalls. The findings from this review highlight the possibility of wide-spread application of the web-based dietary assessment in the future.

The creation of the Global Scales for Early Development (GSED) for children aged 0–3 years: combining subject matter expert judgements with big data

IntroductionWith the ratification of the Sustainable Development Goals, there is an increased emphasis on early childhood development (ECD) and well-being. The WHO led Global Scales for Early Development (GSED) project...

Saudi EFL Primary School Teachers’ and Parents’ Perceptions of Online Assessment During COVID-19 Pandemic

Since March 2020, the world has been impacted by the COVID-19 pandemic, which resulted in sudden school closures and a rapid transition from traditional face-to-face education to a new model of online learning and assessment. The present study analyzes two cohorts—primary-school EFL teachers and students’ parents—regarding the perceptions they have had and the challenges they have faced when assessing young EFL learners online during COVID-19. A specific aim of the study is to identify primary-school EFL teachers’ perceptions of the online methods used in assessing young EFL learners in the Riyadh region of Saudi Arabia. The research follows a quantitative method involving a convenience-sampling method for the selection of the study’s participants. A total of 34 primary-school EFL teachers and 20 parents of young learners who are studying online in primary public schools were the main participants of the study. The researcher used a survey-based method involving a five-point Likert scale to collect data from the participants. The surveys were distributed online via the social-media application WhatsApp. The statistically analyzed responses yielded two types of descriptive statistics: frequencies, and percentages. The results show that both the teachers and the parents perceived online assessments as more convenient, fun and interactive than traditional paper-based assessments. Furthermore, both the teachers and the parents associated online assessments with serious challenges, such as cheating and technical problems. And teachers held positive views of various online methods and techniques for the assessment of young EFL learners.
  

Development and acceptance of online assessment in higher education: Recommendations for further research

Online assessment is now used commonly in higher education institutions. While this approach to assessment has several advantages over paper-based assessments, its introduction often precipitates concerns from users, in particular, students and academic staff members. This paper traces the development of online assessment and reviews studies published on student and teacher perceptions of online assessment over a 15-year period. Studies suggest that while students’ perceptions are generally found to be positive, academic staff members’ perceptions have been more mixed. Recommendations for future research into teachers’ responses to online assessment tools are made.

CrossQuestion Game: A Group-Based Assessment for Gamified Flipped Classroom Experience Using the ARCS Model

The importance of formal online education has been felt like never before because of the ongoing nature of the COVID-19 pandemic. Researchers and academics are continuously experimenting with combinations of established pedagogies, tools, and technologies to enhance or sustain the student learning process and motivation. For this study, we have designed and developed a multi-player game called CrossQuestion to explore the combined effect of applying gamification and flipped classroom pedagogies using the ARCS (Attention, Relevance, Confidence, and Satisfaction) model guidelines and its application in the obligatory IT foundation online course during the pandemic. We delivered this course to 79 undergraduate (mostly) non-IT students in Ajman University, UAE. Students were randomly divided into a non-gamified cohort (35 students gone through lecture-based instructions and paper-based assessments–Spring 2019-20–pre-COVID-19 face to face environment) and a gamified flipped classroom cohort (44 students, played the CrossQuestion game–Spring 2020-21–during COVID-19 online environment). We collected the survey data anonymously through our customized Instructional Materials Motivation Survey (IMMS) tool and then performed descriptive analysis, t-test, and regression analysis to address the research hypothesis. We found a significant relationship between learning motivational factors and learning effectiveness. We also found the positive influence of our game on students’ motivation.

Developing a Web-Based App to Assess Mental Health Difficulties in Secondary School Pupils: Qualitative User-Centered Design Study.

BackgroundSecondary schools are an ideal setting to identify young people experiencing mental health difficulties such as anxiety or depression. However, current methods of identification rely on cumbersome paper-based assessments, which are lengthy and time-consuming to complete and resource-intensive for schools to manage. Artemis-A is a prototype web app that uses computerized adaptive testing technology to shorten the length of the assessment and provides schools with a simple and feasible solution for mental health assessment.ObjectiveThe objectives of this study are to coproduce the main components of the Artemis-A app with stakeholders to enhance the user interface, to carry out usability testing and finalize the interface design and functionality, and to explore the acceptability and feasibility of using Artemis-A in schools.MethodsThis study involved 2 iterative design feedback cycles—an initial stakeholder consultation to inform the app design and user testing. Using a user-centered design approach, qualitative data were collected through focus groups and interviews with secondary school pupils, parents, school staff, and mental health professionals (N=48). All transcripts were thematically analyzed.ResultsInitial stakeholder consultations provided feedback on preferences for the user interface design, school administration of the assessment, and outcome reporting. The findings informed the second iteration of the app design and development. The unmoderated usability assessment indicated that young people found the app easy to use and visually appealing. However, school staff suggested that additional features should be added to the school administration panel, which would provide them with more flexibility for data visualization. The analysis identified four themes relating to the implementation of the Artemis-A in schools, including the anticipated benefits and drawbacks of the app. Actionable suggestions for designing mental health assessment apps are also provided.ConclusionsArtemis-A is a potentially useful tool for secondary schools to assess the mental health of their pupils that requires minimal staff input and training. Future research will evaluate the feasibility and effectiveness of Artemis-A in a range of UK secondary schools.

WebPGA: An Educational Technology That Supports Learning by Reviewing Paper-Based Programming Assessments

Providing feedback to students is one of the most effective ways to enhance their learning. With the advancement of technology, many tools have been developed to provide personalized feedback. However, these systems are only beneficial when interactions are done on digital platforms. As paper-based assessment is still a dominantly preferred evaluation method, particularly in large blended-instruction classes, the sole use of electronic educational systems presents a gap between how students learn the subject from the physical and digital world. This has motivated the design and the development of a new educational technology that facilitates the digitization, grading, and distribution of paper-based assessments to support blended-instruction classes. With the aid of this technology, different learning analytics can be readily captured. A retrospective analysis was conducted to understand the students’ behaviors in an Object-Oriented Programming and Data Structures class from a public university. Their behavioral differences and the associated learning impacts were analyzed by leveraging their digital footprints. Results showed that students made significant efforts in reviewing their examinations. Notably, the high-achieving and the improving students spent more time reviewing their mistakes and started doing so as soon as the assessment became available. Finally, when students were guided in the reviewing process, they were able to identify items where they had misconceptions.

Digital Health Training Programs for Medical Students: Scoping Review.

BackgroundMedical schools worldwide are accelerating the introduction of digital health courses into their curricula. The COVID-19 pandemic has contributed to this swift and widespread transition to digital health and education. However, the need for digital health competencies goes beyond the COVID-19 pandemic because they are becoming essential for the delivery of effective, efficient, and safe care.ObjectiveThis review aims to collate and analyze studies evaluating digital health education for medical students to inform the development of future courses and identify areas where curricula may need to be strengthened.MethodsWe carried out a scoping review by following the guidance of the Joanna Briggs Institute, and the results were reported in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. We searched 6 major bibliographic databases and gray literature sources for articles published between January 2000 and November 2019. Two authors independently screened the retrieved citations and extracted the data from the included studies. Discrepancies were resolved by consensus discussions between the authors. The findings were analyzed using thematic analysis and presented narratively.ResultsA total of 34 studies focusing on different digital courses were included in this review. Most of the studies (22/34, 65%) were published between 2010 and 2019 and originated in the United States (20/34, 59%). The reported digital health courses were mostly elective (20/34, 59%), were integrated into the existing curriculum (24/34, 71%), and focused mainly on medical informatics (17/34, 50%). Most of the courses targeted medical students from the first to third year (17/34, 50%), and the duration of the courses ranged from 1 hour to 3 academic years. Most of the studies (22/34, 65%) reported the use of blended education. A few of the studies (6/34, 18%) delivered courses entirely digitally by using online modules, offline learning, massive open online courses, and virtual patient simulations. The reported courses used various assessment approaches such as paper-based assessments, in-person observations, and online assessments. Most of the studies (30/34, 88%) evaluated courses mostly by using an uncontrolled before-and-after design and generally reported improvements in students’ learning outcomes.ConclusionsDigital health courses reported in literature are mostly elective, focus on a single area of digital health, and lack robust evaluation. They have diverse delivery, development, and assessment approaches. There is an urgent need for high-quality studies that evaluate digital health education.

Examining the use of Kahoot to support digital game-based formative assessments in UAE higher education

Attempts to sustain economic growth and diversification within the Middle East have seen governments invested in various educational initiatives. In alignment with this ethos, the United Arab Emirates offers citizens, free education in government-funded higher educational institutes, and bursaries linked to grade point average scores.It is believed that incentivised learning environments provide a powerful motivation to stimulate academic excellence in summative assessments⁠—linked to grade point average scores. An unfortunate outcome of these environments may, however, be that non-grade point reliant tasks are not prioritised by learners. Thus, identifying pedagogical tools which engage student agency towards such tasks, is of obvious interest to educators working within these settings.One such tool which may engage student agency towards formative assessments, is the digital game-based learning platform Kahoot. Kahoot is known to have permeated many educational domains due to its claimed ability to transform classrooms into fun, competitive environments, where students are engaged and motivated to learn. There is, however, a sparsity of literature evidencing the effects that contextual or demographic influences may have upon this and other digital game-based learning tools effectiveness. As such, an explanatory case-based study situated in the United Arab Emirates incentivised learning environment, was undertaken to evaluate female students’ perceptions of Kahoot, as a formative assessment tool. To achieve this Mwanza's eight-step model for translating data into activity theory components was utilised to develop semi-structured interview questions (Mwanza, 2002). These questions permitted analyses of the social, the individual, and socio-economic structures influencing user engagement with this technology.Upon introduction of Kahoot as a formative assessment tool, into the incentivised learning context of the UAE, students described contradictions between classroom and institutional rules, community expectations and the divisions of labour. The transformed classroom environments resultant from resolution of these contradictions produced, greater learner collaboration, superior knowledge retention, reduced test anxiety, and increased levels of student satisfaction.In contrast with other contexts, this study evidenced that digital game-based learning did not predominantly influence student’s motivation to study outside of the classroom for formative assessments. Instead, conventional paper-based assessments provided the greater impetus to engage in out of class learning. This finding may have resulted from the socio-economic and socio-cultural perceptions of the students within this specific learning context.This study, therefore, urges researchers and educators to undertake shifts in their consciousness to acknowledge socio-cultural and intersubjective factors, which may impact upon the effectiveness of digital game-based learning as a formative assessment tool. This is in recognition of this study’s findings that such technologies, may be heavily dependent upon both contextual and demographic influences of those utilising them.Keywords: digital game-based learning; Kahoot; formative assessments; Mwanza’s eight-step model; student motivation; United Arab EmiratesPart of the Special Issue Technology enhanced learning in the MENA region

Virtual reality and mental health in older adults: a systematic review.

Virtual reality (VR) is a promising tool with the potential to enhance care of cognitive and affective disorders in the aging population. VR has been implemented in clinical settings with adolescents and children; however, it has been less studied in the geriatric population. The objective of this study is to determine the existing levels of evidence for VR use in clinical settings and identify areas where more evidence may guide translation of existing VR interventions for older adults. We conducted a systematic review in PubMed and Web of Science in November 2019 for peer-reviewed journal articles on VR technology and its applications in older adults. We reviewed article content and extracted the number of study participants, study population, goal of the investigation, the level of evidence, and categorized articles based on the indication of the VR technology and the study population. The database search yielded 1554 total results, and 55 articles were included in the final synthesis. The most represented study design was cross-sectional, and the most common study population was subjects with cognitive impairment. Articles fell into three categories for VR Indication: Testing, Training, and Screening. There was a wide variety of VR environments used across studies. Existing evidence offers support for VR as a screening and training tool for cognitive impairment in older adults. VR-based tasks demonstrated validity comparable to some paper-based assessments of cognition, though more work is needed to refine diagnostic specificity. The variety of VR environments used shows a need for standardization before comparisons can be made across VR simulations. Future studies should address key issues such as usability, data privacy, and confidentiality. Since most literature was generated from high-income countries (HICs), it remains unclear how this may be translated to other parts of the world.

An Interactive Text Message Survey as a Novel Assessment for Bedtime Routines in Public Health Research: Observational Study.

BackgroundTraditional research approaches, especially questionnaires and paper-based assessments, limit in-depth understanding of the fluid dynamic processes associated with child well-being and development. This includes bedtime routine activities such as toothbrushing and reading a book before bed. The increase in innovative digital technologies alongside greater use and familiarity among the public creates unique opportunities to use these technical developments in research.ObjectiveThis study aimed to (1) examine the best way of assessing bedtime routines in families and develop an automated, interactive, text message survey assessment delivered directly to participants’ mobile phones and (2) test the assessment within a predominately deprived sociodemographic sample to explore retention, uptake, feedback, and effectiveness.MethodsA public and patient involvement project showed clear preference for interactive text surveys regarding bedtime routines. The developed interactive text survey included questions on bedtime routine activities and was delivered for seven consecutive nights to participating parents’ mobile phones. A total of 200 parents participated. Apart from the completion of the text survey, feedback was provided by participants, and data on response, completion, and retention rates were captured.ResultsThere was a high retention rate (185/200, 92.5%), and the response rate was high (160/185, 86.5%). In total, 114 participants provided anonymized feedback. Only a small percentage (5/114, 4.4%) of participants reported problems associated with completing the assessment. The majority (99/114, 86.8%) of participants enjoyed their participation in the study, with an average satisfaction score of 4.6 out of 5.ConclusionsThis study demonstrated the potential of deploying SMS text message–based surveys to capture and quantify real-time information on recurrent dynamic processes in public health research. Changes and adaptations based on recommendations are crucial next steps in further exploring the diagnostic and potential intervention properties of text survey and text messaging approaches.

Student epistemological framing on paper-based assessments

Assessments are usually thought of as ways for instructors to get information from students. In this work, we flip this perspective and explore how assessments communicate information to students. Specifically, we consider how assessments may provide information about what faculty and/or researchers think it means to know and do physics, i.e. their epistemologies. Using data from students completing assessment questions during one-on-one think aloud interviews, we explore how assessment features impact (or did not impact) student engagement with the assessment problems. We analyze video recordings and transcripts to infer the epistemological framings and resources students use while completing introductory-level physics problems. Students' framings tended to be fairly stable, but when shifts did occur - they were triggered by a shift in resource, which can be activated by assessment feature. This work extends existing work on epistemological framing into the realm of assessment and allows us to consider the effects of assessments on our students' understanding of physics teaching and learning.

When didactics meet data science: process data analysis in large-scale mathematics assessment in France

During this digital era, France, like many other countries, is undergoing a transition from paper-based assessments to digital assessments in education. There is a rising interest in technology-enhanced items which offer innovative ways to assess traditional competencies, as well as addressing problem solving skills, specifically in mathematics. The rich log data captured by these items allows insight into how students approach the problem and their process strategies. Educational data mining is an emerging discipline developing methods suited for exploring the unique and increasingly large-scale data that come from such settings. Data-driven methods can be helpful when trying to make sense of process data. However, studies have shown that didactically meaningful findings are most likely generated when data mining techniques are guided by theoretical principles on subjects’ skills. In this study, theoretical didactical grounding has been essential for developing and describing interactive mathematical tasks as well as defining and identifying strategic behaviors from the log data. Interactive instruments from France’s national large-scale assessment in mathematics have been pilot tested in May 2017. Feature engineering and classical machine learning analysis were then applied to the process data of one specific technology-enhanced item. Supervised learning was implemented to determine the model’s predictive power of students’ achievement and estimate the weight of the variables in the prediction. Unsupervised learning aimed at clustering the samples. The obtained clusters are interpreted by the mean values of the important features. Both the analytical model and the clusters enable us to identify among students two conceptual approaches that can be interpreted in theoretically meaningful ways. If there are limitations to relying on log data analysis in order to determine learning profiles, one of them is the fact that this information remains partial when it comes to describing the complete cognitive activity at play, the potential of technology-enriched problem solving situations in large-scale assessments is nevertheless obvious. The type of findings this study produced is actionable from teachers’ perspective in order to address students’ specific needs.