Technology-based Assessment Research Articles

TECHNOLOGY-BASED EVALUATION IN PARKINSONISM

Parkinson's disease is a neurodegenerative disorder characterized by motor and non-motor symptoms that worsen over time. Today, traditional clinical assessment methods are used to monitor disease progression and evaluate treatment responses. However, these methods are subjective and may fail to measure specific conditions. In recent years, thanks to advances in wearable technologies, smart sensors, and data analysis, technology-based approaches to the assessment of patients with Parkinson's disease have gained more attention. With these technologies, objective data can be obtained by monitoring patients' daily activities, motor functions, and symptoms. Motor symptoms such as tremor severity, rigidity, bradykinesia, postural instabilities, freezing phenomenon, and motor parameters of speech impairment can be objectively measured through these technologies. Furthermore, the ability to remotely transmit these data allows patients to be assessed in their own homes and provides continuous feedback to healthcare professionals. This review highlights the importance and potential of technology-based assessment methods in Parkinson's patients and aims to guide future research.

PENGEMBANGAN EVALUASI PEMBELAJARAN MATEMATIKA BERBASIS WORDWALL PADA MATERI BANGUN RUANG SISI DATAR

In order to maximize the learning interest capacity of class VIII MTSS Al-Washliyah Tembung pupils for the 2023–2024 academic year, this research attempts to deepen the mathematics learning assessment technique utilizing Wordwall on flat-sided geometric material. Students might find the evaluation process more engaging and enjoyable when technology-based assessment formats like Wordwall are used. Using the ADDIE model approach (Analysis, Design, Development, Implementation and assessment), a development study was employed as the research technique to create a Wordwall-based mathematics learning assessment. Observation, surveys, interviews, and documentation are some of the techniques used to collect data. Techniques for analyzing data include validity, usefulness, and effectiveness analysis. The project yielded a Wordwall-based evaluation product for mathematics learning on flat-sided geometric material. Four validators gave this product an average score of 93%, which was used to declare it legitimate. This product received an 88% practicality grade after being provided practically by practitioners. The N-gain test scored 0.66 in the medium group based on the outcomes of the Wordwall-based learning assessment test. In summary, the assessment of Wordwall-based mathematics instruction using flat-sided geometric materials for MTSS Al-Washliyah Tembung's class VIII.

Integration of Cognitive Technology in Learning Assessment and Evaluation

The integration of technology in education, especially cognitive technology, is important to increase the efficiency and effectiveness of the learning assessment and evaluation process. Cognitive technology, which combines artificial intelligence and machine learning, offers new possibilities for adapting and personalizing learning experiences. However, its use in learning assessment is still limited and requires further investigation to identify its effectiveness. This research aims to analyze and evaluate the role of cognitive technology in learning assessment and evaluation. The main objective is to determine the extent to which this technology can improve the accuracy and relevance of feedback provided to students and explore the potential for improving learning outcomes through adapting learning content based on assessment data. This research uses quantitative methods with an experimental design. The research sample involved students at a university who used a cognitive technology-based assessment system during one academic semester. The results show that using cognitive technology in assessment can improve accuracy in diagnosing learning weaknesses and provide more personalized and timely feedback. Additionally, students who took assessments with cognitive technology showed greater improvements in academic achievement compared to a control group that did not use similar technology. This research concludes that integrating cognitive technology in learning assessment and evaluation offers great potential to improve the educational process by providing more relevant and personalized feedback. This technology supports teachers in identifying individual learning needs and helps students understand their weaknesses, facilitating more effective and efficient learning. Keywords: The integration of technology in education, especially cognitive technology, is important to increase the efficiency and effectiveness of the learning assessment and evaluation process. Cognitive technology, which combines artificial intelligence and machine learning, offers new possibilities for adapting and personalizing learning experiences. However, its use in learning assessment is still limited and requires further investigation to identify its effectiveness. This research aims to analyze and evaluate the role of cognitive technology in learning assessment and evaluation. The main objective is to determine the extent to which this technology can improve the accuracy and relevance of feedback provided to students and explore the potential for improving learning outcomes through adapting learning content based on assessment data. This research uses quantitative methods with an experimental design. The research sample involved students at a university who used a cognitive technology-based assessment system during one academic semester. The results show that using cognitive technology in assessment can improve accuracy in diagnosing learning weaknesses and provide more personalized and timely feedback. Additionally, students who took assessments with cognitive technology showed greater improvements in academic achievement compared to a control group that did not use similar technology. This research concludes that integrating cognitive technology in learning assessment and evaluation offers great potential to improve the educational process by providing more relevant and personalized feedback. This technology supports teachers in identifying individual learning needs and helps students understand their weaknesses, facilitating more effective and efficient learning.

Demystifying Classroom Assessment: Unleashing the Power of Technology Tools

Background: Technology is a tool for instruction, not a replacement for teachers. Teachers are crucial for our nation's future and are essential implementers and integrators of technology in classrooms. They are responsible for innovative use of technology, as it enhances efficiency and effectiveness in everyday life. ICT comprehensively combines computer and telecommunication technology. Aim: They are also commonly used for the certification of competence, to monitor educational standards, and to provide learners with feedback. The qualitative aspect of assessment and evaluation in the field of education has a cutting-edge relationship with students’ academics. Method: Teachers no doubt are saddled with many responsibilities which range from writing lesson plans and notes to instructional delivery and assessment. Hence there is a need for teachers to span to the innovation technology is offering by embracing technological-based assessment. This paper explored the roles of technology in bridging the gap between the paper-pencil assessment to technology-enhanced assessment and proffered strategies on how these assessment apps can be used in the classroom. The study revealed that there are free assessment applications teachers can use to grade students, thereby reducing their workloads. It was therefore recommended among others that training and seminars should be organised for teachers to enlighten them on technology-based assessment. Result and Discussions: Teachers need to take cognizance of the relevancy of the technological tools. It was further pointed out that while observing the relevance of the technology resources, there should be congruence between the tech tools and instructional curriculum. Other factors to be considered are the; appropriateness of the content and alignment with the learning goals. Teachers need to have clear-cut goals before selecting any technology for assessment. Among the common goals to be considered are; efficiency, cost, end-user proficiency, collaboration, and capacity. Conclusıon: Assessment is vital to the education process. Online assessment can help learners engage more effectively and flexibly in knowledge and can help teachers manage assessment more proficiently and efficiently.

Students’ Perception of Receiving Immediate Assessment Feedback Towards Self-efficacy in Learning: Basis for Implementing Technology-Assisted Assessment Strategy

This study has investigated the students’ perception on receiving immediate feedback on students’ self-efficacy in their learning. Students generally reported high self-efficacy (average score of 3.58), with strengths in areas like adapting learning strategies and overcoming challenges. Nevertheless, there were exceptions to this case. Some students experienced test anxiety and had no certainty that they fully understood the test given which suggests a need for things like relaxation techniques for test anxiety and metacognitive skill development that would enable students to reflect upon their own learning processes. To sum up, attention should be given to every student’s needs in any study of this kind. From all angles of self-efficacy, educators can modify instruction and create a supportive environment to increase student’s self-confidence more than ever before. Also worth noting is that technology-based assessments accompanied by immediate feedback can help improve the understanding of the learning process among students and in general boost their self-efficacy in life.

Revolutionizing Primary Education Assessment: Opportunities and Challenges in Bandar Lampung's Technology-Driven Landscape

This study aims to analyze the opportunities and challenges associated with implementing a technology and information-based educational assessment system in elementary schools in Bandar Lampung. The study recruited ten respondents who were IT staff members, teachers, and principals from upper-grade primary levels (grades IV, V, and VI). It used a qualitative descriptive research design. The research focused on identifying the opportunities and challenges related to IT-based assessment systems. Data collection methods included observations, interviews, and documentation, while data analysis was conducted using the Miles and Huberman model, encompassing data reduction, data display, and data verification. The findings reveal significant opportunities in adopting a technology-based assessment system, such as using Google Forms for flexible assessment scheduling and enhanced data analysis capabilities afforded to teachers through IT tools. However, notable challenges were also identified, including insufficient IT proficiency among teachers and staff, concerns over data privacy and protection, and inadequate internet connectivity within school premises, which hampers the effective use of IT-based applications. The study recommends that schools provide intensive training for teachers and staff to enhance their IT skills. The conclusion is that while technology-based assessment systems offer considerable advantages in facilitating teachers' tasks, the identified challenges must be addressed to optimize the benefits and overcome the obstacles encountered.

Opportunities and Applications of Educational Technologies in Surgical Education and Assessment.

Surgical education is challenged by continuously increasing clinical content, greater subspecialization, and public scrutiny of access to high quality surgical care. Since the last Blue Ribbon Committee on surgical education, novel technologies have been developed including artificial intelligence and telecommunication. The goals of this Blue Ribbon Sub-Committee were to describe the latest technological advances and construct a framework for applying these technologies to improve the effectiveness and efficiency of surgical education and assessment. An additional goal was to identify implementation frameworks and strategies for centers with different resources and access. All sub-committee recommendations were included in a Delphi consensus process with the entire Blue Ribbon Committee (N=67). Our sub-committee found several new technologies and opportunities that are well poised to improve the effectiveness and efficiency of surgical education and assessment (see Tables 1-3). Our top recommendation was that a Multidisciplinary Surgical Educational Council be established to serve as an oversight body to develop consensus, facilitate implementation, and establish best practices for technology implementation and assessment. This recommendation achieved 93% consensus during the first round of the Delphi process. Advances in technology-based assessment, data analytics, and behavioral analysis now allow us to create personalized educational programs based on individual preferences and learning styles. If implemented properly, education technology has the promise of improving the quality and efficiency of surgical education and decreasing the demands on clinical faculty.

Identifying secondary science teachers’ adoption styles of technology-based assessments and examining the patterns of teachers’ beliefs, attitudes, and intention: A latent class analysis approach

Identifying secondary science teachers’ adoption styles of technology-based assessments and examining the patterns of teachers’ beliefs, attitudes, and intention: A latent class analysis approach

Constructing and validating authentic assessments: the case of a new technology-based assessment of economic literacy

BackgroundAuthentic situations are considered a source of learning due to their real world relevance. This can encourage learners to acquire new knowledge. Increasing digitisation and associated resources, such as professional development opportunities for teachers, technology tools, or digital equipment for schools enable the development and implementation of authentic assessments. The basic academic principles for acquiring economic literacy are already provided in lower secondary school. This article examines, using the example of a new authentic technology-based assessment (TBA)—Economic Literacy—Assessing the Status Quo in Grade 8 (ECON 2022) -, the processes involved in constructing a TBA. The purpose is to develop a curricular valid measurement instrument for surveying the current state of economic literacy in the 8th grade of a German federal state. This study explores which economic competencies students—typically between 14 and 15 years of age—possess in Grade 8, and what level of competence can therefore be expected of them at the beginning of a vocational training programme. The assessment is geared toward the curriculum of the subject of economics and is based on a domain model. This article presents the background and construction process for the development of ECON 2022 as a TBA.MethodsTo check the validity of test construction with a focus on the implementation of the authentic assessment and an analysis of difficulty-generating characteristics, the ECON 2022 test items were validated with an expert survey (N = 25). The two-stage data analysis comprised a descriptive quantifying analysis of the rating from the difficulty-generating characteristics specificity, cognitive demand and modelling and the design criterion authenticity. A set of experts rated the criteria. The expert survey was then compared with a previously conducted rating by the research team. The analysis of free-text comments on individual items was carried out discursively and qualitatively by the research team. Both sources of information were used to adapt the test items to measured item difficulties from the field test. For this purpose, items of great difficulty were changed to slightly easier items. In this context, the paper focuses on two central research questions: - How does the authenticity of a test environment relate to difficulty-generating criteria at item level? - Does the authenticity of a test environment have an impact on test results?ResultsResults are as follows. (1) The ECON 2022 assessment offers an example of a test design in which the use of TBAs can support innovative and interactive item development. (2) Using the ECON 2022 assessment enabled the implementation of an assessment of economic literacy using authentic situations and the mapping of different facets of economic competence. (3) The validation study showed that the actual item difficulty did not correlate significantly with the authenticity of the assessment, and authenticity thus did not contribute to item difficulty.ConclusionThe results of the study show that we were successful in developing an authentic TBA in ECON 2022. ECON 2022 allows us to examine economic literacy in schools with a high degree of curricular validity and relevance and to analyse what level of competence and knowledge can be expected of students when they enter a vocational training occupation.

Technology-Based Assessment of Phonological Awareness in Kindergarten

AbstractPrevious research has shown that phonological awareness is one of the most important prerequisites for early reading. Monitoring its development requires reliable, easy-to-use instruments especially in the last years of kindergarten. The present study aims to explore the potential for assessing phonological awareness and some of its subskills through online testing. Participants of the study were 317 kindergarteners (Mage = 6.61, SD = .54 years). The instruments developed for this study within an online assessment platform in two assessment dimensions (syllable and phoneme awareness) contain nine subtests (syllable synthesis, segmentation, deletion; phoneme identification in different sound environments, identification of phoneme position, identification of initial phonemes, phoneme synthesis and segmentation). The results of the study show that: (1) the test is a reliable assessment tool for kindergarteners’ phonological awareness skills; (2) according to the underlying measurement model of phonological awareness, the tasks are separated based on particular operational components independently of the size of the language element involved; (3) segmentation tasks proved to be the most difficult parts of the test; and (4) the media effect is insignificant. The online test aims to emphasize the importance of online testing and the inseparable relationship between measuring and developing phonological awareness, prompting teachers to rethink their teaching methods. It also introduces a new tool for educators to use, tailored to children's needs but potentially challenging for teachers with lower ICT literacy, requiring methodological support, ultimately providing a new opportunity for kindergartens.

Information and Communication Technology-based Assessment for Children with Developmental Needs: Kids Brain Balancer

Information and Communication Technology-based Assessment for Children with Developmental Needs: Kids Brain Balancer

Editorial for "Information and Communication Technology-based Assessment for Children with Developmental Needs: Kids Brain Balancer"

Editorial for "Information and Communication Technology-based Assessment for Children with Developmental Needs: Kids Brain Balancer"

Commonly Used Techniques in Testing Foreign Language Skills and Communicative Language Teaching

Teaching and testing foreign language skills are long and challenging processes that cannot be separated from each other. Many traditional test items and alternative methods are utilized for various testing purposes. However, for a successful evaluation, alternative method and traditional assessment types should be handled in communicative language teaching for a successful evaluation. The basics of Communicative Language Teaching are important to increase foreign language learners’ communicative competence. In this sense, this review illustrates traditional testing item types after defining the difference between test and assessment. Secondly, testing language skills and components are put under a critical perspective. Then, the basics of Communicative Language Teaching are explained. Eventually, suggestions about adopting all testing types and methods for an effective assessment are discussed. Also, technology-based assessment tools are recommended for an authentic assessment. Correspondingly, the review mainly tries to shed light on testing language skills and language components. Moreover, it aims to help foreign language teachers prepare exams that follow the basic principles of Communicative Language Teaching.

Upper limb sensorimotor recovery in Asian stroke survivors: a study protocol for the development and implementation of a Technology-Assisted dIgitaL biOmaRker (TAILOR) platform.

Stroke is a leading cause of lifelong disability worldwide, partially driven by a reduced ability to use the upper limb in daily life causing increased dependence on caregivers. However, post-stroke functional impairments have only been investigated using limited clinical scores, during short-term longitudinal studies in relatively small patient cohorts. With the addition of technology-based assessments, we propose to complement clinical assessments with more sensitive and objective measures that could more holistically inform on upper limb impairment recovery after stroke, its impact on upper limb use in daily life, and on overall quality of life. This paper describes a pragmatic, longitudinal, observational study protocol aiming to gather a uniquely rich multimodal database to comprehensively describe the time course of upper limb recovery in a representative cohort of 400 Asian adults after stroke. Particularly, we will characterize the longitudinal relationship between upper limb recovery, common post-stroke impairments, functional independence and quality of life. Participants with stroke will be tested at up to eight time points, from within a month to 3 years post-stroke, to capture the influence of transitioning from hospital to community settings. We will perform a battery of established clinical assessments to describe the factors most likely to influence upper limb recovery. Further, we will gather digital health biomarkers from robotic or wearable sensing technology-assisted assessments to sensitively characterize motor and somatosensory impairments and upper limb use in daily life. We will also use both quantitative and qualitative measures to understand health-related quality of life. Lastly, we will describe neurophysiological motor status using transcranial magnetic stimulation. Descriptive analyses will be first performed to understand post-stroke upper limb impairments and recovery at various time points. The relationships between digital biomarkers and various domains will be explored to inform key aspects of upper limb recovery and its dynamics using correlation matrices. Multiple statistical models will be constructed to characterize the time course of upper limb recovery post-stroke. Subgroups of stroke survivors exhibiting distinct recovery profiles will be identified. This is the first study complementing clinical assessments with technology-assisted digital biomarkers to investigate upper limb sensorimotor recovery in Asian stroke survivors. Overall, this study will yield a multimodal data set that longitudinally characterizes post-stroke upper limb recovery in functional impairments, daily-life upper limb use, and health-related quality of life in a large cohort of Asian stroke survivors. This data set generates valuable information on post-stroke upper limb recovery and potentially allows researchers to identify different recovery profiles of subgroups of Asian stroke survivors. This enables the comparisons between the characteristics and recovery profiles of stroke survivors in different regions. Thus, this study lays out the basis to identify early predictors for upper limb recovery, inform clinical decision-making in Asian stroke survivors and establish tailored therapy programs. ClinicalTrials.gov, identifier: NCT05322837.

(Mis)Perceiving Apnea and Insomnia in Germany: A Tale of Two Disorders

ABSTRACT In Germany, both apnea and insomnia are highly prevalent sleep disorders. But while there is an extensive and growing infrastructure to deal with apnea, there is very little support for insomnia patients. I argue that this is due to various interrelated factors: the role of evidence and experience in diagnosis, the availability of treatment, and—importantly—how evidence, experience, and treatment can (or cannot) be materialized in the medical economy. Drawing on phenomenology and affordance theory, and based on fieldwork among German sleep doctors and their patients, I analyze how different sleep disorders are perceived, evaluated, and acted upon. I use different examples to reflect on the possibilities of “objectively” knowing and “subjectively” experiencing (disordered) sleep, and on how different perspectives (patient versus doctor, first-person versus third-person) and modes of perception (direct or indirect, narrative-based anamnesis or technology-based assessment) matter (or not) for the diagnosis and treatment of sleep disorders.

Use of technology-based assessments: a systematic review covering over 30 countries

ABSTRACT The widespread adoption of technology-based assessments during the COVID-19 pandemic has exacerbated concerns about the digital divide, given global disparities in digital access, connectivity, and coping strategies. This systematic review was intended to assess how the use of technology-based assessments has affected the education system’s functioning in pre-college education, compared to traditional assessments. It covered 34 countries from 34 full-text English sources in 2018–2022. A total of 12 assumptions were tested, corresponding to six hypotheses about the learning, educating, and management facets associated with the use of technology-based assessments. Our findings revealed mixed evidence about technology-based assessments in reducing cheating, enhancing learning, supporting monitoring, improving instruction, and reducing non-teaching workload. However, strong evidence supported the assumptions that technology-based assessments improve measurement precision, interpretability, engagement, interaction, and teacher-parent communication. Limited but positive evidence supported the assumptions that technology-based assessments may reduce the cost and time of test administration.

Technology-based balance performance assessment can eliminate floor and ceiling effects

Many clinical measurement tools for balance have ceiling effects. Technology-based assessments using virtual reality systems such as the Computer-Assisted Rehabilitation Environment (CAREN) may provide a way to develop objective, quantitative measures that scale from low to high levels of difficulty. Our objective was to: (1) develop a performance assessment tool (PAT) for the CAREN; (2) quantify the reliability of the tool; (3) validate the scores against clinical balance measures; and (4) compare the scores from a population with balance impairments to those from able-bodied individuals in a cross-sectional validation study. Three games were developed on the CAREN and tested on 49 participants (36 able-bodied and 13 with impaired mobility). For each module, the corresponding measures were transformed into scores using a series of functions such that ceiling and flooring effects would be minimized. The results showed an association between scores and age, an overlap in scores from impaired high-performance individuals and able-bodied low performance individuals, and a correlation of PAT scores with other clinical tests. Several of the limitations of current clinical tools, including floor and ceiling effects, were overcome by the PAT, suggesting that the PAT can be used to monitor the effect of rehabilitation and training.

Why multimedia might matter: The impact of animations and images on item performance and test‐taker behaviour

AbstractThe use of animations and images in technology‐based assessments (TBAs) represents a significant change in assessment design. To ensure that appropriate inferences can be drawn from assessments that use multimedia stimuli, their impact on test‐taker performance and behaviour must be investigated. To achieve this, an experiment was conducted with 251 Irish post‐primary students using an animated and text‐image version of the same TBA of scientific literacy. Eye movement (n = 33) and interview data (n = 12) were also collected as a measure of test‐taker attentional behaviour. Overall, there was no significant difference in test‐taker performance when identical items used animated or text‐image stimuli. However, items with dynamic stimuli often had higher discrimination indices indicating that these items were better at distinguishing between those with differing levels of knowledge. Eye movement data also revealed that dynamic item stimuli encouraged longer average fixation durations on the response area of an item. These findings indicate that multimedia stimuli may potentially affect how test‐takers interact with online assessments. This has implications for what claims can be made about a learner's performance on an assessment. Recommendations for policy, practice and future research are considered. Practitioner notesWhat is already known about this topic: The use of multimedia stimuli in the form of diagrams, high‐resolutions images, animations, and simulations are becoming more commonplace in technology‐based assessments (TBAs) for post‐primary aged learners. It is unclear what impact the use of multimedia stimuli can have on an individual's performance and behaviour in assessment/testing contexts. Eye movement data can be used to support our understanding of test‐takers' interactions with TBAs. What this paper adds: By comparing the use of different types of multimedia stimuli (animations vs images), this study has responded to calls for a more in‐depth examination of test items involving multimedia stimuli for TBAs. While there was no difference in test‐taker performance between those who saw test items with different forms of multimedia stimuli, key differences in attentional behaviour were noted. Test‐takers interacted with the assessment differently pending the multimedia stimuli used. The current study showed that dynamic stimuli may be a way to improve item discrimination, which is generally desirable in assessments. Implications for practice and/or policy: Certain design features appeared to add additional ‘assessment load’. This information could be leveraged to improve test item design and test specifications. It may also encourage test‐developers to reconsider what claims they make about individuals who complete assessments with these features. Eye‐tracking technology has huge potential to support research in online assessment environments.

A Viscoelastic Model to Evidence Reduced Upper-Limb-Swing Capabilities during Gait for Parkinson’s Disease-Affected Subjects

Parkinson’s disease (PD) is a chronic neurodegenerative disorder with high worldwide prevalence that manifests with muscle rigidity, tremor, postural instability, and slowness of movement. These motor symptoms are mainly evaluated by clinicians via direct observations of patients and, as such, can potentially be influenced by personal biases and inter- and intra-rater differences. In order to provide more objective assessments, researchers have been developing technology-based systems aimed at objective measurements of motor symptoms, among which are the reduced and/or trembling swings of the lower limbs during gait tests, resulting in data that are potentially prone to more objective evaluations. Within this frame, although the swings of the upper limbs during walking are likewise important, no efforts have been made to reveal their support significance. To fill this lack, this work concerns a technology-based assessment of the forearm-swing capabilities of PD patients with respect to their healthy counterparts. This was obtained by adopting a viscoelastic model validated via measurements during gait tests tackled as an inverse dynamic problem aimed at determining the torque forces acting on the forearms. The obtained results evidence differences in the forearm movements during gait tests of healthy subjects and PD patients with different pathology levels, and, in particular, we evidenced how the worsening of the disease can cause the worsening of the mechanical support offered by the forearm’s swing to the walking process.

Effectiveness of Collaboration in VET: Measuring Skills for Solving Complex Vocational Problems With a Multidimensional Authentic Technology-Based Assessment

Context:Collaboration and digitization in work and production process have been of scientific interest in the research field of vocational education and training for decades. So far, there is a lack of empirical evidence on the extent to which the performance of collaborative problem-solving can outperform individual problem-solving. An appropriate instrument for measuring collaborative performance appropriately has been lacking to this point: It is challenging to (1) collect data on interaction in collaboration in a standardized way and (2) to assign the individual performances to the group performance. In a web-based office simulation, this study examined the two-dimensional construct of collaborative problem-solving ability, which consists of cognitive and social components, and explored the effectiveness of collaboration.
 Methods:The test settings are systematically varied between individual and collaborative problem-solving in a domain-typical problem scenario in the economic domain. A theory-driven two-dimensional authentic instrument with a complex commercial problem in a web-based office simulation was applied to measure the performance in these two test settings. Multi-Group Confirmatory Factor Analysis was conducted to test for measurement invariance across test settings, the two-dimensional RASCH-Model incorporating between-item multidimensionality with correlated dimensions was used for ability estimation. ANOVA test was conducted to determine if there is a statistically significant difference with respect to the ability between the individual and collaborative test settings.
 Findings:The two-dimensional construct of collaborative problem-solving ability with correlated dimensions showed better fit than the unidimensional construct. The scalar measurement invariance was established after excluding three items. ANOVA tests confirmed that the collaborative setting enhances the problem-solving ability of learners significantly in both the cognitive component and social component, and the ability in cognitive component was more fostered.
 Conclusion:The findings suggested the validation of this multidimensional authentic instrument to measure the performance in a collaborative work environment in economic domains. The results showed the effectiveness of collaboration with a computer-based agent. In the practice of vocational training and education, teachers can use this instrument for the evaluation of the effectiveness of collaboration.