Program Assessment Systems Research Articles

ProgEdu4Web: An automated assessment tool for motivating the learning of web programming course

AbstractThe expanding software industry has increased the demand for proficient programming professionals. However, more than technical expertise is needed for successful software development; code quality assurance and effective teamwork are also indispensable. Unfortunately, recent graduates often exhibit deficiencies in programming, code quality assurance, and teamwork skills, highlighting the need for improvements in these areas. In response to these challenges, this study presents ProgEdu4Web, an innovative Automated Programming Assessment System designed to evaluate code quality and teamwork proficiency in web programming. The system integrates automatic source code evaluation tools for static code analysis, ensuring timely feedback on coding quality issues. To assess the effectiveness of ProgEdu4Web, a questionnaire and hypothesis model was employed, surveying a cohort of 111 students enrolled in web programming classes. The study's findings demonstrate that the system significantly alleviated instructor workload while motivating students to enhance their web programming with good code quality and teamwork abilities.

ANALYZING GIT LOG IN AN CODE-QUALITY AWARE AUTOMATED PROGRAMMING ASSESSMENT SYSTEM: A CASE STUDY

Automated programming assessment systems have transformed the evaluation of programming assignments, providing detailed feedback and reducing instructors' workload. This paper explores the benefits of Git log analysis in ProgEdu, a code-quality aware automated programming assessment system. ProgEdu was utilized for assessing Java homework assignments and web programming projects over two semesters. The integration of Git log analysis in ProgEdu highlights its potential in tracking student progress, predicting performance, determining student groups based on submission behaviors, identifying inequality in group projects, and facilitating instructors' intervention. The study emphasizes the importance of enhancing software industrial practices in programming courses, including code version control, static code quality checking, unit testing, and automation tools. By incorporating these practices, students benefit from hands-on learning and situated learning experiences. Embracing these practices enhances the learning experience, improves student performance, and fosters a collaborative programming environment. It highlights the benefits for students and instructors, urging institutions to invest in software industrial practices and demonstrating the potential impact on programming education.

TEACHING DEVOPS AND SOFTWARE ENGINEERING PRACTICES USING AN AUTOMATED PROGRAMMING ASSESSMENT SYSTEM

The software industry has seen a shift in recent years, with the adoption of agile development practices, the deployment of cloud-based, and service-oriented applications using DevOps. This has led to increased pressure on educators in higher education institutions to provide students with the best possible preparation for their careers, based on industry best practices. The research proposed a new approach to bridge the gap between in-class exercises and industrial practices by introducing an automated programming assessment system, ProgEdu. ProgEdu simulates the continuous integration pipeline of DevOps, incorporating unit testing and code quality assessment. It is deployed as a service-oriented application on Docker, allowing students to gain practical experience in a real-world setting. Our approach goes beyond simply assessing code, as we have also incorporated the service-oriented architecture, continuous integration pipeline, and Docker of the system to illustrate various DevOps practices in our software engineering classes. By providing hands-on exercises that utilize industry-standard tools and techniques, we can bridge the gap between theoretical knowledge and practical application, helping students to extensively understand fundamental concepts and prepare for the demands of the modern workforce.

An automated system with a versatile test oracle for assessing student programs

AbstractAutomated program assessment systems have been widely adopted in many universities. Many of these systems judge the correctness of student programs by comparing their actual outputs with predefined expected outputs for selected test inputs. A common weakness of such systems is that student programs would be marked as incorrect as long as their outputs deviate from the predefined ones, even if the deviations are only minor, insignificant, and considered acceptable by a human assessor that the programs have satisfied the specifications. This critical weakness caused undue frustration to students and undesirable pedagogical consequences that undermine these systems’ benefits. To address this issue, we developed an improved mechanism for program output comparison to serve as a versatile test oracle that brings the results of automated assessment much closer to those of human assessors. We evaluated the new mechanism in real programming classes using an existing automated program assessment system. We found that the new mechanism achieved zero false‐positive error (did not wrongly accept any incorrect output) and very low (0%–0.02%) false‐negative error (that wrongly rejected correct outputs), with very high accuracy (99.8%–100%) in correctly recognizing outputs deemed acceptable by instructors. This represents a major improvement over an existing assessment mechanism, which had 56.4%–64.1% false‐negative error with an accuracy of 25.4%–40.9%. Moreover, about 67%–96% of students achieved their best results in their first attempt, which could be encouraging to them and reduce their frustration. Furthermore, students generally welcomed the new assessment mechanism and agreed it was beneficial to their learning.

Code-quality evaluation scheme for assessment of student contributions to programming projects

Project-based learning is the most common approach to software engineering education, due to its emphasis on the teamwork skills essential to real-world collaborations. This study developed an automated programming assessment system (APAS) featuring a code-quality evaluation scheme to overcome difficulties in assessing the contribution of individual team members. Team participation is visualized on a weekly basis to provide insights pertaining to team dynamics, and metrics based on code quality allow the segmentation of students by level of contribution. Insights provided by the proposed system were also shown to facilitate interventions aimed at improving code quality. Empirical analysis based on submission data from 146 students (41 teams) demonstrated the feasibility of the proposed system in monitoring group-based learning projects at the university level, particularly in detecting free-riders.

Identifying nonconformities in contributions to programming projects: from an engagement perspective in improving code quality

ABSTRACT Project-based learning is among the most common learning approaches aimed at conveying professional standards and best practices to students in programming courses. However, team projects commonly impose problems related to responsibility sharing, such as low effort or inequality in contributions. This paper presents a collaborative programming assessment system featuring a code quality assessment function with specific metrics to measure individual contributions. Student engagement data is used to detect nonconformities in collaboration using a learning analytical approach. Latent profile analysis was used to detect four theoretical team profiles differentiated by team effort (2 levels) and within-team collaboration (2 levels). We demonstrated the efficacy of assessing code to evaluate team dynamics and student behaviour, wherein efforts to resolve coding style failures can be used as a proxy by which to estimate the taskwork awareness of team members. Submission data from 146 students in 41 web-programming projects revealed four behavioural patterns that could potentially hinder the effective functioning of programming teams: free-riding, social loafing, the bystander effect, and lone wolves. We also demonstrated the applicability of automated programming assessment systems to the monitoring of learning progress, thereby facilitating timely interventions to correct difficulties at the team level.

Effects of an automated programming assessment system on the learning performances of experienced and novice learners

ABSTRACT Programming ability is the core ability of this era and can be obtained and improved through practice. In this paper, an Automated Programming Assessment system based on Mastery learning and Peer competition (APAMP) was proposed and developed. APAMP allows students to practice repeatedly by providing immediate feedback after their programs are submitted. It also presents an analytical dashboard as a competition mechanism for students to visualize their learning performance and compare their performance with peers. By incorporating APAMP into programming courses, students can master programming skills through repeated practice, and their enthusiasm for learning programming can also be encouraged by peer competition. To evaluate the effects of APAMP on the learning performance of students with programming language learning and novice students, a quasi-experiment was conducted in a high school. The experimental results showed that the learning achievement of the two groups of students improved significantly. Moreover, the learning attitude of students in the experienced group improved significantly, with the experienced students benefiting more from the system than the novice students. In fact, the novice students showed a significant decline in learning attitude and learning motivation, which was contrary to our intuition.

Establishing Rigor and Quality in Doctoral Programs Through Program Assessment

This essay describes the development and implementation of a CPED-grounded program assessment system and the ways in which it contributes to quality assurance in Ed.D. programs broadly. We begin by articulating program quality and describing the contextual factors that guide our approach to program assessment. Next, we overview major components and processes of our program assessment system. Specific emphasis is placed on describing the development and evaluation of program effectiveness based on CPED-influenced student learning outcomes. We then briefly describe how we leverage an existing learning management system to implement program assessment efficiently, and outline continuous monitoring and improvement efforts that are based on our program assessment work. Finally, we describe our experiences with academic program review and discuss lessons learned and suggestions to promote program rigor and success.

Navigating Academic Law in Competency Decisions.

Navigating Academic Law in Competency Decisions.

Методология оценки компетентностно-ориентированных образовательных программ

A methodology for evaluating educational programs has been developed. The role of the educational program assessment system in the ESM quality strategy and the implementation of the assessment process of competence-based educational programs in this system are described. Key user roles and functions are indicated. The main evaluation criteria are presented.

Analysis of Learning Behavior in an Automated Programming Assessment Environment: A Code Quality Perspective

Automated programming assessment systems are useful tools to track the learning progress of students automatically and thereby reduce the workload of educators. They can also be used to gain insights into how students learn, making it easier to formulate strategies aimed at enhancing learning performance. Rather than functional code which is always inspected, code quality remains an essential aspect to which not many educators consider when designing an automated programming assessment system. In this study, we applied data mining techniques to analyze the results of an automated assessment system to reveal unexpressed patterns in code quality improvement that are predictive of final achievements in the course. Cluster analysis is first utilized to categorize students according to their learning behavior and outcomes. Cluster profile analysis is then leveraged to highlight actionable factors that could affect their final grades. Finally, the same factors are employed to construct a classification model by which to make early predictions of the students' final results. Our empirical results demonstrate the efficacy of the proposed scheme in providing valuable insights into the learning behaviors of students in novice programming courses, especially in code quality assurance, which could be used to enhance programming performance at the university level.

Building a Comprehensive Automated Programming Assessment System

Automated Programming Assessment Systems (APAS) are used for overcoming problems associated with manually managed programming assignments, such as objective and efficient assessing in large classes and providing timely and helpful feedback. In this paper we survey the literature and software in this field and identify the set of necessary features that make APAS comprehensive - such that it can support all key stages in the assessment process. Put differently, comprehensive APAS is generic enough to meet the demands of “any” computer science course. Despite the vast number of publications, the choice of software turns out to be very limited. We contribute by developing Edgar, a comprehensive open-source APAS which, to the best of our knowledge, exceeds any other similar free and/or open-source tool. Edgar is the result of three years of development and usage in, for the time being, eight courses dealing with various programming languages and paradigms (C, Java, SQL, etc.). Edgar supports various text-based programming languages, multi-correct multiple-choice questions, provides rich exam logging and monitoring infrastructure to prevent potential fraudulent behaviour, and subsequent data analysis and visualization of students' scores, exams, question quality, etc. It can be deployed on all major operating systems and is written in a modular fashion so that it can be adjusted and scaled to a custom fit. We comment on the architecture and present data from real-world use-cases to support these claims. Edgar is in active use today (1000+ students per semester) and it is being constantly developed with new features.

Analysis of Inland Immersed Tunnel Health Monitoring System

Based on Nanchang Hong-gu tunnel, which was the largest inland immersed tube tunnel besides it’s tremendous water level difference, large scale, complex geological conditions, different longitudinal gradients, and other characteristics, accompanied with structure analysis by finite element model, monitoring scheme of the real-time health monitoring program and tunnel health assessment system was analyzed, and the monitoring system was established, then emphasized on the monitoring methods of tube joints open and dislocation, tube uneven settlement, shear key pressure, health assessment indexes and method of index weight determining in health assessment system. This system can make tunnel displacements, internal forces and other health indicators be measured and transmitted automatic in real-time. Research result was applied to Nanchang Honggu tunnel health monitoring system and would be guidance for other tunnel projects as well.

Using Continuous Quality-Improvement Techniques to Evaluate and Enhance an Internal Medicine Residency Program's Assessment System

Using Continuous Quality-Improvement Techniques to Evaluate and Enhance an Internal Medicine Residency Program's Assessment System

A Study on Assessment System for Nursing Bachelor Degree Program Outcomes: Focused on Communication Ability Improvement

Purpose: The core of accreditation for nursing education is to improve the ability and quality of graduates through a CQI system based on outcomes. For program outcomes of nursing education accreditation criterion, it is necessary to understand the relationship among education assessment, objective and curriculum. Methods: The focus of this study was on program outcomes and the assessment. First measures for improving the general problems of program outcomes associated with the assessment system were stated. Detailed measure for practicing program outcomes, the assessment system and the assessment tool using Rubrics were discussed. Results: Detailed measure were established for practicing the program outcomes, assessment system and assessment tool, which makes use of rubrics, program outcome No 3 of the Korean Accreditation Board of Nursing Education. Conclusion: Therefore it is expected that the assessment system established in this study will allow universities to conduct program outcomes and assessment.

An Exercise Management System for Teaching Programming

An effective learning activity in a computer programming course is to study and practice computer programs. In order to help students to submit exercises and to assist instructors to mark programming exercises, a number of program submissions and assessment systems have been developed. However, these systems do not provide sufficient support for instructors to design exercises that can help students to study and practice computer programs in an incremental manner. With the primary aim to improve the teaching and learning of computer programming, we have developed a programming exercise management system, namely Programming Learning Web (PLWeb), to assist instructors to design computer programming exercises and to help students to study and practice programming exercises. PLWeb provides an integrated development environment (IDE) which is used not only as an authoring tool for instructors to compose exercises but also as a novice-friendly editor for students to study programs and to submit solutions. In addition, PLWeb allows instructors to use visualized learning status to assist students with difficulties. A plagiarism detection tool is also provided to deter students from plagiarism.

Assessment with an “End in View”

This article uses Dewey's (1938) concept of “end in view” to frame one California State University's purposeful action in implementing the Performance Assessment for California Teachers (PACT). The authors provide a chronology of events that reveal the ways in which teacher education faculty were engaged in examining PACT outcomes and candidate performance through informal, “mock scoring” events, and the design of formative, embedded signature assessments, and the resulting changes in the program assessment system, curriculum, and instructional practices. The article closes with concerns and suggestions for sustaining the “end in view” during high-stakes and low-budget circumstances.

An Examination of Institutional Reports: Use of Data for Program Improvement

Owing to national emphasis on accountability, teacher education programs are under close scrutiny by federal and state-level entities to justify their purpose and usefulness in preparing future teachers. Accrediting agencies have set high standards, which ask for systematic assessment of teacher education programs and their candidates. The purpose of this study was to examine teacher preparation program assessment systems and unit evaluations—specifically, the use of data for program improvement. Using the interpretive/constructivist paradigm, we used a qualitative approach to compare and contrast 80 institutional reports from teacher education programs based on National Council for Accreditation of Teacher Education's Standard 2, Element 3 (“Assessment System and Unit Evaluation: Use of Data for Program Improvement”). The findings revealed that the most common areas of program improvement were (1) changes in curriculum, (2) assessment systems, (3) unit and program structures, and (4) admissions and advising policies and procedures. In addition, programs throughout the country generally identified similar problems and proposed common solutions, no matter the institution size, type, or geographic location.

An innovative outcomes-based medical education program built on adult learning principles

An innovative medical curriculum at the University of New South Wales (UNSW) has been developed through a highly collaborative process aimed at building faculty ownership and ongoing sustainability. The result is a novel capability-based program that features early clinical experience and small-group teaching, which offers students considerable flexibility and achieves a high degree of alignment between graduate outcomes, learning activities and assessments. Graduate capabilities that focus student learning on generic outcomes are described (critical evaluation, reflection, communication and teamwork) along with traditional outcomes in biomedical science, social aspects, clinical performance and ethics. Each two-year phase promotes a distinctive learning process to support and develop autonomous learning across six years. The approaches emphasize important adult education themes: student autonomy; learning from experience; collaborative learning; and adult teacher–learner relationships. Teaching in each phase draws on stages of the human life cycle to provide an explicit organization for the vertical integration of knowledge and skills. A learning environment that values the social nature of learning is fostered through the program's design and assessment system, which supports interdisciplinary integration and rewards students who exhibit self-direction. Assessment incorporates criterion referencing, interdisciplinary examinations, a balance between continuous and barrier assessments, peer feedback and performance assessments of clinical competence. A portfolio examination in each phase, in which students submit evidence of reflection and achievement for each capability, ensures overall alignment.

Ludwig

An online programming tutoring and assessment system is described, and the results of a preliminary study are presented. Ten students in an introductory C++ programming course used the system; both qualitative and quantitative data were collected and suggest that a future large-scale implementation will yield beneficial results