This paper presents the development and deployment of a web-based platform designed to improve the process of managing and evaluating programming assignments in academic setting. The system integrates three components: assignment creation, automated grading, and code similarity detection — all within a unified, user-friendly interface. Its primary objective is to reduce the time and effort required by teachers to assess large volumes of student submissions while maintaining fairness, consistency, and transparency in grading. The platform allows teachers to define assignments and corresponding test cases, which are then used for automatic evaluation of submitted code. The grading engine executes student programs in a secure environment and compares the outputs against expected results, providing detailed feedback for each test case. In addition, the system features a built-in similarity detection module that highlights pairs of potentially plagiarized solutions. This mechanism supports teachers in identifying dishonest practices while preserving students’ privacy and autonomy. The system was successfully implemented and tested in real university courses, where it demonstrated its practical value in improving both the efficiency of course management and the quality of student feedback. Teachers benefited from significantly reduced manual workload, and students gained faster access to consistent evaluations. While the platform currently supports a limited set of programming languages and uses a relatively simple similarity detection approach, it lays a solid foundation for further improvement. Overall, the developed solution contributes to modernizing programming education and addressing challenges in large-scale course delivery.

MENTOR: Fixing introductory programming assignments with formula-based fault localization and LLM-driven program repair

This study presents a new structured approach to AI-assisted learning designed to support students in actively engaging with programming tasks, progressively developing independent problem-solving skills, and effectively utilizing Generative Artificial Intelligence (GenAI) in programming education. The framework, based on constructivist and experiential learning theories, aims to guide students in interacting with GenAI tools to enhance algorithmic, critical, and analytical reasoning rather than replace cognitive effort. The research employs a mixed-methods design, incorporating quantitative data from laboratory performance assignments and paper-based assessments of programming skills, as well as qualitative data from semi-structured expert interviews. Three experts from well-known Omani universities verified and confirmed the model's consistency with established learning theories and instructional design principles. Thirty-two undergraduate students at Sultan Qaboos University were divided into experimental and control groups. Quantitative analysis indicated that the experimental group, which adopted the new approach, significantly outperformed the control group, which used GenAI without restrictions, on assignments assessing code analysis, debugging, optimization, and problem-solving skills (p = 0.009). These findings suggest that the proposed model effectively balances creativity and academic integrity.

ABSTRACT Background and context Some skills taught in introductory programming courses are categorized into 1) explaining code, 2) arranging lines of code in correct sequence, 3) tracing through the execution of a program and 4) writing code from scratch. Objective Knowing if a programming skill is a prerequisite to another would benefit teachers in properly planning the course and structuring the order in which they present activities relating to new content. Prior attempts to establish a skill hierarchy have suffered from methodological issues. Method In this study, we used the conviction measure from association rule mining to perform pair-wise comparisons of five skills: Write, Trace, Reverse trace, Sequence and Explain code. We used the data from four exams with more than 600 participants where students solved programming assignments of different skills for several programming topics. Findings Our findings matched the previous finding that tracing is a prerequisite for students to learn to write code. Contradicting the previous claims, our analysis showed that using the mean threshold writing code is a prerequisite to explaining code. However, there is no clear relationship when we change the threshold to the median. Unlike prior work, we did not find a clear prerequisite relationship between sequencing code and writing or explaining code. Implications Our research can help instructors by systematically arranging the skills students exercise when encountering a new topic. The goal is to help instructors properly teach and assess programming in a fashion most effective for learning by leveraging the relationship between skills.

ABSTRACT Computational thinking (CT) and programming are essential competencies in the Fourth Industrial Revolution, spurring interest in learner‐centered and artificial intelligence (AI)‐based education. Combining the active environment of flipped learning with the personalized support of generative AI is a promising new strategy for programming education; however, research integrating these elements remains limited. This study investigated the effects of programming education that incorporates both flipped learning and generative AI on students' CT and problem‐solving skills, with particular attention to variations by university size and educational setting. Participants were students enrolled in introductory software courses at two universities (K and J). Programming lessons integrating generative AI and flipped learning were developed, and changes in students' CT and problem‐solving skills were evaluated using pre‐ and post‐surveys and analyses of programming assignments. The study also examined the relationship between programming‐related CT components (problem decomposition, abstraction, algorithm design, automation, and simulation) and students' programming difficulties, as well as key factors contributing to differences in learning outcomes between institutes. The results revealed significant improvements in the CT and problem‐solving skills of students. Based on these findings, pedagogical strategies were proposed to optimize the use of generative AI and flipped learning in diverse educational contexts. This study provides valuable insights into the application of active, personalized learning in various settings, including introductory software education and online learning.

The integration of large language models (LLMs) and fuzzy metrics offers new possibilities for improving automated grading in programming education. While LLMs enable efficient generation and semantic evaluation of programming assignments, traditional crisp grading schemes fail to adequately capture partial correctness and uncertainty. This paper proposes a grading framework in which LLMs assess student solutions according to predefined criteria and output fuzzy grades represented by trapezoidal membership functions. Defuzzification is performed using the centroid method, after which fuzzy distance measures and fuzzy C-means clustering are applied to correct grades based on cluster centroids corresponding to linguistic performance levels (poor, good, excellent). The approach is evaluated on several years of real course data from an introductory programming course with approximately 800 students per year called “Programski jezici i strukture podataka” in the first year of studies of multiple study programs at the Faculty of Technical Sciences, University of Novi Sad, Serbia. Experimental results show that direct fuzzy grading tends to be overly strict compared to human grading, while fuzzy metric correction significantly reduces grading deviation and improves alignment with human assessment, particularly for higher-performing students. Combining LLM-based semantic analysis with fuzzy metrics yields a more nuanced, interpretable, and adaptable grading process, with potential applicability across a wide range of educational assessment scenarios.

Due to the vast number of students enrolled in programming courses, there has been an increasing number of automated program repair techniques focused on introductory programming assignments ( IPAs ). Typically, such techniques use program clustering to take advantage of previous correct student implementations to repair a new incorrect submission. These repair techniques use clustering methods since analyzing all available correct submissions to repair a program is not feasible. However, conventional clustering methods rely on program representations based on features such as abstract syntax trees ( ASTs ), syntax, control flow, and data flow. This paper proposes InvAASTCluster , a novel approach for program clustering that uses dynamically generated program invariants to cluster semantically equivalent IPAs . InvAASTCluster ’s program representation uses a combination of the program’s semantics, through its invariants, and its structure through its anonymized abstract syntax tree ( AASTs ). Invariants denote conditions that must remain true during program execution, while AASTs are ASTs devoid of variable and function names, retaining only their types. Our experiments show that the proposed program representation outperforms syntax-based representations when clustering a set of correct IPAs . Furthermore, we integrate InvAASTCluster into a state-of-the-art clustering-based program repair tool. Our results show that InvAASTCluster advances the current state-of-the-art when used by clustering-based repair tools by repairing around 13% more students’ programs, in a shorter amount of time. • We cluster programming assignments (IPAs) using invariants and anonymized ASTs. • We present our invariant-based program clustering tool, InvAASTCluster. • InvAASTCluster demonstrates the effectiveness of invariant-based clustering for IPAs. • We compare InvAASTCluster with Clara, a leading clustering-based repair tool. • InvAASTCluster outperforms Clara, allowing Clara to fix more IPAs and faster.

Exams including practical programming are a crucial component of evaluating students' coding and problem-solving capabilities. However, it might be challenging to objectively evaluate a candidate's skills because standard programming exam approaches are prone to cheating. It takes a lot of time to review programming assignments. These drawbacks emphasize the need for a new coding evaluation method that may offer a more impartial and precise evaluation of a candidate's coding abilities. By using an automated approach, the initiative seeks to revolutionize the way practical tests are administered using cutting-edge technologies. Exam scheduling, assigning distinct problem statements, automatically evaluating submissions, providing a fair evaluation, and preventing malpractice with secure exam controls are all made easier by the platform. Additionally, the system is adaptable to all pupils because it supports a variety of programming languages.

The paper presents a case study on using artificial intelligence (AI) for preliminary grading of student programming assignments. By integrating our previously introduced learning programming interface Verificator with the Gemini 2.5 large language model via Google AI Studio, C++ student submissions were evaluated automatically and compared with teacher-assigned grades. The results showed moderate to high correlation, although the AI was stricter. The study demonstrates that AI tools can improve grading speed and consistency while highlighting the need for human oversight due to limitations in interpreting non-standard solutions. It also emphasizes ethical considerations such as transparency, bias, and data privacy in educational AI use. A hybrid grading model combining AI efficiency and human judgment is recommended.

Providing consistent summative assessment to students is important, as the grades they are awarded affect their progression through university and future career prospects. While small cohorts are typically assessed by a single assessor, such as the module/class leader, larger cohorts are often assessed by multiple assessors, typically teaching assistants, which increases the risk of inconsistent grading. To investigate the consistency of human grading of programming assignments, we asked 28 participants to each grade 40 CS1 introductory Java assignments, providing grades and feedback for correctness, code elegance, readability and documentation; the 40 assignments were split into two batches of 20. The 28 participants were divided into seven groups of four (where each group graded the same 40 assignments) to allow us to investigate the consistency of a group of assessors. In the second batch of 20, we duplicated one assignment from the first to analyse the internal consistency of individual assessors. We measured the inter-rater reliability of the groups using Krippendorff’s \(\alpha\) —an \(\alpha > 0.667\) is recommended to make tentative conclusions based on the rating. Our groups were inconsistent, with an average \(\alpha=0.2\) when grading correctness and an average \(\alpha < 0.1\) for code elegance, readability and documentation. To measure the individual consistency of graders, we measured the distance between the grades they awarded for the duplicated assignment in batch one and batch two. Only one participant of the 22 who didn’t notice that the assignment was a duplicate was awarded the same grade for correctness, code elegance, readability and documentation. The average grade difference was 1.79 for correctness and less than 1.6 for code elegance, readability and documentation. Our results show that human graders in our study cannot agree on the grade to give a piece of student work and are often individually inconsistent, suggesting that the idea of a ‘gold standard’ of human grading might be flawed. This highlights that a shared rubric alone is not enough to ensure consistency, and other aspects such as assessor training and alternative grading practices should be explored to improve the consistency of human grading further when grading programming assignments.

Programming education traditionally requires extensive manual assessment of student assignments, which is both time-consuming and resource-intensive for instructors. Recent advances in large language models (LLMs) open opportunities for automating this process and providing timely feedback. This paper investigates the application of artificial intelligence (AI) tools for preliminary assessment of undergraduate programming assignments. A multi-phase experimental study was conducted across three computer science courses: Introduction to Programming, Programming 2, and Advanced Programming Concepts. A total of 315 Python assignments were collected from the Moodle learning management system, with 100 randomly selected submissions analyzed in detail. AI evaluation was performed using ChatGPT-4 (GPT-4-turbo), Claude 3, and Gemini 1.5 Pro models, employing structured prompts aligned with a predefined rubric that assessed functionality, code structure, documentation, and efficiency. Quantitative results demonstrate high correlation between AI-generated scores and instructor evaluations, with ChatGPT-4 achieving the highest consistency (Pearson coefficient 0.91) and the lowest average absolute deviation (0.68 points). Qualitative analysis highlights AI’s ability to provide structured, actionable feedback, though variability across models was observed. The study identifies benefits such as faster evaluation and enhanced feedback quality, alongside challenges including model limitations, potential biases, and the need for human oversight. Recommendations emphasize hybrid evaluation approaches combining AI automation with instructor supervision, ethical guidelines, and integration of AI tools into learning management systems. The findings indicate that AI-assisted grading can improve efficiency and pedagogical outcomes while maintaining academic integrity.

This paper presents a comprehensive analysis of artificial intelligence-powered automated grading systems (AI AGSs) in STEM education, systematically examining their algorithmic foundations, mathematical modeling approaches, and quantitative evaluation methodologies. AI AGSs enhance grading efficiency by providing large-scale, instant feedback and reducing educators’ workloads. Compared to traditional manual grading, these systems improve consistency and scalability, supporting a wide range of assessment types, from programming assignments to open-ended responses. This paper provides a structured taxonomy of AI techniques including logistic regression, decision trees, support vector machines, convolutional neural networks, transformers, and generative models, analyzing their mathematical formulations and performance characteristics. It further examines critical challenges, such as user trust issues, potential biases, and students’ over-reliance on automated feedback, alongside quantitative evaluation using precision, recall, F1-score, and Cohen’s Kappa metrics. The analysis includes feature engineering strategies for diverse educational data types and prompt engineering methodologies for large language models. Lastly, we highlight emerging trends, including explainable AI and multimodal assessment systems, offering educators and researchers a mathematical foundation for understanding and implementing AI AGSs into educational practices.

Abstract Recent advances in artificial intelligence (AI), machine learning (ML), and natural language processing (NLP) have led to the development of a new generation of Large Language Models (LLMs) trained on massive amounts of data. Commercial applications (e.g., ChatGPT) have made this available to the general public, enabling the use of LLMs to produce high-quality texts for academic and professional purposes. Educational institutions are increasingly aware of students’ use of AI-generated content and are researching its impact and potential misuse. Computer Science (CS) and related fields are particularly affected, as LLMs can also generate programming code in various languages. To understand the potential impact of publicly available LLMs in CS education, we extend our previously introduced (Raihan et al. 2024), a framework comprising hundreds of programming exercise prompts and multiple-choice questions from introductory CS and programming courses. We provide experimental results on , evaluating the performance of several LLMs in generating Python code and answering basic computer science and programming questions, offering insights into the implications of this technology for CS education.

In today's digital context, artificial intelligence-based tools such as Blackbox AI are increasingly being used by students to complete programming assignments, especially in the field of web development. Blackbox AI is one form of AI that is widely used by students. This study aims to uncover concerns about students' dependence on Blackbox AI, which is considered to have a negative impact on their ability to code (do programming) during lectures or when completing assignments. In addition, there is concern that students may experience a decline in critical thinking skills due to frequent reliance on assistance in completing each existing assignment. The purpose of this study was to evaluate the extent to which students depend on Blackbox AI and its impact on critical thinking and coding skills. The method used is a quantitative descriptive approach, with data collected through a questionnaire using a Likert scale from 30 sixth semester students in the Informatics Education Study Program, Citra Bangsa University. The results of the descriptive analysis, most students felt significant benefits from utilizing Blackbox AI with an average of 77.6%, but they also faced a dependence of 63.8% and a negative effect on understanding and independence of 57.1%. Concerns about the decline in coding skills due to reliance on AI were also quite high, reaching 48.9%. This finding highlights the importance of using Blackbox AI wisely and integrated with traditional learning, so as not to hinder the development of critical thinking skills and the basics of programming in students.

This paper explores the application of large language models (LLMs) to automate the evaluation of programming assignments in an undergraduate “Introduction to Programming” course. This study addresses the challenges of manual grading, including time constraints and potential inconsistencies, by proposing a system that integrates several LLMs to streamline the assessment process. The system utilizes a graphic interface to process student submissions, allowing instructors to select an LLM and customize the grading rubric. A comparative analysis, using LLMs from OpenAI, Google, DeepSeek and ALIBABA to evaluate student code submissions, revealed a strong correlation between LLM-generated grades and those assigned by human instructors. Specifically, the reduced model using statistically significant variables demonstrates a high explanatory power, with an adjusted R2 of 0.9156 and a Mean Absolute Error of 0.4579, indicating that LLMs can effectively replicate human grading. The findings suggest that LLMs can automate grading when paired with human oversight, drastically reducing the instructor workload, transforming a task estimated to take more than 300 h of manual work into less than 15 min of automated processing and improving the efficiency and consistency of assessment in computer science education.

This paper examines the problem of automated testing of modified programming tasks for future computer science teachers. It is recommended that GitHub Copilot be used to generate tests based on the code. This approach makes it possible to solve the following tasks: reducing the time and effort required for manual checking of programming tasks completed by students; promoting better assimilation of the material of the relevant disciplines by students; promoting the development and improvement of students' skills in algorithmisation and programming; compliance by students with academic integrity; effective use of GitHub Copilot to generate baseline tests to test modified programming assignments completed by students; ensuring the flexibility and scalability of the approach to the development of various training courses in programming; development of students' software product testing skills. In the process of research, we found the following disadvantages of using the GitHub Copilot system for generating basic tests: GitHub Copilot does not always generate perfect code or tests; for complex tasks, GitHub Copilot may require additional correction of the generated code. Therefore, it is important to check and refine the generated tests carefully, if necessary. Therefore, at the moment, we recommend using GitHub Copilot as a template generator for writing tests. The proposed approach is a promising solution for facilitating the verification of modified programming tasks and increasing the effectiveness of the education process of future informatics teachers. The conducted research opens up new prospects for effective improvement of the verification of modified tasks performed by students and the generation of tests for verification. In particular, the integration of the proposed system based on GitHub Copilot with learning management systems (LMS) and automated task verification systems. Another area of research could be exploring the possibilities of using other tools for generating tests instead of GitHub Copilot or combining them in order to obtain better results.

IntroductionGenerative AI tools can produce programming code that looks very similar to human-written code, which creates challenges in programming education. Students may use these tools inappropriately for their programming assignments, and there currently are not reliable methods to detect AI-generated code. It is important to address this issue to protect academic integrity while allowing the constructive use of AI tools. Previous studies have explored ways to detect AI-generated text, such as analyzing structural differences, embedding watermarks, examining specific features, or using fine-tuned language models. However, certain techniques, like prompt engineering, can make AI-generated code harder to identify.MethodsTo tackle this problem, this article suggests a new approach for instructors to handle programming assignment integrity. The idea is for instructors to use generative AI tools themselves to create example AI-generated submissions (pseudo-AI submissions) for each task. These pseudo-AI submissions, shared along with the task instructions, act as reference solutions for students. In the presence of pseudo-AI submissions, students are made aware that submissions resembling these examples are easily identifiable and will likely be flagged for lack of originality. On one side, this transparency removes the perceived advantage of using generative AI tools to complete assignments, as their output would closely match the provided examples, making it obvious to instructors. On the other side, the presence of these pseudo-AI submissions reinforces the expectation for students to produce unique and personalized work, motivating them to engage more deeply with the material and rely on their own problem-solving skills.ResultsA user study indicates that this method can detect AI-generated code with over 96% accuracy.DiscussionThe analysis of results shows that pseudo-AI submissions created using AI tools do not closely resemble student-written code, suggesting that the framework does not hinder students from writing their own unique solutions. Differences in areas such as expression assignments, use of language features, readability, efficiency, conciseness, and clean coding practices further distinguish pseudo-AI submissions from student work.

Large Language Models (LLMs) are a type of artificial intelligence (AI) that have emerged as powerful tools for a wide range of tasks, paving the way for new applications previously unhandled. The use of LLMs is increasing, especially among students. This study aimed to understand how students perceive and use these technologies for programming tasks. We surveyed students and recent graduates of a Master's degree in Data Science for Health regarding their programming assignments. Among respondents (n=77), 84.4% (n=65) reported using LLMs for tasks such as debugging, generating code, understanding error messages, optimizing code and providing detailed explanations. Of these users, 55.4% (n=36) reported that LLM usage has become a daily habit, while 46.1% (n=30) noted a growing trend in their usage of LLMs. Furthermore, 87.7% (n=57) engaged in monitoring to keep up to date with the latest developments. LLMs are considered reliable by the majority of participants, however most of them still carried out verifications on their answers. Although 81.5% (n=53) of LLM users were satisfied with the tools, citing their speed, ease of use, and debugging potential, concerns about tool dependency, data confidentiality, and the precision of references were also raised. The results highlighted the uptake of these technologies by students, indicating that the integration of LLMs into educational settings is essential to promote best practices while maintaining a focus on the importance of fundamental skills such as problem-solving and critical thinking, which are indispensable in professional life. Therefore, educators are encouraged to adapt their teaching methods and assessment strategies accordingly.

Automated Program Repair (APR) for introductory programming assignments (IPAs) is motivated by the large number of student enrollments in programming courses each year. Since providing feedback on programming assignments requires substantial time and effort from faculty, personalized automated feedback often involves suggesting repairs to students' programs. Symbolic semantic repair approaches, which rely on Formal Methods (FM), check a program's execution against a test suite or reference solution, are effective but limited. These tools excel at identifying buggy parts but can only fix programs if the correct implementation and the faulty one share the same control flow graph. Conversely, Large Language Models (LLMs) are used for program repair but often make extensive rewrites instead of minimal adjustments. This tends to lead to more invasive fixes, making it harder for students to learn from their mistakes. In summary, LLMs excel at completing strings, while FM-based fault localization excel at identifying buggy parts of a program. In this paper, we propose a novel approach that combines the strengths of both FM-based fault localization and LLMs, via zero-shot learning, to enhance APR for IPAs. Our method uses MaxSAT-based fault localization to identify buggy parts of a program, then presents the LLM with a program sketch devoid of these buggy statements. This hybrid approach follows a Counterexample Guided Inductive Synthesis (CEGIS) loop to iteratively refine the program. We ask the LLM to synthesize the missing parts, which are then checked against a test suite. If the suggested program is incorrect, a counterexample from the test suite is fed back to the LLM for revised synthesis. Our experiments on 1,431 incorrect student programs show that our counterexample guided approach, using MaxSAT-based bug-free program sketches, significantly improves the repair capabilities of all six evaluated LLMs. This method allows LLMs to repair more programs and produce smaller fixes, outperforming other configurations and state-of-the-art symbolic program repair tools.

Artificial Intelligence (AI) literacy is increasingly important across many fields, yet caregivers remain underrepresented in AI-related fields due to a combination of systemic and individual barriers. To address this, the Caregivers and Machine Learning (C&ML) program developed and delivered an accessible AI education program to caregivers on parental leave. Two cohorts participated in this 6-week interprofessional program, featuring fundamental machine learning concepts, hands-on programming assignments, and a capstone project. This study examines the program's impact on participants, focusing on their motivations and barriers before, during, and after the program as outcomes after completion. Post-program surveys and semi-structured interviews highlight that caregivers often face barriers such as the rapid pace of AI, discrimination, and balancing caregiving responsibilities with learning new skills. The C&ML program's flexible structure and personalized support network were critical in enabling participants to fully engage in the program, leading to significant improvements in their knowledge of ML and increased confidence in applying these skills. After completing the program, 20\% of participants transitioned into AI-related roles or pursued further education. This research highlights the value of targeted, inclusive educational programs for underrepresented groups and provides practical recommendations for refining future AI training programs for caregivers.