Abstract
In some situations, it is necessary to measure personal programming skills. For example, often students must be divided according to skill level and motivation to learn or companies recruiting employees must rank candidates by evaluating programming skills through programming tests, programming contests, etc. This process is burdensome because teachers and recruiters must prepare, implement, and evaluate a placement examination. This paper tries to predict the placement and ranking results of programming contests via machine learning without such an examination. Explanatory variables used for machine learning are classified into three categories: Psychological Scales, Programming Tasks, and Student-answered Questionnaires. The participants are university students enrolled in a Java programming class. One target variable is the placement result based on an examination by a teacher of a class and the ranking results of the programming contest. Our best classification model with a decision tree has an F-measure of 0.912, while our best ranking model with an SVM-rank has an nDCG of 0.962. In both prediction models, the best explanatory variable is from the Programming Task followed in order by Psychological Sale and Student-answered Questionnaire. Our classification model uses 9 explanatory variables, while our ranking model uses 20 explanatory variables. These include all three types of explanatory variables. The source code complexity, which is a source code metrics from Programming Task, shows best performance when the prediction uses only one explanatory variable. Contribution (1), this method can automate some of the teacher’s workload, which may improve educational quality and increase the number of acceptable students in the course. Contribution (2), this paper shows the potential of using difficult-to-formulate information for an evaluation such as a Psychological Scale is demonstrated. These are the contributions and implications of this paper.
Highlights
Sometimes it is necessary to measure a person’s programming skills1
This paper uses a traditional method, if this method is applied, our machine learning model may further improve the performance in the future
The explanatory variables of the measured metrics show high normalized Discounted Cumulated Gain (nDCG). These results show that the score of the programming contest is not related to the Psychological Scales or class attitudes, but it is related to the quality of the written source code, which can be measured by the source code metrics
Summary
In education, often students must be divided into advanced and intermediate classes based on skill level, motivation to learn, etc. As another example, a company recruiting and placing employees must rank candidates by evaluating programming skills through programming tests, programming contests, etc. A company recruiting and placing employees must rank candidates by evaluating programming skills through programming tests, programming contests, etc These processes are burdensome because the Ishizue et al Research and Practice in Technology Enhanced Learning (2018) 13:7 evaluator (teacher or recruiter) must prepare, implement, and assess the examination (e.g., placement test or questionnaire regarding class level) to determine programming ability. Some students only memorize the answers of past examinations, while other students cram the night before a test
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
