Effect of grading schemes on outcomes in query writing experiments

Abstract

There have been many experiments comparing query languages. Their findings are difficult to combine as the experiments have used different settings and procedures. Before a meta-analysis combining the experiments, it is proposed that these differences be checked for any possible effect. The most important measure of user performance in these experiments is query accuracy, which has been determined using many different grading schemes. The different schemes are therefore checked for possible effects on hypothesis rejection. These grading schemes are applied to two sets of queries from two different experiments. The outcomes are examined to identify any effects resulting from the grading schemes. The results show that the experimental outcomes are robust and immune to the grading schemes.