Experimental design is a fundamental skill for scientists, but it is often not explicitly taught in large introductory biology classes. We have designed two pencil-and-paper in-class activities to increase student understanding of experimental design: an analyze activity, in which students are asked to evaluate data, and a design activity, in which students are asked to propose a novel experiment. We found that students who completed the design activity but not the analyze activity performed significantly better on the Expanded Experimental Design Ability Tool (E-EDAT) than did students who attended a didactic lecture about experimental design. By using grounded theory on student responses on the in-class activities, we have identified a novel set of accurate and inaccurate conceptions focused on two aspects of experimental design: sample size and the repetition of experiments. These findings can be used to help guide science majors through mastering the fundamental skill of designing rigorous experiments.