How confident a student is about how they answer a question has important education implications. Participants answered 10 mathematics questions and provided their estimates of how likely they got each individual item correct and how many, in total, they answered correctly. They were overconfident in these metacognitive judgments. Some of the participants were asked to justify why their answers were either correct or incorrect prior to making these judgments. This lowered their confidence ratings. They were still overconfident, but less than those in the control group. The instruction also affected the association between the confidence ratings and accuracy. No differences were observed between those asked to justify why their responses were correct versus those asked to justify why their responses were incorrect. Those asked to think about the accuracy of a response had lower confidence. This has important implications for understanding how we construct confidence judgments and within education how student confidence can be affected during assessments.