Mathematics anxiety (MA) refers to the experience of negative affect when engaging in mathematical activity. According to Ashcraft and Kirk (2001), MA selectively affects calculation with high working memory (WM) demand. On the other hand, Maloney, Ansari, and Fugelsang (2011) claim that MA affects all mathematical activities, including even the most basic ones such as magnitude comparison. The two theories make opposing predictions on the negative effect of MA on magnitude processing and simple calculation that make minimal demands on WM. We propose that MA has a selective impact on mathematical problem solving that likely involves processing of magnitude representations. Based on our hypothesis, MA will impinge upon magnitude processing even though it makes minimal demand on WM, but will spare retrieval-based, simple calculation, because it does not require magnitude processing. Our hypothesis can reconcile opposing predictions on the negative effect of MA on magnitude processing and simple calculation. In the present study, we observed a negative relationship between MA and performance on magnitude comparison and calculation with high but not low WM demand. These results demonstrate that MA has an impact on a wide range of mathematical performance, which depends on one's sense of magnitude, but spares over-practiced, retrieval-based calculation.