The goal of measuring conceptual processing in numerical cognition is distanced by the possibility that neural responses to symbolic numerals are influenced by physical stimulus confounds. Here, we targeted conceptual responses to parity (even versus odd), using electroencephalogram (EEG) frequency-tagging with a symmetry/asymmetry design. Arabic numerals (2-9) were presented at 7.5 Hz in 50 s sequences; odd and even numbers were alternated to target differential, 'asymmetry' responses to parity at 3.75 Hz (7.5 Hz/2). Parity responses were probed with four different stimulus sets, increasing in intra-numeral stimulus variability, and with two control conditions composed of non-conceptual numeral alternations. Significant asymmetry responses were found over the occipitotemporal cortex to all conditions, even for the arbitrary controls. The large physical-differences control condition elicited the largest response in the stimulus set with the lowest variability (one font). Only in the stimulus set with the highest variability (20 drawn, coloured exemplars/numeral) did the response to parity surpass both control conditions. These findings show that physical differences across small sets of Arabic numerals can strongly influence, and even account for, automatic brain responses. However, carefully designed control conditions and highly variable stimulus sets may be used towards identifying truly conceptual neural responses.