We examined whether math anxiety is related to a response inhibition deficit and, if so, whether it is a domain-specific inhibition deficit in numerical tasks or a general inhibition deficit. Behavioral performance and electroencephalogram activity were recorded while 28 highly math-anxious (HMA) and 28 low math-anxious (LMA) individuals performed both a numerical and a non-numerical Go/Nogo task. In the numerical task, single-digit numbers were presented, and participants were asked to press a button if the number was even. In the non-numerical task, letters were presented, and the button had to be pressed if the letter was a vowel. Nogo trials were answered less accurately and elicited larger Nogo-N2 and Nogo-P3 than Go trials in both tasks and both groups. Importantly, behavioral and brain response differences between tasks were only found in the HMA group. First, they were more error-prone in numerical Nogo than in non-numerical Nogo trials; and second, their Nogo-N2 and N2d (Nogo-Go difference) were smaller in the numerical task than in the non-numerical task. No differences were found in the LMA group. These results suggest that HMA individuals' response inhibition is impaired specifically when dealing with numbers, which could contribute to their low achievement in math tasks.