Some individuals experience more difficulties with math than others, in particular when arithmetic problems get more complex. Math ability, on one hand, and arithmetic complexity, on the other hand, seem to partly share neural underpinnings. This study addresses the question of whether this leads to an interaction of math ability and arithmetic complexity for multiplication and division on behavioral and neural levels. Previously screened individuals with high and low math ability solved multiplication and division problems in a written production paradigm while brain activation was assessed by combined functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG). Arithmetic complexity was manipulated by using single-digit operands for simple multiplication problems and operands between 2 and 19 for complex multiplication problems and the corresponding division problems. On the behavioral level, individuals with low math ability needed more time for calculation, especially for complex arithmetic. On the neural level, fNIRS results revealed that these individuals showed less activation in the left supramarginal gyrus (SMG), superior temporal gyrus (STG) and inferior frontal gyrus (IFG) than individuals with high math ability when solving complex compared to simple arithmetic. This reflects the greater use of arithmetic fact retrieval and also the more efficient processing of arithmetic complexity by individuals with high math ability. Oscillatory EEG analysis generally revealed theta and alpha desynchronization with increasing arithmetic complexity but showed no interaction with math ability. Because of the discovered interaction for behavior and brain activation, we conclude that the consideration of individual differences is essential when investigating the neurocognitive processing of arithmetic.