Reading and arithmetic are distinct academic skills that share similarities in skill acquisition and use. Previous research investigated the cognitive basis of associations and dissociations between reading and arithmetic by using either subtyping or dimensional approaches. In the current study, we aim to bridge the gap between these two approaches by investigating common and distinct predictors of reading and arithmetic at different performance levels with quantile regression models. This allowed us to look more closely at the lower tail of the ability distributions, and to test whether predictions for children with low reading and arithmetic fluency differed from the typical performance range. We analyzed longitudinal data of 357 children speaking English or German. Outcome variables were reading and arithmetic fluency assessed at the end of Grade 1, 2, and 3. Predictors were assessed in Grade 1. Results confirmed nonverbal IQ and working memory as domain-general predictors of reading and arithmetic. The association of reading and arithmetic was mainly explained by nonverbal IQ, phonological awareness, RAN and multi-digit transcoding. Across grades and performance levels, phonological awareness and RAN made a specific contribution to reading. Magnitude processing and multi-digit transcoding were specific predictors of arithmetic. Counting also made a specific prediction to arithmetic in Grade 3, but only in the low performance range. Our findings indicate partly distinct underlying cognitive mechanisms for reading and arithmetic. Shared predictors are involved in retrieval efficiency, language processing and cross-format integration. These results have important implications, as they suggest that most predictors are equally relevant for children with low, typical or even excellent reading and arithmetic fluency.