In semiconductor quantum dots, the electron hyperfine interaction with the nuclear spin bath is the leading source of spin decoherence at cryogenic temperature. Using high-resolution two-color differential transmission spectroscopy, we demonstrate that such electron-nuclear coupling also imposes a lower limit for the positively charged exciton dephasing rate, \gamma, in an ensemble of InAs/GaAs quantum dots. We find that the dephasing rate is sensitive to the strength of the hyperfine interaction, which can be controlled through the application of an external magnetic field in the Faraday configuration. At zero applied field, strong electron-nuclear coupling induces additional dephasing beyond the radiative limit and \gamma = 230 MHz (0.95 \mu eV). Screening of the hyperfine interaction is achieved for an external field of ~1 T, resulting in \gamma = 172 MHz (0.71 \mu eV) limited only by spontaneous recombination. On the other hand, application of a Voigt magnetic field mixes the spin eigenstates, which increases the dephasing rate by up to 75%. These results are reproduced with a simple and intuitive model that captures the essential features of the electron hyperfine interaction and its influence on \gamma.