This study is aimed at exploiting the potential of graphene oxide (GO) as inorganic filler of hybrid membranes for efficient proton transfer. A series of GO nanosheets with different lateral size and oxidation degree are incorporated into Nafion. Atomic force microscopy (AFM) and small-angle X-ray scattering (SAXS) characterization of the obtained membranes demonstrates the effects of the size and oxidation degree of GO on the nanophase separation of Nafion: (i) large GO sheets can strongly disturb the nanophase separation, while the thoroughly sonicated ones produce moderate interruption; (ii) increasing the oxidation degree of GO can also restrict nanophase separation, but the effect is not pronounced as that of GO size. Furthermore, GO sheets with small sheet size and high oxidation degree can effectively increase the water uptake and proton conductivity of Nafion membrane. For instance, the incorporation of 5 wt% optimal GO sheets into Nafion results in about 1.5-fold increase of proton conductivity under 40–80% RH and 80 °C. By comparison, embedding the same GO sheets into sulfonated poly (ether ether ketone) (SPEEK) leads to comparable proportion of proton conductivity increment under the same conditions, indicating that GO in these membranes is actually a structure modifier rather than a proton accelerator. This feature is beneficial to pushing the limits of phase-separated polymers with intrinsically fast proton transfer characteristics.