Compact optically pumped passively modelocked semiconductor disk lasers (SDLs) based on active quantum wells (QWs) such as vertical external-cavity surface-emitting lasers (VECSELs) or modelocked integrated external-cavity surface-emitting lasers (MIXSELs) are wavelength-versatile sources that offer a unique combination of gigahertz pulse repetition rates and short pulse durations. In this paper, we present record-short pulses of 184 fs from a gigahertz MIXSEL emitting at a center wavelength of 1048 nm. This result comes at the expense of low optical-to-optical pump efficiency (<1%) and average output power limited to 115 mW. We experimentally observe that shorter pulses significantly reduce the macroscopic gain saturation fluence and develop a QW model based on rate equations to reproduce the gain saturation behavior and quantitatively explain the VECSEL and MIXSEL modelocking performances. We identify spectral hole burning as the main cause of the reduced gain at shorter pulse durations, which in combination with the short lifetime of the excited carriers strongly reduces the optical pump efficiency. Our better understanding will help to address these limitations in future ultrafast SDL designs.