We theoretically and experimentally analyze Q-switched cladding pumped ytterbium-doped fiber lasers designed for high pulse energies. We compare the extractable energy from two high-energy fiber designs: (1) single- or few-moded low-NA large mode area (LMA) fibers and (2) large-core multimode fibers, which may incorporate a fiber taper for brightness enhancement. Our results show that the pulse energy is proportional to the effective core area and, therefore, LMA fibers and multimode fibers of comparable core size give comparable results. However, the energy storage in multimode fibers is mostly limited by strong losses due to amplified spontaneous emission (ASE) or even spurious lasing between pulses. The ASE power increases with the number of modes in a fiber. Furthermore, spurious feedback is more difficult to suppress with a higher NA, and Rayleigh back-scattering increases with higher NA, too. These effects are smaller in low-NA LMA fibers, allowing for somewhat higher energy storage. For the LMA fibers, we found that facet damage was a more severe restriction than ASE losses or spurious lasing. With a modified laser cavity, we could avoid facet damage in the LMA fiber, and reached output pulse energies as high as 2.3 mJ, limited by ASE. Theoretical estimates suggest that output pulse energies around 10 mJ are feasible with a larger core fiber, while maintaining a good beam quality.