Context. The precise mechanism that forms jets and large-scale vortices on the giant planets is unknown. An inverse cascade has been suggested by several studies. Alternatively, energy may be directly injected by small-scale convection. Aims. Our aim is to clarify whether an inverse cascade feeds zonal jets and large-scale eddies in a system of rapidly rotating, deep, geostrophic spherical-shell convection. Methods. We analyze the nonlinear scale-to-scale transfer of kinetic energy in such simulations as a function of the azimuthal wave number, m. Results. We find that the main driving of the jets is associated with upscale transfer directly from the small convective scales to the jets. This transfer is very nonlocal in spectral space, bypassing large-scale structures. The jet formation is thus not driven by an inverse cascade. Instead, it is due to a direct driving by Reynolds stresses, statistical correlations of velocity components of the small-scale convective flows. Initial correlations are caused by the effect of uniform background rotation and shell geometry on the flows and provide a seed for the jets. While the jet growth suppresses convection, it increases the correlation of the convective flows, which further amplifies the jet growth until it is balanced by viscous dissipation. To a much smaller extent, energy is transferred upscale to large-scale vortices directly from the convective scales, mostly outside the tangent cylinder. There, large-scale vortices are not driven by an inverse cascade either. Inside the tangent cylinder, the transfer to large-scale vortices is even weaker, but more local in spectral space, leaving open the possibility of an inverse cascade as a driver of large-scale vortices. In addition, large-scale vortices receive kinetic energy from the jets via forward transfer. We therefore suggest a jet instability as an alternative formation mechanism of large-scale vortices. Finally, we find that the jet kinetic energy scales approximatively as ℓ−5, the same as for the so-called zonostrophic regime.