We use three-dimensional contact dynamics simulations to analyze the rheology of polydisperse packings of spherical particles subjected to simple shear. The macroscopic and microstructural properties of several packings are analyzed as a function of their size span (from nearly monodisperse to highly polydisperse). Consistently with previous two-dimensional simulations, we find that the shear strength is independent of the size span despite the increase of packing fraction with size polydispersity. At the grain scale, we analyze the particle connectivity, force transmission, and the corresponding anisotropies of the contact and force networks. We show that force distributions become increasingly broader as the size span increases. In particular, stronger forces are captured by large particles, which are also better connected creating the so-called granular backbone. Throughout this backbone friction mobilization is rare and compressive forces control the stability of such structure. In return, small particles create an important population of rattlers discarded of the strength and granular structure analysis. As a consequence, the contact anisotropy declines with size span, whereas the force and branch anisotropies increase. These microstructural compensations allow us to explain the independence of the shear strength from particle size polydispersity.