Rutting is one of the main sources of distress in asphalt pavements and has been widely studied because it influences safety and maintenance costs. To mitigate this distress, the selection of adequate asphalt binder and aggregates and an appropriate balance between them are key elements. This paper analyzed seven asphalt binders (2 neat, 3 polymer-modified, and 2 rubber-asphalt binders) and asphalt mixtures with different gradations (19-mm dense-graded and 12.5-mm gap-graded asphalt mixtures). The asphalt binders were evaluated in terms of rheology with the use of a dynamic shear rheometer (DSR) and a bending beam rheometer (BBR) to obtain the high- and low-performance grade (PG) temperatures. Additionally, multiple stress creep and recovery (MSCR) tests were performed to analyze the rutting potential of the materials at different temperatures. To evaluate the effect of the mineral skeleton on the asphalt mixtures, the Laboratoire Central des Ponts et Chaussees (LCPC) wheel tracking rutting test was performed. The polymer-modified binders (PMBs) and the asphalt mixtures produced with them were the most resistant to permanent deformation with low values of rut depth (< 2.5%). In general, the gap-graded mixtures had better resistance than the dense-graded mixtures; however, this comparison is dependent on the type of asphalt binder. Relationships between the binder and mixture parameters were also proposed and they demonstrated a good relationship between the nonrecoverable creep compliance of the binders (Jnr) and the rut depth of the dense mixtures (R2 = 0.81); however, no significant relationship was found for the gap-graded mixtures (R2 = 0.52).