Elastomers, plastomers, and Reactive Ethylene Terpolymers (RETs) are some of the most commonly used polymers for modification of bitumen. Each type of modifier can have a considerably different influence on the rheological behavior of the binders and therefore, the performance of bituminous mixtures in pavements. In this study, for comparing the effect of different modifiers on the resistance to permanent deformation of asphalt binders, two elastomer-modified binders, and two RET-modified binders were evaluated using laboratory characterization tests. In order to have a comparison between the different modifiers, all the modified binders were prepared using the same base binder. A range of binder and mixture characterization tests were conducted including the Superpave high-temperature Performance Grading (PG) test, Multiple Stress Creep and Recovery test (MSCR) and steady shear tests on the binders, and flow number, flow time and dry rut wheel tests on the mixtures. Each binder characterization test was found to yield a different ranking of binders with respect to their resistance to permanent deformation. However, the results from mixture characterization tests were more conclusive; all tests on mixtures yielded nearly identical ranking of binders. When the results of binder and mixture characterization tests were compared, viscosity from steady shear tests was found to correlate the most with the permanent deformation observed in the mixture characterization tests.