The use of recycled (hereafter called secondary) aluminum alloys is increasing more and more in light of sustainability as the energy needed for their production is much lower than in the case of the primary alloys and they may have mechanical properties comparable to those of the latter. Most of the secondary aluminum alloys are used to fabricate parts through casting processes, which may need further machining operations to get the part’s final shape. While the mechanical properties of the secondary aluminum alloys have been comprehensively addressed in the literature and correlated to the different intermetallic particles that characterize their microstructure, the same is not true when addressing machinability. In this framework, the paper investigates the machinability of one primary and two secondary aluminum alloys in terms of cutting forces and surface finish after turning trials carried out at fixed cutting parameters. A detailed characterization of the alloys’ microstructure was carried out making use of both optical and scanning electron microscopy to identify the size, morphology, and distribution of intermetallics. The highest cutting force was registered when machining the primary alloy, being characterized by the highest specific cutting energy. The surface damage in terms of tearings induced by cutting was comparable between the primary and secondary alloys. In contrast, the different roughness features that characterize the machined surfaces of the considered alloys can be partly ascribed to the different intermetallics they present. Nevertheless, the surface topography analysis results must be interpreted based on the specific application.