The microstructure and texture of an AlFeMn alloy were studied under different intermediate annealing processes, and the changes in microhardness during cold rolling were analyzed. After annealing at 420 °C with a slow heating rate, the alloy showed a high number of small dispersed particles and recrystallization textures dominated by R texture, with deformation textures of 23.5%. Annealing at 610 °C with a rapid heating rate resulted in a significant decrease in the number of small-sized particles and an increase in recrystallization texture contents, with CubeND being the majority. The deformation texture contents decreased to 14.9%. The electrical conductivity of the 420 °C annealed sheet was higher than before annealing, whereas the sheet annealed at 610 °C showed a decrease in electrical conductivity after annealing. This indicated that annealing at 610 °C led to a higher degree of recrystallization and the development of Cube/CubeND due to the dissolution of dispersed particles. During the subsequent cold rolling process, the microhardness of both annealed sheets initially increased and then decreased. However, the microhardness of the 420 °C annealed sheet with varying cold rolling reductions consistently remained lower than that of the 610 °C annealed sheet, as was the cold rolling reduction corresponding to the peak microhardness. The results showed that the precipitation at 420 °C facilitated work softening, while the dissolution at 610 °C promoted work hardening.