Secondary copper flue dust (SCFD), produced by secondary copper reduction smelting in blast furnaces, contains valuable metal elements (e.g., Zn, Pb, Cu, Sn, and Bi) and various toxic elements (e.g., As, Cd, F, Cl, Br, and Tl). The H2SO4 roasting-water leaching process is widely employed to pre-separate target elements. However, due to the complexity of raw materials, the mineralogical factors that affect the removal efficiency or leaching separation of some elements are unclear. In this study, the quantitative characterization of valuable and toxic elements in SCFD was first investigated through mineralogy methods. Then, the distribution and occurrence of all elements in products from the low-temperature H2SO4 roasting-water leaching process were clarified. The results indicated that the phases in SCFD were complex, including ZnO, PbCl2, Cu2S, metal Pb, Zn2SnO4, Cu-sn-Ni alloy, and C. They also indicated that the formation of Pb3(AsO4)2 was negatively correlated with the removal of As. The encapsulation of newly formed PbSO4 and copper film to large-grained Cu2S affected its transformation to CuSO4. Furthermore, the adsorption of siliceous colloid on Cd2+ impeded the leaching of Cd. Based on the mineralogical factors, guidance for process optimization was proposed. The optimized process realizes the deep removal of toxic elements and efficiently selective leaching of valuable metals.