The conventional walnut drying method by using hot air exhibits low drying efficiency. Prolonged drying processes at low temperatures decrease drying performance and lead to high energy use, cost and product loss. This research employed sequential infrared (IR) pre-drying followed by hot air drying to investigate the drying and quality characteristics of intact-shell, cracked-shell, and kernel walnuts. The obtained results revealed that the drying time for conventional hot air-dried intact-shell walnuts, IR pre-dried intact-shell walnuts, IR pre-dried cracked-shell walnuts, and IR pre-dried kernels were 240, 130, 70, and 40 min, respectively. The quality characteristics of all dried walnuts conformed to industrial quality standards (with lightness, L > 40; peroxide value, PV < 1.0 meq/kg oil; and acidity value, AV < 1.2%), thereby improving the microbial safety of the dried products in comparison to the fresh walnuts. The conventional hot air drying resulted in increased specific energy consumption (31.56 MJ/kg) and kernel energy cost (0.041 $/kg), while the sequential IR pre-drying followed by hot air drying ranged between 26.81 and 7.76 MJ/kg and 0.022–0.013 $/kg, respectively. In conclusion, IR pre-dried cracked-shell walnuts and IR pre-dried kernels maintained the product quality and safety with reduced drying time and energy use. This work provided valuable insights for the walnut industry to improve the drying efficiency and produce walnuts with preserved quality and safety.