In this study, the impact toughness and deformation mechanism of base metal (BM), heat affected zone (HAZ) and weld zone (WZ) of Ti6321 alloy joint at different temperatures (20 °C, 0 °C, −20 °C, −40 °C, −60 °C and −196 °C) are studied. The microstructure of BM consists of primary α (αp), acicular α and β phase. The HAZ is composed of αp, acicular α, acicular martensite (α′) and β phase. The WZ is composed of acicular α′, lath α′ and β phase. All the BM, HAZ and WZ obtain the highest impact toughness at 20 °C, which are 51 J, 75 J and 72 J, respectively. Under −196 °C, brittle fractures occurs in the BM, HAZ and WZ, and their impact toughness are 18 J, 22 J and 30 J, respectively. Compared with the impact test at −196 °C, the fracture surface of 20 °C by impact test has higher kernel average misorientation and geometrically necessary dislocation. The fracture impact resistance at 20 °C is greater. Grain deformation at 20 °C consumes more energy and the crack propagation path is more tortuous.