Powder quality in additive manufacturing (AM) electron beam melting (EBM) of Ti-6Al-4V components is crucial in determining the critical material properties of the end item. In this study, we report on the effect of powder oxidation on the Charpy impact energy of Ti-6Al-4V parts manufactured using EBM. In addition to oxidation, the effects on impact energy due to hot isostatic pressing (HIP), specimen orientation, and EBM process defects were also investigated. This research has shown that excessive powder oxidation (oxygen mass fraction above 0.25% and up to 0.46%) dramatically decreases the impact energy. It was determined that the room temperature impact energy of the parts after excessive oxidation was reduced by about seven times. We also report that HIP post-processing significantly increases the impact toughness, especially for specimens with lower or normal oxygen content. The specimen orientation effect was found to be more significant for low oxidation levels.