Heusler-type Fe2VAl thermoelectric materials have been processed by high-pressure torsion (HPT) to improve their thermoelectric performance through a drastic reduction of the thermal conductivity. While the thermal conductivity for Fe2VTa0.05Al0.95 is about 10 W/m K at 300 K, it is reduced to about 5.0 W/m K due to HPT processing. Furthermore, even after annealing at 873 K for recurrence of the Heusler-type structure, Ta-doped Fe2VTa0.05Al0.95 possesses an ultra-fine grained structure with an average grain size of about 80 nm, in contrast to the presence of large grains of about 270 nm for non-doped Fe2V1.05Al0.95. Since scanning transmission electron microscopy combined with energy-dispersive X-ray analysis reveals the solute segregation of Ta along grain boundaries and recrystallization interfaces, the retardation of recrystallization could be caused by the solute drag effect, leading to a suppression of grain coarsening. Thus, it is possible to retain the low thermal conductivity of about 5.0 W/m K at 300 K for Fe2VTa0.05Al0.95, in parallel with the restoration of a large power factor, so that the dimensionless figure of merit reaches ZT = 0.30 at around 500 K.