The effect of minor additions of substitutional elements such as Al, Cu, Mn, and Fe on the room-temperature (RT) and elevated-temperature hardness of a single bcc phase Hf20Nb31Ta31Ti18 refractory high entropy alloy is studied here. Interestingly, 2.5 at. % nominal addition of Fe hardened the base Hf20Nb31Ta31Ti18 alloy in the temperature range from RT to 800 °C, while the same nominal content of addition of Al, Cu, and Mn softened the base alloy from RT to 1000 °C. Regardless of solid solution hardening or solid solution softening, the hardness variation with temperature essentially showed the same three-stage pattern for all studied alloys here: a temperature-dependent decrease in hardness below 300 °C/400 °C, followed by a temperature-independent hardness plateau between 300/400 and 800 °C, and finally a temperature-dependent decrease in hardness at temperatures higher than 800 °C. The mechanism for solid solution hardening or softening in bcc-structured refractory high entropy alloys is discussed, together with their temperature dependence.