Two volatility forecasting evaluation measures are considered; the squared one-day ahead forecast error and its standardized version. The mean squared forecast error is the widely accepted evaluation function for the realized volatility forecasting accuracy. Additionally, we explore the forecasting accuracy based on the squared distance of the forecast error standardized with its volatility. The statistical properties of the forecast errors point the standardized version as a more appropriate metric for evaluating volatility forecasts. We highlight the importance of standardizing the forecast errors with their volatility. The predictive accuracy of the models is investigated for the FTSE100, DAX30 and CAC40 European stock indices and the exchange rates of Euro to British Pound, US Dollar and Japanese Yen. Additionally, a trading strategy defined by the standardized forecast errors provides higher returns compared to the strategy based on the simple forecast errors. The exploration of forecast errors is paving the way for rethinking the evaluation of ultra-high frequency realized volatility models.