Tracking systems with stringent performance are required to take advantage of solar energy produced through the use of high concentration photovolatics and concentrating solar power technologies. Closed-loop tracking systems use measurements provided by a sun sensor, which picks up direct and diffuse sunlight. High levels of diffuse sunlight produce increased levels of measurement noise at the output of the sensor, which in turn limits the maximal achievable gains in the closed-loop controller and consequently the tracking performance deteriorates. The goal of this paper is to present the advantages of equipping a four-quadrant photodiode sun sensor with infrared and linear polarizing optical filters. Experimental results show that these filters reduce the effect of diffuse radiation and allow increasing the performance of sun tracking systems regardless of the type of controller used in the solar tracker. Tracking experiments are presented under indoor and outdoor conditions employing algorithms currently used in solar trackers, including proportional-integral (PI), and proportional-integral-derivative (PID) controllers, and a recently proposed cascade control law. In all the cases, the best performance is obtained with the sun sensor equipped with an infrared filter. Experiments show that an infrared optical filter reduces the filtered mean squared tracking error up to 75% under artificial light conditions and 90% under direct sunlight.