Infrared (IR) small target detection has recently been widely used in civil and military applications. In IR small target detection, IR images probed at long distances are easily disturbed by complex backgrounds, light changes, and other noises, which makes it pretty challenging. Most existing literature focuses on low false alarms and high detection accuracy rather than computational efficiency. We propose a fast and reliable IR small target detection method that deals with spatial and temporal domains based on the Hadamard product for spatial-temporal matrices (HPSTM). In the spatial domain, a weighted tri-layer window is used to convolute the IR image and obtain the gradient matrices. In the temporal domain, three interval frames were used instead of conventional adjacent frames and contrast matrices were extracted. Finally, the two spatial-temporal matrices use the Hadamard product to multiply and a simple threshold to transfer into a binary image for target detection. We also proposed the optimal mask size selection (OMSS) method, which can adjust the optimal tri-layer window size in the spatial domain to obtain the best detection result. The experimental results show that the proposed HPSTM method has a high detection accuracy and fast running time compared with other methods, indicating that the proposed HPSTM method is reliable and suitable for real-time IR small target detection.