This paper presents a fire detection algorithm using color and motion models from video sequences. The proposed method detects change in color and motion of overall regions for detecting fire, and thus, it can be implemented in both fixed and pan/tilt/zoom (PTZ) cameras. The proposed algorithm consists of three parts. The first part exploits color models of flames and smoke. The candidate regions in the video frames are extracted with the hue-saturation-value (HSV) color model. The second part models the motion information of flames and smoke. Optical flow in the fire candidate region is estimated, and the spatial–temporal distribution of optical flow vectors is analyzed. The final part accumulates the probability of fire in successive video frames, which reduces false-positive errors when fire-like color objects appear. Experimental results from 100 fire videos are shown, where various types of smoke and flames appear in indoor and outdoor environments. According to the experiments and the comparison, the proposed fire detection algorithm works well in various situations, and outperforms the conventional algorithms.