Fire accident remains a problem in modern society. This leads great efforts in finding ways to prevent, detect and control it. Conventional fire detection systems are mostly point detectors, which have limitation for early smoke detection, especially in a high-ceiling atrium. A video-based smoke detection system is an interesting alternative approach. It has better area coverage and detecting smoke faster. In this work, a video-based smoke detection system was developed with two main processes, i.e. moving objects segmentation with Gaussian Mixture Models (GMM) and smoke classifications with Mathematical Model of Meaning (MMM). In the MMM model, the interpretation of dangerous smoke is based on the context provided. Then the classification results are compared with conventional smoke detector. The results show that MMM can recognize the dangerous smoke faster than conventional smoke detectors.