A novel algorithm based on Eulerian magnification and empirical mode decomposition (EM-EMD) is proposed to classify emotional stress and physical stress. Different from previous stress recognition algorithms, EM-EMD does not model the relationship between physiological stress parameters and thermal imprints, but establishes the classification model using different thermal signal features under different types and statuses of stress. It first amplifies blood vessel signals in the human forehead. Later, the proposed algorithm performs frequency division processing on the emotional stress signal and the physical stress signal according to time scale characteristics of the data. Finally, it establishes a classification model of emotional and physical stress using the Gaussian mixture model classifier. Experimental results demonstrated that the EM-EMD algorithm could achieve 85% classification accuracy and could provide a practical method model for future industrial applications. It also shows that the classification rate of the proposed algorithm is better than the conventional classification method. As far as we know, the proposed EM-EMD algorithm is a successful classification model of emotional and physical stress through non-contact imaging.