Facial expression plays an important role in non-verbal social communication, emotion expression and affective recognition. To make the reorganization of facial expression more effectively, researchers try to recognize facial expression by the recognition of facial action units. In this paper, in order to identify lip AUs, we adopt Gabor wavelet transformation as the feature extraction method and Adaboost-SVM (combination of Adaboost and SVM) as the pattern classifier. Compared with the traditional non-cascaded Adboost algorithms in which the number of weak classifiers actually is fixed beforehand, a process of number optimization is added to ensure the minimum number of the weak classifies combination with the highest recognition rate. To test the effect of the given solution, 150 images from frontal faces for training and 100 images for prediction are collected. Compared with traditional SVM, the proposed system with Adaboost-SVM can not only improve the recognition accuracy from 79.0% to 83.0%, but also speed up AU classification process obviously.