Lettuce is one of the most widely planted leafy vegetables in plant factories. The lack of trace elements in nutrient solutions has caused huge losses to the lettuce industry. Non-obvious symptoms of trace element deficiency, the inconsistent size of the characteristic areas, and the difficulty of extraction in different growth stages are three key problems affecting lettuce deficiency symptom identification. In this study, a batch of cream lettuce (lactuca sativa) was planted in the plant factory, and its nutrient elements were artificially controlled. We collected images of the lettuce at different growth stages, including all nutrient elements and three nutrient-deficient groups (potassium deficiency, calcium deficiency, and magnesium deficiency), and performed feature extraction analysis on images of different defects. We used traditional algorithms (k-nearest neighbor, support vector machine, random forest) and lightweight deep-learning models (ShuffleNet, SqueezeNet, andMobileNetV2) for classification, and we compared different feature extraction methods (texture features, color features, scale-invariant feature transform features). The experiment shows that, under the optimal feature extraction method (color), the random-forest recognition results are the best, with an accuracy rate of 97.6%, a precision rate of 97.9%, a recall rate of 97.4%, and an F1 score of 97.6%. The accuracies of all three deep-learning models exceed 99.5%, among which ShuffleNet is the best, with the accuracy, precision, recall, and F1 score above 99.8%. It also uses fewer floating-point operations per second and less time. The proposed method can quickly identify the trace elements lacking in lettuce, and it can provide technical support for the visual recognition of the disease patrol robot in the plant factory.