The definition of recognition of farming behavior is by embedding deep learning algorithms into industrial communication devices such as cameras to capture the farming actions of agricultural workers, and analyze their actions in the videos. Aiming at the problem, the lack of information about the agricultural workers' labor process in agricultural production, this paper proposes to realize the automatic classification of farming behavior by embedding deep learning algorithms in the cameras. In algorithm, as a result of insufficient calculation speed, timing duration and resolution of existing behavior recognition models, an efficient and lightweight 3D attention mechanism named embedded position coordinate information (EPCI attention) suitable for videos is proposed. In the experiment, EPCI attention is connected with ConvLSTM to form an end-to-end deep learning model EPCI-LSTM. The experimental comparison demonstrates EPCI-LSTM achieves clear improvement on FBD (Farming Behavior Dataset), a dataset containing 905 short videos involving 4 typical farming behaviors: spraying pesticides, hoeing the ground, weeding, and planting seedlings, against P3DConvLSTM and ConvLSTM by 2.58%, 6.93% in F1 score respectively. By connecting EPCI attention, the ability of ConvLSTM is significantly ameliorated with accuracy of 0.9441 and recall of 0.9485. EPCI-LSTM makes the absolute improvement over ConvLSTM by 5.85%, and 7.99% in precision and recall, and the training time is greatly reduced by 26.78%. It proves that EPCI-LSTM has advantages in visual recognition compared with ConvLSTM that is individually defined or optimized for recognition or generation. Therefore, the experimental statistics verifies the success of the EPCI attention structure, and the EPCI-LSTM network can successfully realize the efficient and accurate discrimination of the labor behavior of agricultural workers. It is of great significance to promote the digital and standardized management of farm workers by agricultural enterprises, and the further transformation of traditional agriculture to automated and smart agriculture.