HighlightsAn evaluation system for agricultural machinery operation was developed.Quantitatively evaluating agricultural machinery operators can support the precision management for agricultural machinery service organizations.The evaluation system can detect and quantify subtle and transient operating behaviors based on smartphone sensors.The evaluation system can distinguish the relative pros and cons of the operating skills of agricultural machinery operators.Abstract. To support the precision management of agricultural machinery operators, scaled agricultural machinery service organizations require comprehensive evaluations for the operating behaviors of each operator, particularly subtle and transient operating behaviors (e.g., sudden acceleration and unstable operating velocities), which greatly influence the operation quality. In this paper, embedded smartphone sensors were utilized to collect high-frequency motion information for agricultural machinery operation, and then, an Agricultural Machinery Operation Evaluation System (AMOES) was designed to evaluate the operating behaviors of the corresponding operators. In AMOES, four evaluation items (the ratio of useful work, the effect on machinery health, the quality of working operation, and the efficiency of U-turn) were defined specifically for the evaluation of subtle and transient operating behaviors, and they were quantified using motion information. Moreover, a case of a scaled agricultural machinery cooperative in Beijing was performed, and AMOES was used to evaluate the operating behaviors of six operators (two autonomous-driving and four manual-driving operators). The case study indicated that the motion data collected by smartphone sensors could capture subtle and transient operating behaviors and that AMOES could effectively detect and quantify subtle operating behaviors. Keywords: Agricultural machinery, Operating behavior, Operation evaluation, Smartphone sensors.