Autonomous Agricultural Machines Research Articles

Heuristic cooperative coverage path planning for multiple autonomous agricultural field machines performing sequentially dependent tasks of different working widths and turn characteristics

Coverage path planning is a central task in agricultural field operations such as tillage, planting, cultivation, and harvesting. In future visions, manual operation will be replaced by fleets of autonomous agricultural vehicles that perform the tasks autonomously. A step towards this transition is to enable simultaneous and safe cooperation of autonomous vehicles on the field. In this article a novel approach is presented for coverage path planning (CPP) for two autonomous tractors that perform sequentially dependent tasks simultaneously on the same area. The approach is based on the idea of computing the coverage solutions for each task by dividing them into short paths that consist of a swath and a turn. The approach ensures collision avoidance by examining that the simultaneous short paths, operated by different tractors, do not collide geometrically, and then schedules them to be operated simultaneously in real-time. The approach was demonstrated successfully in a real-world test environment with two autonomous tractors. The tractor that performed the first task was equipped with a disc cultivator and the second tractor was equipped with a seed drill. A test area of 0.8 ha was used for the demonstration drive, during which the tractors drove 22 swaths simultaneously. Both tractors completed their respective tasks.

Headland Identification and Ranging Method for Autonomous Agricultural Machines

Headland boundary identification and ranging are the key supporting technologies for the automatic driving of intelligent agricultural machinery, and they are also the basis for controlling operational behaviors such as autonomous turning and machine lifting. The complex, unstructured environments of farmland headlands render traditional image feature extraction methods less accurate and adaptable. This study utilizes deep learning and binocular vision technologies to develop a headland boundary identification and ranging system built upon the existing automatic guided tractor test platform. A headland image annotation dataset was constructed, and the MobileNetV3 network, notable for its compact model structure, was employed to achieve binary classification recognition of farmland and headland images. An improved MV3-DeeplabV3+ image segmentation network model, leveraging an attention mechanism, was constructed, achieving a high mean intersection over union (MIoU) value of 92.08% and enabling fast and accurate detection of headland boundaries. Following the detection of headland boundaries, binocular stereo vision technology was employed to measure the boundary distances. Field experiment results indicate that the system’s average relative errors of distance in ranging at distances of 25 m, 20 m, and 15 m are 6.72%, 4.80%, and 4.35%, respectively. This system is capable of meeting the real-time detection requirements for headland boundaries.

Identification of Advantages and Limitations of Current Risk Assessment and Hazard Analysis Methods when Applied on Autonomous Agricultural Machineries

Highlights The three main types of risk assessment and hazard analysis techniques applied on autonomous agricultural machines are: (1) Informal Group Analysis; (2) Hazard Analysis and Risk Assessment (HARA); and (3) Failure Mode and Effects Analysis (FMEA). Replicability is the main advantage of FMEA and HARA, while cost effectiveness is the main advantage of Informal Group Analysis. Subjectivity and the requirement for prior knowledge (data) are the main weaknesses of FMEA, HARA, and Informal Group Analysis when applied to novel and revolutionary autonomous agricultural machines. Abstract. In the last ten years, the development of automated agricultural machinery has seen noteworthy advancements. Nevertheless, the successful commercialization of these technologies depends critically on their ability to operate safely. This study evaluated the advantages and limitations of current risk assessment and hazard analysis methods currently used to ensure the safety of autonomous agricultural machines. An online survey containing 18 questions was distributed to 711 participants identified as potential individuals who are currently working or have worked on autonomous agricultural machines to determine the type and frequency of risk assessment and hazard analysis methods applied on autonomous agricultural machines, examine the advantages and limitations of each method, and investigate the perceived effectiveness of each method. Frequency analysis was used to determine the most and least utilized risk assessment and hazard analysis methods. The advantages and limitations of each risk assessment and hazard analysis approach were compared. Descriptive statistics (counts, means, medians, percent) and frequency analysis of the variables were used. The three main types of risk assessment and hazard analysis techniques applied to autonomous agricultural machines. The methods are (a) Informal Group Analysis (e.g., Brainstorming), (b) Hazard Analysis and Risk Assessment (HARA), and (c) Failure Mode and Effects Analysis (FMEA). Replicability is perceived as the main advantage of FMEA and HARA, while cost-effectiveness is the main advantage of Informal Group Analysis. The need to have pre-existing data of the autonomous agricultural machine at hand to be able to perform risk assessment and subjectivity are the main limitations of FMEA, HARA, and Informal Group Analysis dealing with novel and revolutionary autonomous agricultural machines. Industry experts do not believe that the risk assessment and hazard analysis procedures now used are reliable and efficient enough to guarantee the safety of autonomous agricultural tractors. This study reveals important information about the current state of risk assessment and hazard analysis methods in the context of autonomous agricultural machinery. This knowledge can inform future research, policy development, and industry practices to ensure the safety of autonomous agricultural machines. Keywords: Agricultural machine, Autonomous, FMEA, HARA, Hazard analysis, Informal Group Analysis, Risk assessment, Safety, Survey.

Safety Risk Assessment of an Autonomous Agricultural Machine

Highlights The safety guidelines outlined in ISO 18497 are not sufficient to guarantee the safe operation of autonomous agricultural machines. Since the risk assessment techniques specified in ISO 12100:2012 require historical failure data of the machine at hand, they cannot be used to effectively mitigate the risk associated with autonomous agricultural machines where such data are not readily available. Analysis from the perspective of ergonomics can potentially increase the safety of autonomous agricultural machines. Abstract. The goal of this study was to analyze the safety implications of an autonomous agricultural machine (TerraPreta) using the standards ISO 18497 (ISO, 2018) and ISO 12100:2012 (ISO, 2012), as well as to investigate the ergonomics associated with the use of the autonomous agricultural machine. First, three engineers involved in the robot's manufacturing process were asked to evaluate the robot's functionalities compliance with the applicable safety standards and protective measures outlined in standard ISO 18497 (ISO, 2018). Second, while the robot was planting cover crop seeds, an attempt was made to identify and evaluate every risk connected to the robot using the risk assessment techniques outlined in ISO 12100:2012 (ISO, 2012). (1) Half (50%) of the functionalities of the autonomous agricultural machine complied with the safety requirements and protective measures described within the standard ISO 18497 (ISO, 2018). (2) The heavy reliance on past incident data of the risk assessment procedure described within the standard ISO 12100:2012 (ISO, 2012) makes it ineffective for new and revolutionary technologies such as autonomous agricultural machines where such data are not available. (3) Lifting a bag to fill the robot hopper with seeds was found to be a moderately hazardous activity associated with human-robot interaction. Multiple tentative solutions were provided to avoid this moderately hazardous activity. Keywords: Autonomous agricultural machine, Ergonomics, Machine safety, NIOSH lifting equation, Risk assessment, Standards.

Discussion on the Current Situation and Future Development of Unmanned Smart Farmland

At present, unmanned smart farmland in China is in the early stage of demonstration and promotion, and technological research mainly focuses on single machine intelligent control technologies such as autonomous driving and precision operation. However, related technologies such as multi machine collaboration, host machine and machine operation collaboration, as well as collaboration between agricultural production scenarios and autonomous agricultural machine operation, are insufficient. Building unmanned smart farmland is an important way to alleviate the shortage of rural labor and promote the modernization of agriculture and rural areas. It provides precise planting, visual management, and intelligent decision-making for agricultural production, and also allows agriculture and rural areas to share the achievements of technological development in the era.

Modeling and parameter identification for agricultural machines

AbstractA series of technical and social aspects have led the pressure to increase energy efficiency and reduce the level of environmental impact in agricultural production. Crop producers and equipment manufacturers have identified computerized assistance in operation management as an indispensable tool to optimize agricultural processes. A central place is occupied in this regard by the so called digital twins, computer models of a specific instance of an object or system, which are also capable of exchanging data with the physical counterpart and updating their state accordingly. The core of a digital twin is given by the digital mock‐up, or offline digital twin, that is the computer model of the system's physics. The goal of this work is to present the development of an offline digital twin for a novel gantry‐tractor type autonomous agricultural machine. The peculiarity of this system resides in its until now scarcely adopted mechanical design, and its ability to perform a range of processes by attaching different modules to its frame. After a brief introduction into the characteristics of a gantry‐type tractor and its role in the context of precision autonomous agriculture and controlled traffic farming, different components of the machine and dynamic phenomena are separately investigated and mathematically modeled. Particular focus is placed thereby on the study of the vehicle's longitudinal dynamics. Based on the software implementation of the mathematical models, the unknown parameters of the system model are estimated by means of nonlinear optimization techniques, using sensor data collected during field operation as the measurement reference. The accuracy of different estimation methods is then evaluated against experimental samples collected by sensors.

Lidar-Based 3D Obstacle Detection Using Focal Voxel R-CNN for Farmland Environment

With advances in precision agriculture, autonomous agricultural machines can reduce human labor, optimize workflow, and increase productivity. Accurate and reliable obstacle-detection and avoidance systems are essential for ensuring the safety of automated agricultural machines. Existing LiDAR-based obstacle detection methods for the farmland environment process the point clouds via manually designed features, which is time-consuming, labor-intensive, and weak in terms of generalization. In contrast, deep learning has a powerful ability to learn features autonomously. In this study, we attempted to apply deep learning in LiDAR-based 3D obstacle detection for the farmland environment. In terms of perception hardware, we established a data acquisition platform including LiDAR, a camera, and a GNSS/INS on the agricultural machine. In terms of perception method, considering the different agricultural conditions, we used our datasets to train an effective 3D obstacle detector, known as Focal Voxel R-CNN. We used focal sparse convolution to replace the original 3D sparse convolution because of its adaptable ability to extract effective features from sparse point cloud data. Specifically, a branch of submanifold sparse convolution was added to the upstream of the backbone convolution network; this adds weight to the foreground point and retains more valuable information. In comparison with Voxel R-CNN, the proposed Focal Voxel R-CNN significantly improves the detection performance for small objects, and the AP in the pedestrian class increased from 89.04% to 92.89%. The results show that our model obtains an mAP of 91.43%, which is 3.36% higher than the base model. The detection speed is 28.57 FPS, which is 4.18 FPS faster than the base model. The experiments show the effectiveness of our model, which can provide a more reliable obstacle detection model for autonomous agricultural machines.

Risk Assessment Methods for Autonomous Agricultural Machines: A Review of Current Practices and Future Needs

Highlights Risk assessment for highly automated and autonomous agricultural machines must consider risks beyond operator risk. Engineering standards are a starting point for autonomous equipment risk assessment but are not yet adequate. Engineers designing highly automated equipment now assess risk holistically but need more tools and support. Education in accredited engineering programs and professional development should include risk assessment. Abstract. Technology continues to advance in agricultural machines and includes the development of highly automated, robotic, autonomous, and other types of machines used in fields, farmsteads, buildings, and other farm production locations. New engineering design and safety-related standards have been developed in the past half-decade, but safety remains a concern of key stakeholders and is a barrier that could influence widespread adoption. A survey of practicing engineers and researchers involved with highly automated and autonomous agricultural machine design will be presented that shows the methods for risk assessment and control currently in use including different frameworks for hazard and failure identification, prediction, and quantification. The use of engineering design standards (ASABE, ISO, and others) among practitioners is discussed including some important needs that go beyond obstacle detection and injury prevention for operators. These include safety and risk issues connected to animals, property, civic infrastructure, downtime, cyber, and environmental risk. Commonly used risk assessment methods such as the related failure modes and effects analysis (FMEA) or hazard analysis and risk assessment (HARA) are a useful starting point but are based on historical data and experience that can be used to estimate the probability and severity levels of undesirable failures or incidents such as injuries. These data do not yet exist as compared to risk assessment data that can be used to assess incident occurrence probability, failure, detectability, or controllability in more traditional machines. Suggestions are presented for further development of standards and practice recommendations including software needs and operational data that might be used by autonomous machines that is informed by what we do know about past farm incidents that could include accidents, injuries, and other unexpected failures. Keywords: Automation, Autonomous agricultural machinery, Engineering design standards, Farm equipment, Risk assessment, Robotics, Safety.

3D point cloud density-based segmentation for vine rows detection and localisation

The adoption of new sensors for crop monitoring is leading to the acquisition of large amounts of data, which usually are not directly usable for agricultural applications. The 3D point cloud maps of fields and parcels, generated from remotely sensed data, are examples of such big data, which require the development of specific algorithms for their processing and interpretation, with the final aim to obtain valuable information about crop status.This manuscript presents an innovative 3D point cloud processing algorithm for vine row detection and localisation within vineyard maps, based on the detection of key points and a density-based clustering approach. Vine row localisation is a crucial phase in the interpretation of the complex and huge 3D point clouds of agricultural environments, which makes it possible to move the focus from a macro level (parcel and plot scale) to a micro level (plants, fruits and branches). The algorithm outputs fully describe the spatial location of each vine row within the whole 3D model of the agricultural environment by a set of key points and an interpolating curve. The algorithm is specifically conceived to be robust and: (i) independent of the adopted airborne sensor used to acquire the in-field data (not requiring a model with colour or spectral information); (ii) able to manage vineyards with any vine row layout or orientation (such as curvilinear) and (iii) not hindered by the occurrence of missing plants. The experimental results, obtained by processing the models of seven case study parcels, proved the algorithm’s reliability and accuracy: the automatic vine row detection was found to be 100% in accordance with the manual one; and the obtained localisation indices showed an average error of 12 cm and standard deviation of 10 cm, which is fully compatible with the considered agricultural applications. In addition, the algorithm outputs can be profitably exploited for enhanced path planning of autonomous agricultural machines adopted for in-field operations.

Optimal vision-based guidance row locating for autonomous agricultural machines

Optimal vision-based guidance row locating for autonomous agricultural machines

Remote supervision of autonomous agricultural machines: Concepts and feasibility

The scientific literature provides a description of various models depicting autonomous agricultural machines working to complete typical field operations. Many of the models involve some form of automation interface that is used by the machine owner to supervise the operation of the machine from a remote location. The objective of this study was to interview experts in the design of autonomous agricultural machines (university researchers, entrepreneurs, and leaders in the agricultural machinery sector) to ascertain their opinions about future autonomous agricultural machines, particularly related to how such machines will be supervised by the machine’s owner. Of the four remote supervision concepts described by participants (within the field, close to the field, from the farm office, and outside the farm site), the close-to-the-field remote supervision concept was determined to be the most viable concept. Designers were divided on the idea of providing real-time live video on the automation interface, however, most of them believed that having live video would reassure the farmer that everything was going well. Desktop computer, tablet and phone were the main devices recommended as tools for remote supervision (i.e., the hardware on which to display the automation interface), with tablet perhaps being the preferred alternative.

Evaluation of Warning Methods for Remotely Supervised Autonomous Agricultural Machines.

Humans who supervise autonomous agricultural machines require some type of warning to perceive abnormal conditions in the machine or its environment. Visual and tactile warnings were the most suitable warning methods for in-field and close-to-field remote supervision. This study will help improve the performance of remote supervisors and minimize unexpected incidents or liabilities during operation of autonomous machines. As agricultural machinery moves toward full autonomy, human supervisors will need to monitor the autonomous machines during operation and minimize system failures or malfunctions. However, to intervene in an emergency, the supervisor must first recognize the emergency in a timely manner. Existing warning devices rely on the human visual, auditory, and tactile senses. However, these warning methods vary in their ability to attract attention. Hence, it is important to determine which warning method is best suited to draw the attention of a remote supervisor of an autonomous machine in an emergency. To achieve this objective, participants were recruited and asked to interact with a simulation of an autonomous sprayer. Seven warning methods (presented alone or in combinations of visual, auditory, and tactile sensory cues) and four remote supervision scenarios (in-field, close-to-field, farm office, outside the farmland) were considered in this study. The findings revealed that a combination of tactile and visual methods was most suitable for in-field and close-to-field remote supervision, in comparison to the other warning methods. However, there was insufficient evidence to recommend the best warning methods for supervisors at the farm office or outside the farmland. This study will help improve the performance of remote supervisors and minimize unexpected incidents during field operations with autonomous agricultural machines.

Design and Evaluation of a User Interface for an Autonomous Agricultural Sprayer

This paper describes the design and evaluation of a user interface for a remotely supervised autonomous agricultural sprayer. The interface was designed to help the remote supervisor to instruct the autonomous sprayer to commence operation, monitor the status of the sprayer and its operation in the field, and intervene when needed (i.e., to stop or shut down). Design principles and guidelines were carefully selected to help develop a human-centered automation interface. Evaluation of the interface using a combination of heuristic, cognitive walkthrough, and user testing techniques revealed several strengths of the design as well as areas that needed further improvement. Overall, this paper provides guidelines that will assist other researchers to develop an ergonomic user interface for a fully autonomous agricultural machine.

Measurement of latency during real-time wireless video transmission for remote supervision of autonomous agricultural machines

As the prominence of autonomous vehicles continues to rise within agriculture, remote surveillance of the equipment is likely to become a key aspect of operation. Transmission latency during the relay of video from vehicle to viewer is not well explored and is an important part of communications which should be assessed. A riding mower was equipped with a Raspberry Pi using GStreamer and an open-source latency measurement library to assemble a real-time streaming system to evaluate transmission latency in different environments using cellular and radio transmission. In most locations, measured latencies were under 200–300 ms. In areas where cellular connection quality was adequate, cellular latency and variance were reduced compared to that of radio transmission for higher qualities of video. In areas of poor cellular network coverage, cellular transmission latency increased while radio transmission latencies remained constant. Overall, latency tends to increase with video quality to a statistically significant degree. It is recommended that real-time video can be transmitted over short distances for edge-of-field surveillance of autonomous agricultural machines using existing cellular networks in areas where adequate cellular signal strength is available. If adequate cellular signal strength is not available in a specific field location, it is recommended that video should be transmitted using radio transmission.

Concept and Realization of a Novel Test Method Using a Dynamic Test Stand for Detecting Persons by Sensor Systems on Autonomous Agricultural Robotics.

As an essential part for the development of autonomous agricultural robotics, the functional safety of autonomous agricultural machines is largely based on the functionality and robustness of non-contact sensor systems for human protection. This article presents a new step in the development of autonomous agricultural machine with a concept and the realization of a novel test method using a dynamic test stand on an agricultural farm in outdoor areas. With this test method, commercially available sensor systems are tested in a long-term test around the clock for 365 days a year and 24 h a day on a dynamic test stand in continuous outdoor use. A test over a longer period of time is needed to test as much as possible all occurring environmental conditions. This test is determined by the naturally occurring environmental conditions. This fact corresponds to the reality of unpredictable/determinable environmental conditions in the field and makes the test method and test stand so unique. The focus of the developed test methods is on creating own real environment detection areas (REDAs) for each sensor system, which can be used to compare and evaluate the autonomous human detection of the sensor systems for the functional safety of autonomous agricultural robots with a humanoid test target. Sensor manufacturers from industry and the automotive sector provide their sensor systems to have their sensors tested in cooperation with the TÜV.

Local Motion Planner for Autonomous Navigation in Vineyards with a RGB-D Camera-Based Algorithm and Deep Learning Synergy

With the advent of agriculture 3.0 and 4.0, in view of efficient and sustainable use of resources, researchers are increasingly focusing on the development of innovative smart farming and precision agriculture technologies by introducing automation and robotics into the agricultural processes. Autonomous agricultural field machines have been gaining significant attention from farmers and industries to reduce costs, human workload, and required resources. Nevertheless, achieving sufficient autonomous navigation capabilities requires the simultaneous cooperation of different processes; localization, mapping, and path planning are just some of the steps that aim at providing to the machine the right set of skills to operate in semi-structured and unstructured environments. In this context, this study presents a low-cost, power-efficient local motion planner for autonomous navigation in vineyards based only on an RGB-D camera, low range hardware, and a dual layer control algorithm. The first algorithm makes use of the disparity map and its depth representation to generate a proportional control for the robotic platform. Concurrently, a second back-up algorithm, based on representations learning and resilient to illumination variations, can take control of the machine in case of a momentaneous failure of the first block generating high-level motion primitives. Moreover, due to the double nature of the system, after initial training of the deep learning model with an initial dataset, the strict synergy between the two algorithms opens the possibility of exploiting new automatically labeled data, coming from the field, to extend the existing model’s knowledge. The machine learning algorithm has been trained and tested, using transfer learning, with acquired images during different field surveys in the North region of Italy and then optimized for on-device inference with model pruning and quantization. Finally, the overall system has been validated with a customized robot platform in the appropriate environment.

Visual Information Requirements for Remotely Supervised Autonomous Agricultural Machines

A study to determine the visual requirements for a remote supervisor of an autonomous sprayer was conducted. Observation of a sprayer operator identified 9 distinct “look zones” that occupied his visual attention, with 39% of his time spent viewing the look zone ahead of the sprayer. While observation of the sprayer operator was being completed, additional GoPro cameras were used to record video of the sprayer in operation from 10 distinct perspectives (some look zones were visible from the operator’s seat, but other look zones were selected to display other regions of the sprayer that might be of interest to a sprayer operator). In a subsequent laboratory study, 29 experienced sprayer operators were recruited to view and comment on video clips selected from the video footage collected during the initial ride-along. Only the two views from the perspective of the operator’s seat were rated highly as providing important information even though participants were able to identify relevant information from all ten of the video clips. Generally, participants used the video clips to obtain information about the boom status, the location and movement of the sprayer within the field, the weather conditions (especially the wind), obstacles to be avoided, crop conditions, and field conditions. Sprayer operators with more than 15 years of experience provided more insightful descriptions of the video clips than their less experienced peers. Designers can influence which features the user will perceive by positioning the camera such that those specific features are prominent in the camera’s field of view. Overall, experienced sprayer operators preferred the concept of presenting visual information on an automation interface using live video rather than presenting that same information using some type of graphical display using icons or symbols.

Augmented Reality for Supervising Multirobot System in Agricultural Field Operation

Agriculture is shifting from farmers manually operating machines to monitoring autonomous machines. Thus, the task of a farmer is fleet management and taking care of safe operation. Situational awareness during the operation is important. Augmented reality (AR) is a powerful tool for visualizing information in real-time. To demonstrate the use of AR in agricultural fleet management, in this paper we present a novel AR system to help the farmer supervise the operation of two autonomous agricultural machines. The paper discusses the requirements for AR application, and we present the architecture of the system and the results of a demonstration carried out in a test field.

Remote supervision of autonomous agricultural sprayers: The farmer’s perspective

The autonomous agricultural machine may soon be a reality. For it to be commercially viable, designers must also consider the design of the interface that will be used to remotely supervise the operation of the autonomous machine. Before the interface can be designed, however, it is necessary to understand the information required by the individual supervising the machine from a remote location Hence, the goal of this study was to determine the expectations of farmers regarding autonomous agricultural machines (specifically, an autonomous sprayer), and to determine the essential information that should be provided on the interface to enable the farmer to supervise the operation of an autonomous sprayer remotely. A questionnaire was developed and distributed to farmers and agriculture students attending western Canadian universities. In a follow-up study, experienced, operators were then asked to rank each parameter that was recommended from the survey using a five-point Likert scale. Findings indicated that sprayer operators are willing to accept an autonomous sprayer. However, to increase the ease of adoption, certain conditions need to be met. Parameters and features that were recommended varied in their level of importance and frequency of usage. These results will help designers minimize the chance of overcrowding the interface with irrelevant information. It will also assist designers with organization of information based on the needs of the farmer and in a way that will increase the situation awareness of the farmer.

Autonomous agricultural machines

Autonomous agricultural machines