3D-TOF Camera Research Articles

Technology of Automatic Evaluation of Dairy Herd Fatness

The global recent development trend in dairy farming emphasizes the automation and robotization of milk production. The rapid development rate of dairy farming requires new technologies to increase the economic efficiency and improve production. The research goal was to increase the milk production efficiency by introducing the technology to automatically assess the fatness of a dairy herd in 0.25-point step on a 5-point scale. Experimental data were collected on the 3D ToF camera O3D 303 installed in a walk-through machine on robotic free-stall farms in the period from August 2020 to November 2022. The authors collected data on 182 animals and processed 546 images. All animals were between 450 and 700 kg in weight. Based on the regression analysis, they developed software to find and identify the main five regions of interest: the spinous processes of the lumbar spine and back; the transverse processes of the lumbar spine and the gluteal fossa area; the malar and sciatic tuberosities; the tail base; and the vulva and anus region. The adequacy of the proposed technology was verified by means of a parallel expert survey. The developed technology was tested on 3 farms with a total of 1810 cows and is helpful for the non-contact evaluation of the fatness of a dairy herd within the herd’s life cycle. The developed method can be used to evaluate the tail base area with 100% accuracy. The hungry hole can be determined with a 98.9% probability; the vulva and anus area—with a 95.10% probability. Protruding vertebrae—namely, spinous processes and transverse processes—were evaluated with a 52.20% and 51.10% probability. The system’s overall accuracy was assessed as 93.4%, which was a positive result. Animals in the condition of 2.5 to 3.5 at 5–6 months were considered healthy. The developed system makes it possible to divide the animals into three groups, confirming their physiological status: normal range body condition, exhaustion, and obesity. By means of a correlation dependence equal to R = 0.849 (Pearson method), the authors revealed that animals of the same breed and in the same lactation range have a linear dependence of weight-to-fatness score. They have developed an algorithm for automated assessment of the fatness of animals with further staging of their physiological state. The economic effect of implementing the proposed system has been demonstrated. The effect of increasing the production efficiency of a dairy farm by introducing the technology of automatic evaluation of the fatness of a dairy herd with a 0.25-point step on a 5-point scale had been achieved. The overall accuracy of the system was estimated at 93.4%.

Development of a Modular System for Digitalization of Appraisal Process

It is noted that the existing weighing platforms for weighing dairy cattle demonstrate certain shortcomings during the grading process. It was proposed to determine the fatness of animals by scanning their sacrum with optical systems, which allows monitoring their physiological state and, in case of deterioration, promptly change the diet. (Research purpose) To develop a modular system for the digitalization of appraisal process ensuring the autonomy and independence of modules. (Materials and methods) The emphasis is paid on the necessity of integrating into the whole farm system by using a unified Internet space. The development of a modular system for digitalization of appraisal was divided into 4 modules. A grading algorithm was worked out and compared with the layout diagram of the modules in order to assess the volume of work on the grading digitalization. Four strain gauges were installed in the weighing module. (Results and discussion) It was proved that the module provides high-precision weighing of animals (C3 accuracy), regardless of the external influence factors such as cow movements and the asymmetrical position of the cow inside the system box. The results can be displayed on the display of a single control unit. The camera of the vision module was placed at a height of 2,200 millimeters, which eliminates it being damaged by the animals and provides an unobstructed view for the 3D ToF camera. The transportation dimensions of the system prototype are as follows: length – 2,500 millimeters, width – 1,564, height – 2,118 millimeters. The total mass of the manufactured system, including the original platform, is approximately 620-640 kilograms. (Conclusions) A modular system for digitalization of appraisal work has been successfully developed and created. The weighing module was tested revealing an error of less than 1 percent, when testing the system with a reference mass of 655 kilograms and that of 1,200 kilograms.

USING A CONVOLUTIONAL NEURAL NETWORK TO DETERMINE THE TEAT LIMITS OF A COW’S UDDER

In dairy farming, there is an active use of systems for the accurate recognizing of cow’s teats, which is necessary for robotic milking. Object detection consists both in assessing the exact position of the object (localization of the object) and in determining the category to which it belongs (classification of the object). (Research purpose) The research purpose is developing a method for searching for clusters of points to determine the limits of the teat in three-dimensional maps obtained with a 3D ToF camera. (Materials and methods) An IFM O3D303 camera with a resolution of 352 × 264 pixels was used. Authors have developed a structure for analyzing static and dynamic scenes, the model consists of five blocks of convolutional layers, three fully connected layers of neurons and a classifier layer. The first two blocks contain two layers, the other three blocks contain three convolution layers. (Results and discussion) The software highlights the detected teats in color. The method is considered suitable if at 10 repetitions of the experiment the limits of four teats are recognized and there are no artifacts, the experiment can be considered successful, the permissible error of the experiment is no more than 10 percent. During the search for limits of teats, the algorithm repeatedly searches for points in the bounding cylinder of the refined cluster of points. (Conclusions) A method for searching for clusters of points to identify teats limits in three-dimensional maps processing data from a 3D ToF camera using a neural network with «anchor rectangles». Data from 36 animals and 6000 images were collected for training the neural network, field experiments confirmed the accuracy of the neural network in 90 percent. The proposed method allows determining length X, diameter Y and inclination angle Z of the teat in automatic mode.

РАЗРАБОТКА АЛГОРИТМА ОЦЕНКИ ФИЗИОЛОГИЧЕСКОЙ УПИТАННОСТИ МОЛОЧНЫХ КОРОВ

Cow’s body condition scoring (BCS) based on neural network algorithms is necessary to monitor the health and productivity of the animals. To develop an algorithm for assessing the physiological fatness of dairy cows according to the parameters of the fermur heads, the sacrum and the hunger hollow, the authors studied eighty animals on a farm of the state unitary enterprise “Grigorievskoe”, which were divided into fatness groups (from 1 to 5). The studies were carried out in November 2021 during the morning milking. Data were collected using a 3D ToF camera O3D303. Previously, the installation of a three-dimensional camera on the farm was sim ulated, and an algorithm was developed that takes into account the cow height and the distance from the highest point of the spine to the three-dimensional camera. An algorithm for assessing the physiological fatness of dairy cows (BCS) has been developed to take into account the condition of the fermur heads, the sacrum, the hunger hollow, which determines the highest point of the withers, the proportions between the body length and width, as well as the depth of the hunger hollows and the severity of the caudal ligament. Software has been developed to register the unique number of a cow and determine the BCS, as well as show the dynamics of changes in the animal’s fatness. Data were processed in accordance with the developed algorithm. The image was processed using the regression method. Comparison results of the BCS of cows, obtained according to the developed algorithm, and the experts’ assessment showed that the algorithm error in the fatness range of 2…4 points averaged 10%. When determining the BCS of cows with borderline and limit fatness state (1 and 5 points), the measurement error by the proposed algorithm increased to 25%. Based on the results obtained, the authors recommend pre-setting a neural network for further research; determine the correction factor for fatness points 1 and 5; fi nalize the software, develop customized software and an automatic system for body condition scoring, and conduct tests.

БЕСКОНТАКТНАЯ ОЦЕНКА УПИТАННОСТИ МОЛОЧНЫХ КОРОВ С ИСПОЛЬЗОВАНИЕМ TOF-ТЕХНОЛОГИИ

The paper presents the results of applying the body condition score (BCS) least squares algorithm used to assess the body condition of dairy cows with a 3D ToF camera. The authors propose a method for collecting field data on the body condition of dairy cows in industrial milk production using a 3D ToF camera. The camera was installed at a height of 2200 mm from the floor at an angle of 5° towards the sacrum. Four areas of the body of 34 dairy cows were examined: the ischial tuberosities, the roundness of the maclugs, the sacral ligament, and the caudal ligament. Data were collected during milking. 136 images were processed. Digital data were processed in three types of images: in the RGB-D color spectrum, Point Cloud and binary. The assessment took into account five groups of fatness: 1 - lean; 2 - thin; 3 - well-fed; 4 - very well-fed; 5 - obese dairy cow. The resulting images were analyzed using software developed in the Matlab environment. The results of the algorithm were compared with the expert assessment of four specialists. According to the results of the BCS assessment of the algorithm, the fatness scores of groups 1 and 5 coincided with the opinion of experts with a probability of 73 and 67%, in groups 2, 3, 4 the coincidence was 61, 52 and 55%, respectively. The authors suppose that the inaccuracy in determining the fatness of groups 2, 3 and 4 is associated with their implicit differences. It is concluded that of all the BCS groups, the system more accurately determines the fatness of cows of groups 1 and 5, as well as from three image options (Point cloud 3D, RGB-D, binary) RGB-D most accurately determined the fatness of cows.

Mathematical model of a milking machine manipulator based on stepper motors

The automation of agriculture increases labour productivity and contributes to an increase in the output of agricultural products, an increase in their quality. These processes are closely related to the application of industrial production technology in agriculture, the improvement of planning and management. Also, the improvement of product quality and the growth of labour productivity are associated with the development of automated control systems that contribute to the autonomous operation of entire agricultural complexes. This paper deals with the development of a mathematical model of a milking installation based on stepper motors. The authors of the work recreated the operation of the milking installation, taking into account the operation of the kinematic scheme of the robot and the dynamics of the stepping motors, positioning of its working body near the tops of the teats of the udder of the animal, which are automatically recognized using image analysis from the 3D TOF camera. The uniqueness of this model lies in the fact that various stages of the operation of an autonomous milking installation are concentrated in one software product. This technology makes it possible to study the effect of each stage separately on the static positioning error of the manipulator working body. The model has been tested, an acceptable accuracy of the results of the model operation is achieved. This model can be recommended for use for engineering calculations and the development of control algorithms for the movement of the working body of the manipulator.

Hybrid assembled MEMS scanner array with large aperture for fast scanning LIDAR systems

Abstract This article presents a large aperture micro scanning mirror (MSM) array especially developed for the panoramic 3D-ToF camera Fovea-3D. The Fovea-3D system uses a fiber amplified pulsed laser ToF technique at λ = 1550 nm \lambda =1550\hspace{0.1667em}\text{nm} with 1 MVoxel distance measuring rate. It targets for real time 3D imaging with a panoramic optical field of view (FOV) of 360 ° × 60 ° 360^\circ \times 60^\circ (horizontal × vertical) combined with a large distance measurement range of 100 m and a video-like frame rate of 10 Hz. For fast vertical scan axis a MEMS scanner module with large receiver aperture was especially developed. It increases the scanning rate to 3200 Hz which is four times faster in comparison to state-of-the-art fast macroscopic polygon scanning systems used for LIDAR systems. To guarantee at the same time a large reception aperture of D eff = 23 mm {D_{\mathit{eff}}}=23\hspace{0.1667em}\text{mm} , large FOV of 60 ° 60^\circ and high vertical scanning rate of 3200 Hz, a hybrid assembled MSM array was developed consisting of 22 reception mirrors and a separate emitting mirror for laser scanning of the target. For Fovea3D hybrid assembly of frequency selected scanner elements was chosen instead of a monolithic MEMS scanner array to guaranty a high yield of MEMS fabrication. All MSM are driven in parametric resonance to enable a fully synchronized operation of all individual MEMS scanner elements. Therefore, piezo-resistive position sensors are integrated on each MEMS chip for position feedback of driving control. The paper discusses details of the MEMS system integration including the synchronized operation of multiple MEMS scanning elements.

Principle and Robotic Applications of Conical Scanning Method

We have proposed a surrounding environmental recognition method using conical scanning distance measurement for mobile robots. The conical scanning method has a high robustness against ambient light noise and its calculation cost is low. This report describes the principle of this method and the two implementations. The first application is the quadrupedal wheeled robot. The robot can recognize a stair in sight, approach to it and climb. A second application is the indoor-outdoor wheeled mobile robot. The robot can recognize its position and obstacles on floor from information of a laser range finder and a 3D ToF camera. The validity of this method was confirmed by these robotic applications.

In-pixel time-to-digital converter for 3D TOF cameras with time amplifier

In this paper, a novel in-pixel time-to-digital converter (TDC) for 3D time-of-flight cameras is presented. Coarse---fine architecture has been used to guarantee both wide dynamic range and high time resolution. The proposed TDC exploits pulse-train time amplifier (TA) to further improve the time resolution and optimize time error performance. The 11-bit TDC provides 62.5 ps resolution and 126 ns full-scale range. Simulation results indicate the TDC achieves a DNL of under 0.52 LSB and INL less than 0.8 LSB. Both the TA and TDC are designed in GSMC CIS 0.18 μm 1P6M technology.

Non-invasive 3D time-of-flight imaging technique for tumour volume assessment in subcutaneous models

Subcutaneous tumour xenograft volumes are generally measured using callipers. This method is susceptible to inter- and intra-observer variability and systematic inaccuracies. Non-invasive 3D measurement using ultrasound and magnetic resonance imaging (MRI) have been considered, but require immobilization of the animal. An infrared-based 3D time-of-flight (3DToF) camera was used to acquire a depth map of tumour-bearing mice. A semi-automatic algorithm based on parametric surfaces was applied to estimate tumour volume. Four clay mouse models and 18 tumour-bearing mice were assessed using callipers (applying both prolate spheroid and ellipsoid models) and 3DToF methods, and validated using tumour weight. Inter-experimentalist variability could be up to 25% in the calliper method. Experimental results demonstrated good consistency and relatively low error rates for the 3DToF method, in contrast to biased overestimation using callipers. Accuracy is currently limited by camera performance; however, we anticipate the next generation 3DToF cameras will be able to support the development of a practical system. Here, we describe an initial proof of concept for a non-invasive, non-immobilized, morphology-independent, economical and potentially more precise tumour volume assessment technique. This affordable technique should maximize the datapoints per animal, by reducing the numbers required in experiments and reduce their distress.

Non-visual 2D representation of obstacles

Mobility of blind and visually impaired people always suffers from various hindrances while walking indoor and outdoor, due to unaware distributions and properties of obstacles. In our research, we propose to use a 3D TOF camera to detect the surroundings, which not only obtains precise distance and orientation of obstacles, but also analyzes more properties (e.g. size, hanging height) through a 3D segmentation algorithm. Furthermore, a novel touch-sensitive tactile display is employed to present the 3D spatial scene into a 2D perspective. The non-visual representation let visually impaired users sense the distribution of obstacles precisely and explicitly, as well as the surroundings environments. An evaluation plan has been addressed to compare the effectiveness of audio and tactile representation, and investigate the feasibility of extending the while cane.

Neuro-evolutionary mobile robot egomotion estimation with a 3D ToF camera

An innovative neuro-evolutionary approach for mobile robot egomotion estimation with a 3D ToF camera is proposed. The system is composed of two main modules following a preprocessing step. The first module is a Neural Gas network that computes a Vector Quantization of the preprocessed camera 3D point cloud. The second module is an Evolution Strategy that estimates the robot motion parameters by performing a registration process, searching on the space of linear transformations, restricted to the translation and rotation, between the codebooks obtained for successive camera readings. The fitness function is the matching error between the predicted and the observed codebook corresponding to the next camera readings. In this paper, we report results of an implementation of this system tested on data from a real mobile robot, and provide several comparisons between our and other well-known registration algorithms.