Keypoint Detection Research Articles

Robust Keypoint Detection and Matching on Fisheye Images by Self-Supervised Learning.

Accurate image feature point detection and matching are essential to computer vision tasks such as panoramic image stitching and 3D reconstruction. However, ordinary feature point approaches cannot be directly applied to fisheye images due to their large distortion, which makes the ordinary camera model unable to adapt. To address such a problem, this paper proposes a self-supervised learning method for feature point detection and matching on fisheye images. This method utilizes a Siamese network to automatically learn the correspondence of feature points across transformed image pairs to avoid high annotation costs. Due to the scarcity of the fisheye image dataset, a two-stage viewpoint transform pipeline is also adopted for image augmentation to increase the data variety. Furthermore, this method adopts both deformable convolution and contrastive learning loss to improve the feature extraction and description of distorted image regions. Compared with traditional feature point detectors and matchers, this method has been demonstrated with superior performance on fisheye images.

Estimating Ground Reaction Forces from Two-Dimensional Pose Data: A Biomechanics-Based Comparison of AlphaPose, BlazePose, and OpenPose.

The adoption of computer vision pose estimation approaches, used to identify keypoint locations which are intended to reflect the necessary anatomical landmarks relied upon by biomechanists for musculoskeletal modelling, has gained increasing traction in recent years. This uptake has been further accelerated by keypoint use as inputs into machine learning models used to estimate biomechanical parameters such as ground reaction forces (GRFs) in the absence of instrumentation required for direct measurement. This study first aimed to investigate the keypoint detection rate of three open-source pose estimation models (AlphaPose, BlazePose, and OpenPose) across varying movements, camera views, and trial lengths. Second, this study aimed to assess the suitability and interchangeability of keypoints detected by each pose estimation model when used as inputs into machine learning models for the estimation of GRFs. The keypoint detection rate of BlazePose was distinctly lower than that of AlphaPose and OpenPose. All pose estimation models achieved a high keypoint detection rate at the centre of an image frame and a lower detection rate in the true sagittal plane camera field of view, compared with slightly anteriorly or posteriorly located quasi-sagittal plane camera views. The three-dimensional ground reaction force, instantaneous loading rate, and peak force for running could be estimated using the keypoints of all three pose estimation models. However, only AlphaPose and OpenPose keypoints could be used interchangeably with a machine learning model trained to estimate GRFs based on AlphaPose keypoints resulting in a high estimation accuracy when OpenPose keypoints were used as inputs and vice versa. The findings of this study highlight the need for further evaluation of computer vision-based pose estimation models for application in biomechanical human modelling, and the limitations of machine learning-based GRF estimation models that rely on 2D keypoints. This is of particular relevance given that machine learning models informing athlete monitoring guidelines are being developed for application related to athlete well-being.

Kidney tracking for live augmented reality in stereoscopic mini-invasive partial nephrectomy

ABSTRACT Mini-invasive partial nephrectomy (MIPN) has many advantages. However, it is challenging for the surgeon to localise the hidden anatomical structures to be spared or resected during surgery. Augmented reality (AR) is a promising localisation assistance approach. Existing AR-MIPN methods augment the endoscopic view with 3D models from preoperative CT scan. However, they do not track the kidney in real-time, which considerably reduces usability, as AR is only temporarily available on isolated images. We propose an approach to achieve continuous live AR-MIPN. It uses classical camera calibration and manual initial registration. Its key novelty is a keypoint-based automatic kidney tracking module, with three main technical contributions. First, it performs stereo tracking-by-detection from stereo keyframes, exploiting left-right consistency to maximise robustness. Second, it only considers keypoints within the parenchyma, as segmented by a specifically trained neural network. Third, it improves keypoint detection and matching by a new process that we call stereo perspective correction (SPC). It uses the stereo depth-map and surface flattening to generate an image warp that cancels the perspective effect, improving the performance of keypoint detection and matching. We carried out experiments on semi-synthetic and real surgical datasets to compare several tracking methods, showing that our method outperforms.

IMAGE ORIENTATION BY EMBEDDING IN A GAN LATENT SPACE

Abstract. Estimation of an image exterior orientation is required in multiple tasks including 3D reconstruction, autonomous driving and navigation, and single-photo 3D reconstruction. The problem can be easily solved if some reference points (keypoints) with known coordinates in the reference frame are detected in the image. While multiple robust keypoint detectors were developed, estimation of an image exterior orientation from a single image remains challenging in many cases. For example, repeating structures in the scene or absence of textures can reduce the performance of keypoint detectors. In this paper, we propose an algorithm for estimation of an image exterior orientation that leverages the latent space of Generative Adversarial Network (GAN). We propose a modification of the StyleGAN2 model that we term ExteriorGAN. Unlike the StyleGAN2 that generates random images from a random noise vector z, we aim training a mapping from a random vector z and a given image exterior orientation p G : (z,p) → A. Our model generates random images for a constant exterior orientation p and random z that have constant geometry but differs in the scene appearance (e.g. different light direction or intensity). We perform embedding of the image A into the latent space w and reconstruct the input noise vector z and exterior orientation parameters w using a stochastic gradient descent. We developed a dedicated dataset with 50k images and corresponding orientation parameters to train and validate our ExteriorGAN model. The results of evaluation demonstrate that our algorithm allows estimation of the exterior orientation of an image with respect to a known 3D scene. The accuracy of the exterior orientation is comparable with modern state-of-the-art methods. The camera pose can be recovered with a mean error of 50 mm for a working space of 5 by 5 meters.

Merging three‐dimensional occupancy grid maps to support multi‐UAVs cooperative navigation systems

AbstractUnmanned aerial vehicles (UAVs) have been widely used in many applications due to, among other features, their versatility, reduced operation cost, and reduced size. These applications increasingly require that features related to autonomous navigation be employed, such as mapping. However, the reduced capacity of resources such as battery life and hardware (memory and processing capacity) can hinder the development of these applications in UAVs. Thus, the collaborative use of multi‐UAVs for mapping can be used as an alternative to solve this problem, creating a cooperative navigation system. This system requires that individual local maps will be merged into a global map in a distributed way. In this context, this work proposes a system for merging three‐dimensional occupancy grid maps for use in UAVs that compose a cooperative navigation system. The proposed solution consists of a keypoint detector, a keypoint descriptor, and filters for keypoints and correspondences. The keypoint properties, such as orientation, are acquired from potential field gradients. Image processing techniques are applied to remove the map noise and better calculate the transformation parameters. The proposed system is validated by a set of simulation experiments performed in six different environments (indoor and outdoor). First, each component of the proposed system is individually evaluated. Then, the complete pairwise map merging system is evaluated. Finally, the appropriate functioning of the proposed system is validated through a visual evaluation.

A simple and effective multi-person pose estimation model for low power embedded system

In recent years, algorithms based on human pose estimation have been applied more and more in low power embedded system. However, the keypoints detection under occlusion is not well solved, resulting in poor effect in practical application on embedded devices. In this paper, we propose a novel Simple and Effective Network (SEN) to deal with the multi-person pose estimation problem of low power embedded system via detecting occlusion keypoints quite well to a certain extent. This model is easy to apply to embedded devices, and has the characteristics of simplicity, strong expansibility and wide application, which are very important in the world of Internet of things. Our model contains three novel modules: Feature Fusion Module (FFM), Channel Enhancement Attention Module (CEAM), and Feature Enhancement Module (FEM). The FFM fuses the shallow and deep feature maps, bringing rich context information to the model. At the same time, it can alleviate the problem of information loss caused by downsampling operations and locate the keypoints more accurately. The FEM and the CEAM act on the deep feature maps of the network, which helps to infer the keypoints of occlusion or invisibility. Related experiments explain that the raised means is effective and achieves the superior performance over two benchmark datasets: the COCO keypoints detection dataset and the MPII Human Pose dataset.

Detecting and grouping keypoints for multi-person pose estimation using instance-aware attention

Bottom-up human pose estimation models detect keypoints and learn associative information between keypoints, usually requiring human predefined offset fields or embeddings for keypoints grouping (clustering). In this paper, we present a brand new method that can entirely solve these problems based on Transformer, making the grouping process free of the human-defined associative signals. Specifically, the self-attention in vision Transformer measures feature similarity between any pair of locations, which provides a metric space to associate keypoints together into corresponding human instances. However, the naive attention patterns formed in Transformer are still not subjectively controlled, so there is no guarantee that the keypoints only attend to the instances to which they belong. To address it we propose a novel approach of supervising self-attention to be instance-aware, simultaneously accomplishing multi-person keypoint detection and clustering. By doing so, we can group the detected keypoints to their corresponding instances, according to the pairwise attention scores.An additional benefit of our method is that the instance segmentation results of any number of people can be directly obtained from the supervised attention matrix, thereby simplifying the pixel assignment pipeline. The qualitative and quantitative results on the COCO shows that, with a very simple architecture design, our method can achieve comparable performance against the CNN-based bottom-up counterparts with fewer parameters, which also demonstrate a promising way to control self-attention mechanism behavior for specific purposes.

Factors of Influence for Transfer Learning Across Diverse Appearance Domains and Task Types.

Transfer learning enables to re-use knowledge learned on a source task to help learning a target task. A simple form of transfer learning is common in current state-of-the-art computer vision models, i.e., pre-training a model for image classification on the ILSVRC dataset, and then fine-tune on any target task. However, previous systematic studies of transfer learning have been limited and the circumstances in which it is expected to work are not fully understood. In this paper we carry out an extensive experimental exploration of transfer learning across vastly different image domains (consumer photos, autonomous driving, aerial imagery, underwater, indoor scenes, synthetic, close-ups) and task types (semantic segmentation, object detection, depth estimation, keypoint detection). Importantly, these are all complex, structured output tasks types relevant to modern computer vision applications. In total we carry out over 2000 transfer learning experiments, including many where the source and target come from different image domains, task types, or both. We systematically analyze these experiments to understand the impact of image domain, task type, and dataset size on transfer learning performance. Our study leads to several insights and concrete recommendations: (1) for most tasks there exists a source which significantly outperforms ILSVRC'12 pre-training; (2) the image domain is the most important factor for achieving positive transfer; (3) the source dataset should include the image domain of the target dataset to achieve best results; (4) at the same time, we observe only small negative effects when the image domain of the source task is much broader than that of the target; (5) transfer across task types can be beneficial, but its success is heavily dependent on both the source and target task types.

One Metric to Measure Them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks

Despite being widely used as a performance measure for visual detection tasks, Average Precision (AP) is limited in (i) reflecting localisation quality, (ii) interpretability and (iii) robustness to the design choices regarding its computation, and its applicability to outputs without confidence scores. Panoptic Quality (PQ), a measure proposed for evaluating panoptic segmentation (Kirillov et al., 2019), does not suffer from these limitations but is limited to panoptic segmentation. In this paper, we propose Localisation Recall Precision (LRP) Error as the average matching error of a visual detector computed based on both its localisation and classification qualities for a given confidence score threshold. LRP Error, initially proposed only for object detection by Oksuz et al. (2018), does not suffer from the aforementioned limitations and is applicable to all visual detection tasks. We also introduce Optimal LRP (oLRP) Error as the minimum LRP Error obtained over confidence scores to evaluate visual detectors and obtain optimal thresholds for deployment. We provide a detailed comparative analysis of LRP Error with AP and PQ, and use nearly 100 state-of-the-art visual detectors from seven visual detection tasks (i.e. object detection, keypoint detection, instance segmentation, panoptic segmentation, visual relationship detection, zero-shot detection and generalised zero-shot detection) using ten datasets to empirically show that LRP Error provides richer and more discriminative information than its counterparts. Code available at: https://github.com/kemaloksuz/LRP-Error.

A Novel Deep Learning-Based Pose Estimation Method for Robotic Grasping of Axisymmetric Bodies in Industrial Stacked Scenarios

A vision-based intelligent robotic grasping system is essential for realizing unmanned operations in industrial manufacturing, and pose estimation plays an import role in this system. In this study, deep learning was used to obtain the 6D pose of an axisymmetric body which was optimal for robotic grasping in industrial stacked scenarios. We propose a method to obtain the 6D pose of an axisymmetric body by detecting the pre-defined keypoints on the side surface. To realize this method and solve other challenges in industrial stacked scenarios, we propose a multitask real-time convolutional neural network (CNN), named Key-Yolact, which involves object detection, instance segmentation, and multiobject 2D keypoint detection. A small CNN as a decision-making subsystem was designed to score multiple predictions of Key-Yolact, and the body with the highest score is considered the best for grasping. Experiments on a self-built stacked dataset showed that Key-Yolact has a practical tradeoff between inference speed and precision. The inference speed of Key-Yolact is higher by 10 FPS, whereas its precision is decreased by only 7% when compared with the classical multitask Keypoint R-CNN. Robotic grasping experiments showed that the proposed design is effective and can be directly applied to industrial scenarios.

PosePipe: Open-Source Human Pose Estimation Pipeline for Rehabilitation Research

<h3>Research Objectives</h3> While computer vision algorithms for human pose estimation (HPE) are advancing rapidly, both in terms of accuracy and the types of understanding they enable, using these cutting edge approaches for clinical research presents many challenges including the range of algorithms, algorithmic interdependencies and data management. In order to advance the use of HPE in clinical research, we have developed an open-source pipeline based that reduces these challenges. <h3>Design</h3> Retrospective observation study on the use of HPE pipeline on videos of people involved in rehabilitation. <h3>Setting</h3> Videos acquired in both inpatient and outpatient contexts. <h3>Participants</h3> Convenience sample of videos including from a clinical gait analysis laboratory and newly collected data including gait from people with a range of diagnoses. <h3>Interventions</h3> None. <h3>Main Outcome Measures</h3> Ability to efficiently analyze large volumes of videos acquired in clinic and qualitative review of the outputs. <h3>Results</h3> We have used the pipeline to analyze tens of thousands of videos acquired in clinical settings. If includes wrappers for numerous specific implementations of classes of algorithms, making exploratory analysis and comparison feasible. Among the classes of algorithms are tracking algorithms, 2D keypoint detection, 3D lifting layers, and body model parameter estimation. The movement of specific subjects of interest is isolated using tracking algorithms, which perform well even with other visible people. However, it occasionally swaps identities between patients and therapists and detection failures are more common for people using assistive devices. Identifying 2D keypoints is fairly reliable, but struggles with some prosthetic limbs. Lifting methods perform well when lifting these keypoints into 3D joint locations. Approaches that estimate the parameters of a body mesh are promising, but often brittle and produce implausible estimates. A custom neural network allows accurately detecting temporal gait parameters of rehabilitation participants. <h3>Conclusions</h3> The performance of HPE is advancing rapidly, but more tools will be developed to widely disseminate this to the clinical and research communities. Additionally, these algorithms need validation on clinical populations, who can be different than the population the algorithms are trained upon. <h3>Author(s) Disclosures</h3> None

A framework for robotic arm pose estimation and movement prediction based on deep and extreme learning models

Human-robot collaboration has gained a notable prominence in Industry 4.0, as the use of collaborative robots increases efficiency and productivity in the automation process. However, it is necessary to consider the use of mechanisms that increase security in these environments, as the literature reports that risk situations may exist in the context of human-robot collaboration. One of the strategies that can be adopted is the visual recognition of the collaboration environment using machine learning techniques, which can automatically identify what is happening in the scene and what may happen in the future. In this work, we are proposing a new framework that is capable of detecting robotic arm keypoints commonly used in Industry 4.0. In addition to detecting, the proposed framework is able to predict the future movement of these robotic arms, thus providing relevant information that can be considered in the recognition of the human-robot collaboration scenario. The proposed framework has two main modules. The first one contains a convolutional neural network based on self-calibrated convolutions enabling better discriminative feature extraction and the support of extreme learning machine neural networks with different kernels for predicting robotic arm keypoints. The second module is composed of deep recurrent learning models, such as long short-term memory and gated recurrent unit. These models are able to predict future robotic arm keypoints. All experiments were evaluated using the mean squared error metric. Results show that the proposed framework is capable of detecting and predicting with low error, contributing to the mitigation of risks in human-robot collaboration. In addition, it was possible to verify that the use of convolutional neural networks in conjunction with extreme learning machines can offer a lower detection error in a regression task (e.g., keypoint detection), something that, as far as the authors are aware of, is not yet known, nor had been evaluated previously in the literature.

Research on conflict detection model for taxi‐in process on the apron based on aircraft wingtip keypoint detection

AbstractApron safety is critical to aviation operations and aprons are prone to wingtip scraping accidents. However, few studies have been conducted on wingtip conflicts. In this paper, a class of aircraft conflict detection models based on wingtip keypoint detection is devised to solve the problem of wingtip scraping that may occur during the aircraft taxi‐in process on the apron. Firstly, a concise camera calibration technique is designed for the conversion between the pixel coordinate and the airport actual coordinate, which allows for the acquisition of high accuracy activity posture within the apron. Secondly, the static and dynamic safety protection zone delineation methods for wingtip conflict detection are given, taking the apron standard operating procedures and aircraft kinematic model into consideration. Then, the corresponding conflict detection models and algorithms are designed for the possible scraping between the vehicle and aircraft wingtips and between the aircraft and aircraft wingtips during the aircraft taxi‐in process on the apron. Finally, HRNet is selected to detect the keypoint of the wingtip by comparison, and the corresponding models and algorithm validation are carried out based on the apron physical sandbox. The verification results indicate that the models have strong accuracy and real‐time performance.

2D Human pose estimation: a survey

Human pose estimation aims at localizing human anatomical keypoints or body parts in the input data (e.g., images, videos, or signals). It forms a crucial component in enabling machines to have an insightful understanding of the behaviors of humans, and has become a salient problem in computer vision and related fields. Deep learning techniques allow learning feature representations directly from the data, significantly pushing the performance boundary of human pose estimation. In this paper, we reap the recent achievements of 2D human pose estimation methods and present a comprehensive survey. Briefly, existing approaches put their efforts in three directions, namely network architecture design, network training refinement, and post processing. Network architecture design looks at the architecture of human pose estimation models, extracting more robust features for keypoint recognition and localization. Network training refinement tap into the training of neural networks and aims to improve the representational ability of models. Post processing further incorporates model-agnostic polishing strategies to improve the performance of keypoint detection. More than 200 research contributions are involved in this survey, covering methodological frameworks, common benchmark datasets, evaluation metrics, and performance comparisons. We seek to provide researchers with a more comprehensive and systematic review on human pose estimation, allowing them to acquire a grand panorama and better identify future directions.

An Automatic Recognition Method for Fish Species and Length Using an Underwater Stereo Vision System

Developing new methods to detect biomass information on freshwater fish in farm conditions enables the creation of decision bases for precision feeding. In this study, an approach based on Keypoints R-CNN is presented to identify species and measure length automatically using an underwater stereo vision system. To enhance the model’s robustness, stochastic enhancement is performed on image datasets. For further promotion of the features extraction capability of the backbone network, an attention module is integrated into the ResNeXt50 network. Concurrently, the feature pyramid network (FPN) is replaced by an improved path aggregation network (I-PANet) to achieve a greater fusion of effective feature maps. Compared to the original model, the mAP of the improved one in object and key point detection tasks increases by 4.55% and 2.38%, respectively, with a small increase in the number of model parameters. In addition, a new algorithm is introduced for matching the detection results of neural networks. On the foundation of the above contents, coordinates of head and tail points in stereo images as well as fish species can be obtained rapidly and accurately. A 3D reconstruction of the fish head and tail points is performed utilizing the calibration parameters and projection matrix of the stereo camera. The estimated length of the fish is acquired by calculating the Euclidean distance between two points. Finally, the precision of the proposed approach proved to be acceptable for five kinds of common freshwater fish. The accuracy of species identification exceeds 94%, and the relative errors of length measurement are less than 10%. In summary, this method can be utilized to help aquaculture farmers efficiently collect real-time information about fish length.

Target-free 3D tiny structural vibration measurement based on deep learning and motion magnification

• A novel target-free tiny vibration displacement measurement approach is proposed. • It is based on deep learning and motion magnification techniques. • Binocular vision system is used for measuring 3D vibration measurement. • Deep learning techniques are used for key-points detection, matching and tracking. • Experimental and in-field studies are conducted to validate the accuracy. In the field of structural health monitoring (SHM), computer vision based methods have been usually developed for 2-dimensional (2D) vibration displacement measurement and crack identification of civil engineering structures. However, the accurate measurement of tiny 3-dimensional (3D) vibrations for real civil engineering structures remains a very difficult task. To overcome this challenge, this paper proposes a target-free full-field 3D tiny vibration measurement approach for civil engineering structures by using a binocular vision system. The proposed approach is based on deep learning and motion magnification. A phase-based video motion magnification algorithm is employed to achieve a high measurement accuracy of tiny vibrations at the submillimeter level. The advanced key point detection, matching and tracking algorithms via deep learning techniques are employed to achieve target-free tiny vibration displacement measurement. The accuracy and performance of the proposed approach are evaluated through experimental tests on a steel cantilever beam in the laboratory. In-field experimental tests are conducted on a pedestrian bridge on a university campus to investigate the accuracy of the proposed approach in practical applications. The measured 3D tiny vibration displacement from the proposed approach is compared with those measured by laser displacement sensors, and the derived acceleration responses are compared with those measured from the installed accelerometers on the testing structures. The results demonstrate that the 3D tiny vibration measurements are obtained accurately by the proposed approach.

Active 3-D Thermography Based on Feature-Free Registration of Thermogram Sequence and 3-D Shape Via a Single Thermal Camera

Active three-dimensional(3-D) thermography combines 3-D shape and active thermography. Thus, it enables the provision of the intuitive thermographic inspection results for composite parts with a complex geometry. However, conventional 3-D thermography acquires thermographic information and 3-D shape through at least two independent sensors and requires complex cross-modal image registration algorithm based on the keypoint detection and matching. In this article, an active 3-D thermography system is proposed using only one thermal camera for moving objects. This system does not require an independent 3-D sensor while the thermal camera acts as the 3-D sensor. Nature behind that is that a mathematical model is proposed to unify the line scanning thermography and laser triangulation in the dynamic scanning process, taking a line laser as both heat excitation for active thermography and spatial coding for 3-D reconstruction. Furthermore, the model enables the feature-free registration of thermogram sequence and 3-D shape, so that the registration is fast and robust without keypoint features. The experiments on standard height specimens, 3-D printing glass fiber composites, and carbon fiber intake tube have shown the error is calibrated within 0.25 mm in the range of 1 to 150 mm and evidenced the capability for subsurface defects detection. The ease and robustness of the proposed active 3-D thermography have a bright future for 3-D measurement, defects detection, and quality control in the production line.

KTN: Knowledge Transfer Network for Learning Multiperson 2D-3D Correspondences

Human densepose estimation, aiming at establishing dense correspondences between 2D pixels of human body and 3D human body template, is a key technique in enabling machines to have an understanding of people in images. It still poses several challenges due to practical scenarios where real-world scenes are complex and only partial annotations are available, leading to incompelete or false estimations. In this work, we present a novel framework to detect the densepose of multiple people in an image. The proposed method, which we refer to Knowledge Transfer Network (KTN), tackles two main problems: 1) how to refine image representation for alleviating incomplete estimations, and 2) how to reduce false estimation caused by the low-quality training labels (i.e., limited annotations and class-imbalance labels). Unlike existing works directly propagating the pyramidal features of regions for densepose estimation, the KTN uses a refinement of pyramidal representation, where it simultaneously maintains feature resolution and suppresses background pixels, and this strategy results in a substantial increase in accuracy. Moreover, the KTN enhances the ability of 3D based body parsing with external knowledges, where it casts 2D based body parsers trained from sufficient annotations as a 3D based body parser through a structural body knowledge graph. In this way, it significantly reduces the adverse effects caused by the low-quality annotations. The effectiveness of KTN is demonstrated by its superior performance to the state-of-the-art methods on DensePose-COCO dataset. Extensive ablation studies and experimental results on representative tasks (e.g., human body segmentation, human part segmentation and keypoints detection) and two popular densepose estimation pipelines (i.e., RCNN and fully-convolutional frameworks), further indicate the generalizability of the proposed method.

One Stage Monocular 3D Object Detection Utilizing Discrete Depth and Orientation Representation

On-road object detection is a critical component in an autonomous driving system. The safety of the vehicle can only be as good as the reliability of the on-road object detection system. Thus, developing a fast and robust object detection algorithm has been the primary goal of many automotive industries and institutes. In recent years, multi-purpose vision-based driver assistance systems have gained popularity with the emergence of a deep neural network. A monocular camera has been developed to locate an object in the image plane and estimate the distance of the said object in the real world or the vehicle plane. In this work, we present a monocular 3D object detection method that utilizes the discrete depth and orientation representation. Our proposed method strives to predict object locations on 3D space utilizing keypoint detection on the object’s center point. To improve the point detection, we employ center regression on the objects segmentation mask, reducing the detection offset significantly. The simplicity of our proposed network architecture and its one-stage approach allows our algorithm to achieve competitive speed compared with prior methods. Our proposed method is able to achieve 26.93% detection score on the Cityscapes 3D object detection dataset, outperforming the preceding monocular method by a margin of 2.8 points.

Tree detection and diameter estimation based on deep learning

AbstractTree perception is an essential building block toward autonomous forestry operations. Current developments generally consider input data from lidar sensors to solve forest navigation, tree detection and diameter estimation problems, whereas cameras paired with deep learning algorithms usually address species classification or forest anomaly detection. In either of these cases, data unavailability and forest diversity restrain deep learning developments for autonomous systems. Therefore, we propose two densely annotated image datasets—43 k synthetic, 100 real—for bounding box, segmentation mask and keypoint detections to assess the potential of vision-based methods. Deep neural network models trained on our datasets achieve a precision of 90.4 % for tree detection, 87.2 % for tree segmentation and centimeter accurate keypoint estimations. We measure our models’ generalizability when testing it on other forest datasets, and their scalability with different dataset sizes and architectural improvements. Overall, the experimental results offer promising avenues toward autonomous tree felling operations and other applied forestry problems. The datasets and pre-trained models in this article are publicly available on GitHub (https://github.com/norlab-ulaval/PercepTreeV1).