Ground Truth Annotations Research Articles

Physics-Informed Hyperspectral Remote Sensing Image Synthesis With Deep Conditional Generative Adversarial Networks

High-resolution hyperspectral remote sensing images are of great significance to agricultural, urban, and military applications. However, collecting and labeling hyperspectral images are time-consuming, expensive, and usually heavily rely on domain knowledge. In this article, we propose a new method for generating high-resolution hyperspectral images and subpixel ground-truth annotations from RGB images. Given a single high-resolution RGB image as its conditional input, unlike previous methods that directly predict spectral reflectance and ignores the physics behind it, we consider both imaging mechanism and spectral mixing, introduce a deep generative network that first recovers the spectral abundance for each pixel, and then generate the final spectral data cube with the standard USGS spectral library. In this way, our method not only synthesizes high-quality spectral data existing in the real world but also generates subpixel-level spectral abundance with well-defined spectral reflectance characteristics. We also introduce a spatial discriminative network and a spectral discriminative network to improve the fidelity of the synthetic output from both spatial and spectral perspectives. The whole framework can be trained end-to-end in an adversarial training paradigm. We refer to our method as “Physics-informed Deep Adversarial Spectral Synthesis (PDASS).” On the IEEE <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">grss_dfc_2018</i> dataset, our method achieves an MPSNR of 47.56 on spectral reconstruction accuracy and outperforms other state-of-the-art methods. As latent variables, the generated spectral abundance and the atmospheric absorption coefficients of sunlight also suggest the effectiveness of our method.

A CNN-transformer hybrid approach for decoding visual neural activity into text

Background and ObjectiveMost studies used neural activities evoked by linguistic stimuli such as phrases or sentences to decode the language structure. However, compared to linguistic stimuli, it is more common for the human brain to perceive the outside world through non-linguistic stimuli such as natural images, so only relying on linguistic stimuli cannot fully understand the information perceived by the human brain. To address this, an end-to-end mapping model between visual neural activities evoked by non-linguistic stimuli and visual contents is demanded. MethodsInspired by the success of the Transformer network in neural machine translation and the convolutional neural network (CNN) in computer vision, here a CNN-Transformer hybrid language decoding model is constructed in an end-to-end fashion to decode functional magnetic resonance imaging (fMRI) signals evoked by natural images into descriptive texts about the visual stimuli. Specifically, this model first encodes a semantic sequence extracted by a two-layer 1D CNN from the multi-time visual neural activity into a multi-level abstract representation, then decodes this representation, step by step, into an English sentence. ResultsExperimental results show that the decoded texts are semantically consistent with the corresponding ground truth annotations. Additionally, by varying the encoding and decoding layers and modifying the original positional encoding of the Transformer, we found that a specific architecture of the Transformer is required in this work. ConclusionsThe study results indicate that the proposed model can decode the visual neural activities evoked by natural images into descriptive text about the visual stimuli in the form of sentences. Hence, it may be considered as a potential computer-aided tool for neuroscientists to understand the neural mechanism of visual information processing in the human brain in the future.

Artificial Intelligence for Text-Based Vehicle Search, Recognition, and Continuous Localization in Traffic Videos

The concept of searching and localizing vehicles from live traffic videos based on descriptive textual input has yet to be explored in the scholarly literature. Endowing Intelligent Transportation Systems (ITS) with such a capability could help solve crimes on roadways. One major impediment to the advancement of fine-grain vehicle recognition models is the lack of video testbench datasets with annotated ground truth data. Additionally, to the best of our knowledge, no metrics currently exist for evaluating the robustness and performance efficiency of a vehicle recognition model on live videos and even less so for vehicle search and localization models. In this paper, we address these challenges by proposing V-Localize, a novel artificial intelligence framework for vehicle search and continuous localization captured from live traffic videos based on input textual descriptions. An efficient hashgraph algorithm is introduced to compute valid target information from textual input. This work further introduces two novel datasets to advance AI research in these challenging areas. These datasets include (a) the most diverse and large-scale Vehicle Color Recognition (VCoR) dataset with 15 color classes—twice as many as the number of color classes in the largest existing such dataset—to facilitate finer-grain recognition with color information; and (b) a Vehicle Recognition in Video (VRiV) dataset, a first of its kind video testbench dataset for evaluating the performance of vehicle recognition models in live videos rather than still image data. The VRiV dataset will open new avenues for AI researchers to investigate innovative approaches that were previously intractable due to the lack of annotated traffic vehicle recognition video testbench dataset. Finally, to address the gap in the field, five novel metrics are introduced in this paper for adequately accessing the performance of vehicle recognition models in live videos. Ultimately, the proposed metrics could also prove intuitively effective at quantitative model evaluation in other video recognition applications. T One major advantage of the proposed vehicle search and continuous localization framework is that it could be integrated in ITS software solution to aid law enforcement, especially in critical cases such as of amber alerts or hit-and-run incidents.

Implementation and Practice of Deep Learning-Based Instance Segmentation Algorithm for Quantification of Hepatic Fibrosis at Whole Slide Level in Sprague-Dawley Rats.

Exponential development in artificial intelligence or deep learning technology has resulted in more trials to systematically determine the pathological diagnoses using whole slide images (WSIs) in clinical and nonclinical studies. In this study, we applied Mask Regions with Convolution Neural Network (Mask R-CNN), a deep learning model that uses instance segmentation, to detect hepatic fibrosis induced by N-nitrosodimethylamine (NDMA) in Sprague-Dawley rats. From 51 WSIs, we collected 2011 cropped images with hepatic fibrosis annotations. Training and detection of hepatic fibrosis via artificial intelligence methods was performed using Tensorflow 2.1.0, powered by an NVIDIA 2080 Ti GPU. From the test process using tile images, 95% of model accuracy was verified. In addition, we validated the model to determine whether the predictions by the trained model can reflect the scoring system by the pathologists at the WSI level. The validation was conducted by comparing the model predictions in 18 WSIs at 20× and 10× magnifications with ground truth annotations and board-certified pathologists. Predictions at 20× showed a high correlation with ground truth (R2 = 0.9660) and a good correlation with the average fibrosis rank by pathologists (R2 = 0.8887). Therefore, the Mask R-CNN algorithm is a useful tool for detecting and quantifying pathological findings in nonclinical studies.

3D Quantum Cuts for automatic segmentation of porous media in tomography images

Binary segmentation of volumetric images of porous media is a crucial step towards gaining a deeper understanding of the factors governing biogeochemical processes at minute scales. Contemporary work primarily revolves around primitive techniques based on global or local adaptive thresholding that have known common drawbacks in image segmentation. Moreover, the absence of a unified benchmark prohibits quantitative evaluation, which further undermines the impact of existing methodologies. In this study, we tackle the issue on both fronts. First, by drawing parallels with natural image segmentation, we propose a novel, and automatic segmentation technique, 3D Quantum Cuts (QCuts-3D) grounded on a state-of-the-art spectral clustering technique. Secondly, we curate and present a publicly available dataset of 68 multiphase volumetric images of porous media with diverse solid geometries, along with voxel-wise ground truth annotations for each constituting phase. We provide comparative evaluations between QCuts-3D and the current state-of-the-art over this dataset across a variety of evaluation metrics. The proposed systematic approach achieves a 26% increase in AUROC (Area Under Receiver Operating Characteristics) while achieving a substantial reduction of the computational complexity over state-of-the-art competitors. Moreover, statistical analysis reveals that the proposed method exhibits significant robustness against the compositional variations of porous media.

Probabilistic 4D predictive model from in-room surrogates using conditional generative networks for image-guided radiotherapy.

Shape and location organ variability induced by respiration constitutes one of the main challenges during dose delivery in radiotherapy. Providing up-to-date volumetric information during treatment can improve tumor tracking, thereby increasing treatment efficiency and reducing damage to healthy tissue. We propose a novel probabilistic model to address the problem of volumetric estimation with scalable predictive horizon from image-based surrogates during radiotherapy treatments, thus enabling out-of-plane tracking of targets. This problem is formulated as a conditional learning task, where the predictive variables are the 2D surrogate images and a pre-operative static 3D volume. The model learns a distribution of realistic motion fields over a population dataset. Simultaneously, a seq-2-seq inspired temporal mechanism acts over the surrogate images yielding extrapolated-in-time representations. The phase-specific motion distributions are associated with the predicted temporal representations, allowing the recovery of dense organ deformation in multiple times. Due to its generative nature, this model enables uncertainty estimations by sampling the latent space multiple times. Furthermore, it can be readily personalized to a new subject via fine-tuning, and does not require inter-subject correspondences. The proposed model was evaluated on free-breathing 4D MRI and ultrasound datasets from 25 healthy volunteers, as well as on 11 cancer patients. A navigator-based data augmentation strategy was used during the slice reordering process to increase model robustness against inter-cycle variability. The patient data was used as a hold-out test set. Our approach yields volumetric prediction from image surrogates with a mean error of 1.67 ± 1.68mm and 2.17 ± 0.82mm in unseen cases of the patient MRI and US datasets, respectively. Moreover, model personalization yields a mean landmark error of 1.4 ± 1.1mm compared to ground truth annotations in the volunteer MRI dataset, with statistically significant improvements over state-of-the-art.

Texture appearance model, a new model-based segmentation paradigm, application on the segmentation of lung nodule in the CT scan of the chest

Lung cancer causes more than one million deaths worldwide each year. Averages of 5-year survival rate of patients with Non-small cell lung cancer (NSCLC), which is the most common type of lung cancer, is 15%. Computer-Aided Detection (CAD) is a very important tool for identifying lung lesions in medical imaging. In general, the process line of a CAD system can be divided into four main stages: preprocessing, localization, feature extraction, and classification. As segmentation is required for localization in computer vision and medical image analysis, this step has become a major and challenging problem, and much research has been done on new segmentation techniques. In recent years, interest in model-based segmentation methods has increased, and the reason for this is even if some object information is lost, such gaps can be filled by using the previous information in the model. This paper proposed Texture Appearance Model (TAM), which is a new model-based method and segments all types of nodule areas accurately and efficiently, including juxta-pleural nodules, without separating the lung from the surrounding area in a CT scan of the lung. In this method, Texture Representation of Image (TRI) is obtained using tissue texture feature extraction and feature selection algorithms. The proposed method has been evaluated in 85 nodules of the dataset, received from the Iranian hospital, in which the ground-truth annotation by physicians and CT imaging data were provided. The results show that the proposed algorithm has an encouraging performance for distinguishing different types of nodules, including pleural, cavity and non-solid nodules, achieving an average dice similarity coefficient (DSC) of 84.75%.

Automatic detection of attachment sites for knee ligaments and tendons on CT images.

The diseases and injuries of the knee joint are the most common orthopedic disorders. Personalized knee models can be helpful in the process of early intervention and lasting treatment techniques development. Fully automatic reconstruction of knee joint anatomical structures from medical images (CT, MRI, ultrasound) remains a challenge. For this reason, most of state-of-the-art knee joint models contain simplifications such as representation of muscles and ligaments as line segments connecting two points which replace attachment areas. The paper presents algorithms for automatic detection of such points on knee CT images. This paper presents three approaches to automatic detection of ligaments and tendons attachment sites on the patients CT images: qualitative anatomical descriptions, analysis of bones curvature, and quantitative anatomical descriptions. Combinations of these approaches result in new automatic detection algorithms. Each algorithm exploits anatomical peculiarities of each attachment site, e.g., bone curvature and number of other attachments in a neighborhood of the site. The experimental dataset consisted of 26 anonymized CT sequences containing right and left knee joints in different resolutions. The proposed algorithms take into account bone surface curvatures and spatial differences in locations of medial and lateral parts of both knees. The algorithms for detection of quadriceps femoris, popliteus, biceps femoris tendons, and lateral collateral and medial collateral ligaments attachment sites are provided, as well as examples of their application. Two algorithms are validated by comparison with known statistics of ligaments lengths and also using ground truth annotations for anatomical landmarks approved by clinical experts. The algorithms simplify generation of patient-specific knee joint models demanded in personalized biomechanical models. The algorithms in the current implementation have two important limitations. First, the correctness of the produced results depends on the bones segmentation quality. Second, the presented algorithms detect a point of the attachment site, which is not necessarily its center. Therefore, manual correction of the attachment site location may be required for attachments with relatively large area.

Inter-Variability Study of COVLIAS 1.0: Hybrid Deep Learning Models for COVID-19 Lung Segmentation in Computed Tomography.

Background: For COVID-19 lung severity, segmentation of lungs on computed tomography (CT) is the first crucial step. Current deep learning (DL)-based Artificial Intelligence (AI) models have a bias in the training stage of segmentation because only one set of ground truth (GT) annotations are evaluated. We propose a robust and stable inter-variability analysis of CT lung segmentation in COVID-19 to avoid the effect of bias. Methodology: The proposed inter-variability study consists of two GT tracers for lung segmentation on chest CT. Three AI models, PSP Net, VGG-SegNet, and ResNet-SegNet, were trained using GT annotations. We hypothesized that if AI models are trained on the GT tracings from multiple experience levels, and if the AI performance on the test data between these AI models is within the 5% range, one can consider such an AI model robust and unbiased. The K5 protocol (training to testing: 80%:20%) was adapted. Ten kinds of metrics were used for performance evaluation. Results: The database consisted of 5000 CT chest images from 72 COVID-19-infected patients. By computing the coefficient of correlations (CC) between the output of the two AI models trained corresponding to the two GT tracers, computing their differences in their CC, and repeating the process for all three AI-models, we show the differences as 0%, 0.51%, and 2.04% (all < 5%), thereby validating the hypothesis. The performance was comparable; however, it had the following order: ResNet-SegNet > PSP Net > VGG-SegNet. Conclusions: The AI models were clinically robust and stable during the inter-variability analysis on the CT lung segmentation on COVID-19 patients.

Self-Supervised Feature Learning of 1D Convolutional Neural Networks with Contrastive Loss for Eating Detection Using an In-Ear Microphone.

The importance of automated and objective monitoring of dietary behavior is becoming increasingly accepted. The advancements in sensor technology along with recent achievements in machine-learning-based signal-processing algorithms have enabled the development of dietary monitoring solutions that yield highly accurate results. A common bottleneck for developing and training machine learning algorithms is obtaining labeled data for training supervised algorithms, and in particular ground truth annotations. Manual ground truth annotation is laborious, cumbersome, can sometimes introduce errors, and is sometimes impossible in free-living data collection. As a result, there is a need to decrease the labeled data required for training. Additionally, unlabeled data, gathered in-the-wild from existing wearables (such as Bluetooth earbuds) can be used to train and fine-tune eating-detection models. In this work, we focus on training a feature extractor for audio signals captured by an in-ear microphone for the task of eating detection in a self-supervised way. We base our approach on the SimCLR method for image classification, proposed by Chen et al. from the domain of computer vision. Results are promising as our self-supervised method achieves similar results to supervised training alternatives, and its overall effectiveness is comparable to current state-of-the-art methods. Code is available at https://github.com/mug-auth/ssl-chewing.

Calibration and three-dimensional reconstruction with a photorealistic simulator based on the omnidirectional vision system

Recent advancements in deep learning require a large amount of the annotated training data containing various terms and conditions of the environment. Thus, developing and testing algorithms for the navigation of mobile robots can be expensive and time-consuming. Motivated by the aforementioned problems, this article presents a photorealistic simulator for the computer vision community working with omnidirectional vision systems. Built using unity, the simulator integrates sensors, mobile robots, and elements of the indoor environment and allows one to generate synthetic photorealistic data sets with automatic ground truth annotations. With the aid of the proposed simulator, two practical applications are studied, namely extrinsic calibration of the vision system and three-dimensional reconstruction of the indoor environment. For the proposed calibration and reconstruction techniques, the processes themselves are simple, robust, and accurate. Proposed methods are evaluated experimentally with data generated by the simulator. The proposed simulator and supporting materials are available online: http://www.ilabit.org .

Automated vessel segmentation in lung CT and CTA images via deep neural networks

The distribution of pulmonary vessels in computed tomography (CT) and computed tomography angiography (CTA) images of lung is important for diagnosing disease, formulating surgical plans and pulmonary research. Based on the pulmonary vascular segmentation task of International Symposium on Image Computing and Digital Medicine 2020 challenge, this paper reviews 12 different pulmonary vascular segmentation algorithms of lung CT and CTA images and then objectively evaluates and compares their performances. First, we present the annotated reference dataset of lung CT and CTA images. A subset of the dataset consisting 7,307 slices for training and 3,888 slices for testing was made available for participants. Second, by analyzing the performance comparison of different convolutional neural networks from 12 different institutions for pulmonary vascular segmentation, the reasons for some defects and improvements are summarized. The models are mainly based on U-Net, Attention, GAN, and multi-scale fusion network. The performance is measured in terms of Dice coefficient, over segmentation rate and under segmentation rate. Finally, we discuss several proposed methods to improve the pulmonary vessel segmentation results using deep neural networks. By comparing with the annotated ground truth from both lung CT and CTA images, most of 12 deep neural network algorithms do an admirable job in pulmonary vascular extraction and segmentation with the dice coefficients ranging from 0.70 to 0.85. The dice coefficients for the top three algorithms are about 0.80. Study results show that integrating methods that consider spatial information, fuse multi-scale feature map, or have an excellent post-processing to deep neural network training and optimization process are significant for further improving the accuracy of pulmonary vascular segmentation.

Automated chart filing on panoramic radiographs using deep learning

ObjectiveThe aim of this study is to automatically detect, segment and label teeth, crowns, fillings, root canal fillings, implants and root remnants on panoramic radiographs (PR(s)). Material and methodsAs a reference, 2000 PR(s) were manually annotated and labeled. A deep-learning approach based on mask R-CNN with Resnet-50 in combination with a rule-based heuristic algorithm and a combinatorial search algorithm was trained and validated on 1800 PR(s). Subsquently, the trained algorithm was applied onto a test set consisting of 200 PR(s). F1 scores, as a measure of accuracy, were calculated to quantify the degree of similarity between the annotated ground-truth and the model predictions. The F1-score considers the harmonic mean of precison (positive predictive value) and recall (specificity). ResultsThe proposes method achieved F1 scores up to 0.993, 0.952 and 0.97 for detection, segmentation and labeling, respectivley. ConclusionThe proposed method forms a promising foundation for the further development of automatic chart filing on PR(s). Clinical significanceDeep learning may assist clinicians in summarizing the radiological findings on panoramic radiographs. The impact of using such models in clinical practice should be explored.

Clinical Validation of Deep Learning Algorithms for Lung Cancer Radiotherapy Targeting

<h3>Purpose/Objective(s)</h3> Automated target segmentation for non-small cell lung cancer (NSCLC) patients has the potential to support radiation treatment planning. Artificial intelligence (AI) has demonstrated great promise in medical image segmentation tasks. However, most studies have been confined to <i>in silico</i> validation in small internal cohorts, lacking data on real-world clinical utility. In this study, we developed primary tumor and involved lymph node segmentation algorithms in computed tomography (CT) images. Validation is performed in multiple large multi-institutional cohorts to assess model generalizability. <h3>Materials/Methods</h3> Simulation CTs and ground truth annotations were collected from multiple public and private sources (total n = 2584). We employed the following benchmarks: Inter-observer (6 radiation oncologists, n = 20, median volumetric dice 0.83, 95% CI 0.82-0.84) and intra-observer (1 radiation oncologist, 3 reads, n = 21, median volumetric dice 0.88, 95% CI 0.84-0.9). We developed two segmentation algorithms: seed-point assisted and fully automated. Model training data (n = 787) comprised NSCLC-Radiomics (stages I-IIIB, n = 422) and LungRT-1 (stages IA-IV, n = 365). Validation was first performed in an internal dataset annotated by a single thoracic radiation oncologist (LungRT-1, n = 136). Additional validation included: (1) an internal dataset annotated by other radiation oncologists, including generalists, in our center (LungRT-2, n = 1075), (2) an external clinical trial dataset from 185 different institutions (RTOG-0617, n = 403), and (3) a dataset of early-stage surgical patients annotated for diagnostic purposes by radiologists (NSCLC-Radiogenomics, n = 142). Volumetric dice, using expert manual segmentations as ground truth, was used as an evaluation metric. <h3>Results</h3> The model performance is comparable to the benchmarks when validated on internal data, with degrading performance in cohorts annotated by other radiation oncologists. <h3>Conclusion</h3> The results highlight the importance of assessing segmentation style among annotators and understanding model generalizability in external cohorts, all while cautioning against degrading performance in increasingly external data. Differences between radiologists and radiation oncologists performing the same segmentation task underscore the importance of clinical context in AI model deployment. Further validation includes studying the dosimetric impact of AI-generated segmentations, and conducting human subject experiments to assess AI output acceptance and time savings.

Deep learning improves macromolecule identification in 3D cellular cryo-electron tomograms.

Cryogenic electron tomography (cryo-ET) visualizes the 3D spatial distribution of macromolecules at nanometer resolution inside native cells. However, automated identification of macromolecules inside cellular tomograms is challenged by noise and reconstruction artifacts, as well as the presence of many molecular species in the crowded volumes. Here, we present DeepFinder, a computational procedure that uses artificial neural networks to simultaneously localize multiple classes of macromolecules. Once trained, the inference stage of DeepFinder is faster than template matching and performs better than other competitive deep learning methods at identifying macromolecules of various sizes in both synthetic and experimental datasets. On cellular cryo-ET data, DeepFinder localized membrane-bound and cytosolic ribosomes (roughly 3.2 MDa), ribulose 1,5-bisphosphate carboxylase-oxygenase (roughly 560 kDa soluble complex) and photosystem II (roughly 550 kDa membrane complex) with an accuracy comparable to expert-supervised ground truth annotations. DeepFinder is therefore a promising algorithm for the semiautomated analysis of a wide range of molecular targets in cellular tomograms.

Fast and High-Quality, GPU-based, Deliberative, Object-Pose Estimation

Pose estimation of recognized objects is fundamental to tasks such as robotic grasping and manipulation. The need for reliable grasping imposes stringent accuracy requirements on pose estimation in cluttered, occluded scenes in dynamic environments. Modern methods employ large sets of training data to learn features and object templates in order to find correspondence between models and observed data. However, these methods require extensive annotation of ground-truth poses. An alternative is to use algorithms, such as PERCH (PErception Via SeaRCH) that seek an optimal explanation of the observed scene in a space of possible rendered versions. While PERCH offers strong guarantees on accuracy, the initial formulation suffers from poor scalability owing to its high runtime. In this work, we present PERCH 2.0, a deliberative approach that takes advantage of GPU acceleration and RGB data by formulating pose estimation as a single-shot, fully parallel approach. We show that PERCH 2.0 achieves a two orders of magnitude speedup (∼100X) over the hierarchical PERCH by evaluating thousands of poses in parallel. In addition, we propose a combined deliberative and discriminative framework for 6-DoF pose estimation that doesn’t require any ground-truth pose-annotation. Our work shows that PERCH 2.0 achieves, on the YCB-Video Dataset, a higher accuracy than DenseFusion, a state-of-the-art, end-to-end, learning-based approach. We also demonstrate that our work leads directly to an extension of deliberative pose estimation methods like PERCH to new domains, such as conveyor picking, which was previously infeasible due to high runtime. Our code is available at https://sbpl-cruz.github.io/perception/

Deep learning for automated detection and numbering of permanent teeth on panoramic images.

This study aimed to evaluate an automated detection system to detect and classify permanent teeth on orthopantomogram (OPG) images using convolutional neural networks (CNNs). In total, 591 digital OPGs were collected from patients older than 18 years. Three qualified dentists performed individual teeth labelling on images to generate the ground truth annotations. A three-step procedure, relying upon CNNs, was proposed for automated detection and classification of teeth. Firstly, U-Net, a type of CNN, performed preliminary segmentation of tooth regions or detecting regions of interest (ROIs) on panoramic images. Secondly, the Faster R-CNN, an advanced object detection architecture, identified each tooth within the ROI determined by the U-Net. Thirdly, VGG-16 architecture classified each tooth into 32 categories, and a tooth number was assigned. A total of 17,135 teeth cropped from 591 radiographs were used to train and validate the tooth detection and tooth numbering modules. 90% of OPG images were used for training, and the remaining 10% were used for validation. 10-folds cross-validation was performed for measuring the performance. The intersection over union (IoU), F1 score, precision, and recall (i.e. sensitivity) were used as metrics to evaluate the performance of resultant CNNs. The ROI detection module had an IoU of 0.70. The tooth detection module achieved a recall of 0.99 and a precision of 0.99. The tooth numbering module had a recall, precision and F1 score of 0.98. The resultant automated method achieved high performance for automated tooth detection and numbering from OPG images. Deep learning can be helpful in the automatic filing of dental charts in general dentistry and forensic medicine.

ADAADepth: Adapting Data Augmentation and Attention for Self-Supervised Monocular Depth Estimation

Self-supervised learning of depth has been a highly studied topic of research as it alleviates the requirement of having ground truth annotations for predicting depth. Depth is learnt as an intermediate solution to the task of view synthesis, utilising warped photometric consistency. Although it gives good results when trained using stereo data, the predicted depth is still sensitive to noise, illumination changes and specular reflections. Also, occlusion can be tackled better by learning depth from a single camera. We propose ADAA, utilising depth augmentation as depth supervision for learning accurate and robust depth. We propose a relational self-attention module that learns rich contextual features and further enhances depth results. We also optimize the auto-masking strategy across all losses by enforcing L1 regularisation over mask. Our novel progressive training strategy first learns depth at a lower resolution and then progresses to the original resolution with slight training. We utilise a ResNet18 encoder, learning features for prediction of both depth and pose. We evaluate our predicted depth on the standard KITTI driving dataset and achieve state-of-the-art results for monocular depth estimation whilst having significantly lower number of trainable parameters in our deep learning framework. We also evaluate our model on Make3D dataset showing better generalization than other methods.

Weakly-Supervised Single-view Dense 3D Point Cloud Reconstruction via Differentiable Renderer

In recent years, addressing ill-posed problems by leveraging prior knowledge contained in databases on learning techniques has gained much attention. In this paper, we focus on complete three-dimensional (3D) point cloud reconstruction based on a single red-green-blue (RGB) image, a task that cannot be approached using classical reconstruction techniques. For this purpose, we used an encoder-decoder framework to encode the RGB information in latent space, and to predict the 3D structure of the considered object from different viewpoints. The individual predictions are combined to yield a common representation that is used in a module combining camera pose estimation and rendering, thereby achieving differentiability with respect to imaging process and the camera pose, and optimization of the two-dimensional prediction error of novel viewpoints. Thus, our method allows end-to-end training and does not require supervision based on additional ground-truth (GT) mask annotations or ground-truth camera pose annotations. Our evaluation of synthetic and real-world data demonstrates the robustness of our approach to appearance changes and self-occlusions, through outperformance of current state-of-the-art methods in terms of accuracy, density, and model completeness.

A powerful image synthesis and semi-supervised learning pipeline for site-specific weed detection

Precise and efficient weed detection in agricultural fields is the key for robotic weed control. Recent developments in convolutional neural networks (CNNs) have achieved significant success in this regard. CNNs that simultaneously localize and classify objects in images are the predominant forms that have been widely used to detect crops and weeds. However, the use of CNNs in agriculture, particularly for weed detection, has been impeded by a lack of large training dataset with ground truth annotations. Cut-and-paste image synthesis approach and semi-supervised learning are popular methods to alleviate the training data deficiency problem. In this paper, we propose a novel image synthesis and semi-supervised learning pipeline to train site-specific weed detection models without the need for manually labeled images. The CNN models trained by this pipeline achieve performance levels close to that of the supervised models. We investigated the behavior of the proposed pipeline by varying several key components and showed that color match between the training and testing images, training-time color augmentation, and iterative semi-supervised learning largely improve the model performance. These promising results can be used to guide the construction of a weed image database applicable to different weed detection scenarios.