Quality Of Reference Data Research Articles

Data-Driven Wildfire Spread Modeling of European Wildfires Using a Spatiotemporal Graph Neural Network

Wildfire spread models are an essential tool for mitigating catastrophic effects associated with wildfires. However, current operational models suffer from significant limitations regarding accuracy and transferability. Recent advances in the availability and capability of Earth observation data and artificial intelligence offer new perspectives for data-driven modeling approaches with the potential to overcome the existing limitations. Therefore, this study developed a data-driven Deep Learning wildfire spread modeling approach based on a comprehensive dataset of European wildfires and a Spatiotemporal Graph Neural Network, which was applied to this modeling problem for the first time. A country-scale model was developed on an individual wildfire time series in Portugal while a second continental-scale model was developed with wildfires from the entire Mediterranean region. While neither model was able to predict the daily spread of European wildfires with sufficient accuracy (weighted macro-mean IoU: Portugal model 0.37; Mediterranean model 0.36), the continental model was able to learn the generalized patterns of wildfire spread, achieving similar performances in various fire-prone Mediterranean countries, indicating an increased capacity in terms of transferability. Furthermore, we found that the spatial and temporal dimensions of wildfires significantly influence model performance. Inadequate reference data quality most likely contributed to the low overall performances, highlighting the current limitations of data-driven wildfire spread models.

Evaluating capabilities of machine learning algorithms for aquatic vegetation classification in temperate wetlands using multi-temporal Sentinel-2 data

Different perspectives use of machine learning (ML) algorithms have proven their performance depends on the quality of reference data. This is particularly true when targets are complex environments, such as wetlands, on which a vast majority of studies are site-specific and based on a single date. With this work, an extensive reference dataset of about 400,000 samples was collected, covering nine different sites and multiple seasons, to be considered representative of temperate wetland vegetation communities at continental scale. Starting from this dataset, the performance of selected ML classifiers was compared for detailed wetland vegetation type mapping, using spectral indices (SI) derived from multi-temporal composites of Sentinel-2 as input. Global and per-class accuracy metrics were computed based on four independent training and testing subsets, extracted from the overall dataset, and the impacts of input features variation in number and sites covered were assessed. Our results show a generally higher predictive power for ensemble methods, such as Random Forest (RF) and eXtreme Gradient Boosting (XGBoost), compared to standalone ones, with the notable exception of Support Vector Machine (SVM); the latter in fact, in the algorithm that scored the highest overall accuracy (0.977 ± 0.001) and F-score for all the target classes. Decreasing the number of input features generally resulted in classification accuracy losses, less marked for RF than for SVM, while site-specific algorithms training showed more stability of SVM, thus indicating SVM stronger transferability than RF and XGBoost.

Milk infrared spectra from multiple instruments improve performance of prediction models

Milk data predicted from infrared spectra are the basis of selection programmes for dairy species, but several factors influence the robustness of prediction models, e.g., (i) variability of collected samples, (ii) number and quality of reference data, and (iii) influence traits on milk spectral fingerprint. Use of single or multiple instruments spectra from the same set of samples to develop mid infrared calibration models was investigated. Fitting statistics for milk coagulation properties were low to fair. Better results were achieved for titratable acidity and cheese yield; coefficients of determination in cross validation ranged from 0.52 to 0.77, depending on the trait analysed and training set. Merging spectral data from two instruments did not improve calibration performance but provided the best fitting statistics in external validation for most of the traits on different validation sets. Merging spectral data from different instruments is a cost-effective method to improve calibration performance and robustness.

Thematic accuracy assessment of the NLCD 2016 land cover for the conterminous United States

The National Land Cover Database (NLCD) is an operational land cover monitoring program providing updated land cover and related information for the United States at five-year intervals. NLCD2016 extends temporal coverage to 15 years (2001–2016). We collected land cover reference data for the 2011 and 2016 nominal dates to report land cover accuracy for the NLCD2016 database 2011 and 2016 land cover components at Level II and Level I and for Level I 2011–2016 land cover change using two definitions of agreement. For both the 2011 and 2016 land cover components, single-date Level II overall accuracies (OA) were 72% (standard error of ±0.9%) when agreement was defined as match between the map label and primary reference label only and 86% (± 0.7%) when agreement also included the alternate reference label. The corresponding level I OA for both dates were 79% (± 0.9%) and 91% (± 1.0%). The 2011–2016 user's and producer's accuracies (UA and PA) were ~ 75% for forest loss and PA for water loss, grassland loss, and grass gain were > 70% when agreement included a match between the map label and either the primary or alternate reference label. Depending on agreement definition and level of the classification hierarchy, OA for the 2011 land cover component of the NLCD2016 database was about 4% to 7% higher than OA for the 2011 land cover component of the NLCD2011 database, suggesting that the changes in mapping methodologies initiated for production of the NLCD2016 database have led to improved product quality. Additionally, we used the reference dataset collected for assessment of the NLCD2011 database to assess the 2001 and 2006 land cover components of the NLCD2016 database. OA for the 2001 and 2006 land cover components was 1% - 5% lower than OA for the 2011 and 2016 land cover components of the NLCD2016 database. Higher OA for 2011 and 2016 land cover components of the NLCD2016 database relative to OA for its 2001 and 2006 components may be attributable to differences in reference data quality.

Accuracy testing of a new optical device for noninvasive estimation of systolic and diastolic blood pressure compared to intra-arterial measurements.

The objective of this study was to compare the systolic (S) and diastolic (D) blood pressure (BP) estimations from a new optical device at the wrist with invasive measurements performed on patients scheduled for radial arterial catheterization in the ICU. Optical signals were automatically processed by a library of algorithms from Aktiia SA (OBPM - optical blood pressure monitoring algorithms). A total of 31 participants from both sexes, aged 32-87 years, were enrolled in the study (NCT03837769). The measurement protocol consisted of the simultaneous recording of reflective photoplethysmographic signals (PPG) from the cuffless optical device and the reference BP values recorded by a contralateral radial arterial catheter. From the 31 participants, 23 subjects whose reference data quality requirements were adequate were retained for further analysis. The PPG signals from these patients were then automatically processed by the Aktiia OBPM library of algorithms, which generated uncalibrated estimates of SBP and DBP. After the automatic assessment of optical signal quality, 326 pairs of uncalibrated SBP and DBP determinations from 16 patients were available for analysis. These values were finally transformed into calibrated estimations (in mmHg) using arterial catheter SBP and DBP values, respectively. For SBP, a mean difference (±SD) of 0.0 ± 7.1 mmHg between the arterial catheter and the optical device values was found, with 95% limits of agreement in the Bland-Altman method of -11.9 to + 12.2 mmHg (correlation of r = 0.87, P < 0.001). For DBP, a mean difference (±SD) of 0.0 ± 2.9 mmHg between arterial catheter and the optical device values was found, with 95% limits of agreement in the Bland-Altman method of -4.8 to + 5.5 mmHg (correlation of r = 0.98, P < 0.001). SBP and DBP values obtained by radial artery catheterization and those obtained from optical measurements at the wrist were compared. The new optical technique appears to be capable of replacing more traditional methods of BP estimation.

The t-SNE Algorithm as a Tool to Improve the Quality of Reference Data Used in Accurate Mapping of Heterogeneous Non-Forest Vegetation

Supervised classification methods, used for many applications, including vegetation mapping require accurate “ground truth” to be effective. Nevertheless, it is common for the quality of this data to be poorly verified prior to it being used for the training and validation of classification models. The fact that noisy or erroneous parts of the reference dataset are not removed is usually explained by the relatively high resistance of some algorithms to errors. The objective of this study was to demonstrate the rationale for cleaning the reference dataset used for the classification of heterogeneous non-forest vegetation, and to present a workflow based on the t-distributed stochastic neighbor embedding (t-SNE) algorithm for the better integration of reference data with remote sensing data in order to improve outcomes. The proposed analysis is a new application of the t-SNE algorithm. The effectiveness of this workflow was tested by classifying three heterogeneous non-forest Natura 2000 habitats: Molinia meadows (Molinion caeruleae; code 6410), species-rich Nardus grassland (code 6230) and dry heaths (code 4030), employing two commonly used algorithms: random forest (RF) and AdaBoost (AB), which, according to the literature, differ in their resistance to errors in reference datasets. Polygons collected in the field (on-ground reference data) in 2016 and 2017, containing no intentional errors, were used as the on-ground reference dataset. The remote sensing data used in the classification were obtained in 2017 during the peak growing season by a HySpex sensor consisting of two imaging spectrometers covering spectral ranges of 0.4–0.9 μm (VNIR-1800) and 0.9–2.5 μm (SWIR-384). The on-ground reference dataset was gradually cleaned by verifying candidate polygons selected by visual interpretation of t-SNE plots. Around 40–50% of candidate polygons were ultimately found to contain errors. Altogether, 15% of reference polygons were removed. As a result, the quality of the final map, as assessed by the Kappa and F1 accuracy measures as well as by visual evaluation, was significantly improved. The global map accuracy increased by about 6% (in Kappa coefficient), relative to the baseline classification obtained using random removal of the same number of reference polygons.

A Cluster Graph Approach to Land Cover Classification Boosting

When it comes to land cover classification, the process of deriving the land classes is complex due to possible errors in algorithms, spatio-temporal heterogeneity of the Earth observation data, variation in availability and quality of reference data, or a combination of these. This article proposes a probabilistic graphical model approach, in the form of a cluster graph, to boost geospatial classifications and produce a more accurate and robust classification and uncertainty product. Cluster graphs can be characterized as a means of reasoning about geospatial data such as land cover classifications by considering the effects of spatial distribution, and inter-class dependencies in a computationally efficient manner. To assess the capabilities of our proposed cluster graph boosting approach, we apply it to the field of land cover classification. We make use of existing land cover products (GlobeLand30, CORINE Land Cover) along with data from Volunteered Geographic Information (VGI), namely OpenStreetMap (OSM), to generate a boosted land cover classification and the respective uncertainty estimates. Our approach combines qualitative and quantitative components through the application of our probabilistic graphical model and subjective expert judgments. Evaluating our approach on a test region in Garmisch-Partenkirchen, Germany, our approach was able to boost the overall land cover classification accuracy by 1.4% when compared to an independent reference land cover dataset. Our approach was shown to be robust and was able to produce a diverse, feasible and spatially consistent land cover classification in areas of incomplete and conflicting evidence. On an independent validation scene, we demonstrated that our cluster graph boosting approach was generalizable even when initialized with poor prior assumptions.

Marker Development for Phylogenomics: The Case of Orobanchaceae, a Plant Family with Contrasting Nutritional Modes.

Phylogenomic approaches, employing next-generation sequencing (NGS) techniques, have revolutionized systematic and evolutionary biology. Target enrichment is an efficient and cost-effective method in phylogenomics and is becoming increasingly popular. Depending on availability and quality of reference data as well as on biological features of the study system, (semi-)automated identification of suitable markers will require specific bioinformatic pipelines. Here, we established a highly flexible bioinformatic pipeline, BaitsFinder, to identify putative orthologous single copy genes (SCGs) and to construct bait sequences in a single workflow. Additionally, this pipeline has been constructed to be able to cope with challenging data sets, such as the nutritionally heterogeneous plant family Orobanchaceae. To this end, we used transcriptome data of differing quality available for four Orobanchaceae species and, as reference, SCG data from monkeyflower (Erythranthe guttata, syn. Mimulus g.; 1,915 genes) and tomato (Solanum lycopersicum; 391 genes). Depending on whether gaps were permitted in initial blast searches of the four Orobanchaceae species against the reference, our pipeline identified 1,307 and 981 SCGs with average length of 994 bp and 775 bp, respectively. Automated bait sequence construction (using 2× tiling) resulted in 38,170 and 21,856 bait sequences, respectively. In comparison to the recently published MarkerMiner 1.0 pipeline BaitsFinder identified about 1.6 times as many SCGs (of at least 900 bp length). Skipping steps specific to analyses of Orobanchaceae, BaitsFinder was successfully used in a group of non-parasitic plants (three Asteraceae species and, as reference, SCG data from Arabidopsis thaliana based on previously compiled SCGs). Thus, BaitsFinder is expected to be broadly applicable in groups, where only transcriptomes or partial genome data of differing quality are available.

Urban area thermal monitoring: Liepaja case study using satellite and aerial thermal data

The aim of this study is to explore large (60m/pixel) and small scale (individual building level) temperature distribution patterns from thermal remote sensing data and to conclude what kind of information could be extracted from thermal remote sensing on regular basis. Landsat program provides frequent large scale thermal images useful for analysis of city temperature patterns. During the study correlation between temperature patterns and vegetation content based on NDVI and building coverage based on OpenStreetMap data was studied. Landsat based temperature patterns were independent from the season, negatively correlated with vegetation content and positively correlated with building coverage.Small scale analysis included spatial and raster descriptor analysis for polygons corresponding to roofs of individual buildings for evaluating insulation of roofs. Remote sensing and spatial descriptors are poorly related to heat consumption data, however, thermal aerial data median and entropy can help to identify poorly insulated roofs. Automated quantitative roof analysis has high potential for acquiring city wide information about roof insulation, but quality is limited by reference data quality and information on building types, and roof materials would be crucial for further studies.

The Sensitivity of Mapping Methods to Reference Data Quality: Training Supervised Image Classifications with Imperfect Reference Data

The accuracy of a map is dependent on the reference dataset used in its construction. Classification analyses used in thematic mapping can, for example, be sensitive to a range of sampling and data quality concerns. With particular focus on the latter, the effects of reference data quality on land cover classifications from airborne thematic mapper data are explored. Variations in sampling intensity and effort are highlighted in a dataset that is widely used in mapping and modelling studies; these may need accounting for in analyses. The quality of the labelling in the reference dataset was also a key variable influencing mapping accuracy. Accuracy varied with the amount and nature of mislabelled training cases with the nature of the effects varying between classifiers. The largest impacts on accuracy occurred when mislabelling involved confusion between similar classes. Accuracy was also typically negatively related to the magnitude of mislabelled cases and the support vector machine (SVM), which has been claimed to be relatively insensitive to training data error, was the most sensitive of the set of classifiers investigated, with overall classification accuracy declining by 8% (significant at 95% level of confidence) with the use of a training set containing 20% mislabelled cases.

Validation of Uncertainty in IEC Damage Calculations Based on Measurements from Alpha Ventus

The aim of this paper is to validate environmental assumptions used in the design of offshore wind turbines. This is done by comparing the load uncertainty (load variation) resulting from assumptions to full scale measurement data from a Senvion 5M turbine on a jacket substructure positioned in the offshore test field Alpha Ventus in the North Sea. The focus is put on fatigue loads occurring during power production. Sensors at the tower base and tower top of the turbine are evaluated. Simulations are performed by using the coupled simulation tool Flex5-Poseidon. Measurements over a period of 10 months are selected so that a high quality of reference data is ensured. Uncertainty from both measurements and simulations are determined by Monte Carlo experiments and evaluated by Bootstrap methods. Results show that a variation of damage from simulations is significant and that the presented method could be used for evaluation of assumptions used in the design process.

Wildlife forensic science: A review of genetic geographic origin assignment

Wildlife forensic science has become a key means of enforcing legislation surrounding the illegal trade in protected and endangered species. A relatively new dimension to this area of forensic science is to determine the geographic origin of a seized sample. This review focuses on DNA testing, which relies on assignment of an unknown sample to its genetic population of origin. Key examples of this are the trade in timber, fish and ivory and these are used only to illustrate the large number of species for which this type of testing is potentially available. The role of mitochondrial and nuclear DNA markers is discussed, alongside a comparison of neutral markers with those exhibiting signatures of selection, which potentially offer much higher levels of assignment power to address specific questions. A review of assignment tests is presented along with detailed methods for evaluating error rates and considerations for marker selection. The availability and quality of reference data are of paramount importance to support assignment applications and ensure reliability of any conclusions drawn. The genetic methods discussed have been developed initially as investigative tools but comment is made regarding their use in courts. The potential to compliment DNA markers with elemental assays for greater assignment power is considered and finally recommendations are made for the future of this type of testing.

Spatial characterization of bark beetle infestations by a multidate synergy of SPOT and Landsat imagery

Biological infestations in forests, e.g. the insect outbreaks, have been shown as favoured by future climate change trends. In Europe, the European spruce bark beetle (Ips typographus L.) is one of the main agents causing substantial economic disturbances in forests. Therefore, studies on spatio-temporal characterization of the area affected by bark beetle are of major importance for rapid post-attack management. We aimed at spatially detecting damage classes by combining multidate remote sensing data and a non-parametric classification. As study site served a part of the Bavarian Forest National Park (Germany). For the analysis, we used 10 geometrically rectified scenes of Landsat and SPOT sensors in the period between 2001 and 2011. The main objective was to explore the potential of medium-resolution data for classifying the attacked areas. A further aim was to explore if the temporally adjacent infested areas are able to be separated. The random forest (RF) model was applied using the reference data drawn from high-resolution aerial imagery. The results indicate that the sufficiently large patches of visually identifiable damage classes can be accurately separated from non-attacked areas. In contrast to those, the other mortality classes (current year, current year 1 and current year 2 infested classes) were mostly classified with higher commission or omission errors as well as higher classification biases. The available medium-resolution satellite images, combined with properly acquired reference data, are concluded to be adequate tools to map area-based infestations at advanced stages. However, the quality of reference data, the size of infested patches and the spectral resolution of remotely sensed data are the decisive factors in case of smaller areas. Further attempts using auxiliary height information and spatially enhanced data may refine such an approach.

Tree Species Classification with Random Forest Using Very High Spatial Resolution 8-Band WorldView-2 Satellite Data

Tree species diversity is a key parameter to describe forest ecosystems. It is, for example, important for issues such as wildlife habitat modeling and close-to-nature forest management. We examined the suitability of 8-band WorldView-2 satellite data for the identification of 10 tree species in a temperate forest in Austria. We performed a Random Forest (RF) classification (object-based and pixel-based) using spectra of manually delineated sunlit regions of tree crowns. The overall accuracy for classifying 10 tree species was around 82% (8 bands, object-based). The class-specific producer’s accuracies ranged between 33% (European hornbeam) and 94% (European beech) and the user’s accuracies between 57% (European hornbeam) and 92% (Lawson’s cypress). The object-based approach outperformed the pixel-based approach. We could show that the 4 new WorldView-2 bands (Coastal, Yellow, Red Edge, and Near Infrared 2) have only limited impact on classification accuracy if only the 4 main tree species (Norway spruce, Scots pine, European beech, and English oak) are to be separated. However, classification accuracy increased significantly using the full spectral resolution if further tree species were included. Beside the impact on overall classification accuracy, the importance of the spectral bands was evaluated with two measures provided by RF. An in-depth analysis of the RF output was carried out to evaluate the impact of reference data quality and the resulting reliability of final class assignments. Finally, an extensive literature review on tree species classification comprising about 20 studies is presented.

Impacts of imperfect reference data on the apparent accuracy of species presence-absence models and their predictions

Aim To explore the impacts of imperfect reference data on the accuracy of species distribution model predictions. The main focus is on impacts of the quality of reference data (labelling accuracy) and, to a lesser degree, data quantity (sample size) on species presence–absence modelling. Innovation The paper challenges the common assumption that some popular measures of model accuracy and model predictions are prevalence independent. It highlights how imperfect reference data may impact on a study and the actions that may be taken to address problems. Main conclusions The theoretical independence of prevalence of popular accuracy measures, such as sensitivity, specificity, true skills statistics (TSS) and area under the receiver operating characteristic curve (AUC), is unlikely to occur in practice due to reference data error; all of these measures of accuracy, together with estimates of species occurrence, showed prevalence dependency arising through the use of a non-gold-standard reference. The number of cases used also had implications for the ability of a study to meet its objectives. Means to reduce the negative effects of imperfect reference data in study design and interpretation are suggested.

Thematic accuracy of the NLCD 2001 land cover for the conterminous United States

The land-cover thematic accuracy of NLCD 2001 was assessed from a probability-sample of 15,000 pixels. Nationwide, NLCD 2001 overall Anderson Level II and Level I accuracies were 78.7% and 85.3%, respectively. By comparison, overall accuracies at Level II and Level I for the NLCD 1992 were 58% and 80%. Forest and cropland were two classes showing substantial improvements in accuracy in NLCD 2001 relative to NLCD 1992. NLCD 2001 forest and cropland user's accuracies were 87% and 82%, respectively, compared to 80% and 43% for NLCD 1992. Accuracy results are reported for 10 geographic regions of the United States, with regional overall accuracies ranging from 68% to 86% for Level II and from 79% to 91% at Level I. Geographic variation in class-specific accuracy was strongly associated with the phenomenon that regionally more abundant land-cover classes had higher accuracy. Accuracy estimates based on several definitions of agreement are reported to provide an indication of the potential impact of reference data error on accuracy. Drawing on our experience from two NLCD national accuracy assessments, we discuss the use of designs incorporating auxiliary data to more seamlessly quantify reference data quality as a means to further advance thematic map accuracy assessment.