Data Reduction Algorithm Research Articles

Mapping forest tree species and its biodiversity using EnMAP hyperspectral data along with Sentinel-2 temporal data: An approach of tree species classification and diversity indices

Forests play a crucial role in maintaining ecological balance and biodiversity, making the accurate mapping of tree species and assessment of biodiversity indices essential for informed management decisions. This study introduces an innovative methodology that integrates EnMAP (Environmental Mapping and Analysis Program) hyperspectral data with Sentinel-2 multitemporal data to classify tree species in the biodiverse landscapes of Kampinos National Park and its surrounding regions in Poland.We extract essential vegetation indices such as NDVI, NDMI, SAVI, and EVI from Sentinel-2 data to assess forest health and dynamics. The Sentinel-2 data is upscaled from 10 m to 30 m to align with EnMAP’s spatial resolution, followed by precise co-registration of the images using QGIS. Utilizing a rich dataset from the National Forest Inventory, we extract spectral signatures of nine distinct tree species from both data sources. We employ five machine learning algorithms—Support Vector Machines (SVM), Random Forest (RF), CatBoost (CAT), Gradient Boosting Classifier (GBC), and XGBoost (XGB)—to enhance classification accuracy.Through iterative experimentation with data reduction techniques and algorithm tuning, we achieve optimal performance across needle-leaved and broad-leaved species. The resulting tree species maps are validated through quantitative accuracy assessments against mixed-species polygons from the National Forest Inventory and ground truthing in the Kampinos National Park. Achieving an overall accuracy of 85% to 93%, our study demonstrates the efficacy of this integrated approach in tree species mapping. Furthermore, the tree species maps serve as a foundation for deriving key biodiversity indices—species richness, Shannon-Wiener Diversity Index, Simpson’s Diversity Index, and a composite Biodiversity Index—providing insights into spatial biodiversity patterns and informing targeted conservation strategies. This study exemplifies the potential of combining advanced remote sensing techniques with field validation to enhance our understanding of forest ecosystems and guide sustainable management practices.

Boosting interclass boundary preservation (BIBP): a KD-tree enhanced data reduction algorithm

Boosting interclass boundary preservation (BIBP): a KD-tree enhanced data reduction algorithm

A modified sequential wavenumber selection-discriminant analysis with Bayesian optimization strategy for detection and identification of chia seed oil adulteration using Raman spectroscopy

The detection of oil fraud can be accomplished through the use of Raman spectroscopy, which is a potent analytical technique for identifying the adulteration of edible oils with inferior or less expensive oils. However, appropriate data reduction and classification methods are required to achieve high accuracy and reliability in the analysis of Raman spectra. In this study, data reduction algorithms such as principal component analysis (PCA) and modified sequential wavenumber selection (MSWS) were applied, along with discriminant analysis (DA) as a classifier for detecting oil fraud. The parameters of DA, such as the discriminant type, the amount of regularization, and the linear coefficient threshold, were optimized using Bayesian optimization. The methods were tested on a dataset of chia oil mixed with 5–40 % sunflower oil, which is a common form of fraud in the market. The results showed that MSWS-DA achieved 100 % classification accuracy, while PCA-DA achieved 91.3 % accuracy. Therefore, it was demonstrated that Raman spectroscopy combined with MSWS-DA and Bayesian optimization can effectively detect oil fraud with high accuracy and robustness.

New data reduction algorithms based on the fusion of instance and feature selection

This paper presents two novel data reduction algorithms that combine instance selection and feature selection methods to simultaneously reduce the number of records and features. These algorithms are rigorously defined and thoroughly explained. They are applied to reduce twelve data sets selected from a commonly available database repository and single representative of big database. Their reduction efficiency is tested using three well-known machine learning models: classification and regression tree, convolutional neural network, and support vector machines. The results are extensively analyzed, comparing the classification performance of the models on the reduced data sets with that on the original ones. The computational complexity of algorithms is theoretically estimated and practically verified. The obtained performance results are compared with those available in the literature. It is shown that implementing the proposed data reduction algorithms consistently improves model accuracy in the vast majority of classification scenarios.

An efficiency-improved GPU algorithm for the 2 + 2 + 1 method in nonlinear beamforming

Abstract Nonlinear beamforming (NLBF) has emerged as a highly effective technology for enhancing seismic data quality. The crux of NLBF's success lies in its ability to robustly estimate local traveltime operators directly from input data, a process that entails solving millions or even billions of nonlinear optimization problems per input gather. Among the solvers utilized for estimating these operators is the 2 + 2 + 1 method, for which we have previously introduced algorithmic implementations on both the CPU and GPU platforms. In this paper, we present an efficiency-improved GPU algorithm for the 2 + 2 + 1 method, particularly beneficial when dealing with small data apertures in NLBF. Our enhanced GPU algorithm brings significant improvements in computation efficiency through several strategic measures, which include leveraging Horner's method to minimize the mathematical overhead of traveltime calculation, implementing a GPU-friendly data reduction algorithm to exploit GPU computational power, and optimizing shared GPU memory usage as the primary workspace whenever feasible. To demonstrate the tangible efficiency enhancement achieved by our new GPU algorithm, via two illustrative examples, we compare its performance with that of our previous implementation.

Application of clustering strategy for automatic segmentation of tissue regions in mass spectrometry imaging.

Mass spectrometry imaging (MSI) has been widely used in biomedical research fields. Each pixel in MSI consists of a mass spectrum that reflects the molecule feature of the tissue spot. Because MSI contains high-dimensional datasets, it is highly desired to develop computational methods for data mining and constructing tissue segmentation maps. To visualize different tissue regions based on mass spectrum features and improve the efficiency in processing enormous data, we proposed a computational strategy that consists of four procedures including preprocessing, data reduction, clustering, and quantitative validation. In this study, we examined the combination of t-distributed stochastic neighbor embedding (t-SNE) and hierarchical clustering (HC) for MSI data analysis. Using publicly available MSI datasets, one dataset of mouse urinary bladder, and one dataset of human colorectal cancer, we demonstrated that the generated tissue segmentation maps from this combination were superior to other data reduction and clustering algorithms. Using the staining image as a reference, we assessed the performance of clustering algorithms with external and internal clustering validation measures, including purity, adjusted Rand index (ARI), Davies-Bouldin index (DBI), and spatial aggregation index (SAI). The result indicated that SAI delivered excellent performance for automatic segmentation of tissue regions in MSI. We used a clustering algorithm to construct tissue automatic segmentation in MSI datasets. The performance was evaluated by comparing it with the stained image and calculating clustering validation indexes. The results indicated that SAI is important for automatic tissue segmentation in MSI, different from traditional clustering validation measures. Compared to the reports that used internal clustering validation measures such as DBI, our method offers more effective evaluation of clustering results for MSI segmentation. We envision that the proposed automatic image segmentation strategy can facilitate deep learning in molecular feature extraction and biomarker discovery for the biomedical applications of MSI.

Learning prevalent patterns of co-morbidities in multichronic patients using population-based healthcare data

The prevalence of longstanding chronic diseases has increased worldwide, along with the average age of the population. As a result, an increasing number of people is affected by two or more chronic conditions simultaneously, and healthcare systems are facing the challenge of treating multimorbid patients effectively. Current therapeutic strategies are suited to manage each chronic condition separately, without considering the whole clinical condition of the patient. This approach may lead to suboptimal clinical outcomes and system inefficiencies (e.g. redundant diagnostic tests and inadequate drug prescriptions). We develop a novel methodology based on the joint implementation of data reduction and clustering algorithms to identify patterns of chronic diseases that are likely to co-occur in multichronic patients. We analyse data from a large adult population of multichronic patients living in Tuscany (Italy) in 2019 which was stratified by sex and age classes. Results demonstrate that (i) cardio-metabolic, endocrine, and neuro-degenerative diseases represent a stable pattern of multimorbidity, and (ii) disease prevalence and clustering vary across ages and between women and men. Identifying the most common multichronic profiles can help tailor medical protocols to patients’ needs and reduce costs. Furthermore, analysing temporal patterns of disease can refine risk predictions for evolutive chronic conditions.

HDRA: A Haybird Data Reduction and Routing Algorithm

Presently, wireless sensor networks (WSNs) are emerging as a vibrant field of research due to various challenging aspects such as energy consumption, routing strategies, effectiveness, among others. Despite unresolved issues within WSNs, a substantial array of applications has already been developed. For any application design, a primary objective is to optimize the WSN in terms of its lifecycle and functionality. Recent studies on data reduction methods have shown that sensor nodes often transmit data directly (single hop) to the base station (BS). However, a significant concern is that most existing multi-hop routing protocols do not address data reduction before forwarding data to the BS. Consequently, this study introduces a Hybrid Data Reduction and Routing Algorithm (HDRA). The principal aim of HDRA is to prolong the lifespan of cluster-based WSNs. It strives to decrease the packet transmission by sensor nodes, especially when there's minimal change in sensor readings. The findings indicate that HDRA outperforms the LEACH protocol in terms of energy efficiency in sensor networks, irrespective of network type (T, H, or TH) or deployment scenarios (200x200m or 400x400m). Overall, the proposed algorithm enhances network performance by conserving energy and extending network lifespan.

Serial and parallel kernelization of Multiple Hitting Set parameterized by the Dilworth number, implemented on the GPU

The NP-hard Multiple Hitting Set problem is the problem of finding a minimum-cardinality set intersecting each of the sets in a given input collection a given number of times. Generalizing a well-known data reduction algorithm due to Weihe, we show a problem kernel for Multiple Hitting Set parameterized by the Dilworth number, a graph parameter introduced by Foldes and Hammer in 1978 yet seemingly so far unexplored in the context of parameterized complexity theory. Using matrix multiplication, we speed up the algorithm to quadratic sequential time and logarithmic parallel time. We experimentally evaluate our algorithms. By implementing our algorithm on GPUs, we show the feasibility of realizing kernelization algorithms on SIMD (Single Instruction, Multiple Data) architectures.

Static aeroelasticity analysis of a rotor blade using a Gauss-Seidel fluid-structure interaction method

The present study introduces a Gauss-Seidel fluid-structure interaction (FSI) method including the flow solver, structural statics solver and a fast data transfer technique, for the research of structural deformation and flow field variation of rotor blades under the combined influence of steady aerodynamic and centrifugal forces. The FSI method is illustrated and validated by the static aeroelasticity analysis of a transonic compressor rotor blade, NASA Rotor 37. An improved local interpolation with data reduction (LIWDR) algorithm is introduced for fast data transfer on the fluid-solid interface of blade. The results of FSI calculation of NASA Rotor 37 show that when compared with the radial basis function (RBF) based interpolation algorithm, LIWDR meets the interpolation accuracy requirements, while the calculation cost can be greatly improved. The data transmission time is only about 1% of that of RBF. Moreover, the iteration step of steady flow computation within one single FSI has little impact on the converged aerodynamic and structural results. The aerodynamic load-caused deformation accounts for nearly 50% of the total. The effects of blade deformation on the variations of aerodynamic performance are given, demonstrating that when static aeroelasticity is taken into account, the choke mass flow rate increases and the peak adiabatic efficiency slightly decreases. The impact mechanisms on performance variations are presented in detail.

JWST MIRI Flight Performance: Detector Effects and Data Reduction Algorithms

The detectors in the Mid-Infrared Instrument (MIRI) of the James Webb Space Telescope (JWST) are arsenic-doped silicon impurity band conduction (Si:As IBC) devices and are direct descendants of the Spitzer IRAC long wavelength arrays (channels 3 and 4). With appropriate data processing, they can provide excellent performance. In this paper we discuss the various non-ideal behaviors of these detectors that need to be addressed to realize their potential. We have developed a set of algorithms toward this goal, building on experience with previous similar detector arrays. The MIRI-specific stage 1 pipeline algorithms, of a three stage JWST calibration pipeline, were developed using pre-flight tests on the flight detectors and flight spares and have been refined using flight data. This paper describes these algorithms, which are included in the first stage of the JWST Calibration Pipeline for the MIRI instrument.

On data reduction for dynamic vector bin packing

We study a dynamic vector bin packing (DVBP) problem. We show hardness for shrinking arbitrary DVBP instances to size polynomial in the number of request types or in the maximal number of requests overlapping in time. We also present a simple polynomial-time data reduction algorithm that allows to recover (1+ε)-approximate solutions for arbitrary ε>0. It shrinks instances from Microsoft Azure and Huawei Cloud by an order of magnitude for ε=0.02.

Performance Enhancement of CAN/Ethernet Automotive Gateway with a CAN Data Reduction Algorithm

Data reduction (DR) techniques for the controller area network (CAN) are being developed to reduce the increased bus load caused by the growing number of electronic control units (ECUs) and automotive software complexity in modern automobiles. DR techniques enable the transmission of the same information with less bandwidth, effectively reducing the busload in CAN-based networks. Modern vehicles are composed of various in-vehicle network (IVN) protocols, such as CAN, local interconnect network (LIN), and Ethernet. However, existing DR techniques only consider the communication between CAN nodes. The application of DR techniques to a CAN bus may lead to compatibility issues when communicating with heterogeneous IVN protocols. This paper proposed a CAN/Ethernet gateway system for seamless communication with CAN DR. The proposed gateway system was implemented on a TC275-based embedded system, and its performance was evaluated and analyzed. The experimental results revealed that CAN DR compression effectively improves both the CAN bus load and end-to-end processing time of the gateway system. The CAN bus load was reduced by up to 33.68%, and the average end-to-end processing time was reduced to 336 μs.

Interclass boundary preservation (IBP): a data reduction algorithm

Interclass boundary preservation (IBP): a data reduction algorithm

An experimental study of acoustic emissions from active surface degradation in planetary gears

In this article, the detection of high frequency elastic waves, known as acoustic emission (AE), is used to correlate the events derived from the gear mesh with the surface wear and the operational behaviour in a planetary gearbox. In this regard, AE monitoring is a method that is able to provide information about the friction in the contact between surfaces in relative motion, which enables to employ a monitoring technology that overcomes the lack of sensitivity of traditional technologies under certain working conditions. Therefore, the investigation of the AE generated during the degradation of active gear surfaces can clarify how this process affects the performance in gearboxes. From the results, it has been stated that modifications in the active surfaces are barely reflected through traditional condition indicators (CI), such as AE frequency spectrum and AE signal envelope frequency spectrum, and same processing with accelerometry signals. It has been also observed that AE event width, or event lifetime, related with the gear meshing its a suitable indicator to monitor gear active surfaces. In this respect, a CI related to the AE event width and based in data reduction algorithm is proposed for gears monitoring with AE. For this work, characteristics of the complex studied system, as sensor position or cyclic periods inherent to the planetary gearbox, are assessed and highlighted as important factors in order to develop a valid monitoring.

Prediction-based spatial correlation clustering algorithm for efficient data reduction in wireless sensor network

Prediction-based spatial correlation clustering algorithm for efficient data reduction in wireless sensor network

Prediction-based spatial correlation clustering algorithm for efficient data reduction in wireless sensor network

Prediction-based spatial correlation clustering algorithm for efficient data reduction in wireless sensor network

The implementation of the ‘Testing of Phonological Skills (TOPS)’ tool: a computer-based phonological analysis algorithm

ABSTRACT A computer-based algorithm was used for phonological analysis of Cypriot Greek (CG) word productions elicited from a sample of typically developing (TD) CG-speaking children. Specific focus was on (a) creating a complete CG phonetic and phonological test; (2) implementing a novel computer-based algorithm in the analysis of selected speech stimuli from CG-speaking TD children; and (3) examining independent and relational speech patterns based on the computer-based algorithm. The probe word list consisted of 182 target words and included all CG consonants in all word positions. Normative data were collected on a representative sample of 20 TD CG-speaking children aged 3;0 to 6;0 years. Productions were phonetically transcribed and inserted into the computer algorithm for automatic data reduction of measurable dependent variables including the percentage of consonants correct (PCC), phonetic inventory size (PIS), word-initial onset deletion (IOD), and Regressive Assimilation (RASS). Correlation analyses revealed significant interactions among phonetic and phonological patterns as well as predictive relationships of PCC and segmental word-position. The implementation of this specific probe word list permitted a homogeneous and inclusive sampling analysis. Participants presented specific developmental trajectories of segmental and phonological patterns as a function of age.

Multimodal X-ray nano-spectromicroscopy analysis of chemically heterogeneous systems.

Understanding the nanoscale chemical speciation of heterogeneous systems in their native environment is critical for several disciplines such as life and environmental sciences, biogeochemistry, and materials science. Synchrotron-based X-ray spectromicroscopy tools are widely used to understand the chemistry and morphology of complex material systems owing to their high penetration depth and sensitivity. The multidimensional (4D+) structure of spectromicroscopy data poses visualization and data-reduction challenges. This paper reports the strategies for the visualization and analysis of spectromicroscopy data. We created a new graphical user interface and data analysis platform named XMIDAS (X-ray multimodal image data analysis software) to visualize spectromicroscopy data from both image and spectrum representations. The interactive data analysis toolkit combined conventional analysis methods with well-established machine learning classification algorithms (e.g. nonnegative matrix factorization) for data reduction. The data visualization and analysis methodologies were then defined and optimized using a model particle aggregate with known chemical composition. Nanoprobe-based X-ray fluorescence (nano-XRF) and X-ray absorption near edge structure (nano-XANES) spectromicroscopy techniques were used to probe elemental and chemical state information of the aggregate sample. We illustrated the complete chemical speciation methodology of the model particle by using XMIDAS. Next, we demonstrated the application of this approach in detecting and characterizing nanoparticles associated with alveolar macrophages. Our multimodal approach combining nano-XRF, nano-XANES, and differential phase-contrast imaging efficiently visualizes the chemistry of localized nanostructure with the morphology. We believe that the optimized data-reduction strategies and tool development will facilitate the analysis of complex biological and environmental samples using X-ray spectromicroscopy techniques.

An algorithm for U–Pb geochronology by secondary ion mass spectrometry

Abstract. Secondary ion mass spectrometry (SIMS) is a widely used technique for in situ U–Pb geochronology of accessory minerals. Existing algorithms for SIMS data reduction and error propagation make a number of simplifying assumptions that degrade the precision and accuracy of the resulting U–Pb dates. This paper uses an entirely new approach to SIMS data processing that introduces the following improvements over previous algorithms. First, it treats SIMS measurements as compositional data using log-ratio statistics. This means that, unlike existing algorithms, (a) its isotopic ratio estimates are guaranteed to be strictly positive numbers, (b) identical results are obtained regardless of whether data are processed as normal ratios (e.g. 206Pb / 238U) or reciprocal ratios (e.g. 238U / 206Pb), and (c) its uncertainty estimates account for the positive skewness of measured isotopic ratio distributions. Second, the new algorithm accounts for the Poisson noise that characterises secondary electron multiplier (SEM) detectors. By fitting the SEM signals using the method of maximum likelihood, it naturally handles low-intensity ion beams, in which zero-count signals are common. Third, the new algorithm casts the data reduction process in a matrix format and thereby captures all sources of systematic uncertainty. These include significant inter-spot error correlations that arise from the Pb / U–UO(2) / U calibration curve. The new algorithm has been implemented in a new software package called simplex. The simplex package was written in R and can be used either online, offline, or from the command line. The programme can handle SIMS data from both Cameca and SHRIMP instruments.