Preprocessing Tool Research Articles

From Manual to Intelligent: A Review of Input Data Preparation Methods for Geographic Modeling

One of the key concerns in geographic modeling is the preparation of input data that are sufficient and appropriate for models. This requires considerable time, effort, and expertise since geographic models and their application contexts are complex and diverse. Moreover, both data and data pre-processing tools are multi-source, heterogeneous, and sometimes unavailable for a specific application context. The traditional method of manually preparing input data cannot effectively support geographic modeling, especially for complex integrated models and non-expert users. Therefore, effective methods are urgently needed that are not only able to prepare appropriate input data for models but are also easy to use. In this review paper, we first analyze the factors that influence data preparation and discuss the three corresponding key tasks that should be accomplished when developing input data preparation methods for geographic models. Then, existing input data preparation methods for geographic models are discussed through classifying into three categories: manual, (semi-)automatic, and intelligent (i.e., not only (semi-)automatic but also adaptive to application context) methods. Supported by the adoption of knowledge representation and reasoning techniques, the state-of-the-art methods in this field point to intelligent input data preparation for geographic models, which includes knowledge-supported discovery and chaining of data pre-processing functionalities, knowledge-driven (semi-)automatic workflow building (or service composition in the context of geographic web services) of data preprocessing, and artificial intelligent planning-based service composition as well as their parameter-settings. Lastly, we discuss the challenges and future research directions from the following aspects: Sharing and reusing of model data and workflows, integration of data discovery and processing functionalities, task-oriented input data preparation methods, and construction of knowledge bases for geographic modeling, all assisting with the development of an easy-to-use geographic modeling environment with intelligent input data preparation.

Application of total variation denoising in nuclear power plant signal pre-processing

Pre-processing of generated signals from sensors installed in Nuclear Power Plant (NPP) components is a critical step towards enhancing signal characteristics and reliability of extracted data used in diagnosing the operational state of components or systems. Denoising is one of the essential features of the signal pre-processing tools in the instrument control system of a NPP. A number of denoising methods have been applied in filtering out noisy signals from sensors in NPP. This paper presents the study on the applicability of the Total Variation (TV) denoising method in NPP signal preprocessing. The Majorization-Minimization optimization framework was firstly applied to determine the optimization function suitable for the TV denoising model. An iterative algorithm was subsequently derived to solve the TV denoising model. The model was applied to denoise the water level signal of a NPP pressurizer and a simulation signal of loose parts. The results showed that the denoising method used can effectively filter out the noise mixed in the signal by selecting appropriate regularization parameter that enhances the accuracy of signal feature extraction. The results of the study indicate that the TV denoising model is applicable to NPP components.

Dynamic Adaptive Piecewise Linear Representation Approach Based on Streaming Time Series

With the burgeoning of IoE (Internet of Everything), massive numbers of IoT devices in extensive fields are continuously producing huge number of streaming time series. The high dimensionality and dynamic uncertainty of this kind of data lead to the main challenge on traditional time series data mining research. Accordingly, time series representation methods have been regarded as a necessary pre-processing tool to provide data support for the follow-up time series data mining research. In this paper, we propose a novel dynamic time series representation approach called dynamic adaptive piecewise linear representation (DAPLR) for streaming time series, which can automatically provide a series of piecewise linear representation results to meet the diverse needs of different users.

Fuzzy-Rough Set Bireducts for Data Reduction

Data reduction is an important step that helps ease the computational intractability for learning techniques when data are large. This is particularly true for the huge datasets that have become commonplace in recent times. The main problem facing both data preprocessors and learning techniques is that data are expanding both in terms of dimensionality and also in terms of the number of data instances. Approaches based on fuzzy-rough sets offer many advantages for both feature selection and classification, particularly for real-valued and noisy data; however, the majority of recent approaches tend to address the task of data reduction in terms of either dimensionality or training data size in isolation. This paper demonstrates how the notion of fuzzy-rough bireducts can be used for the simultaneous reduction of data size and dimensionality. It also shows how bireducts and, therefore, reduced subtables of data can be used not only as a preprocessing tool but also for the learning of compact and robust classifiers. Furthermore, the ideas can also be extended to the unsupervised domain when dealing with unlabeled data. Experimental evaluation of various techniques demonstrate that high levels of simultaneous reduction of both dimensionality and data size can be achieved whilst maintaining robust performance.

Evaluation of simulation models to mimic the distortions introduced into squiggles by nanopore sequencers and segmentation algorithms.

Nucleotides ratcheted through the biomolecular pores of nanopore sequencers generate raw picoamperage currents, which are segmented into step-current level signals representing the nucleotide sequence. These ‘squiggles’ are a noisy, distorted representation of the underlying true stepped current levels due to experimental and algorithmic factors. We were interested in developing a simulation model to support a white-box approach to identify common distortions, rather than relying on commonly used black box neural network techniques for basecalling nanopore signals. Dynamic time warped-space averaging (DTWA) techniques can generate a consensus from multiple noisy signals without introducing key feature distortions that occur with standard averaging. As a preprocessing tool, DTWA could provide cleaner and more accurate current signals for direct RNA or DNA analysis tools. However, DTWA approaches need modification to take advantage of the a-priori knowledge regarding a common, underlying gold-standard RNA / DNA sequence. Using experimental data, we derive a simulation model to provide known squiggle distortion signals to assist in validating the performance of analysis tools such as DTWA. Simulation models were evaluated by comparing mocked and experimental squiggle characteristics from one Enolase mRNA squiggle group produced by an Oxford MinION nanopore sequencer, and cross-validated using other Enolase, Sequin R1_71_1 and Sequin R2_55_3 mRNA studies. New techniques identified high inserted but low deleted base rates, generating consistent x1.7 squiggle event to base called ratios. Similar probability density and cumulative distribution functions, PDF and CDF, were found across all studies. Experimental PDFs were not the normal distributions expected if squiggle distortion arose from segmentation algorithm artefacts, or through individual nucleotides randomly interacting with individual nanopores. Matching experimental and mocked CDFs required the assumption that there are unique features associated with individual raw-current data streams. Z-normalized signal-to-noise ratios suggest intrinsic sensor limitations being responsible for half the gold standard and noisy squiggle DTW differences.

CatSent: a Catalan sentiment analysis website

In this paper we investigate, analyze and compare sentimental analysis methodologies in Catalan tweets. The main goal is to develop a high-performance Catalan classifier. There are three main steps: Catalan language preprocessing tool, classification model and corpus training. The preprocessing tool is used for cleaning and extracting features from a document (or tweet). This is a key step due to the great morphological complexity of the Catalan language. The tool will remove empty words from the text and find the roots of other words. The classification algorithm will divide the tweet into “positive” and “negative” classes. To choose the best algorithm, five models are compared: Naive Bayes, Maximum Entropy, Support Vector Machine, Decision Tree and Neural Networks. Finally, the corpus will be used for training and testing these methods. There is no known public corpus in Catalan, so we created one using a lexicon-based approach. This work aims to enable the tools to carry out sentiment analysis studies in the Catalan language. The last step is to develop a public web service with the best classification model achieved where users will be able to check its effectiveness.

Three-step coreference-based summarizer for Polish news texts

Abstract This article addresses the problem of automatic summarization of press articles in Polish. The main novelty of this research lays in the proposal of a three-step summarization algorithm which benefits from using coreference information. In related work section, all coreference-based approaches to summarization are presented. Then we describe in detail all publicly available summarization tools developed for Polish language. We state the problem of single-document press article summarization for Polish, describing the training and evaluation dataset: the POLISH SUMMARIES CORPUS. Next, a new coreference-based extractive summarization system NICOLAS is introduced. Its algorithm utilises advanced third-party preprocessing tools to extract the coreference information from the text to be summarized. This information is transformed into a complex set of features related to coreference concepts (mentions and coreference clusters) that are used for training the summarization system (on the basis of a manually prepared gold summaries corpus). The proposed solution is compared to the best publicly available summarization systems for Polish language and two state-of-the-art tools, developed for English language, but adapted to Polish for this article. NICOLAS summarization system obtains best scores, for selected metrics outperforming other systems in a statistically significant way. The evaluation also contains calculation of interesting upper-bounds: human performance and theoretical upper-bound.

Performance comparison of continuous Wavelet-Fuzzy and discrete Wavelet-Fuzzy models for water level predictions at northern and southern boundary of Bosphorus

In this study, combined Discrete Wavelet Transform-Fuzzy (DWT-Fuzzy) and combined Continuous Wavelet Transform-Fuzzy (CWT-Fuzzy) models are developed for predicting the daily water levels at northern and southern boundary of Bosphorus Strait. The observed daily water level data is decomposed into spectral bands (sub-series) by using wavelet transformation as a pre-processing tool in order to achieve more accurate daily water level predictions with extended lead-times up to 7 days. The time series of daily water level data is decomposed into spectral bands, which are used as inputs into the Fuzzy model and the daily water levels are predicted from the sum of the predicted components (spectral bands). A predictive model is developed using combined DWT-Fuzzy and combined CWT-Fuzzy models to predict water level fluctuations. It is found that CWT-Fuzzy model performed better than DWT-Fuzzy and stand-alone Fuzzy models for prediction lead-times up to 7 days at northern and southern boundary of Bosphorus based on RMSE and CE evaluation criteria. It is concluded that CWT is a better pre-processing technique as it yields more accurate daily water level predictions with improved prediction lead-times than DWT.

FastProNGS: fast preprocessing of next-generation sequencing reads

BackgroundNext-generation sequencing technology is developing rapidly and the vast amount of data that is generated needs to be preprocessed for downstream analyses. However, until now, software that can efficiently make all the quality assessments and filtration of raw data is still lacking.ResultsWe developed FastProNGS to integrate the quality control process with automatic adapter removal. Parallel processing was implemented to speed up the process by allocating multiple threads. Compared with similar up-to-date preprocessing tools, FastProNGS is by far the fastest. Read information before and after filtration can be output in plain-text, JSON, or HTML formats with user-friendly visualization.ConclusionsFastProNGS is a rapid, standardized, and user-friendly tool for preprocessing next-generation sequencing data within minutes. It is an all-in-one software that is convenient for bulk data analysis. It is also very flexible and can implement different functions using different user-set parameter combinations.

Objective Assessment of Beat Quality in Transcranial Doppler Measurement of Blood Flow Velocity in Cerebral Arteries.

Transcranial Doppler (TCD) ultrasonography measures pulsatile cerebral blood flow velocity in the arteries and veins of the head and neck. Similar to other real-time measurement modalities, especially in healthcare, the identification of high-quality signals is essential for clinical interpretation. Our goal is to identify poor quality beats and remove them prior to further analysis of the TCD signal. We selected objective features for this purpose including Euclidean distance between individual and average beat waveforms, cross-correlation between individual and average beat waveforms, ratio of the high-frequency power to the total beat power, beat length, and variance of the diastolic portion of the beat waveform. We developed an iterative outlier detection algorithm to identify and remove the beats that are different from others in a recording. Finally, we tested the algorithm on a dataset consisting of more than 15 h of TCD data recorded from 48 stroke and 34 in-hospital control subjects. We assessed the performance of the algorithm in the improvement of estimation of clinically important TCD parameters by comparing them to that of manual beat annotation. The results show that there is a strong correlation between the two, that demonstrates the algorithm has successfully recovered the clinically important features. We obtained significant improvement in estimating the TCD parameters using the algorithm accepted beats compared to using all beats. Our algorithm provides a valuable tool to clinicians for automated detection of the reliable portion of the data. Moreover, it can be used as a pre-processing tool to improve the data quality for automated diagnosis of pathologic beat waveforms using machine learning.

CAD-based design and pre-processing tools for additive manufacturing

This paper discusses a set of geometry based computational pre-processing algorithms developed for Powder Bed Fusion Additive Manufacturing (PBFAM) processes. To start with, based on an initial part design, an automatic support structure generation module generates customized CAD-based support structures for a given part build orientation. Various additive manufacturing (AM) parameters and Design for Additive Manufacturing (DFAM) metrics are calculated on the fly for assigning producibility scores at different part build orientations. A set of stand-alone computational geometry-based algorithms with associated graphical user interfaces (GUI) are developed for calculating support structure parameters, as well as for detecting and highlighting DFAM features that are difficult to manufacture. These stand-alone tools provide a quantified output for each of the parameters or features, which are then used downstream during the producibility index (PI) calculation. An algorithm that evaluates ease of removing supports during the post-processing phase, and suggests the optimum number of setups needed to remove support structures is developed. Finally, Producibility Index, which is a weighted optimization metric, brings together the quantified outputs of the DFAM analysis, support structure parameters, accessibility analysis and suggests the best build orientations for the given part geometry. All the algorithms are implemented within the Siemens NX modelling environment utilizing C++ and NX API functions. The developed algorithm and tools have been succesfully demonstrated on two sample parts.

Pre-processing the nuclear many-body problem

The solution of the nuclear A-body problem encounters severe limitations from the size of many-body operators. These limitations are typically related to both the (iterative) storing of the associated tensors and to the computational time related to their multiple contractions in the calculation of various quantities of interest. However, not all the degrees of freedom encapsulated into these tensors equally contribute to the description of many-body observables. Identifying systematic and dominating patterns, a relevant objective is to achieve an \emph{a priori} reduction to the most relevant degrees of freedom via a pre-processing of the A-body problem. The present paper is dedicated to the analysis of two different paradigms to do so. The factorization of tensors in terms of lower-rank ones, whose know-how has been recently transferred to the realm of nuclear structure, is compared to a reduction of the tensors' index size based on an importance truncation. While the objective is to eventually utilize these pre-processing tools in the context of non-perturbative many-body methods, benchmark calculations are presently performed within the frame of perturbation theory. More specifically, we employ the recently introduced Bogoliubov many-body perturbation theory that is systematically applicable to open-shell nuclei displaying strong correlations. This extended perturbation theory serves as a jumpstart for non-perturbative Bogoliubov coupled cluster and Gorkov self-consistent Green's function theories. Results obtained in "small" model spaces are equally encouraging for tensor factorization and importance truncation techniques. While the former requires significant numerical developments to be applied in large model spaces, the latter is presently applied in this context and demonstrates great potential to enable high-accuracy calculations at a much reduced computational cost.

Web-rMKL: a web server for dimensionality reduction and sample clustering of multi-view data based on unsupervised multiple kernel learning.

More and more affordable high-throughput techniques for measuring molecular features of biomedical samples have led to a huge increase in availability and size of different types of multi-omic datasets, containing, for example, genetic or histone modification data. Due to the multi-view characteristic of the data, established approaches for exploratory analysis are not directly applicable. Here we present web-rMKL, a web server that provides an integrative dimensionality reduction with subsequent clustering of samples based on data from multiple inputs. The underlying machine learning method rMKL-LPP performed best for clinical enrichment in a recent benchmark of state-of-the-art multi-view clustering algorithms. The method was introduced for a multi-omic cancer subtype discovery setting, however, it is not limited to this application scenario as exemplified by a presented use case for stem cell differentiation. web-rMKL offers an intuitive interface for uploading data and setting the parameters. rMKL-LPP runs on the back end and the user may receive notifications once the results are available. We also introduce a preprocessing tool for generating kernel matrices from tables containing numerical feature values. This program can be used to generate admissible input if no precomputed kernel matrices are available. The web server is freely available at web-rMKL.org.

FastqPuri: high-performance preprocessing of RNA-seq data

BackgroundRNA sequencing (RNA-seq) has become the standard means of analyzing gene and transcript expression in high-throughput. While previously sequence alignment was a time demanding step, fast alignment methods and even more so transcript counting methods which avoid mapping and quantify gene and transcript expression by evaluating whether a read is compatible with a transcript, have led to significant speed-ups in data analysis. Now, the most time demanding step in the analysis of RNA-seq data is preprocessing the raw sequence data, such as running quality control and adapter, contamination and quality filtering before transcript or gene quantification. To do so, many researchers chain different tools, but a comprehensive, flexible and fast software that covers all preprocessing steps is currently missing.ResultsWe here present FastqPuri, a light-weight and highly efficient preprocessing tool for fastq data. FastqPuri provides sequence quality reports on the sample and dataset level with new plots which facilitate decision making for subsequent quality filtering. Moreover, FastqPuri efficiently removes adapter sequences and sequences from biological contamination from the data. It accepts both single- and paired-end data in uncompressed or compressed fastq files. FastqPuri can be run stand-alone and is suitable to be run within pipelines. We benchmarked FastqPuri against existing tools and found that FastqPuri is superior in terms of speed, memory usage, versatility and comprehensiveness.ConclusionsFastqPuri is a new tool which covers all aspects of short read sequence data preprocessing. It was designed for RNA-seq data to meet the needs for fast preprocessing of fastq data to allow transcript and gene counting, but it is suitable to process any short read sequencing data of which high sequence quality is needed, such as for genome assembly or SNV (single nucleotide variant) detection. FastqPuri is most flexible in filtering undesired biological sequences by offering two approaches to optimize speed and memory usage dependent on the total size of the potential contaminating sequences. FastqPuri is available at https://github.com/jengelmann/FastqPuri. It is implemented in C and R and licensed under GPL v3.

Efficient Third-Party Abstraction of Lithium-Ion Batteries for Asset Modeling

As part of international strategies towards high penetration of renewable energy sources, the implementation of energy storage systems in power grids becomes increasingly necessary for short- and long-term energy deferral. Due to technical advantages in short-term and overall practical advantages, Lithium-Ion batteries are excellent storage mediums and already serve today widespread in a variety of applications both stationary and mobile. However, most storage systems are privately funded. To investors, battery storage is considered an asset with high capital expenditure and slow, only partially predictable return on investment. The value estimation of storage systems is intercepted by two major difficulties. The first issue is the inherent complexity of batteries regarding performance and aging, which requires in-depth knowledge for accurate statements. The second issue is the inability to determine the optimal operation strategy due to the sizable portfolio of potential applications, reaching from frequency response over arbitrage to peak-shaving, and the potential impact of each on the storage system. Investors are commonly provided with very few details on the battery itself and don’t always have the background knowledge necessary to accurately predict battery behavior and value generation in long term due to this complexity. This makes battery storage less tangible and less attractive than other assets. This presentation introduces a tool for potential customers of battery storage that can be applied to any market-available carbon-based Li-Ion battery technology. It aims to provide an approximation for the battery’s lifetime behavior based on a multi-physics model. Electrical, thermal and aging models are constructed from literature and scaled and enhanced through experimental data. The tool, which is developed in Python, takes as input the maximum amount of information available to customers in advance while making the minimum amount of assumptions about the storage system. The tool produces an approximation for the degradation of the storage system on an absolute and linear scale and can therefore be used inside optimization algorithms or as pre-processing tool. It is designed to be easily transferable and computationally efficient enough to allow for flexible parameter adjustment and multi-case comparison between storage providers and applications on any scale. Figure 1

A novel framework for MR image segmentation and quantification by using MedGA

Background and Objectives: Image segmentation represents one of the most challenging issues in medical image analysis to distinguish among different adjacent tissues in a body part. In this context, appropriate image pre-processing tools can improve the result accuracy achieved by computer-assisted segmentation methods. Taking into consideration images with a bimodal intensity distribution, image binarization can be used to classify the input pictorial data into two classes, given a threshold intensity value. Unfortunately, adaptive thresholding techniques for two-class segmentation work properly only for images characterized by bimodal histograms. We aim at overcoming these limitations and automatically determining a suitable optimal threshold for bimodal Magnetic Resonance (MR) images, by designing an intelligent image analysis framework tailored to effectively assist the physicians during their decision-making tasks.Methods: In this work, we present a novel evolutionary framework for image enhancement, automatic global thresholding, and segmentation, which is here applied to different clinical scenarios involving bimodal MR image analysis: (i) uterine fibroid segmentation in MR guided Focused Ultrasound Surgery, and (ii) brain metastatic cancer segmentation in neuro-radiosurgery therapy. Our framework exploits MedGA as a pre-processing stage. MedGA is an image enhancement method based on Genetic Algorithms that improves the threshold selection, obtained by the efficient Iterative Optimal Threshold Selection algorithm, between the underlying sub-distributions in a nearly bimodal histogram.Results: The results achieved by the proposed evolutionary framework were quantitatively evaluated, showing that the use of MedGA as a pre-processing stage outperforms the conventional image enhancement methods (i.e., histogram equalization, bi-histogram equalization, Gamma transformation, and sigmoid transformation), in terms of both MR image enhancement and segmentation evaluation metrics.Conclusions: Thanks to this framework, MR image segmentation accuracy is considerably increased, allowing for measurement repeatability in clinical workflows. The proposed computational solution could be well-suited for other clinical contexts requiring MR image analysis and segmentation, aiming at providing useful insights for differential diagnosis and prognosis.

Multi-resolution representation with recurrent neural networks application for streaming time series in IoT

Nowadays, with the proliferation of IoT (Internet of Things), we have gradually entered into a new IoE (Internet of Everything) era, in which billions of connected devices in widespread fields are constantly producing oceans of streaming time series. In order to conduct in-depth data mining researches (similarity searching, classification, clustering, prediction, etc.) based on streaming time series efficiently and effectively, time series representation should be done as the first step. In this paper, we propose a novel multi-resolution hybrid representation approach for streaming time series, which can not only generate different types of representation results in a more flexible way to meet diverse needs of users, but also be utilized as a useful preprocessing tool for the subsequent time series data mining researches. Extensive experiments on different kinds of typical time series datasets have been conducted to demonstrate the superiorities of our method.

Texture classification using optimized local ternary patterns with nonlinear diffusion as pre-processing

The main focus of this paper is to improve the performance of the texture model Optimized Local Ternary Patterns (OLTP), which is known as one of the successful variants of the texture model Local Binary Patterns (LBP), a well known method for texture analysis and its applications. Generally preprocessing is used in digital image processing for reducing the unwanted noise and disturbances in such a way that it improves the quality of the image. Preprocessing not only removes the distortions but also enhances the features of the image for further processing. To achieve better recognition accuracy, the texture model OLTP was combined with a preprocessing method that uses nonlinear diffusion method as a preprocessing tool in this paper, with the hope that this idea will surely improve the local feature description and texture classification process. This nonlinear diffusion method uses two newly developed edge stopping functions for preprocessing. This proposed method is tested with two standard texture datasets namely Brodatz Dataset and Usptex dataset. The results show that the use of the preprocessing step really improved the texture classification accuracy.

A Hybrid Proposed Fundus Image Enhancement Framework for Diabetic Retinopathy

Diabetic retinopathy (DR) is a complication of diabetes and is known as visual impairment, and is diagnosed in various ethnicities of the working-age population worldwide. Fundus angiography is a widely applicable modality used by ophthalmologists and computerized applications to detect DR-based clinical features such as microaneurysms (MAs), hemorrhages (HEMs), and exudates (EXs) for early screening of DR. Fundus images are usually acquired using funduscopic cameras in varied light conditions and angles. Therefore, these images are prone to non-uniform illumination, poor contrast, transmission error, low brightness, and noise problems. This paper presents a novel and real-time mechanism of fundus image enhancement used for early grading of diabetic retinopathy, macular degeneration, retinal neoplasms, and choroid disruptions. The proposed system is based on two folds: (i) An RGB fundus image is initially taken and converted into a color appearance module (called lightness and denoted as J) of the CIECAM02 color space model to obtain image information in grayscale with bright light. Afterwards, in step (ii), the achieved J component is processed using a nonlinear contrast enhancement approach to improve the textural and color features of the fundus image without any further extraction steps. To test and evaluate the strength of the proposed technique, several performance and quality parameters—namely peak signal-to-noise ratio (PSNR), contrast-to-noise ratio (CNR), entropy (content information), histograms (intensity variation), and a structure similarity index measure (SSIM)—were applied to 1240 fundus images comprised of two publicly available datasets, DRIVE and MESSIDOR. It was determined from the experiments that the proposed enhancement procedure outperformed histogram-based approaches in terms of contrast, sharpness of fundus features, and brightness. This further revealed that it can be a suitable preprocessing tool for segmentation and classification of DR-related features algorithms.

SmartCube: An Adaptive Data Management Architecture for the Real-Time Visualization of Spatiotemporal Datasets.

Interactive visualization and exploration of large spatiotemporal data sets is difficult without carefully-designed data pre-processing and management tools. We propose a novel architecture for spatiotemporal data management. The architecture can dynamically update itself based on user queries. Datasets is stored in a tree-like structure to support memory sharing among cuboids in a logical structure of data cubes. An update mechanism is designed to create or remove cuboids on it, according to the analysis of the user queries, with the consideration of memory size limitation. Data structure is dynamically optimized according to different user queries. During a query process, user queries are recorded to predict the performance increment of the new cuboid. The creation or deletion of a cuboid is determined by performance increment. Experiment results show that our prototype system deliveries good performance towards user queries on different spatiotemporal datasets, which costing small memory size with comparable performance compared with other state-of-the-art algorithms.