Data Filtering Techniques Research Articles

Unsupervised anomaly detection of multivariate time series based on multi-standard fusion

With the wide application of the Internet of Things, industrial systems generate a large amount of multivariate time series data every day. Industrial data often contains a wide variety of anomalies. The existing anomaly detection methods have some shortcomings in dealing with large-scale, high-dimensional, and real-time industrial data. Particularly in the absence of prior knowledge, unsupervised learning methods are easily influenced by noise. Therefore, we propose a new unsupervised anomaly detection algorithm of multivariate time series based on multi-standard fusion (MSAD). Specifically, MSAD first classifies the data by analyzing the data density and sample spacing, thereby converting unsupervised anomaly detection into weakly supervised anomaly detection. Then, based on the characteristics of the data and deep learning technology, the degree of abnormality in the sample is comprehensively measured from four perspectives: holistic, local, time, and deep feature learning. At this stage, MSAD 's ability to learn edge features of data has been improved. Finally, based on MSAD, we design an anomaly detection method for industrial data streams in combination with data filtering techniques. The experimental results show that MSAD can detect anomalies more accurately and show stronger robustness to the influence of noise.

Measurement and quantification of low frequency noise emissions from utility-scale wind turbines

In wind turbine project permit hearings and other proceedings, concerns are often raised regarding low frequency noise (LFN). As the size of utility-scale wind turbines continues to grow, the concern over LFN may only intensify. This paper describes the measurement and quantification of low frequency noise emissions from utility-scale wind turbines, specifically in the 16, 31.5, and 63 Hertz octave bands. Accurate measurement of wind turbine noise emissions, particularly in the 16 Hertz octave band, is challenging due to interference from wind-induced noise. Topics covered include choice of windscreen, the effectiveness of shutdown tests, data filtering techniques, and the comparison of measured levels at multiple wind turbine sites to levels predicted using manufacturer's data and the ISO 9613-2:1996 method.

Energy Forecasting in Buildings Using Deep Neural Networks

Abstract The building sector is responsible for the largest portion of the total energy consumption in the United States. Conventional physics-based building energy models (BEMs) consider all of the building characteristics in order to accurately simulate their energy usage, requiring an extensive, complex, and costly process, particularly for existing buildings. In recent years, data-driven models have emerged as an additional path toward the prediction of energy consumption in buildings. The purpose of this work is to present a methodology for predicting the energy consumption of buildings using deep neural networks (NNs). Three machine learning algorithms, including a linear regression model, a multilayer perceptron NN, and a convolutional NN (CNN) model, are proposed to solve an energy consumption regression problem using outside dry-bulb temperature as the only input. To assess these methods, a building in Melbourne, FL, is considered and modeled in EnergyPlus. Ten years of data were used as inputs to the EnergyPlus model, and the energy consumption was calculated accordingly. The input to the machine learning algorithm (average daily dry-bulb temperature) and the output (daily total energy consumption) are used for training. Cross-validation was performed on the trained model using actual weather data measured onsite at the building location. The results showed that all three proposed machine learning algorithms were trained successfully and were able to solve the regression problem with high accuracy. However, the CNN model provided the best results when compared with the other two methods. This work also investigates different data filtering techniques that provide the best positive correlation between inputs and outputs for a similar type of problem. Results from this work aim to be used toward accurate energy forecasting that facilitates achieving higher energy efficiency in the building sector. The presented framework provides a readily simple model that allows accurate prediction of outputs when supplied with new inputs and can be used by a wide range of end users.

Recommendation System using Data Mining a Review

Recommendation systems are widely used for suggesting products, social media content, and web content to users. These systems utilize information filtering techniques to predict the preferences or ratings that a user would give to a particular item. This paper presents an overview of various data filtering techniques, algorithms, and application areas utilized in recommendation systems. It also includes a comparison between different algorithms used for recommendation systems.

On the role of user segmentation in recommender systems’ performances

Recommender systems are designed to suggest items for purchase or examination based on user preferences. In a variety of industries, such as e-commerce, tourism, social media, etc., these systems are frequently employed. However, they do have certain limits because of the vast amount of data they use. To enhance suggestion quality, artificial learning models have become prevalent in this field, along with data filtering techniques. In this study, as a data filtering technique, a user segmentation similar to well-known Recency-Frequency-Monetary is presented with the goal of examining how the use of user-segmented data affects the precision and caliber of recommendations produced by various algorithms. From the most fundamental popularity-based techniques to the most recent and sophisticated graph neural network approaches, the experiments are performed with eight different approaches. The results of the comparative analysis demonstrate that recommendation performance depends not only on user segmentation but also on other factors of the chosen method or data set. Nevertheless, user segmentation can improve the performance of some algorithms.

Geophysical surveying for the detection of ferrous-based objects: possibilities for depth estimative combining analytic signal and vertical integral of the anomalous magnetic field

The detection of buried clandestine objects challenges forensic and archeologic search group teams on varying terrains, and variable scales of research. Therefore, the study of controlled buried objects is useful for trainings in geophysical acquisition and processing. In this study, we applied ground survey data for testing the magnetic method at controlled geophysical sites for the location of ordinary objects and firearms. We used data filtering techniques in order to facilitate the location of magnetic targets. Also, we experienced the 3D inversion of analytic signal of the vertically integrated magnetic field (ASVI), for the location of targets in depth. As a result, the study determined the location of four magnetic targets, and a three-dimensional view was constructed from the estimated magnetic susceptibility. We concluded that modeling transformed magnetic data is an affordable technique for application in near-surface investigations. Also, this experiment exemplifies the relevance of magnetic methods for location of excavation sites on the basis of geophysical methods.

Diagnosing predated tags in telemetry survival studies of migratory fishes in river systems

BackgroundAcoustic telemetry is a powerful tool for studying fish behavior and survival that relies on the assumption that tag detection reflects the presence of live study subjects. This assumption is violated when tag signals continue to be recorded after consumption by predators. When such tag predation is possible, it is necessary for researchers to diagnose and remove these non-representative detections. Past studies have employed a variety of data-filtering techniques to address the issue, ranging from rule-based algorithms that rely on expert judgements of behavior and movement capabilities of study subjects and their predators to automated pattern-recognition techniques using multivariate analyses. We compare four approaches for flagging suspicious tracks or detection events: two rule-based expert-opinion approaches of differing complexity and two unsupervised pattern-recognition approaches with and without data from deliberately tagged predators. We compare alternative approaches by applying these four filters to a case study of survival estimation of acoustic-tagged juvenile Chinook salmon (Oncorhynchus tshawytscha) in the San Joaquin River, California, United States.ResultsFiltering approaches differed in the number and composition of tags suspected of being consumed by predators; the largest differences occurred between the two broad categories, rule-based versus pattern recognition. All methods required some investigator judgement and all flagged a small subset (5%) of suspicious tags that had exceptionally long residence times and evidence of upstream transitions; 27% of tags showed evidence of predation based on at least one filter. The complex rule-based filter deemed the most tags suspicious (21%) and the simpler pattern-recognition method the fewest (10%). Reach-specific survival estimates from the four filters were mostly within 2% of the unfiltered estimates, but differences up to 11% were observed.ConclusionsSensitivity of survival results to tag predation and predator filtering depends on the study setting, spatiotemporal scale of inference, and habitat use of predators. Choice of filtering technique depends on the data available and knowledge of the study system. We recommend that survival studies include clear documentation of filtering methods and report on robustness of results to the filtering approach selected.

Maximizing Multi‐Decadal Water Surface Elevation Estimates With Landsat Imagery and Elevation/Bathymetry Datasets

AbstractIdentifying patterns and trends in long‐term lake dynamics is essential to establish effective water management procedures and boost our understanding of inland water's role in the global water cycle. This research leverages Google Earth Engine to estimate multi‐decadal water surface elevations for 52 lakes and reservoirs with varying physical properties. Water surface elevation was estimated using the entire Landsat 4, 5, 7, and 8 Landsat Top‐of‐Atmosphere Tier‐1 Collection‐1 archive from August 1982 through December 2017 via shoreline boundary statistics extracted from the National Elevation Dataset merged with lake bathymetry. Image contamination was identified and removed to provide elevation estimates for images with varying levels of image contamination. To improve accuracy, data filtering techniques were identified which retained over 70% of images with detectable water boundaries producing 26 lakes with sub‐meter root‐mean‐square‐error accuracy and 40 lakes with sub‐meter mean‐absolute‐error‐accuracy using a general overall parameter model. Additionally, lake‐specific locally optimized models were also determined with 45 of the 52 lakes producing sub‐meter root‐mean‐square‐error accuracies and 49 with sub‐meter mean‐absolute‐errors with individual lake accuracy as low as 0.191 m RMSE CI95%[0.129, 0.243]. In general, individual lake accuracy is highly correlated with the mean slope of the surrounding terrain with low‐slope shorelines having greater accuracy than high‐slope shorelines. Seasonal patterns in estimate accuracy were also identified. This research extends our ability to track lake dynamics over long time periods to lakes lacking traditional in‐situ monitoring, enables rapid assessment of lake dynamics across large areas, and balances a need for both high‐accuracy measurements and maximum temporal resolution.

Managing vessel-generated underwater noise to reduce acoustic impacts to killer whales

Launched in 2014, the Enhancing Cetacean Habitat and Observation (ECHO) Program, led by the Vancouver Fraser Port Authority, is a regional, collaborative program designed to better understand and reduce the cumulative effects from commercial shipping activities on at-risk whales. The program advances a number of research projects and operational initiatives, with a key focus on reduction of vessel-generated underwater noise to support recovery of the endangered Southern Resident Killer Whale (SRKW). To reduce vessel noise in key killer whale habitat, the ECHO Program manages seasonal voluntary slowdowns of large commercial traffic and lateral displacement of inshore traffic, and advances projects to better characterize vessel-generated noise. Determining the efficacy of vessel noise reduction efforts requires careful consideration of confounding factors to assess acoustic reductions, and estimate the resulting benefits to SRKW. This presentation will highlight results of noise mitigation efforts conducted to date, and identify some of the technical challenges faced in the acoustic analysis, including decisions on data filtering techniques. Methods used by the ECHO Program for evaluating potential effects of noise exposure on killer whale behavior and foraging will also be discussed.

DATA FILTERING TECHNIQUES IN DECISION SUPPORT SYSTEMS

Background. The majority of modern dynamic processes in economy, finances, ecology, technologies and many other areas of studies exhibit short- and long-term nonlinear and nonstationary behavior. That is why it is required to create for their thorough analysis modern highly developed specialized instrumentation providing for appropriate preliminary statistical data processing, simulation state and parameter estimation and quality forecasting their evolution in time to be used in decision support systems (DSS). Objective. The purpose of the paper is to perform introductory analysis of some modern methods for filtering statistical and experimental data; to consider modern filtering techniques on the basis of probabilistic Bayesian approach, that provide a possibility for preparing the data to adequate simulation, computing high quality state and forecast estimates for dynamic systems in stochastic environment and availability of measurement errors. Methods. To implement modern data filtering techniques appropriate simulation and optimization procedures, probabilistic Bayesian methods of data analysis are utilized; simulation algorithms for parameter estimation, and criteria bases for analyzing quality of intermediate and final results in the frames of DSS are used. Results. A set of data filtering techniques is presented to be used together with the models describing formally selected processes dynamics. The methodology is considered for implementation of probabilistic Bayesian filter based upon modern statistical data analysis techniques including application of appropriate simulation procedures. Conclusions. Development of effective means for simulation, state estimation and forecasting dynamics of nonlinear nonstationary processes in various areas of human activities provides a possibility for high quality state and parameter estimation and compute short and middle term forecasts for their future evolution. The methods of optimal Kalman and probabilistic Bayesian filtering considered in the review provide a possibility for performing appropriate analysis of nonlinear nonstationary processes, compute forecasts and provide for managerial decision support on the basis of the forecast estimates.

Trading off accuracy for speed: Hedge funds' decision-making under uncertainty

Active managers operating quick portfolio adjustments, backed only by some rough estimates and loose predictions, might improve their market timing performances to benefit in turbulent times, but oversimplification can lead to an inability to profit in calm markets. By selecting accuracy levels upfront, through different data-filtering techniques, we expose a trade-off between prediction accuracy and reaction speed across different hedge funds' investment styles. Our empirical analysis shows that less accurate predictions can speed up reactions to unexpected changes in a large set of uncertainty and risk measures. We justify and complement these findings with a simulation exercise.

Data Fusion for the Prediction of Elemental Concentrations in Polymetallic Sulphide Ore Using Mid-Wave Infrared and Long-Wave Infrared Reflectance Data

The increasing availability of complex multivariate data yielded by sensor technologies permits qualitative and quantitative data analysis for material characterization. Multivariate data are hard to understand by visual inspection and intuition. Thus, data-driven models are required to derive study-specific insights from large datasets. In the present study, a partial least squares regression (PLSR) model was used for the prediction of elemental concentrations using the mineralogical techniques mid-wave infrared (MWIR) and long-wave infrared (LWIR) combined with data fusion approaches. In achieving the study objectives, the usability of the individual MWIR and LWIR datasets for the prediction of the concentration of elements in a polymetallic sulphide deposit was assessed, and the results were compared with the outputs of low- and mid-level data fusion methods. Prior to low-level data fusion implementation, data filtering techniques were applied to the MWIR and LWIR datasets. The pre-processed data were concatenated and a PLSR model was developed using the fused data. The mid-level data fusion was implemented by extracting features using principal component analysis (PCA) scores. As the models were applied to the MWIR, LWIR, and fused datasets, an improved prediction was achieved using the low-level data fusion approach. Overall, the acquired results indicate that the MWIR data can be used to reliably predict a combined Pb–Zn concentration, whereas LWIR data has a good correlation with the Fe concentration. The proposed approach could be extended for generating indicative element concentrations in polymetallic sulphide deposits in real-time using infrared reflectance data. Thus, it is beneficial in providing elemental concentration insights in mining operations.

Geophysical delineation of volcanics and intrusives offshore NW Australia using global analogues

The northwestern margin of Australia is a prolific hydrocarbon province, with large areas affected by volcanism and igneous intrusions, such as the outer Exmouth and Browse basins as well as parts of the Canning Basin. Volcanic features can easily be mistaken for potential hydrocarbon traps. Some have subsequently been drilled with disappointing and costly results. Proper delineation of volcanic features is therefore crucial in exploration efforts especially with current NW shelf exploration progressing in ever deeper water closer to the continent-ocean boundary. Most igneous rocks drilled on the northwest shelf consist of doleritic intrusions, basalt flows and volcaniclastics. Normally doleritic intrusions show a sharp contrast on seismic data due to their high acoustic impedance resulting in bright amplitudes on the seismic stack as seen on spectacular sill complexes on the Exmouth plateau as well as in the Canning basin. The high impedance contrast of volcanics also contributes to seismic imaging problems below the volcanic flows or intrusions, this becomes more evident when basalt flows thicken as evidenced on the North Atlantic margin. Similar potential thickening of basalt and volcaniclastic sequences is encountered on the NW edge of the Browse basin. This has important consequences for prospectivity in this region. Another common imaging problem is interbed multiple generation due to the high impedance of the volcanics. Other problems arise when volcanics have been exposed to significant weathering and subsequent burial resulting in lower impedance values, making it more difficult to distinguish them from clastic rocks. Large volcanic features such as igneous complexes and volcanoes, can mimic antiformal structures on seismic data, but can be easily de-risked by integrating gravity and magnetic data, especially by using data filtering techniques. More subtle volcanic features can be more difficult to distinguish and could be mistaken for reefs or potential hydrocarbon traps. Submerged volcanoes might also be prospective due to reefs growing on top of them as evidenced by examples from western India.

Chemometric Analysis of Mid-Wave Infrared Spectral Reflectance Data for Sulphide Ore Discrimination

Despite significant recent advancements in the sensor technologies, the use of sensors for raw material characterization in the mining industry remains limited. The aim of the present study was to assess the utility of applying the mid-wave infrared (MWIR) reflectance data acquired by the use of a handheld Fourier-transform infrared spectrometer (FTIR), combined with partial least squares-discriminant analysis (PLS-DA), for the characterization of a polymetallic sulphide ore deposit. In achieving the study objectives, focus was given to the MWIR portion of the FTIR dataset, as it is the least explored region of the infrared spectrum in mineral characterization studies. Three datasets—covering different wavelength ranges—were generated from the FTIR spectral data, namely the full FTIR range (2.5–15 µm), MWIR (2.5–7 µm) and long-wave infrared (LWIR: 7–15 µm), in order to investigate the associated information level of each defined wavelength region separately. Design of experiment was developed to determine the optimal data filtering techniques. Using the processed data and PLS-DA, a series of calibration and prediction models were developed for ore and waste materials separately. As the models applied to the MWIR data showed a successful classification rate of 86.3% for sulphide ore–waste discrimination, similarly using the full spectral FTIR dataset, a correct classification rate of 89.5% was achieved. This indicates that MWIR spectral range includes informative signals that are sufficient for classifying the material into ore or waste. The proposed approach could be extended for automating the sulphide ore–waste discrimination process, thus greatly benefiting marginally economical mining operations.

Exploration of DNA Microarrays Using Data Mining and a Taboo Search

In this article, we present a hybrid method based on data mining techniques and a taboo search applied in the selection and classification of DNA Microarray genes. The method is divided into two stages, in the first stage all non-relevant information in the database is eliminated using five data filtering techniques. With the subsets of genes obtained by this step, a new stage of gene selection is carried out using a taboo search, for the classification process of the selected genes, the SVM, LDA, KNN classifiers are used separately. The method has been implemented to obtain a small subset of high performance genes, the results obtained are compared with other methods reported in the literature, this method is applied in three databases of public domain.

Guided undersampling classification for automated radiation therapy quality assurance of prostate cancer treatment.

To test the use of well-studied and widely used classification methods alongside newly developed data-filtering techniques specifically designed for imbalanced-data classification in order to demonstrate proof of principle for an automated radiation therapy (RT) quality assurance process on prostate cancer treatment. A series of acceptable (majority class, n = 61) and erroneous (minority class, n = 12) RT plans as well as a disjoint set of acceptable plans used to develop features (n = 273) were used to develop a dataset for testing. A series of five widely used imbalanced-data classification algorithms were tested with a modularized guided undersampling procedure that includes ensemble-outlier filtering and normalized-cut sampling. Hybrid methods including either ensemble-outlier filtering or both filtering and normalized-cut sampling yielded the strongest performance in identifying unacceptable treatment plans. Specifically, five methods demonstrated superior performance in both area under the receiver operating characteristics curve and false positive rate when the true positive rate is equal to one. Furthermore, ensemble-outlier filtering significantly improved results in all but one hybrid method (p < 0.01). Finally, ensemble-outlier filtering methods identified four minority instances that were considered outliers in over 96% of cross-validation iterations. Such instances may be considered distinct planning errors and merit additional inspection, providing potential areas of improvement for the planning process. Traditional imbalanced-data classification methods combined with ensemble-outlier filtering and normalized-cut sampling provide a powerful framework for identifying erroneous RT treatment plans. The proposed methodology yielded strong classification performance and identified problematic instances with high accuracy.

A practical polynomial chaos Kalman filter implementation using nonlinear error projection on a reduced polynomial chaos expansion

SummaryThe polynomial chaos Kalman filter (PCKF) has been gaining popularity as a computationally efficient and robust alternative to sampling methods in sequential data assimilation settings. The PCKF's sampling free scheme and attractive structure to represent non‐Gaussian uncertainties makes it a promising approach for data filtering techniques in nonlinear and non‐Gaussian frameworks. However, the accuracy of PCKF is dependent on the dimension and order of the polynomial chaos expansion used to represent all sources of uncertainty in the system. Thus, when independent sources of errors, like process noise and time independent sensors' errors are incorporated in the system, the curse of dimensionality hinders the efficiency and the applicability of PCKF. This study sheds light on this issue and presents a practical framework to maintain an acceptable accuracy of PCKF without scarifying the computational efficiency of the filter. The robustness and efficiency of the presented implementation is demonstrated on 3 typical numerical examples to illustrate its ability to achieve considerable accuracy at a low computational tax.

Feasibility Studies of a New Event Selection Method to Improve Spatial Resolution of Compton Imaging for Medical Applications.

A Compton imaging method for medical applications that includes new energy determination and data filtering techniques has been tested using several point sources with known emission lines. Using a prototype Compton camera, a distance-of-closest approach technique has been employed to determine the initial energy of the incoming γs and to ensure the reconstructed source position is within an acceptable distance from the known γ source location. Further analysis is done by implementing a Compton line filtering technique, keeping only those interactions whose deposited energy in the first interaction matches the theoretical energy deposition predicted by the Compton equation. Using this new event filtering method, we see improvements in the full width at half maximum in the lateral profiles of γ point sources of up to 70% over standard Compton imaging methods, as well as achievable spatial resolutions in the reconstructed images of better than 2 mm. In addition, this new Compton imaging method was able to reconstruct an extended source of γ rays emitted during irradiation of a water tank with a clinical proton radiotherapy beam.

Data collection model in hybrid network for participatory sensing

Advances in mobile technology make most people have their own mobile devices which contain various sensors such as a smartphone. People produce their own personal data or collect surrounding environment data with their mobile devices at every moment. Recently, a broad spectrum of studies on Participatory Sensing, the concept of extracting new knowledge from a mass of data sent by participants, are conducted. Data collection method is one of the base technologies for Participatory Sensing, so networking and data filtering techniques for collecting a large number of data are the most interested research area. In this paper, we propose a data collection model in hybrid network for participatory sensing. The proposed model classifies data into two types and decides networking form and data filtering method based on the data type to decrease loads on data center and improve transmission speed.

Comparative Analysis of Information Extraction Techniques for Data Mining

Background/Objectives: This paper emphasizes the evolution of data processing adroitness to advanced data processing taxonomy from Mesolithic to recent years and a comparative study of prevailing tools/techniques which are useful for mainly the analysis of the bulky data. Methods/Statistical Analysis: There are various kinds of methods adapted by researchers for analysis of large amount of data. Each method varies on the basis of their different parameters and datasets according to their needs. These methods are implemented on HDFS, Mapreduce and Hadoop environment with integration of R tool. Some Methods are enhanced by the sentimental analysis through NLP which increase the performance of density analysis. Findings: The data or associated facts have been in existence right with the birth of human species. It commenced with manual illustration and gradually advanced through current state-of the art storage and processing. Big data involves novel techniques to manage information within limited run time. Big data is acutely beneficial in ventures growth, society incumbency and scientific research. The paper provides an overview of state of the art and focuses on the usage of conventional tools as well as advanced tools and techniques for effective information extraction. Applications/Improvements: To handle this prodigious data, there is a need to upgrade from the traditional data filtering techniques and adopt the new big data diagnostic tools.