Data Pre-processing Stage Research Articles

Practical issues in signal processing for structural flexibility identification

Compared to ambient vibration testing, impact testing has the merit to extract not only structural modal parameters but also structural flexibility. Therefore, structural deflections under any static load can be predicted from the identified results of the impact test data. In this article, a signal processing procedure for structural flexibility identification is first presented. Especially, practical issues in applying the proposed procedure for structural flexibility identification are investigated, which include sensitivity analyses of three pre-defined parameters required in the data pre-processing stage to investigate how they affect the accuracy of the identified structural flexibility. Finally, multiple-reference impact test data of a three-span reinforced concrete T-beam bridge are simulated by the FE analysis, and they are used as a benchmark structure to investigate the practical issues in the proposed signal processing procedure for structural flexibility identification.

Research on the Data Pre-Processing in the Network Abnormal Intrusion Detection

The data pre-processing is a very important step in network abnormal intrusion detection, and directly affects the accuracy of the subsequent detection. In this paper, there are two issues in the network abnormal intrusion detection based on the hierarchical clustering so that some improvements should be made in the data pre-processing stage: first, there is the redundancy and attribute weight problem, each attribute with the weights should be attributing reduced with the use of rough set theory. Second, Aiming to the problem of the continuous data discretization in the rough set theory, an adaptive discrete algorithm for the data characteristics is proposed, and the algorithm determines the intervals of the discretization on the basis of the distribution of the sample attribute values. At last, the two improved methods are exper- imented and compared with the use of the existing discretization method. The experimental results demonstrate the effec- tiveness and accuracy of the algorithm.

3D Face Recognition based on Radon Transform, PCA, LDA using KNN and SVM

Biometrics (or biometric authentication) refers to the identification of humans by their characteristics or traits. Bio-metrics is used in computer science as a form of identification and access control. It is also used to identify individuals in groups that are under surveillance. Biometric identifiers are the distinctive, measurable characteristics used to label and describe individuals. Three dimensional (3D) human face recognition is emerging as a significant biometric technology. Research interest into 3D face recognition has increased during recent years due to the availability of improved 3D acquisition devices and processing algorithms. Three dimensional face recognition also helps to resolve some of the issues associated with two dimensional (2D) face recognition. In the previous research works, there are several methods for face recognition using range images that are limited to the data acquisition and pre-processing stage only. In the present paper, we have proposed a 3D face recognition algorithm which is based on Radon transform, Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). The Radon transform (RT) is a fundamental tool to normalize 3D range data. The PCA is used to reduce the dimensionality of feature space, and the LDA is used to optimize the features, which are finally used to recognize the faces. The experimentation has been done using three publicly available databases, namely, Bhosphorus, Texas and CASIA 3D face databases. The experimental results are shown that the proposed algorithm is efficient in terms of accuracy and detection time, in comparison with other methods based on PCA only and RT+PCA. It is observed that 40 Eigen faces of PCA and 5 LDA components lead to an average recognition rate of 99.20% using SVM classifier.

FEATURE EXTRACTION OF FRAUDULENT FINANCIAL REPORTING THROUGH UNSUPERVISED NEURAL NETWORKS

The objective of this study is to apply an unsupervised neural network tool to analyze fraudulent financial reporting (FFR) by extracting distinguishing features from samples of groups of companies and converting them into useful in- formation for FFR detection. This methodology can be used as a decision support tool to help build an FFR identification model or other financial fraud or financial distress scenarios. The three stages of the proposed quantitative analysis approach are as follows: the data-preprocessing stage; the clustering stage, which uses an unsupervised neural network tool known as a growing hierarchical self-organizing map (GHSOM) to cluster sample observations into subgroups with hierarchical re- lationships; and the feature-extraction stage, which uncovers common features of each subgroup via principle component analysis. This study uses the hierarchal topology mapping ability of a GHSOM to cluster financial data, and it adopts principal component analysis to determine common embedded features and fraud patterns. The results show that the proposed three-stage approach is helpful in revealing embedded features and fraud patterns, using a set of significant explana- tory financial indicators and the proportion of fraud. The revealed features can be used to distinguish distinctive groups.

The Research on Data Mining of Slim Life Mode Based on Cycle Behavior

In this study, data mining of slim life mode based on cycle behavior is propsed. The mining of the periodic behavior is divided into four stages. The first two stages is data pre-processing stage: Firstly, parsing stay point sequence from data sequence of the original location history. Here stay point represent the geographic area to a person’s stay for some time; Secondly, cluster mining the sequence of stay point, find out the significant places, such as company, supermarket, home location, etc. Thirdly, mining periodic on the significant places. Take a place as a reference point; abstract the original location history data into binary sequence by the location point in or out the place. Then, combination two popular signal processing method fast Fourier and autocorrelation find the periods of every place. Fourthly, mining the periodic behavior of the places with the same periods, in this article, first construct the periodic behavior probabilistic model, then use the method based on the hierarchical clustering to mining the periodic behavior between different places. At last, an example is introduced.

A procedure for the detection of anomalous input-output patterns

Data preprocessing is a main step in data mining because real data can be corrupted for different causes and high performance data mining systems require high quality data. When a database is used for training a neural network, a fuzzy system or a neuro-fuzzy system, a suitable data selection and pre-processing stage can be very useful in order to obtain a reliable result. For instance, when the final aim of a system trained through a supervised learning procedure is to approximate an existing functional relationship between input and output variables, the database that is exploited in the system training phase should not contain input-output patterns for which the same input or similar input sets are associated to very different values of the output variable. In this paper a procedure is proposed for detecting non-coherent associations between input and output patterns: by comparing two distance matrices associated to the input and output patterns, the elements of the available dataset, where similar values of input variables are associated to quite different output values can be pointed out. The efficiency of the proposed algorithm when pre-processing data coming from an industrial database is presented and discussed together with a statistical assessment of the obtained results.

Dataset Quality Assessment: An extension for analogy based effort estimation

Estimation by Analogy (EBA) is an increasingly active research method in the area of software engineering. The fundamental assumption of this method is that the similar projects in terms of attribute values will also be similar in terms of effort values. It is well recognized that the quality of software datasets has a considerable impact on the reliability and accuracy of such method. Therefore, if the software dataset does not satisfy the aforementioned assumption then it is not rather useful for EBA method. This paper presents a new method based on Kendall’s row-wise rank correlation that enables data quality evaluation and providing a data pre-processing stage for EBA. The proposed method provides sound statistical basis and justification for the process of data quality evaluation. Unlike Analogy-X, our method has the ability to deal with categorical attributes individually without the need for partitioning the dataset. Experimental results showed that the proposed method could form a useful extension for EBA as it enables: dataset quality evaluation, attribute selection and identifying abnormal observations.

Conjunction of SOM-based feature extraction method and hybrid wavelet-ANN approach for rainfall–runoff modeling

In rainfall–runoff modeling, the wavelet-ANN model, which includes a wavelet transform to capture multi-scale features of the process, as well as an artificial neural network (ANN) to predict the runoff discharge, is a beneficial approach. One of the essential steps in any ANN-based development process is determination of dominant input variables. This paper presents a two-stage procedure to model the rainfall–runoff process of the Delaney Creek and Payne Creek Basins, Florida, USA. The two-stage procedure includes data pre-processing and model building stages. In the data pre-processing stage, a wavelet transform is used to decompose the rainfall and runoff time series into several sub-series at different scales. Subsequently, independent sub-series are chosen via a self-organizing map (SOM). In the model building stage, selected sub-series are imposed as input data to a feed-forward neural network (FFNN) to forecast runoff discharge. To make a better interpretation of the model efficiency, the proposed model is compared with the Auto Regressive Integrated Moving Average with eXogenous input (ARIMAX) and with the ad hoc FFNN methods, without any data pre-processing. The results proved that the proposed model leads to better outcome especially in term of determination coefficient for detecting peak points (DCpeak).

A Secure Network Detection System against Noisy Unlabeled Data

Today, the Internet along with the corporate network plays a major role in creating and advancing new business avenues. With the ever increasing deployment and usage of gigabit networks, traditional network anomaly detection based intrusion detection systems have not scaled accordingly. Most, if not all, systems deployed assume the availability of complete and clean data for the purpose of intrusion detection. We contend that this assumption is not valid. Factors like noise in the audit data, mobility of the nodes, and the large amount of data generated by the network make it difficult to build a normal traffic profile of the network for the purpose of anomaly detection. From this perspective, the leitmotif of the research effort described in this dissertation is the design of a novel intrusion detection system that has the capability to detect intrusions with high accuracy even when complete audit data is not available. In this dissertation, we take a holistic approach to anomaly detection to address the threats posed by network based denial-of-service attacks by proposing improvements in every step of the intrusion detection process. At the data collection phase, we have implemented an adaptive sampling scheme that intelligently samples incoming network data to reduce the volume of traffic sampled, while maintaining the intrinsic characteristics of the network traffic. A Bloom filters based fast flow aggregation scheme is employed at the data pre-processing stage to further reduce the response time of the anomaly detection scheme. Lastly, this dissertation also proposes an expectation-maximization algorithm based anomaly detection scheme that uses the sampled audit data to detect intrusions in the incoming network traffic. KeywordsNetwork Security, Anomaly Detection, Intrusion detection, Clustering.

A Plausible Comprehensive Web Intelligent System for Investigation of Web User Behaviour Adaptable to Incremental Mining

With the continued increase in the usage of the World Wide Web (WWW) Web mining has been established as an important area of research. The WWW is a vast repository of unstructured information, in the form of interrelated files, distributed on numerous web servers over wide geographical regions. Web mining deals with the discovering and analyzing of useful information from the WWW. Web usage mining focuses on investigating the potential knowledge from the browsing patterns of users and to find the correlation between the pages on analysis. To proceed towards web intelligence, obviating the need for human interaction, need to incorporate and embed artificial intelligence into web tools. Before applying mining techniques, the data in the web log has to be pre-processed, integrated and transformed. The data pre-processing stage is the most important phase in the process of web mining and is critical and complex in successful extraction of useful data. The web log is non scalable, impractical and distributed in nature thus conventional data pre-processing techniques are proved to be not suitable as they assume that the data is static. Hence intelligent system is required for capable of pre processing weblog efficiently. Due to the incremental nature of the web log, it is necessary for web miners to use incremental mining techniques to extract the usage patterns and study the visiting characteristics of user, hence one can require a comprehensive algorithm which reduces the computing cost significantly. This paper introduces an Intelligent System IPS for pre-processing of web log, in addition a learning algorithm IFP-tree model is proposed for pattern recognition. The Intelligent Pre-processing System (IPS) can differentiate human user and web search engine accesses intelligently in less time, and discards search engine accesses. The present system reduces the error rate and improves significant learning performance of the algorithm. The Incremental Frequent Pattern Tree (IFP-Tree) is to suit for continuously growing web log, based on association rule mining with incremental technique. IFP-Tree is to store user-specific browsing path information in a condensed way. The algorithm is more efficient as it avoids the generation of candidates, reduces the number of scans and allows interactive mining with different supports. The experimental results that prove this claim are given in the present paper.