Processing Large Volumes Of Data Research Articles

A Dual Modular Redundancy Scheme for CPU–FPGA Platform-Based Systems

This paper presents a practical view of how to implement a dual modular redundancy (DMR) scheme in a central processing unit–field-programmable gate array (CPU–FPGA) heterogeneous platform-based system, which is also described. FPGAs can be valuable resources when determinism and fast response/acquisition rates are required, while processing large volumes of data. On the other side, CPUs are affordable options for most other processing tasks, especially less frequent tasks, such as data recording. A heterogeneous platform is herein proposed and aims to achieve a reliable, however cost-effective solution. Subsequently, this paper proposes a method to implement a DMR scheme for monitoring systems by means of self-monitoring schemes, health indicators, and a voter that is not a physical switch connected to the monitoring devices, but is software-implemented within the interface system after the data storage system. This proved to be an effective way to deal with disagreements between units when an even number of units are being used, thus not being possible to use a majority-based voting system. The implemented design was thoroughly tested, showing effectiveness in terms of redundancy with improved reliability.

Analysis of large volume data processing using clustering algorithms

The study of large dataset with velocity, variety and volume which is also known as Big data. When the dataset has limited number of clusters, low dimensions and small number of data points the existing traditional clustering algorithms can be used.. As we know this is the internet age, the data is growing very fast and existing clustering algorithms are not giving the acceptable results in terms of time complexity and spatial complexity. So there is a need to develop a new approach of applying clustering of large volume of data processing with low time and spatial complexity through MapReduce and Hadoop frame work applying to different clustering algorithms, k-means, Canopy clustering and proposed algorithm .The analysis shows that the large volume of data processing will take low time and spatial complexity when compared to small volume of data.

An approach to achieve high efficiency for large volume data processing using multiple clustering algorithms

Now a days data is increasing exponentially daily in terms of velocity, variety and volume which is also known as Big data. When the dataset has small number of dimensions, limited number of clusters and less number of data points the existing traditional clustering al- gorithms will give the expected results. As we know this is the Big Data age, with large volume of data sets through the traditional clus- tering algorithms we will not be able to get expected results. So there is a need to develop a new approach which gives better accuracy and computational time for large volume of data processing. The Proposed new System Architecture is a combination of canopy, Kmeans and RK sorting algorithm through Map Reduce Hadoop frame work platform. The analysis shows that the large volume of data processing will take less computational time and higher accuracy, and the RK sorting does not require swapping of elements and stack spaces.

Data Processing Method in Medium Range Coastal Radar Complex

The article presents the basic principles of design and development of integrated middle range Coastal Surveillance System (CSS) used for water surface lookout. It provides solutions for such missions as command and control of maritime forces, border monitoring and control, prevention of illegal activities such as piracy, smuggling, illegal immigration, illegal fishing, supporting search and rescue (SAR) operations, and creates a common situation awareness picture of the Naval Theatre. The system structure diagram is designed to solve computational overload problem when processing large volume of data received from radar stations. The measurement-level fusion algorithm is developed based on the JPDA framework, in which radar data received from a single or group of radars and AIS data is aggregated in a processing center. The servers and workstations make use of local area network (LAN), using standard Gigabit Ethernet technologies for local network communications. Acquisition, analysis, storage and distribution of target data is executed in servers, then the data is sent to automated operator stations (console), where functional operations for managing, identifying and displaying of target on digital situational map are performed.

Mutomvo: Mutation testing framework for simulated cloud and HPC environments

Many current applications provide high performance to process large volumes of data. These applications usually run in highly distributed environments, like cloud and HPC systems. Nevertheless, the large and complex architectures required for deploying these applications may not be available during the development phase. This limitation can be overcome by using simulation platforms to model a wide range of distributed system configurations and execute these applications in the modeled system. Usually, these applications are tested against a small number of test cases that are manually designed by the testers. It is desirable to have effective test suites in order to detect failures in the application models. In this paper we propose a mutation testing framework for detecting errors in distributed applications executed in simulated environments. The execution of a test suite against the set of mutated models allows to determine its effectiveness for detecting different errors. The proposal has been implemented in a tool called MuTomVo. In order to support the feasibility of the proposal, we have carried out a case study over three applications running in different distributed systems: a client/server model, intensive computation and scientific pipeline.

P Wave Arrival Picking and First‐Motion Polarity Determination With Deep Learning

AbstractDetermining earthquake hypocenters and focal mechanisms requires precisely measured P wave arrival times and first‐motion polarities. Automated algorithms for estimating these quantities have been less accurate than estimates by human experts, which are problematic for processing large data volumes. Here we train convolutional neural networks to measure both quantities, which learn directly from seismograms without the need for feature extraction. The networks are trained on 18.2 million manually picked seismograms for the Southern California region. Through cross validation on 1.2 million independent seismograms, the differences between the automated and manual picks have a standard deviation of 0.023 s. The polarities determined by the classifier have a precision of 95% when compared with analyst‐determined polarities. We show that the classifier picks more polarities overall than the analysts, without sacrificing quality, resulting in almost double the number of focal mechanisms. The remarkable precision of the trained networks indicates that they can perform as well, or better, than expert seismologists.

Adaptation of an Iterative PCA to a Manycore Architecture for Hyperspectral Image Processing

This paper presents a study of the adaptation of a Non-Linear Iterative Partial Least Squares (NIPALS) algorithm applied to Hyperspectral Imaging to a Massively Parallel Processor Array manycore architecture, which assembles 256 cores distributed over 16 clusters. This work aims at optimizing the internal communications of the platform to achieve real-time processing of large data volumes with limited computational resources and memory bandwidth. As hyperspectral images are composed of extensive volumes of spectral information, real-time requirements, which are upper-bounded by the image capture rate of the hyperspectral sensor, are a challenging objective. To address this issue, the image size is usually reduced prior to the processing phase, which is itself a computationally intensive task. Consequently, this paper proposes an analysis of the intrinsic parallelism and the data dependency within the NIPALS algorithm and its subsequent implementation on a manycore architecture. Furthermore, this implementation has been validated against three hyperspectral images extracted from both remote sensing and medical datasets. As a result, an average speedup of 17× has been achieved when compared to the sequential version. Finally, this approach has been compared with other state-of-the-art implementations, outperforming them in terms of performance.

Towards a Cloud Computing Paradigm for Big Data Analysis in Smart Cities

In this paper, we present a Big Data analysis paradigm related to smart cities using cloud computing infrastructures. The proposed architecture follows the MapReduce parallel model implemented using the Hadoop framework. We analyse two case studies: a quality-of-service assessment of public transportation system using historical bus location data, and a passenger-mobility estimation using ticket sales data from smartcards. Both case studies use real data from the transportation system of Montevideo, Uruguay. The experimental evaluation demonstrates that the proposed model allows processing large volumes of data efficiently.

Multi-granular electricity consumer load profiling for smart homes using a scalable big data algorithm

With rising electricity prices, there is a need to give consumers greater control over their energy consumption. It is anticipated that such informed consumers in control of their consumption patterns will contribute to reduced energy usage and thus a sustainable environment. Smart meter technology in smart homes provides real-time information to customers through devices such as in-home displays and web portals, and provide half-hourly consumption data to electricity distributors and retailers. Such data enables the profiling of consumers making it possible to understand different life styles and electricity usage behaviours to provide customised electricity billing. To obtain the anticipated benefit from such highly granular and high frequency data, it is essential to have big data technologies which can process such volumes of data in near real time. The research described in this paper focus on addressing the key requirements of large volume data processing and making use of the highly granular nature of the data. Adapting a new scalable algorithm introduced by the authors for big data processing, this work demonstrates the practicality of processing large volumes of data at multiple levels of granularity. The faster processing capacity makes it possible to continuously analyse consumption data at frequent intervals as they are collected and at a highly granular level thus providing a practical solution as a smart home application. The advantages of the technique is demonstrated using electricity consumption data for 10,000 households for a year from an Australian electricity retailer.

Enterprise 4.0 – the emerging digital transformed enterprise?

Information Systems are at the core of all businesses and cover almost all aspects of organizational life. The convergence of computer capabilities, such as the potential to process large volumes of data in a very short time, the growth in Internet usage and the increase in the ability to capture and leverage knowledge in digital format are, in this context, the main enablers of the so called digital transformation of organizations. While the technology enablers allow the production, sharing and management of information and knowledge within the organization between peers and other stakeholders, besides empowering the improvement and innovation of the organizational processes, they also require the updating of the supporting IS. Thus, using technologies in organizations within the context of Digital Transformation (DT) requires an exercise in understanding how to demonstrate their usefulness in relation to the creation, access and sharing of contents and IS improvements in a safe way. This paper provides a comprehensive view of a new context of labour faced within DT of organizations, which we term Enterprise 4.0 and which we propose to be implemented through the m_CSDIT framework, so as to improve the organizational well-being considering the collective intelligence and agility dimensions.

Fuzzy Join for Flexible Combining Big Data Lakes in Cyber-Physical Systems

Cyber-physical systems produce large amounts of data that are stored in domain-related data lakes in a variety of formats. By using the big data technologies that enable efficient data processing, the value of the data increases, as these technologies can turn the data into actionable information that influences important decision-making processes. However, a broader view of the operational environment, an investigated phenomena, and challenges related to them can frequently be obtained after combining data from many data sets located in various big data lakes. This requires contact points in both data lakes that must be flexibly joined because in many cases, data sets do not correspond to one another directly. In this paper, we show fuzzy join operation for flexible combining big data lakes. The fuzzy join transforms numerical values of common attributes of joined data sets into fuzzy sets and uses such a representation in the join operation. We propose two variants of the join operation that transforms crisp numerical values of joining attributes into: 1) fuzzy numbers and 2) linguistic terms. The fuzzy join operation is implemented and tested in the declarative U-SQL language that is used for scalable and parallel querying in big data lakes. The ideas presented here are exemplified by a distributed analysis of cardiac disease data on Microsoft Azure cloud. The results of the conducted experiments confirm that the fuzzy join can enrich data sets that are used in making critical decisions and, as a highly scalable cloud-based solution, can be successfully used in processing large volumes of data delivered by cyber-physical systems.

Predictive Modelling For Credit Card Fraud Detection Using Data Analytics

The finance and banking is very important sector in our present day generation, where almost every human has to deal with bank either physically or online [10]. The productivity and profitability of both public and private sector has tremendously increased because of banking information system. Nowadays most of E-commerce application system transactions are done through credit card and online net banking. These systems are vulnerable with new attacks and techniques at alarming rate. Fraud detection in banking is one of the vital aspects nowadays as finance is major sector in our life. As data is increasing in terms of Peta Bytes (PB) and to improve the performance of analytical server in model building, we have interface analytical framework with Hadoop which can read data efficiently and give to analytical server for fraud prediction. In this paper we have discussed a Big data analytical framework to process large volume of data and implemented various machine learning algorithms for fraud detection and observed their performance on benchmark dataset to detect frauds on real time basis there by giving low risk and high customer satisfaction.

MapReduce‐based parallel GEP algorithm for efficient function mining in big data applications

SummaryGene expression programming (GEP) algorithm is one of the most effective function mining algorithms in enabling the mathematical equation fitting for the input dataset. However, GEP algorithm encounters low efficiency issue in big data processing due to large overhead in its evolution when it handles the large‐scale data. In order to solve the issue, this paper presents two parallelized GEP algorithms using MapReduce. Based on data separation, the first algorithm aims at speeding up the large‐scale classification. However, it is lack of ability to output the mined equation explicitly. Therefore, based on the further improvements of the first algorithm, the second parallelized GEP algorithm aims at mining the equation efficiently and also outputs the equation explicitly and directly. The experimental results show that both algorithms are effective for processing large volume of data.

A distributed platform for big data analysis in smart cities: combining Intelligent Transportation Systems and socioeconomic data for Montevideo, Uruguay

This article proposes a platform for distributed big data analysis in the context of smart cities. Extracting useful mobility information from large volumes of data is crucial to improve decision-making processes in smart cities. This article introduces a framework for mobility analysis in smart cities combining Intelligent Transportation Systems and socioeconomic data for the city of Montevideo, Uruguay. The efficiency of the proposed system is analyzed over a distributed computing infrastructure, demonstrating that the system scales properly for processing large volumes of data for both off-line and on-line scenarios. Applications of the proposed platform and case studies using real data are presented, as examples of the valuable information that can be offered to both citizens and authorities. The proposed model for big data processing can also be extended to allow using other distributed (e.g. grid, cloud, fog, edge) computing infrastructures.

Canary: an atomic pipeline for clinical amplicon assays

BackgroundHigh throughput sequencing requires bioinformatics pipelines to process large volumes of data into meaningful variants that can be translated into a clinical report. These pipelines often suffer from a number of shortcomings: they lack robustness and have many components written in multiple languages, each with a variety of resource requirements. Pipeline components must be linked together with a workflow system to achieve the processing of FASTQ files through to a VCF file of variants. Crafting these pipelines requires considerable bioinformatics and IT skills beyond the reach of many clinical laboratories.ResultsHere we present Canary, a single program that can be run on a laptop, which takes FASTQ files from amplicon assays through to an annotated VCF file ready for clinical analysis. Canary can be installed and run with a single command using Docker containerization or run as a single JAR file on a wide range of platforms. Although it is a single utility, Canary performs all the functions present in more complex and unwieldy pipelines. All variants identified by Canary are 3′ shifted and represented in their most parsimonious form to provide a consistent nomenclature, irrespective of sequencing variation. Further, proximate in-phase variants are represented as a single HGVS ‘delins’ variant. This allows for correct nomenclature and consequences to be ascribed to complex multi-nucleotide polymorphisms (MNPs), which are otherwise difficult to represent and interpret. Variants can also be annotated with hundreds of attributes sourced from MyVariant.info to give up to date details on pathogenicity, population statistics and in-silico predictors.ConclusionsCanary has been used at the Peter MacCallum Cancer Centre in Melbourne for the last 2 years for the processing of clinical sequencing data. By encapsulating clinical features in a single, easily installed executable, Canary makes sequencing more accessible to all pathology laboratories.Canary is available for download as source or a Docker image at https://github.com/PapenfussLab/Canary under a GPL-3.0 License.

Big Data analytics and Computational Intelligence for Cyber–Physical Systems: Recent trends and state of the art applications

Big data is fuelling the digital revolution in an increasingly knowledge driven and connected society by offering big data analytics and computational intelligence based solutions to reduce the complexity and cognitive burden on accessing and processing large volumes of data. In this paper, we discuss the importance of big data analytics and computational intelligence techniques applied to data produced from the myriad of pervasively connected machines and personalized devices offering embedded and distributed information processing capabilities. We provide a comprehensive survey of computational intelligence techniques appropriate for the effective processing and analysis of big data. We discuss a number of exemplar application areas that generate big data and can hence benefit from its effective processing. State of the art research and novel applications in health-care, intelligent transportation and social network sentiment analysis, are presented and discussed in the context of Big data, Cyber–Physical Systems (CPS), and Computational Intelligence (CI). We present a data modelling methodology, which introduces a novel biologically inspired universal generative modelling approach called Hierarchical Spatial–Temporal State Machine (HSTSM). The HSTSM modelling approach incorporates a number of soft computing techniques such as: deep belief networks, auto-encoders, agglomerative hierarchical clustering and temporal sequence processing, in order to address the computational challenges arising from analysing and processing large volumes of diverse data to provide an effective big data analytics tool for diverse application areas. A conceptual cyber–physical architecture, which can accommodate and benefit from the proposed methodology, is further presented.

Strategic eco-design map of the complex products: toward visualisation of the design for environment

With growing sustainability and environmental concerns regarding the products, decision-makers and business managers need to integrate sustainability into business strategies and support it via systematic business processes and decision support tools. With the integration of the eco-efficiency attributes to the product development, more data should be analysed in engineering design. This integration also increases the complexity of the design process due to the large volume of data processing and diversity of the different attributes and features of the products. In this paper, we used the stock market metaphor to develop a visual data mining approach to the strategic eco-design assessment of the complex products. We presented a fresh framework using clustering and visualisation techniques to analyse the eco-efficiency profile of the different modules, components and parts of a complex product, and provide an efficient data exploration tool for decision-makers to facilitate processing of eco-design attributes, and strategic objectives at the same time. An illustrative example is provided to show the procedure of the application and the effectiveness of the proposed approach.

Colt

Most organizations are becoming increasingly data-driven, often processing data from many different sources to enable critical business operations. Beyond the well-addressed challenge of storing and processing large volumes of data, financial institutions in particular are increasingly subject to federal regulations requiring high levels of accountability for the accuracy and lineage of this data. For companies like GE Capital, which maintain data across a globally interconnected network of thousands of systems, it is becoming increasingly challenging to capture an accurate understanding of the data flowing between those systems. To address this problem, we designed and developed a concept lineage tool allowing organizational data flows to be modeled, visualized and interactively explored. This tool has novel features that allow a data flow network to be contextualized in terms of business-specific metadata such as theconcept,business, andproductfor which it applies. Key analysis features have been implemented, including the ability to trace the origination of particular datasets, and to discover all systems where data is found that meets some user-defined criteria. This tool has been readily adopted by users at GE Capital and in a short time has already become a business-critical application, with over 2,200 data systems and over 1,000 data flows captured.

A Compression Algorithm Design and Simulation for Processing Large Volumes of Data from Wireless Sensor Networks

A Compression Algorithm Design and Simulation for Processing Large Volumes of Data from Wireless Sensor Networks

A Review of Infrastructures to Process Big Multimedia Data

In the last years, the volume of information is growing faster than ever before, moving from small to huge, structured to unstructured datasets like text, image, audio and video. The purpose of processing the data is aimed to extract relevant information on trends, challenges and opportunities; all these studies with large volumes of data. The increase in the power of parallel computing enabled the use of Machine Learning (ML) techniques to take advantage of the processing capabilities offered by new architectures on large volumes of data. For this reason, it is necessary to find mechanisms that allow classify and organize them to facilitate to the users the extraction of the required information. The processing of these data requires the use of classification techniques that will be reviewed. This work analyzes different studies carried out on the use of ML for processing large volumes of data (Big Multimedia Data) and proposes a classification, using as criteria, the hardware infrastructures used in works of machine learning parallel approaches applied to large volumes of data.