Data Processing Platform Research Articles

Development of Internet of Vehicle's Information System based on Cloud

As traditional mobile terminals have been unable to meet the needs of vehicle navigation, mobile office and etc, large amount of data from all kinds of sensors need to be acquired, stored and processed with the advent of Internet of vehicle. This paper puts forward vehicle network information processing platform based on Cloud, and designs the vehicle mobile terminal functions and interface functions. And the feasibility and reliability of data processing platform are verified by experiments.

Mining of Web Server Logs in a Distributed Cluster Using Big Data Technologies

Big Data is an emerging growing dataset beyond the ability of a traditional database tool. Hadoop rides the big data where the massive quantity of information is processed using cluster of commodity hardware. Web server logs are semi-structured files generated by the computer in large volume usually of flat text files. It is utilized efficiently by Mapreduce as it process one line at a time. This paper performs the session identification in log files using Hadoop in a distributed cluster. Apache Hadoop Mapreduce a data processing platform is used in pseudo distributed mode and in fully distributed mode. The framework effectively identifies the session utilized by the web surfer to recognize the unique users and pages accessed by the users. The identified session is analyzed in R to produce a statistical report based on total count of visit per day. The results are compared with non-hadoop approach a java environment, and it results in a better time efficiency, storage and processing speed of the proposed work.

Research of Big Data Processing Platform

This paper introduces the operational characteristics of the era of big data and the current era of big data challenges, and exhaustive research and design of big data analytics platform based on cloud computing, including big data analytics platform architecture system, big data analytics platform software architecture , big data analytics platform network architecture big data analysis platform unified program features and so on. The paper also analyzes the cloud computing platform for big data analysis program unified competitive advantage and development of business telecom operators play a certain role in the future.

Quality Measurement for Mobile M-ERP Applications

Mobile business boosting the popularization of the M-ERP(Mobile ERP) in the enterprise: The high penetration of mobile phones enables each employee to become an effective information carrier of the enterprise. The information application of the enterprise becomes more clear-cut. The mobile M-ERP pivots on the frequently changing key indexes of the enterprise operation, and takes the personalized and format-based data collection, analysis and processing platform as tool. It consolidates and unifies quickly diverse people, internal data, and external applications of the enterprise into an integrated whole, and provides real-time, key, and overall data reports to the management personnel. The recent wave of enterprise resource planning (M-ERP) systems adoption is a significant commitment of resource and may affect almost all business processes. M-ERP systems are integrated systems in that they promote cooperation among groups, teamwork, and process expertise and business knowledge. Firm that successfully implements an M-ERP system should raise revenues or decrease costs. The main purpose of this paper is to investigate the relationship among M-ERP system internal control, quality and performance in Korean firms. To investigate the relationship, the questionnaires were collected from 131 M-ERP – adopting Korean manufacturing companies. The following results were determined by verifying six hypotheses using LISREL. Internal control support perceived quality and quality support perceived performance. We expect that the results of the research can be used as the guidance of the implementation strategy of M-ERP systems. And these results provide important insights that complement extant research findings and also raise future research issues. Keyword: Mobile Enterprise Resource planning, Supply Chain Management, ERP Quality measurement, Mobile ERP Architecture

Software and hardware platform for developing laboratory experiment automation systems

The problem of laboratory experiment automation, encountered by almost every experimenter, was considered in the paper. A great range of solutions is proposed by the industry. However, these solutions are rather sophisticated, expensive and do not offer a comprehensive approach to the problem solution. Therefore, the issue of developing software and hardware platform for laboratory experiment automation with minimum time, force and means expenditures is topical nowadays. The ultimate objective of the paper is development and implementation of software and hardware platform for experiment control and data gathering and processing. The principle requirements for the platform include simplicity of design and use, portability and low cost. The platform developed consists of three main blocks: an interface of stepper motors, data gathering system and software for digital signal processing and creation of client application for PC. The interface of stepper motors includes control unit for stepper motors drivers, which is connected to PC via COM port, and stepper motors drivers themselves, which allow implementing a micro step up to 1/32 of nominal step of stepper motor. Any analog-to-digital converter can be used as measuring instrument. In this case, standard PC audio card was chosen as the most affordable and cheap device for measuring signals with admissible metrological characteristics. Software synchronous detector was applied for reducing the influence of background noise on the measured signal. It allows measuring signals with error not exceeding 1% and signal-to-noise ratio 10dB.

Next generation data analytics at IBM research

IBM Research has a rich history of innovation in information management with several revolutionary breakthroughs, including the invention of relational databases, advanced text analytics demonstrated by Watson, and the first data mining algorithms to name a few. IBM Research has been committed to contributing to the community via seminal papers, exemplified by several 10-year awards received by IBM researchers. This short abstract is intended as a quick tour of some of the current information management projects, and not meant to be an exhaustive list by any means. There has been many disruptive technological developments over the last decade. The emergence of cloud computing, and several large scale data processing platforms, advances in on-line social media, the explosion of data volumes, and the advances in hardware have all forced us to rethink the information management architectures and platforms. Today, enterprises are dealing with myriad of data modalities (unstructured, semi-structured, and structured) and very complex and diverse analytics on ever-expanding data volumes to drive business decisions. At IBM Research we are reinventing classical data management solutions to this high speed and volume demand (Section 2), building complex analytical platforms for end-to-end analysis of data (Section 3), building technologies for supporting large scale analytics (Section 4) and graph data (Section 5), as well as solutions that exploit new hardware (Section 6).

Control and Management System of LED Bulb Assembly Line Based on Matlab and WinCC

In order to realize the control and management system of LED bulb assembly line that can exchange and analyze real-time information of production line, first, a monitoring system based on WinCC was built; it can be used not only in on-line real-time monitoring but also in analyzing and diagnosing line’s trouble. Second, we used Matlab as the data processing platform, exchanged data by OPC technology, combined Matlab’s powerful calculate ability with power of WinCC real-time collect data, so the dynamic data exchange between Matlab and WinCC can be realized .In the end , BPNN was used to relate test results of LED Bulb to machine’s running parameters by learning data from host computer. In this way, parameters of machines can be adjusted in real-time and qualified products' rate can be improved.

Parallel Data Processing in the Cloud using Nephele

In recent years, Infrastructure-as-a-Service (IaaS) clouds have become increasingly popular as a flexible and inexpensive platform for ad-hoc parallel data processing. Major players in cloud computing have started to integrate frameworks for parallel data processing in their product portfolio, making it easy for customers to access these services and to deploy their programs. However, currently used processing frameworks have been designed for static, homogeneous cluster systems and do not support the new features which distinguish the cloud platform. In this paper discussion is being done on the research project Nephele. Nephele is the first data processing framework to explicitly exploit the dynamic resource allocation offered by today‟s IaaS clouds for both, task scheduling and execution. First performance results of Nephele are presented and its efficiency is compared with one of the well-known software, MapReduce. MapReduce is chosen for comparison since it is open source software and currently enjoys high popularity in the data processing community.

A platform for eXtreme Analytics

With the rapid increase in the volume of data that enterprises are producing, enterprises are adopting large-scale data processing platforms such as Hadoop® to store, manage, and run deep analytics to gain actionable insights from their big data. At IBM Research - Almaden, we have been helping enterprise customers build solutions exploiting data-intensive analytics. Our deep experience with actual users has led to an extensive understanding of the platform requirements needed to support these solutions, and our goal is to provide a powerful analytics platform, which we call eXtreme Analytics Platform (XAP), that can be used to create solutions for customer problems that have not been economically feasible to solve until now. XAP provides Jaql [i.e., JavaScript® Object Notation (JSON) query language, a scripting language to specify data flows, tools, and techniques to optimize the runtime execution of these flows], an improved task scheduler, connectors to data warehouses, and libraries for advanced analytics. Many of these technologies have been transferred to the IBM InfoSphere BigInsights™ product. In this paper, we describe the overall design principles and technology of XAP.

Big graph mining

How do we find patterns and anomalies in very large graphs with billions of nodes and edges? How to mine such big graphs efficiently? Big graphs are everywhere, ranging from social networks and mobile call networks to biological networks and the World Wide Web. Mining big graphs leads to many interesting applications including cyber security, fraud detection, Web search, recommendation, and many more. In this paper we describe Pegasus, a big graph mining system built on top of MapReduce, a modern distributed data processing platform. We introduce GIM-V, an important primitive that Pegasus uses for its algorithms to analyze structures of large graphs. We also introduce HEigen, a large scale eigensolver which is also a part of Pegasus. Both GIM-V and HEigen are highly optimized, achieving linear scale up on the number of machines and edges, and providing 9.2x and 76x faster performance than their naive counterparts, respectively. Using Pegasus, we analyze very large, real world graphs with billions of nodes and edges. Our findings include anomalous spikes in the connected component size distribution, the 7 degrees of separation in a Web graph, and anomalous adult advertisers in the who-follows-whom Twitter social network.

Approach for streamlining measurement of complex physiological phenotypes of upper airway collapsibility

The critical closing pressure (PCRIT), a quantitative assessment of upper airway collapsibility, is derived from pressure flow relationship during sleep. The analytic generation of the pressure flow relationships are non-standardized due to various regression models (linear, spline, median), breath characteristics (flow limited, non-flow limited) or known covariates (sleep stage, body position). We propose a GUI based PCRIT Analysis Software (PAS) to streamline PCRIT analysis and validate its reliability and accuracy compared to current analysis procedures. MethodsSeventeen subjects underwent a physiology sleep study in which the PCRIT was determined during NREM sleep. Data analysis was performed independently using three paradigms: (1) PAS (Igor Pro; median regression), (2) non-graphic statistics application (SAS; spline regression), and (3) manual spreadsheet calculations (Excel; linear regression). The reliability and accuracy of the PAS was examined through the agreement between each approach using Bland–Altman plots of the mean difference and within-individual variation using intra-class correlation (ICC). ResultsThere was no difference in the group mean values for PCRIT using the PAS (−1.7±0.7cm H2O) compared to spline regression (−1.6±0.7cm H2O; p=0.69) or linear regression (−2.1±0.7cm H2O; p=0.92). The Bland–Altman analysis did not demonstrate a systematic bias between the PAS and either approach. There was a mean difference of 0.39±0.2cm H2O between the PAS and linear regression approaches, with upper and lower limits of agreement of 1.81 and −1.02cm H2O, respectively. The PAS and spline analyses demonstrated an even smaller mean difference of −0.10±0.1cm H2O, with upper and lower limits of 0.90 and −1.08cm H2O, respectively. ConclusionPAS preserves the reliability and accuracy of the original PCRIT analysis methods while vastly improving their efficiency through graphic user interface and automation of analytic processes. Providing a standardized platform for physiologic data processing offers the ability to implement quality assurance and control procedures for multicenter studies as well as cost saving by improving the efficiency of complex repetitive tasks.

Get your chemistry right with KNIME

KNIME (Konstanz Information Miner, [1]) is a user-friendly and comprehensive open-source data integration, processing, analysis, and exploration platform. From day one, KNIME has been developed using rigorous software engineering practices and is used by professionals in both industry and academia in over 60 countries. In the presentation we will show some of the new features in KNIME 2.6 and give an outlook to KNIME 2.7. We also present recent developments in the KNIME Community Contributions [2], where research groups can easily provide their KNIME extensions to the community. We will focus on the freely available chemistry extensions - especially RDKit [3] - and demonstrate their usage in real-world workflows.

Design of Power Harmonic Analyzer Based on LabVIEW and HHT

For shortcomings and deficiencies of FFT and Wavelet Transform, Hilbert Huang Transform is used in power harmonic analysis, EMD method is applied to decompose these signals from the timescale of its characteristics to realize adaptive signal of the band, And then the characteristics of frequency and amplitude of each disturbance signal are extracted on the basis of Hilbert transform. Furthermore, the paper designed a data processing platform based on the integrated software LabVIEW and Matlab with NI PXI-1042 as the carrier. The system can analyze and process the power harmonic data in real-time with the calculation of HHT, compared with the traditional analyzer, with simple operation, processing ability, graphics output etc. Experiments have proved the stability, fast calculation capability, and high precision of the system.

Realization of an Embedded and Automated Performance Testing System for a MEMS Transducer

In this paper, we will discuss the design of an automatic testing system for some performances of micro-energy conversion devices which is manufactured by using MEMS processing techniques. The most important part of the system is based on so-called embedded technology is a microprocessor uploaded operating system, which is the core of the hardware and software platform for data acquisition, data processing and communication via serial ports. By utilizing of this test system, the generators electrical characteristics are evaluated by testing the stored electric quantity of the energy storage units with one or more conversion devices under different vibration test circumstances or different storage capacitors. Finally, we get the experimental results with load or without, which show the test system can measure some parameters precisely. Moreover, those data provide a reference for optimizing the energy conversion device and collecting weak energy of the devices rapidly, and we will make some relevant insights about reliability in the testing field.

A midas plugin to enable construction of reproducible web-based image processing pipelines

Image processing is an important quantitative technique for neuroscience researchers, but difficult for those who lack experience in the field. In this paper we present a web-based platform that allows an expert to create a brain image processing pipeline, enabling execution of that pipeline even by those biomedical researchers with limited image processing knowledge. These tools are implemented as a plugin for Midas, an open-source toolkit for creating web based scientific data storage and processing platforms. Using this plugin, an image processing expert can construct a pipeline, create a web-based User Interface, manage jobs, and visualize intermediate results. Pipelines are executed on a grid computing platform using BatchMake and HTCondor. This represents a new capability for biomedical researchers and offers an innovative platform for scientific collaboration. Current tools work well, but can be inaccessible for those lacking image processing expertise. Using this plugin, researchers in collaboration with image processing experts can create workflows with reasonable default settings and streamlined user interfaces, and data can be processed easily from a lab environment without the need for a powerful desktop computer. This platform allows simplified troubleshooting, centralized maintenance, and easy data sharing with collaborators. These capabilities enable reproducible science by sharing datasets and processing pipelines between collaborators. In this paper, we present a description of this innovative Midas plugin, along with results obtained from building and executing several ITK based image processing workflows for diffusion weighted MRI (DW MRI) of rodent brain images, as well as recommendations for building automated image processing pipelines. Although the particular image processing pipelines developed were focused on rodent brain MRI, the presented plugin can be used to support any executable or script-based pipeline.

EasyProt — An easy-to-use graphical platform for proteomics data analysis

High throughput protein identification and quantification analysis based on mass spectrometry are fundamental steps in most proteomics projects. Here, we present EasyProt (available at http://easyprot.unige.ch), a new platform for mass spectrometry data processing, protein identification, quantification and unexpected post-translational modification characterization. EasyProt provides a fully integrated graphical experience to perform a large part of the proteomic data analysis workflow. Our goal was to develop a software platform that would fulfill the needs of scientists in the field, while emphasizing ease-of-use for non-bioinformatician users. Protein identification is based on OLAV scoring schemes and protein quantification is implemented for both, isobaric labeling and label-free methods. Additional features are available, such as peak list processing, isotopic correction, spectra filtering, charge-state deconvolution and spectra merging. To illustrate the EasyProt platform, we present two identification and quantification workflows based on isobaric tagging and label-free methods.

Mucura: your personal file repository in the cloud

Large-scale distributed data processing platforms for scientific research such as the LHC computing grid include services for transporting, storing and processing massive amounts of data. They often address the data processing needs of a virtual organization but lack the convenience and flexibility required by individual users for their personal data storage needs. This paper presents the motivation, design and implementation status of Mucura, an open source software system for operating multi-tenant cloud-based storage services. The system is specifically intended for building file repositories for individual users, such as those of the scientific research communities, who use distributed computing infrastructures for processing and sharing data. It exposes the Amazon S3-compatible interface, supports both interactive and batch usage and is compatible with the X509 certificates-based authentication mechanism used by grid infrastructures. The system builds on top of distributed persistent key-value stores for storing user's data.

Design and implementation of real-time processing platform for movement error correction of hyperspectrual imaging

The approach that deals with compressed and packed image data transmitted from satellite to the ground is too slow for real-time application occasion, it also has huge image, multi-processing step and complexity recovery arithmetic synchronously, so it is urgent to build accurate and fast data processing platform for real-time processing. For the moment, the platform for data recovery and error correction is much less, the so-called successful platform may directly affect the effect of target detection and identification because of processing speed, precision, flexibility, configuration and upgrade. The platform we build is to set spatial modulation spectrometer as the research goal, We design and implement a hardware platform based on Xilinx Virtex-5 FPGA, It is combined with ISE IP soft-core resources which is configurable, high-precision and flexible by focusing on analyzing key aspects of the hardware platform. And the relevant test data were drawn, then a good way for spectrum recovery and error correction was explored.

From Databases to Big Data

There is a tremendous amount of buzz around the concept of data. In this article, the author discusses the origins of this trend, the relationship between big data and traditional databases and data processing platforms, and some of the new challenges that big data presents.

What's new in KNIME?

KNIME (Konstanz Information Miner, [1]) is a user-friendly and comprehensive open-source data integration, processing, analysis, and exploration platform. From day one, KNIME has been developed using rigorous software engineering practices and is used by professionals in both industry and academia in over 60 countries. In the presentation we will show some of the new features in KNIME 2.4 and give an outlook to KNIME 2.5. We also present the KNIME Community Contributions [2], where research groups can easily provide their KNIME extensions to the community.