Size Of Big Data Research Articles

Cross-species Conservation of context-specific networks.

BackgroundMany large data compendia on context-specific high-throughput genomic and regulatory data have been made available by international research consortia such as ENCODE, TCGA, and Epigenomics Roadmap. The use of these resources is impaired by the sheer size of the available big data and big metadata. Many of these context-specific data can be modeled as data derived regulatory networks (DDRNs) representing the complex and complicated interactions between transcription factors and target genes. These DDRNs are useful for the understanding of regulatory mechanisms and helpful for interpreting biomedical data.ResultsThe Cross-species Conservation framework (CroCo) provides a network-oriented view on the ENCODE regulatory data (CroCo network repository), convenient ways to access and browse networks and metadata, and a method to combine networks across compendia, experimental techniques, and species (CroCo tool suite). DDRNs can be combined with additional information and networks derived from the literature, curated resources, and computational predictions in order to enable detailed exploration and cross checking of regulatory interactions. Applications of the CroCo framework range from simple evidence look-up for user-defined regulatory interactions to the identification of conserved sub-networks in diverse cell-lines, conditions, and even species.ConclusionCroCo adds an intuitive unifying view on the data from the ENCODE projects via a comprehensive repository of derived context-specific regulatory networks and enables flexible cross-context, cross-species, and cross-compendia comparison via a basis set of analysis tools.The CroCo web-application and Cytoscape plug-in are freely available at: http://services.bio.ifi.lmu.de/croco-web. The web-page links to a detailed system description, a user guide, and tutorial videos presenting common use cases of the CroCo framework.

Big data for development: applications and techniques

With the explosion of social media sites and proliferation of digital computing devices and Internet access, massive amounts of public data is being generated on a daily basis. Efficient techniques/algorithms to analyze this massive amount of data can provide near real-time information about emerging trends and provide early warning in case of an imminent emergency (such as the outbreak of a viral disease). In addition, careful mining of these data can reveal many useful indicators of socioeconomic and political events, which can help in establishing effective public policies. The focus of this study is to review the application of big data analytics for the purpose of human development. The emerging ability to use big data techniques for development (BD4D) promises to revolutionalize healthcare, education, and agriculture; facilitate the alleviation of poverty; and help to deal with humanitarian crises and violent conflicts. Besides all the benefits, the large-scale deployment of BD4D is beset with several challenges due to the massive size, fast-changing and diverse nature of big data. The most pressing concerns relate to efficient data acquisition and sharing, establishing of context (e.g., geolocation and time) and veracity of a dataset, and ensuring appropriate privacy. In this study, we provide a review of existing BD4D work to study the impact of big data on the development of society. In addition to reviewing the important works, we also highlight important challenges and open issues.

Cloud Computing Security with Collaborating Encryption

The security of bigger data is the bottleneck in the encryption and decryption because of the big data size. The single encryption technique using one source is not adequate to accomplish the big data cloud computing security. This paper elaborates the working of cloud computing and the collaborating source security system for the Big Data security. A collaborating encryption technique framework is proposed in this paper to meet the futures' faster encryption requirements. The traditional information security system is not capable to provide the complete security during the cloud computing. The method described in this research comprises the parallel and distributed encryption system which gets the benefits the homomorphic encryption technique. The encryption facility during the mobile communication of object is tedious. Every cloud has its own security features and can be working in collaboration with the other cloud servers. Therefore, the parallel and distributed encryption facilities can be possible at every next door of other cloud without breaking the sequence of encryption process. The essential resources become the remote resources and the allocation of these resources can be managed at every cloud. Most of the time while working with the cloud computing is the availability of network and other resources. Providing the information security in the unavailability of resources during encryption and decryption is a difficult task. The collaboration encryption technique is a framework where, different clouds can work in parallel with the distributed processing. The security mechanism is improved by the homomorphic encryption.

Software-Defined Storage-Based Data Infrastructure Supportive of Hydroclimatology Simulation Containers: A Survey

Hydroclimatic research requires highly intensive resources in terms of computation and data to perform simulations. Setting up complex experiment environment and configurations to submit jobs in computational clusters as well as managing user’s limited storage spaces by transferring big size data into the secondary storage are complicated and time-consuming. As a possible answer to address such issues in hydroclimatic research, new technologies, software-defined storage and containers have been introduced. When the two technologies are combined to support hydroclimatic simulations, we discuss how the software-defined storage data infrastructure strengthens containers in terms of flexibility of data handling and storage scalability.

P-Codec: Parallel Compressed File Decompression Algorithm for Hadoop

The increasing amount and size of data being handled by data analytic applications running on Hadoop has created a need for faster data processing. One of the effective methods for handling big data sizes is compression. Data compression not only makes network I/O processing faster, but also provides better utilization of resources. However, this approach defeats one of Hadoop’s main purposes, which is the parallelism of map and reduce tasks. The number of map tasks created is determined by the size of the file, so by compressing a large file, the number of mappers is reduced which in turn decreases parallelism. Consequently, standard Hadoop takes longer times to process. In this paper, we propose the design and implementation of a Parallel Compressed File Decompressor (P-Codec) that improves the performance of Hadoop when processing compressed data. P-Codec includes two modules; the first module decompresses data upon retrieval by a data node during the phase of uploading the data to the Hadoop Distributed File System (HDFS). This process reduces the runtime of a job by removing the burden of decompression during the MapReduce phase. The second P-Codec module is a decompressed map task divider that increases parallelism by dynamically changing the map task split sizes based on the size of the final decompressed block. Our experimental results using five different MapReduce benchmarks show an average improvement of approximately 80% compared to standard Hadoop.

Patent Big Data Analysis using Fuzzy Learning

Big data has had an immense effect on most social and industrial fields. It has three main characteristics, namely volume, variety, and velocity. Volume refers to the tremendous size of big data, variety pertains to its heterogeneous sources including numbers, text, and figures, and velocity refers to the rapid speed of data growth. Patent documents follow the characteristics of big data. A patent contains various results about the developed technology such as title, abstract, citations, figures, and drawings. In general, the volume of patent documents related to a target technology is very large. Moreover, a massive number of patent applications are submitted to the patent offices in every country daily. Patent data are analyzed for R&D planning by many institutes and companies. In this study, we propose a methodology for technology analysis applied to patent big data. Additionally, we employ fuzzy learning based on the fuzzy rule-based system for patent big data analysis. We study the fuzzy models for classification, regression, and clustering and group the patents by the fuzzy classification model. Using a fuzzy regression model, we build a technological relationship between subtechnologies. Lastly, we develop a fuzzy clustering model for technology clustering. To illustrate how our research may be applied to a practical domain, we employ a case study using the patent documents related to the three-dimensional printing technology.

Big data and data reuse: a taxonomy of data reuse for balancing big data benefits and personal data protection

The emergence of Big Data has amounted to the complexity of the discussion on data reuse. The benefits of Big Data lie in the possibilities to discover novel trends, patterns and relationships by combining very large amounts of data from different sources. Current personal data protection requirements like data minimization and purpose specification are potentially inimical to Big Data as they limit the size and use of Big Data. Substantial loss of economic and social benefits of Big Data may be the result. In order to avoid this, the reuse of data could be encouraged. Data reuse, when done properly, may be both privacy preserving and economically and socially beneficial. In this paper, we provide a taxonomy of data reuse from both the data controller’s and the data subject’s perspective that may be useful to determine the extent to which data reuse should be allowed and under which conditions. From the data controller’s perspective we distinguish data recycling, data repurposing and data recontextualisation. From the data subject’s perspective, we distinguish data sharing and data portability. It is argued that forms of data reuse that stay close to the awareness and intentions of data subjects should be approached less tight (for instance, by assuming informed consent), whereas forms of data reuse that are ‘at a distance’, i.e., in which awareness and transparency may be lacking and data subject’s rights may prove more difficult to exercise, more restrictions and additional protection should be considered (for instance, by requiring explicit consent).

Big and high dimensional data in the digital world: statistical, computational and privacy challenges

Big data holds great promises and opportunities for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. Huge size and high dimensionality of big data introduce various statistical, computational, legal and privacy challenges. These challenges are distinguished and formidable, and require new statistical, computational and legal paradigm. In this paper, we present a brief non-technical overview of the salient features of big and high dimensional data; explain how these features impact on paradigm changes on statistical, computational and legal aspects. The paper ends with some concluding remarks about these new challenges and future research directions.

Differential Spatio-temporal Multiband Satellite Image Clustering using K-means Optimization With Reinforcement Programming

Deforestration is one of the crucial issues in Indonesia because now Indonesia has world's highest deforestation rate. In other hand, multispectral image delivers a great source of data for studying spatial and temporal changeability of the environmental such as deforestration area. This research present differential image processing methods for detecting nature change of deforestration. Our differential image processing algorithms extract and indicating area automatically. The feature of our proposed idea produce extracted information from multiband satellite image and calculate the area of deforestration by years with calculating data using temporal dataset. Yet, multiband satellite image consists of big data size that were difficult to be handled for segmentation. Commonly, K- Means clustering is considered to be a powerfull clustering algorithm because of its ability to clustering big data. However K-Means has sensitivity of its first generated centroids, which could lead into a bad performance. In this paper we propose a new approach to optimize K-Means clustering using Reinforcement Programming in order to clustering multispectral image. We build a new mechanism for generating initial centroids by implementing exploration and exploitation knowledge from Reinforcement Programming. This optimization will lead a better result for K-means data cluster. We select multispectral image from Landsat 7 in past ten years in Medawai, Borneo, Indonesia, and apply two segmentation areas consist of deforestration land and forest field. We made series of experiments and compared the experimental results of K-means using Reinforcement Programming as optimizing initiate centroid and normal K-means without optimization process.Keywords: Deforestration, Multispectral images, landsat, automatic clustering, K-means.

An Enriched Ciphering Method to Evaluate Performance of EEA2-algorithm for LTE Security

In order to address vulnerabilities in LTE security and the objective to strengthen the implementation of encryption and decryption algorithms in LTE, we proposed a method for integration of an algorithm with a dynamic matrix generation scheme of 16*16 elements during AES implementation. Implementing such algorithm eradicates the vulnerability that causes a hacker to access plain text. The vulnerability arises within the implementation of AES. This research explores Enriched Ciphering algorithm for number of scenarios based on function calls and time metrics that increases 3.9 times when there is a change in size of the plain text. We evaluated the proposed algorithm referred to as Enriched Ciphering Algorithm based on various use cases, including compression and decompression of cipher text that provides an extra layer of complexity to strengthen the security, and provides performance improvements of 13.9 percent when executing the Enriched Ciphering algorithm with big data size.

Data Outsourcing based on Secure Association Rule Mining Processes

Data mining is the process of extracting information from data warehousing applications. Data outsourcing is the major task in present days, for accessing services and other features of the database processing. But sometimes this process may achieve to split among various parties with recommended data items in analyzing of the data. Data security is one of the key processes in outsourcing data to various outside users. Traditionally Fast Distribution Mining algorithm was proposed for securing distributed data. This paper addresses a problem by secure association rules over partitioned data in both horizontal and vertical representation. A secure frequency developed algorithm is used for doing above process efficiently in partitioned data, which includes services of the data in outsourcing process. Frequent item sets are used to access services in outsourcing data in recent application development data mining. Our proposed work maintains efficient security over vertical and horizontal view of representation in secure mining applications. The result shows that algorithm timing is desirable for big size data for security considerations using association rule mining operations in real time application development.

Active Manifold Learning with Twitter Big Data

Abstract The data produced by Internet applications have increased substantially. Big data is a flaring field that deals with this deluge of data by using storage techniques, dedicated infrastructures and development frameworks for the parallelization of defined tasks and its consequent reduction. These solutions how- ever fall short in online and highly data demanding scenarios, since users expect swift feedback. Reduction techniques are efficiently used in big data online applications to improve classification problems. Reduction in big data usually falls in one of two main methods: (i) reduce the dimensionality by pruning or reformulating the feature set; (ii) reduce the sample size by choosing the most relevant examples. Both approaches have benefits, not only of time consumed to build a model, but eventually also performance-wise, usually by reducing overfitting and improving generalization capabilities. In this paper we investigate reduction techniques that tackle both dimensionality and size of big data. We propose a framework that combines a manifold learning approach to reduce dimensionality and an active learning SVM-based strategy to reduce the size of labeled sample. Results on Twitter data show the potential of the proposed active manifold learning approach.

Lowering Data Dimensionality in Big Data for the Benefit of Precision Agriculture

Abstract Predictive analytics can be used to make smarter decisions in farming by collecting real-time data on weather, soil and air quality, crop maturity and even equipment and labor costs and availability. This is known as precision agriculture. Big data is expected to play an important role in precision agriculture for managing real-time data analysis with massive streaming data. The data analysis efficiency and throughput would be a challenge with the massive increase in size of big data. The unstructured streaming data received from different agricultural sources would contain multiple dimensions and not the entire content is needed for performing analysis. The core data which is small but that alone enough to represent the entire content should be extracted. This paper explains how to systematically reduce the size of big data by applying a tensor based feature reduction model. The data decomposition and core value extraction is done with the help of IHOSVD algorithm. This way it reduces the overall file size by eliminating unwanted data dimensions. The time involved in data analysis and CPU usage will be significantly reduced when dimensionality reduced data is used in place of raw (unprocessed) data.

High-Performance Computing on Very Large-Scale Biological Data

In the past decade, due to the rapid growth of high-throughput instrumentations both of volume and complexity of biomedical data have been increasing dramatically. To process these big size data the high performance computing technologies become very important role. Recently, two interesting computing technologies, cloud computing and GPU computing, have been applied to solve very large-scale problem in bioinformatics. The paper by Hung, et al. entitled “Implementation of a Parallel Protein Structure Alignment Service on Cloud” proposed a protein structure alignment algorithm implemented by Hadoop to efficiently compare a interesting protein structure with a large database. Cloud-based bioinformatics applications can process data that are stored in many places. Therefore, they can make a significant breakthrough to achieve a high quality healthcare.

Challenges of Big Data Analysis.

Big Data bring new opportunities to modern society and challenges to data scientists. On one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data introduce unique computational and statistical challenges, including scalability and storage bottleneck, noise accumulation, spurious correlation, incidental endogeneity, and measurement errors. These challenges are distinguished and require new computational and statistical paradigm. This article gives overviews on the salient features of Big Data and how these features impact on paradigm change on statistical and computational methods as well as computing architectures. We also provide various new perspectives on the Big Data analysis and computation. In particular, we emphasize on the viability of the sparsest solution in high-confidence set and point out that exogeneous assumptions in most statistical methods for Big Data can not be validated due to incidental endogeneity. They can lead to wrong statistical inferences and consequently wrong scientific conclusions.

COFADMM: A Computational Features Selection with Alternating Direction Method of Multipliers

Due to the explosion in size and complexity of Big Data, it is increasingly important to be able to solve problems with very large number of features. Classical feature selection procedures involves combinatorial optimization, with computational time increasing exponentially with the number of features. During the last decade, penalized regression has emerged as an attractive alternative for regularization and high dimensional feature selection problems. Alternating Direction Method of Multipliers (ADMM) optimization is suited for distributed convex optimization and distributed computing for big data. The purpose of this paper is to propose a broader algorithm COFADMM which combines the strength of convex penalized techniques in feature selection for big data and the power of the ADMM for optimization. We show that combining the ADMM algorithm with COFADMM can provide a path of solutions efficiently and quickly. COFADMM is easy to use, is available in C, Matlab upon request from the corresponding author.

Big data summarization using semantic feture for IoT on cloud

Data management is a crucial aspect in the Internet of Things (IoT) on Cloud. Big data is about the processing and analysis of large data repositories on Cloud computing. Big document summarization method is an important technique for data management of IoT. Traditional document summarization methods are restricted to summarize suitable information from the exploding IoT big data on Cloud. This paper proposes a big data (i.e., documents, texts) summarization method using the extracted semantic feature which it is extracted by distributed parallel processing of NMF based cloud technique of Hadoop. The proposed method can well represent the inherent structure of big documents set using the semantic feature by the non-negative matrix factorization (NMF). In addition, it can summarize the big data size of document for IoT using the distributed parallel processing based on Hadoop. The experimental results demonstrate that the proposed method can summarize the big data document comparing with the single node of summarization methods.

Fuzzy Crime Investigation Framework for Tracking Data Theft based on USB Storage

Since the lives of the persons are on the edge after being convicted in digital crimes. The main goal of digital forensics is to extract accurate evidence which determines whether the convict is guilty or not. The recent challenge is due to the big size of data that the investigator may deal with. These data stored in unnoticeable tiny devices such as USB sticks which may lead to a muddled decision because of the tediousness of the investigation. Fortunately, in Windows Operating systems, all users' transactions are stored in a central point which is known as Windows Registry. It stores all hardware and software configurations, user activities, and transactions. Therefore, digital forensics based on Windows registry is considered as a hot research field. This paper presents a proposed framework for digital crime investigation based on Fuzzy logic. It helps the investigator in the decision making phase about the evidence. This deals with the extracted evidence from relevant Windows Registry keys. Also, tracking the usage of USB devices for data theft was presented. Finally the proposed framework was tested on a simulated case study.

Web Mining: Opinion and Feedback Analysis for Educational Institutions

Big amount of data available in the forms of reviews, opinions, feedbacks, remarks, comments, observations, clarifications, and explanations that require a robust mechanism to store, retrieve, analyze, and management. In this paper, we are proposing system that provides review or summary of above mention text data available on web for an educational institute. Due to big data size above system can be extended as real time recommendation system for Big Data analysis as future enhancement.

A Review: Oracle Big Data Aplliance

Big data, which refers to the data sets that are too big to be handled using the existing database management tools, are emerging in many important applications, such as Internet search, business informatics, social networks, social media, genomics, and meteorology. Big data presents a grand challenge for database and data analytics research. The challenges include capture, curation, storage, search, sharing, analysis, and visualization. Big data usually includes data sets with sizes beyond the ability of commonly-used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set. Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead massively parallel software running on tens, hundreds, or even thousands of servers. Keywords: Big Data, Data analytics, terabyte, petabyte, data set, server, social network, social media.