Big Data Community Research Articles

Application of Machine Learning in Stock Market Return Forecasting

With the development of our country's financial market, people pay more attention to the stock market return forecast, which can bring the investors a lot of income and also become a tool of risk management. Machine learning (ML) and artificial intelligence (AI) are becoming increasingly sophisticated and have achieved amazing results in many fields. Machine learning and deep learning algorithm models can process huge amounts of data, from which they can quickly process and analyze laws, many scholars try to apply various machine learning models to the financial field. This paper first expounds the theoretical basis of stock market return prediction, and summarizes the main literature in this field in recent years, from the traditional stock prediction model to the machine learning prediction model. Finally, the traditional stock prediction model ARIMA model and the deep learning LSTM neural network model are selected for comparative empirical research. The sample data are from the Tushare big data community, taking the new energy vehicle stock BYD (002594) as an example. The empirical results show that the prediction accuracy and stability of LSTM neural network are far higher than ARIMA model, and it will have broad application prospects in financial prediction and other fields in the future.

Big Data Quality: A Systematic Literature Review

Big data is the term for large data sets with larger, more varied and complex structures. Difficulties in collecting, analyzing and visualizing for further processes or results. This is the process of exploring large amounts of data to reveal hidden patterns and secrets. A correlation named as big data analytics. One of the most significant problems with Big Data is to gain knowledge through vast amounts of data. Usefulness of extracts Information depends heavily on data quality. In addition to importance, data has quality Recently considered by the big data community and there are none A comprehensive review was conducted in this area. Therefore, this is the main objective of the study to provide a systematic literature review for researchers interested in large Data quality topics. By carefully studying the selected papers, we propose a research tree that divides tasks based on process type, function and technology.

Data-Driven Monitoring and Predictive Maintenance for Engineering Structures: Technologies, Implementation Challenges, and Future Directions

Estimating engineering structures’ health conditions and predicting their future behaviors are fundamental problems for a city’s safe and efficient operations. Data-driven solutions estimate the health conditions using statistical models generated from measurement data. They have attracted growing interest recently because advances in information and communication technologies (ICT) have enabled numerous real-world measurement data, and the flourishing big data community has provided enormous state-of-the-art data analytics algorithms. Nevertheless, most existing studies remain in numerical simulation while neglecting their real-world implementation, restricting the extensive development of data-driven methods. In this survey, we provide a structural overview of the past decade’s data-driven structural health monitoring (SHM) systems and algorithms from the perspective of real-world implementation. Specifically, we cover various aspects of the design and implementation of monitoring systems, including sensing technologies, communication technologies, and processing software. In addition, we classify the used data sources and statistical models with fined details. Under the proposed taxonomy, their limitations and advantages are thoroughly discussed. Based on our insights into existing studies, we clarify two major implementation challenges: the digressive performance in the real-world environment and inefficient computing systems for real-time data analytics. Possible solutions are then proposed to mitigate those challenges and promote the implementation of data-driven methods. Finally, we raise our outlook on future trends and suggest promising directions for further investigation.

Interaction Design Based on Big Data Community Home Care Service Demand Levels

Most of the contemporary models for meeting the majority of the needs of middle-aged and elderly people are community-based, in-home care. Therefore, this paper designs an Interaction model that can meet the need for a rich spiritual and cultural life of the elderly at home. First, the questionnaire content of the Chinese Longitudinal Healthy Longevity Survey (CLHLS) sampling method was designed based on the content of community-based home care services. Then, using the CLHLS sampling method, the survey results of the home care group were collected to form a community of big data consisting of four types of home care service needs. Finally, the Interaction book model was designed based on the hierarchy of service needs obtained from Abraham Maslow’s hierarchy of needs classification method. The experimental results showed that the mean values of the target population’s ratings for the presentation and interface aesthetics of the Interaction mode were 4.34 and 4.19, respectively, the mean value for improving the learning effectiveness of the home-bound population was 4.57, and the mean value for their overall satisfaction was 4.31. It proves that the Interaction model is ideal for practice and can meet the learning needs of the elderly, at-home population from different service demand levels, thus solving the problem of education for the elderly.

Multihoming Big Data Network Using Blockchain-Based Query Optimization Scheme

In this paper, we have proposed the intimate environment of multiblockchain optimization algorithm using big data inquiry to mend the effectiveness of association query handling among numerous multihoming blockchains by implementing the big data system. This technique adds semantic evidences to the old-styled multiblockchain prototype and constructs a semantic model of multiblockchain that delivers a foundation for linking queries among the various blockchain multihoming system in big data. On the base of this model, distributed databases have index arrangement, and a linking index arrangement is proposed among the numerous blockchains, with several attributes linked to these blockchains employed to improve the efficacy of linking calculation. Besides, the communication cost is d for data communication. On this foundation, a multichain linking enquiry algorithm based on optimization is anticipated to progress the productivity of multiblockchain connection queries. To conclude, two genuine big data community sets of data are used to conduct experiments on. The associating index arrangement among blockchains is unchanging and is equated with the old-styled direct linking inquiry operation. The multiblockchain linking query method of optimization shortens the probe processing procedure. It acquires the query outcomes directly by retrieving the linking index, sinking the local calculating ability, system overhead, and illuminating query efficacy.

Technical influence on knowledge grid model to promote knowledge sharing in big data community

The significant role of big data in improving decision making, services and, productivity in organisations is not doubtable, therefore knowledge sharing is necessary to avoid redundancy and reduce time and cost. So a suitable infrastructure such as knowledge grid that can effectively capture, share, publish and manage knowledge resources to answer the questions from masses of data will be essential. This qualitative research aimed to analyse a knowledge grid model, from a technological view, in facilitating knowledge sharing among big data community. In this way, the study reviews the technology effects in the current knowledge sharing model and it also highlights the connection of knowledge grid, technology, and big data. It narrows the findings of knowledge grid components as node and link, the different layers in a knowledge grid's architecture and their condition from a technical view by analysing current knowledge grid models and their represented data. As a result, the paper clears the technological factors, their relationship, and effect in designing a knowledge grid model to promote knowledge sharing in the big data community.

Technical influence on knowledge grid model to promote knowledge sharing in big data community

Technical influence on knowledge grid model to promote knowledge sharing in big data community

A hybrid neuro-fuzzy technique to overcome clustering approach issues in big data

MapReduce is regarded as one of the major enabling methods in the big data community for handling the ever-increasing demands on computer resources imposed by enormous datasets. Currently, Google's MapReduce or its open-source equivalent Hadoop are well-known solutions for these types of applications. Several techniques are developed based on Hadoop such as apriori(M), HFUPM, apriori, EFUPM. The conventional algorithms of clustering schemes suffer from many challenging issues as clustering error, and computational complexity. To overcome the challenges of existing techniques we develop a novel approach to data pre-processing to extract the significant information. After pre-processing the data is processed applying neuro fuzzy-based scheduler to obtain semantic relationship between user queries. The fuzzy logic considers query time, query length, query expiry, total queries, CPU usages, and task activity. The proposed approach is implemented using a Hadoop platform. The comparative study shows a significant improvement in MapReduce performance for huge dataset.

A tensor-network-based big data fusion framework for Cyber–Physical–Social Systems (CPSS)

Big data has been continuously generated from the rapidly developing of cloud/fog/edge computing, Internet of Things (IoT) and 5G technology. This big data not only brings great benefits and opportunities to human beings but also brings many risks and challenges. One major challenge is how to represent and treat higher-order and heterogeneous data from multi-sources. Tensors are emerging as powerful tools for representation and modeling of this data. Tensor decomposition can be used to extract potentially useful information from this data. Thus, it has attracted much attention from the big data community. The main target of this paper is to propose a novel data fusion framework to solve several main challenges of CPSS data applications, including CPSS big data representation, fusion, efficient computing, storage, robustness, and security issues. In this paper, we use many graphics to visualize complex tensor decomposition and transformation processes. The visualization may help the readers better understand tensor and tensor decomposition. It also provides a general guideline and a good starting point for those who are interested in tensor and tensor decomposition. Specifically, we first introduce the most extensively used matrix and tensor decomposition methods. Second, the current popular data fusion methods are reviewed and summarized. Third, we propose a novel tensor-network-conversion-based data fusion approach which can simultaneously analyze multiple matrices and multiple tensors. To better understand this approach, we give a brief review of tensor network. Fourth, based on the proposed approach, a novel CPSS big data fusion framework is proposed in this paper. Meanwhile, we verify it by a concise case study. Finally, some challenges and open problems of the proposed framework are discussed. The discussion also includes some exciting future research directions in the big data fusion field.

An Experimental Model of the Collect/Report Paradigm

In the Big Data community, the “Map/Reduce Paradigm” is one of the key enabling approaches for meeting the continuously increasing demands on computing resources imposed by massive data sets. Today it is implemented in many open source projects. The popularity of Map/Reduce is due to its high scalability, fault-tolerance, simplicity and independence from the programming language or the data storage system. At the same time, Map/Reduce faces a number of obstacles when dealing with Big Data. A possible solution of them may be the Collect/Report Paradigm (CRP) and Natural Language Addressing (NLA) approach. It is suitable for storing Big Data in large information bases located on different storage systems – from personal computers up to cloud servers. An experimental Model of the CRP is presented in this paper. An experimental implementation of the CRP to process and store data is outlined. The structures of the input and output data are in the form of RDF triplets. The ease of implementation of this model and the benefits of its use are discussed.

Bridging Storage Semantics Using Data Labels and Asynchronous I/O

In the era of data-intensive computing, large-scale applications, in both scientific and the BigData communities, demonstrate unique I/O requirements leading to a proliferation of different storage devices and software stacks, many of which have conflicting requirements. Further, new hardware technologies and system designs create a hierarchical composition that may be ideal for computational storage operations. In this article, we investigate how to support a wide variety of conflicting I/O workloads under a single storage system. We introduce the idea of a Label , a new data representation, and, we present LABIOS: a new, distributed, Label- based I/O system. LABIOS boosts I/O performance by up to 17× via asynchronous I/O, supports heterogeneous storage resources, offers storage elasticity, and promotes in situ analytics and software defined storage support via data provisioning. LABIOS demonstrates the effectiveness of storage bridging to support the convergence of HPC and BigData workloads on a single platform.

A novel algorithm for estimation of Twitter users location using public available information

Abstract Social media networks are an attractive and hot research area in the big data community because of their numerous active users. One of the most widely studied topics in social networks is the prediction from the public available data. Recently, researchers have successfully predicted many statistical and human properties from social media networks using different machine learning algorithms. In this paper, a new efficient and accurate algorithm is proposed to predict the country location of a Twitter user using his or her public information only. The proposed algorithm employs the public information of the Twitter user and that of his or her followers and friends to predict his or her location without using GPS data. A convenient data set of Twitter users is gathered and used to test our proposed algorithm using KNIME software. The proposed algorithm is compared with other state-of-the-art algorithms, and results showed that our proposed algorithm significantly outperforms other location detection algorithms by using Twitter users from different countries.

Using GORE in Big Data: A Systematic Mapping Study

Big Data has developed rapidly into a hot topic that generates great attention in academia, industry and governments around the world because of its ability to process large volumes of data at a reasonable speed. The real value of Big Data lies in the help to business decision making, so it requires appropriate methods and techniques to find the requirements of the system to be developed. However, the Big Data community has focused mainly on technologies to obtain information, and not on the goal of the business. The objective of this paper is to offer an overview of how some Goal Oriented Requirements Engineering (GORE) proposals are used to obtain the requirements of the Big Data system, serving as a starting point for future research. The methodology used is the Systematic Mapping. The goal models identified it is used as conceptual models for different situations: understanding the nature of business and analyzing the technical aspects for purposes of functional and non-functional requirements of Big Data system.

Biomolecular Databases and Subnetwork Identification Approaches of Interest to Big Data Community: An Expert Review.

Next-generation sequencing approaches and genome-wide studies have become essential for characterizing the mechanisms of human diseases. Consequently, many researchers have applied these approaches to discover the genetic/genomic causes of common complex and rare human diseases, generating multiomics big data that span the continuum of genomics, proteomics, metabolomics, and many other system science fields. Therefore, there is a significant and unmet need for biological databases and tools that enable and empower the researchers to analyze, integrate, and make sense of big data. There are currently large number of databases that offer different types of biological information. In particular, the integration of gene expression profiles and protein-protein interaction networks provides a deeper understanding of the complex multilayered molecular architecture of human diseases. Therefore, there has been a growing interest in developing methodologies that integrate and contextualize big data from molecular interaction networks to identify biomarkers of human diseases at a subnetwork resolution as well. In this expert review, we provide a comprehensive summary of most popular biomolecular databases for molecular interactions (e.g., Biological General Repository for Interaction Datasets, Kyoto Encyclopedia of Genes and Genomes and Search Tool for The Retrieval of Interacting Genes/Proteins), gene-disease associations (e.g., Online Mendelian Inheritance in Man, Disease-Gene Network, MalaCards), and population-specific databases (e.g., Human Genetic Variation Database), and describe some examples of their usage and potential applications. We also present the most recent subnetwork identification approaches and discuss their main advantages and limitations. As the field of data science continues to emerge, the present analysis offers a deeper and contextualized understanding of the available databases in molecular biomedicine.

User Influence on Knowledge Grid Model among Big Data Community to Promote Knowledge Sharing

User Influence on Knowledge Grid Model among Big Data Community to Promote Knowledge Sharing

On dimension reduction models for functional data

This contribution is part of the recent links between Functional Data and Big Data communities. A selected survey highlights how earlier ideas in high dimensional problems can be adapted in functional setting.

Biomedical Big Data Training Collaborative (BBDTC): An effort to bridge the talent gap in biomedical science and research

The BBDTC (https://biobigdata.ucsd.edu) is a community-oriented platform to encourage high-quality knowledge dissemination with the aim of growing a well-informed biomedical big data community through collaborative efforts on training and education. The BBDTC is an e-learning platform that empowers the biomedical community to develop, launch and share open training materials. It deploys hands-on software training toolboxes through virtualization technologies such as Amazon EC2 and Virtualbox. The BBDTC facilitates migration of courses across other course management platforms. The framework encourages knowledge sharing and content personalization through the playlist functionality that enables unique learning experiences and accelerates information dissemination to a wider community.

A Collaborative Data Library for Testing Prognostic Models

A web-based data management system for use by researchers and industry around the world to access suitable datasets for testing prognostic models is developed. The value of the project is in the provision of, and access to, real-world data for asset failure prediction work. In practice, it is difficult for researchers to obtain data from industrial equipment. Industry datasets are rarely shared and hardly ever published. When such data is made available, very little meta-data about the underlying asset is provided. This restricts the number and type of models that can be applied.The solution is a data management system for three groups: researchers needing datasets, industry and academics with datasets. This paper identifies the data being sought, the system requirements and architecture, and discusses how the design is being implemented using an Agile development approach. Crucially, meta-data is stored in the database and accessed using a secure web-based front-end so as to maximize the available information, whilst obfuscating any corporate-sensitive material. The success of this prognostics data library depends on the support of the prognostic community to contribute and use the data; similar projects have been successful in the Machine Learning and Big Data communities.

Geometrical and topological approaches to Big Data

Modern data science uses topological methods to find the structural features of data sets before further supervised or unsupervised analysis. Geometry and topology are very natural tools for analysing massive amounts of data since geometry can be regarded as the study of distance functions. Mathematical formalism, which has been developed for incorporating geometric and topological techniques, deals with point cloud data sets, i.e. finite sets of points. It then adapts tools from the various branches of geometry and topology for the study of point cloud data sets. The point clouds are finite samples taken from a geometric object, perhaps with noise. Topology provides a formal language for qualitative mathematics, whereas geometry is mainly quantitative. Thus, in topology, we study the relationships of proximity or nearness, without using distances. A map between topological spaces is called continuous if it preserves the nearness structures. Geometrical and topological methods are tools allowing us to analyse highly complex data. These methods create a summary or compressed representation of all of the data features to help to rapidly uncover particular patterns and relationships in data. The idea of constructing summaries of entire domains of attributes involves understanding the relationship between topological and geometric objects constructed from data using various features.A common thread in various approaches for noise removal, model reduction, feasibility reconstruction, and blind source separation, is to replace the original data with a lower dimensional approximate representation obtained via a matrix or multi-directional array factorization or decomposition. Besides those transformations, a significant challenge of feature summarization or subset selection methods for Big Data will be considered by focusing on scalable feature selection. Lower dimensional approximate representation is used for Big Data visualization.The cross-field between topology and Big Data will bring huge opportunities, as well as challenges, to Big Data communities. This survey aims at bringing together state-of-the-art research results on geometrical and topological methods for Big Data.

The RADAR Project—A Service for Research Data Archival and Publication

The aim of the RADAR (Research Data Repository) project is to set up and establish an infrastructure that facilitates research data management: the infrastructure will allow researchers to store, manage, annotate, cite, curate, search and find scientific data in a digital platform available at any time that can be used by multiple (specialized) disciplines. While appropriate and innovative preservation strategies and systems are in place for the big data communities (e.g., environmental sciences, space, and climate), the stewardship for many other disciplines, often called the “long tail research domains”, is uncertain. Funded by the German Research Foundation (DFG), the RADAR collaboration project develops a service oriented infrastructure for the preservation, publication and traceability of (independent) research data. The key aspect of RADAR is the implementation of a two-stage business model for data preservation and publication: clients may preserve research results for up to 15 years and assign well-graded access rights, or to publish data with a DOI assignment for an unlimited period of time. Potential clients include libraries, research institutions, publishers and open platforms that desire an adaptable digital infrastructure to archive and publish data according to their institutional requirements and workflows.