Huge Amount Of Heterogeneous Data Research Articles

An Urban Intelligence Architecture for Heterogeneous Data and Application Integration, Deployment and Orchestration.

This paper describes a novel architecture that aims to create a template for the implementation of an IT platform, supporting the deployment and integration of the different digital twin subsystems that compose a complex urban intelligence system. In more detail, the proposed Smart City IT architecture has the following main purposes: (i) facilitating the deployment of the subsystems in a cloud environment; (ii) effectively storing, integrating, managing, and sharing the huge amount of heterogeneous data acquired and produced by each subsystem, using a data lake; (iii) supporting data exchange and sharing; (iv) managing and executing workflows, to automatically coordinate and run processes; and (v) to provide and visualize the required information. A prototype of the proposed IT solution was implemented leveraging open-source frameworks and technologies, to test its functionalities and performance. The results of the tests performed in real-world settings confirmed that the proposed architecture could efficiently and easily support the deployment and integration of heterogeneous subsystems, allowing them to share and integrate their data and to select, extract, and visualize the information required by a user, as well as promoting the integration with other external systems, and defining and executing workflows to orchestrate the various subsystems involved in complex analyses and processes.

An Exhaustive and Methodical Analysis of Literature Pertaining To Technigues for Managing Big Data within the Context of the Internet of Things

The merging of big data and the Internet of Things (IoT) has brought both exceptional difficulties and possibilities in data management. This study offers a thorough and methodical examination of the current literature on large data management approaches inside the Internet of Things framework. The study encompasses a broad spectrum of inquiry, ranging from fundamental notions to sophisticated approaches. The Internet of Things (IoT) is a powerful force that seeks to improve user experience and lifestyle. It incorporates several essential technologies including human-machine and machine-to machine communications, networking technologies, and sensor technologies. Fundamental to the success of the Internet of Things is the effective management of data transmitted through these technologies. This article explores the issues and challenges associated with data management in the context of the Internet of Things and examines different aspects of data, including its sources, collection processes, processing methods, and transmission devices. The article identifies and discusses problems arising from the need to process huge amounts of heterogeneous data in different systems. In relation to these issues, the logical and physical aspects of data management and communications networks are discussed. Additionally, the article takes an in-depth look at the data models used in IoT and explores data management, cleansing, and indexing techniques that take into account the unique characteristics of IoT data. The final sections of the paper comprehensively discuss the benefits and limitations associated with data management in IoT.

Industrial analytics – An overview

Abstract The digital transformation generates huge amounts of heterogeneous data across the industrial value chain, from simulation data in engineering, over sensor data in manufacturing to telemetry data on product use. Extracting insights from these data constitutes a critical success factor for industrial enterprises, e. g., to optimize processes and enhance product features. This is referred to as industrial analytics, i. e., data analytics for industrial value creation. Industrial analytics is an interdisciplinary subject area between data science and industrial engineering and is at the core of Industry 4.0. Yet, existing literature on industrial analytics is fragmented and specialized. To address this issue, this paper presents a holistic overview of the field of industrial analytics integrating both current research as well as industry experiences on real-world industrial analytics projects. We define key terms, describe typical use cases and discuss characteristics of industrial analytics. Moreover, we present a conceptual framework for industrial analytics that structures essential elements, e. g., data platforms and data roles. Finally, we conclude and highlight future research directions.

KaruBioNet: a network and discussion group for a better collaboration and structuring of bioinformatics in Guadeloupe (French West Indies).

Sequencing and other biological data are now more frequently available and at a lower price. Mutual tools and strategies are needed to analyze the huge amount of heterogeneous data generated by several research teams and devices. Bioinformatics represents a growing field in the scientific community globally. This multidisciplinary field provides a great amount of tools and methods that can be used to conduct scientific studies in a more strategic way. Coordinated actions and collaborations are needed to find more innovative and accurate methods for a better understanding of real-life data. A wide variety of organizations are contributing to KaruBioNet in Guadeloupe (French West Indies), a Caribbean archipelago. The purpose of this group is to foster collaboration and mutual aid among people from different disciplines using a 'one health' approach, for a better comprehension and surveillance of humans, plants or animals' health and diseases. The KaruBioNet network particularly aims to help researchers in their studies related to 'omics' data, but also more general aspects concerning biological data analysis. This transdisciplinary network is a platform for discussion, sharing, training and support between scientists interested in bioinformatics and related fields. Starting from a little archipelago in the Caribbean, we envision to facilitate exchange between other Caribbean partners in the future, knowing that the Caribbean is a region with non-negligible biodiversity which should be preserved and protected. Joining forces with other Caribbean countries or territories would strengthen scientific collaborative impact in the region. Information related to this network can be found at: http://www.pasteur-guadeloupe.fr/karubionet.html. Furthermore, a dedicated 'Galaxy KaruBioNet' platform is available at: http://calamar.univ-ag.fr/c3i/galaxy_karubionet.html. Availability and implementation Information about KaruBioNet is availabe at: http://www.pasteur-guadeloupe.fr/karubionet.html. dcouvin@pasteur-guadeloupe.fr. Supplementary data are available at Bioinformatics Advances online.

Healthcare analytics in big data: current trends and prospects

Healthcare analytics is a research area where many healthcare services operate on huge amounts of heterogeneous data. This leads to the generation of big healthcare-data. This paper performed a survey to answer six questions related to healthcare analytics: what are the different perspectives on the concepts of big healthcare-data? What major big data analytics approaches, tactics, tools, and strategies are being used in healthcare applications? What are the vital sources of big healthcare-data? What are the different architectures for modelling big healthcare-data? What are the leading terms that constitute the research field of healthcare analytics in big data and how rapidly is this research field expanding? Where is healthcare analytics in big data headed to the future? The analysis inferred interesting patterns, identified popular big data analytical techniques, tactics, tools, the vital sources of big healthcare-data, big healthcare-data processing architectures, growth patterns of the research field, and future trends.

Dimensionality Reduction Techniques for IoT Based Data

Background: Internet of Things (IoT) plays a vital role by connecting several heterogeneous devices seamlessly via the Internet through new services. Every second, the scale of IoT keeps on increasing in various sectors like smart home, smart city, health, smart transportation and so on. Therefore, IoT becomes the reason for the massive rise in the volume of data which is computationally difficult to work out on such a huge amount of heterogeneous data. This high dimensionality in data has become a challenge for data mining and machine learning. Hence, with respect to efficiency and effectiveness, dimensionality reduction techniques show the roadmap to resolve this issue by removing redundant, irrelevant and noisy data, making the learning process faster with respect to computation time and accuracy. Methods: In this study, we provide a broad overview on advanced dimensionality reduction techniques to facilitate selection of required features necessary for IoT based data analytics and for machine learning on the basis of criterion measure, training dataset and inspired by soft computation technology followed by significant challenges of dimensionality reduction techniques for IoT generated data that exists as scalability, streaming datasets and features, stability and sustainability. Results and Conclusion: In this survey, the various dimensionality reduction algorithms reviewed delivers the essential information in order to recommend the future prospect to resolve the current challenges in the use of dimensionality reduction techniques for IoT data. In addition, we highlight the comparative study of various methods and algorithms with respect to certain factors along with their pros and cons.

Text And Sentimental Analysis On Big Data

Big Data has become a very important part of all industries and organizations sectors nowadays. All sectors like energy, banking, retail, hardware, networking, etc all generate a huge amount of unstructured data which is processed and analyzed accurately in a structured form. Then the structured data can reveal very useful information for their business growth. Big Data helps in getting useful data from unstructured or heterogeneous data by analyzing them. Big data initially defined by the volume of a data set. Big data sets are generally huge, measuring tens of terabytes and sometimes crossing the sting of petabytes. Today, big data falls under three categories structured, unstructured, and semi-structured. The size of big data is improving in a fast phase from Terabytes to Exabytes Of data. Also, Big data requires techniques that help to integrate a huge amount of heterogeneous data and to process them. Data Analysis which is a big data process has its applications in various areas such as business processing, disease prevention, cybersecurity, and so on. Big data has three major issues such as data storage, data management, and information retrieval. Big data processing requires a particular setup of hardware and virtual machines to derive results. The processing is completed simultaneously to realize results as quickly as possible. These days big data processing techniques include Text mining and sentimental analysis. Text analytics is a very large field under which there are several techniques, models, methods for automatic and quantitative analysis of textual data. The purpose of this paper is to show how the text analysis and sentimental analysis process the unstructured data and how these techniques extract meaningful information and, thus make information available to the various data mining statistical and machine learning) algorithms.

Public Safety Knowledge Graph using Bilstm- Attention-CRF and Multi-Source Data

Public safety has always been a hot topic in the world, yet related data are stored in various forms of structures and have a huge amount. The poor characteristics of these huge amounts of heterogeneous data bring great difficulties to its application. For the purpose of applying these data conveniently and efficiently, knowledge graph was adopted. Public safety knowledge graph can help organize public safety data by an organized and properly utilized form, so that it can be used more optimally in public safety related retrieval, question and answer systems, etc. In this paper, unstructured, as well as structured public safety data were used for constructing, with the support of BiLSTM-Attention-CRF model. To be able to reveal this public safety knowledge graph in an understandable way, Neo4j database was used to store public safety knowledge graph.

Multimedia analysis platform for crime prevention and investigation

Nowadays,the use of digital technologies is promoting three main characteristics of information, i.e. the volume, the modality and the frequency. Due to the amount of information generated by tools and individuals, it has been identified a critical need for the Law Enforcement Agencies to exploit this information and carry out criminal investigations in an effective way. To respond to the increasing challenges of managing huge amounts of heterogeneous data generated at high frequency, the paper outlines a modular approach adopted for the processing of information gathered from different information sources, and the extraction of knowledge to assist criminal investigation. The proposed platform provides novel technologies and efficient components for processing multimedia information in a scalable and distributed way, allowing Law Enforcement Agencies to make the analysis and a multidimensional visualization of criminal information in a single and secure point.

WINFRA: A Web-Based Platform for Semantic Data Retrieval and Data Analytics

Given the huge amount of heterogeneous data stored in different locations, it needs to be federated and semantically interconnected for further use. This paper introduces WINFRA, a comprehensive open-access platform for semantic web data and advanced analytics based on natural language processing (NLP) and data mining techniques (e.g., association rules, clustering, classification based on associations). The system is designed to facilitate federated data analysis, knowledge discovery, information retrieval, and new techniques to deal with semantic web and knowledge graph representation. The processing step integrates data from multiple sources virtually by creating virtual databases. Afterwards, the developed RDF Generator is built to generate RDF files for different data sources, together with SPARQL queries, to support semantic data search and knowledge graph representation. Furthermore, some application cases are provided to demonstrate how it facilitates advanced data analytics over semantic data and showcase our proposed approach toward semantic association rules.

A generic knowledge management approach towards the development of a decision support system

ABSTRACT The fourth industrial revolution is increasingly implemented in most fields and categories of industry. The use of several highly developed technologies generates huge amounts of heterogeneous data and knowledge, hence the need to manage it in order to facilitate its reuse. This article proposes a global approach of knowledge management, from the analysis and development of the knowledge base structuring models, to the implementation phase using knowledge engineering tools. This research work is part of a global knowledge-based decision support framework that ensures several axes of decision support, this article deals with the diagnosis axis. It provides analysis and comprehension of the failures occurring during production. The approach has been implemented as a first demonstrator tested in a real case study: the aeronautical mechanical machining industry. This article details the issues of this implementation and its interest for the validation of the approach.

Application of Big Data Fusion Based on Cloud Storage in Green Transportation: An Application of Healthcare

In the 21st century, transportation brought great convenience to people, but at the same time, automobile transportation is the major factor causing greenhouse gas emissions and climate change. Movements of the world towards green environments, there is hike in use and production of electric vehicles (energy vehicles). However, with the continuous growth in the number of energy vehicles, it is necessary for the government to provide strong support in the construction of charging piles. Real-time and effective management has become a practical problem for the relevant departments which needs to be solved. This paper uses the information research method to fuse the huge amount of heterogeneous data generated by the charging pile resultant to the new energy electric vehicle in the vehicle network and introduces cloud computing as its storage module to facilitate the storage and related expansion of the big data. This paper proposes a system scheme of heterogeneous data fusion based on cloud computing for the acquisition, storage, and fusion of heterogeneous data in the vehicle network. After testing the results, it shows that the system is stable and effective in practical application, which can meet the design requirements of the system. What is the significance of analyzing big data of charging point? Considering from the supply side, obtaining the user’s charging behaviour data is helpful to build a digital map of the charging pile of new energy vehicles, connect the service information between the vehicle enterprises and the charging pile enterprises, and provide the most comprehensive and effective real-time charging information covering the widest range of vehicles, which can solve many problems of information asymmetry in the current charging information service.

Optimal resource selection framework for Internet-of-Things

The fundamental requirement for communication and computation across distinct application areas on Internet-of-Things is the resource discovery that demands appropriate reasoning for the optimal selection. With exponential growth of resources and their produced huge amount of heterogeneous data, various activities with respect to foraging and sense-making loops face challenges due to interoperability. Hence, interoperability emerges as a major bottleneck for the requirement. Therefore, to eliminate the challenge, the paper has proposed an “Optimal Resource Selection Framework for Internet-of-Things” that deals with the interoperability and ease the resource discovery and selection. The framework facilitates formation of semantic knowledge base as Shared Virtual Composite Ontology for capturing dynamic IoT heterogeneous data. Moreover, it supports optimal resource selection through the proposed algorithms, namely, Resource discovery Algorithm and Improved Firefly Algorithm. Both algorithms target coordination and optimization with Shared Ontology, respectively. The feasibility of the framework is checked against data collected from Sutlej river, Ludhiana, Punjab, India. The proposed framework is evaluated using benchmark functions with respect to metrics such as mean, standard deviation, processing and execution time. The obtained results are compared with the existing Nature-Inspired algorithms to confirm the efficiency of the proposed framework.

Integration of the ImageJ Ecosystem in the KNIME Analytics Platform.

Open-source software tools are often used for analysis of scientific image data due to their flexibility and transparency in dealing with rapidly evolving imaging technologies. The complex nature of image analysis problems frequently requires many tools to be used in conjunction, including image processing and analysis, data processing, machine learning and deep learning, statistical analysis of the results, visualization, correlation to heterogeneous but related data, and more. However, the development, and therefore application, of these computational tools is impeded by a lack of integration across platforms. Integration of tools goes beyond convenience, as it is impractical for one tool to anticipate and accommodate the current and future needs of every user. This problem is emphasized in the field of bioimage analysis, where various rapidly emerging methods are quickly being adopted by researchers. ImageJ is a popular open-source image analysis platform, with contributions from a global community resulting in hundreds of specialized routines for a wide array of scientific tasks. ImageJ's strength lies in its accessibility and extensibility, allowing researchers to easily improve the software to solve their image analysis tasks. However, ImageJ is not designed for development of complex end-to-end image analysis workflows. Scientists are often forced to create highly specialized and hard-to-reproduce scripts to orchestrate individual software fragments and cover the entire life-cycle of an analysis of an image dataset. KNIME Analytics Platform, a user-friendly data integration, analysis, and exploration workflow system, was designed to handle huge amounts of heterogeneous data in a platform-agnostic, computing environment and has been successful in meeting complex end-to-end demands in several communities, such as cheminformatics and mass spectrometry. Similar needs within the bioimage analysis community led to the creation of the KNIME Image Processing extension which integrates ImageJ into KNIME Analytics Platform, enabling researchers to develop reproducible and scalable workflows, integrating a diverse range of analysis tools. Here we present how users and developers alike can leverage the ImageJ ecosystem via the KNIME Image Processing extension to provide robust and extensible image analysis within KNIME workflows. We illustrate the benefits of this integration with examples, as well as representative scientific use cases.

Performance Evaluation of IoT Data Management Using MongoDB Versus MySQL Databases in Different Cloud Environments

The Internet of Things (IoT) introduces a new challenge for Database Management Systems (DBMS). In IoT, large numbers of sensors are used in daily lives. These sensors generate a huge amount of heterogeneous data that needs to be handled by the appropriate DBMS. The IoT has a challenge for the DBMS in evaluating how to store and manipulate a huge amount of heterogeneous data. DBMS can be categorized into two main types: The Relational DBMSs and the Non-relational DBMSs. This paper aims to provide a thorough comparative evaluation of two popular open-source DBMSs: MySQL as a Relational DBMS and MongoDB as a Non-relational DBMS. This comparison is based on evaluating the performance of inserting and retrieving a huge amount of IoT data and evaluating the performance of the two types of databases to work on resources with different specifications in cloud computing. This paper also proposes two prediction models and differentiates between them to estimate the response time in terms of the size of the database and the specifications of the cloud instance. These models help to select the appropriate DBMS to manage and store a certain size of data on an instance with particular specifications based on the estimated response time. The results indicate that MongoDB outperforms MySQL in terms of latency and the database size through increasing the amount of tested data. Moreover, MongoDB can save resources better than MySQL that needs resources with high capabilities to work with less performance.

Incorporating process and data heterogeneity in enterprise architecture: Extended AMA4EA in an international manufacturing company

The heterogeneity of production processes is a serious problem faced by international manufacturing companies. The transformation towards Industry 4.0 and the adoption of Internet-of-Things (IoT) have produced huge amounts of heterogeneous data. The production processes and data from sites across the world cannot be shared and compared at the enterprise level. Therefore, companies cannot improve their production processes and the current state-of-the-art of enterprise architecture (EA) cannot address this heterogeneity problem. To mitigate and address this heterogeneity problem, we extend the automated modelling with abstraction for EA (AMA4EA). We demonstrate the extension using the processes and data of an international manufacturing company in Denmark. The results show that the extended AMA4EA addresses the process heterogeneity problem by automatically creating EA models that relate and compare production processes from different sites. In addition, the extended AMA4EA extracts value from heterogeneous data and visualizes them in EA models. The extended AMA4EA exhibits a novel method in EA to incorporate process and data heterogeneity. This is a significant advance to EA research because it supports EA in modelling the different realities of companies. In addition, the extended AMA4EA demonstrates how production managers can jointly analyse production processes from different sites. As a result, managers can identify potential opportunities for improvement across production sites. Through EA models, they can access data and documentation stored on different enterprise systems. These contributions pave the foundation for understanding and improving the performance of heterogeneous production processes for international manufacturing companies.

Big Data Architectures: A Detailed and Application Oriented Analysis

Big Data refers to huge amounts of heterogeneous data from both traditional and new sources, growing at a higher rate than ever. Due to their high heterogeneity, it is a challenge to build systems to centrally process and analyze efficiently such data which are internal and external to organizations. A Big data architecture describes the blueprint of a system handling massive volume of data during its storage, processing, analysis and visualization. Several architectures belonging to different categories have been proposed by academia and industry but the field is still lacking benchmarks. Therefore, a detailed analysis of the characteristics of the existing architectures is required in order to ease the choice between architectures for specific use cases or industry requirements. The types of data sources, the hardware requirements, the maximum tolerable latency, the fitment to industry, the amount of data to be handled are some of the factors that need to be considered carefully before making the choice of an architecture of a Big Data system. However, the wrong choice of architecture can result in huge decline for a company reputation and business. This paper reviews the most prominent existing Big Data architectures, their advantages and shortcomings, their hardware requirements, their open source and proprietary software requirements and some of their real-world use cases catering to each industry. The purpose of this body of work is to equip Big Data architects with the necessary resources to make better informed choices to design optimal Big Data systems.

Application of a Weight of Evidence Approach for Monitoring Complex Environmental Scenarios: the Case-Study of Off-Shore Platforms

Multidisciplinary investigations based on integration of chemical and biological measurements, represent an added value to monitoring and management protocols, and their use is recommended by European Directives to evaluate environmental status of aquatic ecosystems. Nonetheless, assessing the overall significance of results from different studies is still a challenge. This work was aimed to present a quantitative Weight Of Evidence (WOE) model (Sediqualsoft) to integrate huge amounts of heterogeneous data and to validate this approach in complex monitoring scenarios. Using the case-study of off-shore platforms in the Adriatic Sea, procedures are presented to elaborate different typologies of data (lines of evidence, LOEs), including chemical characterization of sediments, bioavailability, biomarkers, ecotoxicological bioassays and benthic communities. These data are initially evaluated by logical flowcharts and mathematical algorithms, which provide specific hazard indices for each considered LOE, before their weighting and integration in an environmental risk index. The monitoring study selected for the WOE elaboration consisted on chemical analyses of trace metals, aliphatic hydrocarbons, polycyclic aromatic hydrocarbons carried out on 60 sediment samples; the same samples were characterized for the status of benthic communities; bioavailability of metals from sediments was assessed at laboratory conditions on the polychaete Hediste diversicolor, while bioaccumulation of inorganic and organic chemicals and biomarkers were measured in native and transplanted mussels; ecotoxicological properties of sediments were evaluated through a battery of bioassays determining algal growth of the diatom Phaeodactylum tricornutum, bioluminescence of the marine bacterium Vibrio fischeri, survival of the copepod Acartia tonsa and embryotoxicity of sea urchin Paracentrotus lividus. Almost 7000 analytical results were elaborated and summarized in specific hazard indices. The WOE integration of multiple typologies of data allowed more robust and weighted conclusions compared to the use of individual LOEs, highlighting the feasibility of this procedure for multidisciplinary monitoring and risk assessment approaches. On a practical side, the WOE evidences also suggested a revision of actual monitoring procedures. The proposed WOE model appears a useful tool to summarize large datasets of complex data in integrative indices, and to simplify the interpretation for stakeholders and decision makers, thus supporting a more comprehensive process of “site-oriented” management decisions.

City brain: practice of large‐scale artificial intelligence in the real world

A city is an aggregate of a huge amount of heterogeneous data. However, extracting meaningful values from that data remains a challenge. City Brain is an end-to-end system whose goal is to glean irreplaceable values from big city data, specifically from videos, with the assistance of rapidly evolving artificial intelligence technologies and fast-growing computing capacity. From cognition to optimisation, to decision-making, from search to prediction and ultimately, to intervention, City Brain improves the way to manage the city, as well as the way to live in it. In this study, the authors introduce current practices of the City Brain platform in a few cities in China, including what they can do to achieve the goal and make it a reality. Then they focus on the system overview and key technical details of each component of the City Brain system, from cognition to intervention. Lastly, they present a few deployment cases of City Brain in various cities in China.

A flexible data acquisition system for storing the interactions on mashup user interfaces

Nowadays, mashups applications are growing in popularity. They are accessible by cross-device applications, supporting multiple forms of interaction in cloud environments. In general, mashups manage a huge amount of heterogeneous data from different sources and handle different kinds of users. In this respect, mashup User Interfaces are becoming one of the most important pieces in many kinds of current management systems, such as for certain geographic or environmental information systems working on the Internet. In this type of systems, the user interface plays a particular role due to the huge variety of components or apps that the users need to manage at the same time. However, currently, there has been scant attention paid to the management of the user’s interaction with mashups interfaces. This goal involves the need of having important, well-constructed tools and methods conducting the data acquisition process for managing properly: (a) the interaction over the mashup user interfaces, at the front-end side; (b) the storage of the interaction in relational databases; and (c) well-supported microservices structures handled in the cloud. The fact of having valuable and flexible data acquisition processes encourages the deployment of others important issues of the interaction management, i.e., data searching, data mining, marketing, security, accessibility, usability or traceability of interaction data, among others. In this article, we present a flexible Data Acquisition System capable of capturing the human-computer interactions performed by users over mashup (User) Interfaces with the aim of storing them in a relational database. Firstly, the morphology of traditional mashup applications, their specifications and the relevant information that surrounds an interaction have been studied. Thereupon, a data acquisition system that stores user interaction on mashup based on such specifications was constructed. To achieve that purpose, an architecture of microservices was also designed in the cloud to detect, acquire, and collect the interactions performed over this kind of interfaces. The whole process is ready for acquiring internal data of the information system as well as context information and location awareness. To validate the data acquisition system, some tests on empirical case studies have been developed. Efficiency and effectiveness have also been determined by evaluating the performance of the acquisition system during different load tests. Finally, in order to ensure the software quality, a continuous integration strategy for software development and an easy management of the code have been used, facilitating the software maintenance alongside the microservice architecture, where functionalities are well encapsulated.