Heterogeneous Distributed Data Research Articles

Dynamic MLML-tree based adaptive object detection using heterogeneous data distribution

We propose a robust object-detector ensemble by introducing a dynamic multi-layer multi-label (MLML)-tree–based adaptive deep learning framework. In many heterogeneous data distributions, deep attributes show latent hierarchical clustering properties. Object detector performance can be enhanced using the dynamic MLML-tree, which can adjust the ambiguities between inter-class nodes and variations between sub-class nodes. In the MLML-tree, dynamic multi-label (DML) trees are configured in two layers and adapt to using a sparse working dataset. First, coarse object clusters are built using an outlier-aware soft-clustering algorithm. Each coarse cluster is denoted by an inter-class node and is associated with an adaptive object detector in DML tree layer 1. It is built by recursively partitioning inter-class nodes until homogeneous object-class leaves are built. DML tree layer 2 is built for each object-class node, which is associated with a convolutional neural network detector, recursively. A novel sub-class can be learned automatically in DML tree layer 2 by applying semi-supervised learning. Extensive experiments show that the proposed method is superior to state-of-the-art techniques using PASCAL Visual Object Classes (VOC) 2007, VOC 2012, and the Microsoft Common Objects in Context (COCO) datasets.

Adaptation of the attribute dependency model for designing a heterogeneous digital oilfield data warehouse

The development of digital oilfield technologies has led us to the fact that there are virtually no production and management tasks left without any automated software solutions. However, the distribution, implementation, and use of individual IT-solutions often occur outside the rational automation strategy. At the same time, due to the complexity of the management object, due to the nature and composition of the tasks being solved, the lack of integration between the multitude of existing software solutions prevents the enterprise single information space formation. Thus, with the actual availability of critical information, the operational and strategic management tasks solution occurs without considering it. As a solution, the article considers an approach to creating a heterogeneous data warehouse based on an attribute dependency model that describes the relationship and interdependencies of data stored in distributed and heterogeneous sources. The article describes the interim results of a project to develop the architecture and software heterogeneous distributed data warehouse prototype for digital oilfield infrastructure.

Fine Grained Access Control on Information Retrieval with Collaborative Fusion and Active Feedback

Enterprise information retrieval is quite challenging than web based information retrieval due to retrieval from heterogeneous distributed data sources and need for higher accuracy in retrieval [17]. In our earlier work, collaborative fusion guided by active feedback is proposed for information retrieval in enterprise environment. Applying three dimensions of user similarity, user-document search history and document similarity, collaborative information fusion based retrieval was proposed in that work However the work did not consider enforcing fine grained access control on the retrieval which is very important requirement in enterprise environment. In this work, fine grained access controlled information retrieval on enterprise environment, with zero leakage assurance thorough direct or inference is proposed. Fine grained access control is ensured with help of CP-ABE. Concept masking is done with CP-ABE and unmasked when user satisfies the query access right criteria.

IndexMEED cases studies using "Omics" data with graph theory

Data produced within marine and terrestrial biodiversity research projects that evaluate and monitor Environmental Status, have a high potential for use by stakeholders involved in environmental management. However, environmental data, especially in ecology, are not readily accessible to various users. The specific scientific goals and the logics of project organization and information gathering lead to a decentralized data distribution. In such a heterogeneous system across different organizations and data formats, it is difficult to efficiently harmonize the outputs. Few tools are available to assist. For instance standards and specific protocols can be applied to interconnect databases. Such semantic approaches greatly increase data interoperability. This communication present the recent results and the consortium IndexMEED (Indexing for Mining Ecological and Environmental Data) activity that aims to build new approaches to investigate complex research questions, and support the emergence of new scientific hypotheses based on graph theory Auber et al. 2014). Current developments in data mining based on graphs, as well as the potential for relevant contributions to environmental research, particularly about strategic decision-making, and new ways of organizing data will be presented (David et al. 2015). In particular, the consortium makes decisions on how i) to analyze heterogeneous distributed data spread throughout different databases combining molecular and habitat characteristics data [3], ii) to create matches and incorporate some approximations, iii) to identify statistical relationships between observed data and the emergence of contextual patterns using a calculation library and distributed calculation center at the European level, iv) to encourage openness and sharing data while complying with the general principles of FAIR (Findable, Accessible, Interoperable, Re-usable and citable) in order to enhance data value and their utilization. IndexMEED participants are now exploring the ability of two scientific communities (ecology sensu lato and computer sciences) to work together, using several studies cases. The ECOSCOPE project aims to meet the need to access structured and complementary omics-datasets to better understand biodiversity state and its dynamics. Indeed, the ECOSCOPE case study targets to visualize, through the graph approach, links between datasets and databases from genetics to ecosystems. Another case study, displaying anthropology fossils and omics on the same graph, will also be presented. DEVOTES (DEVelopment Of innovative Tools for understanding marine biodiversity and assessing good Environmental Status) and CIGESMED (Coralligenous based Indicators to evaluate and monitor the Good Environmental Status of the MEDiterranean coastal water) European projects, conducted by IMBE, are focused on photo quadrats, cartography and omics data of the marine hard bottom in order to discover context patterns helpful to build decision support system building. Study case “65 Millions d’observateurs” French project is testing AskOmics to provide a graph-based querying interface using RDF (Resource Description Framework) and SPARQL technologies. Scientific questions can be resolved by the new data mining approaches that offer new ways to investigate heterogeneous environmental data with graph mining (Munoz et al. 2017). The uses of data from biodiversity research demonstrate the prototype functionalities (David et al. 2016) and introduce new perspectives to analyze environmental and societal responses including decision-making at large scale, both at the information system level and the observing system level than at the observed system level.

Near real-time estimation of ionosphere vertical total electron content from GNSS satellites using B-splines in a Kalman filter

Abstract. Although the number of terrestrial global navigation satellite system (GNSS) receivers supported by the International GNSS Service (IGS) is rapidly growing, the worldwide rather inhomogeneously distributed observation sites do not allow the generation of high-resolution global ionosphere products. Conversely, with the regionally enormous increase in highly precise GNSS data, the demands on (near) real-time ionosphere products, necessary in many applications such as navigation, are growing very fast. Consequently, many analysis centers accepted the responsibility of generating such products. In this regard, the primary objective of our work is to develop a near real-time processing framework for the estimation of the vertical total electron content (VTEC) of the ionosphere using proper models that are capable of a global representation adapted to the real data distribution. The global VTEC representation developed in this work is based on a series expansion in terms of compactly supported B-spline functions, which allow for an appropriate handling of the heterogeneous data distribution, including data gaps. The corresponding series coefficients and additional parameters such as differential code biases of the GNSS satellites and receivers constitute the set of unknown parameters. The Kalman filter (KF), as a popular recursive estimator, allows processing of the data immediately after acquisition and paves the way of sequential (near) real-time estimation of the unknown parameters. To exploit the advantages of the chosen data representation and the estimation procedure, the B-spline model is incorporated into the KF under the consideration of necessary constraints. Based on a preprocessing strategy, the developed approach utilizes hourly batches of GPS and GLONASS observations provided by the IGS data centers with a latency of 1 h in its current realization. Two methods for validation of the results are performed, namely the self consistency analysis and a comparison with Jason-2 altimetry data. The highly promising validation results allow the conclusion that under the investigated conditions our derived near real-time product is of the same accuracy level as the so-called final post-processed products provided by the IGS with a latency of several days or even weeks.

60 Research on the shortest route algorithm for medical rescue vehicles based on big data

Objectives Due to the randomness and uncertainty of city medical rescue vehicles, the scheduling of vehicles need to be dynamically optimized. Based on time window constraints and big data technology, we propose a method to reduce the running time of emergency vehicles and improve punctuality. Methods Big data technology is used to implement unified management and classify the heterogeneous distributed data. The symbolic method is then introduced for encoding and decoding based on the greedy method. In order to improve the convergence, the adaptive mechanism of quantum crossover probability and mutation probability is proposed. Results Experiment contrast analysis shows that the quantum genetic algorithm can quickly converge to the optimal solution compared with other existing intelligent algorithms. Big data technology can manage and analyze complex traffic conditions, which helps to improve emergency vehicle routing, save time, improve journey accuracy and ensure time delays do not affect timely treatment. Conclusions This method can meet the real-time requirements of the dynamic scheduling of medical rescue vehicles using the quantum genetic algorithm and big data technology. It can solve the problem of dynamic scheduling of medical rescue vehicles effectively. Acknowledgments This research was financially supported by Liaoning Social Planning Fund, China (Grant No. L16BGL008), National Natural Science Foundation, China (Grant No. 51579024), and Dr scientific research fund of Liaoning Province (Grant No. 201601244).

Towards a hybrid relational and XML benchmark for loosely-coupled distributed data sources

There are known benchmarks for the performance evaluation of relational and XML databases. However, there is an increasing demand for database applications that require access to heterogeneous loosely-coupled distributed data sources. This paper presents a hybrid benchmark based on TPC-H where the data sources are heterogeneous. Specifically, the paper describes the design of the relational and XML data sources as well as the query redesign in the LINQ query language, which supports queries over heterogeneous data sources. The results of a performance evaluation of the hybrid benchmark over various database products is included for untyped and typed XML with and without clearing the database cache.

Entity resolution based EM for integrating heterogeneous distributed probabilistic data

Distributed computing is linked and equated to the industrial revolution. Its transformational nature is, however, associated with significant instances in the form of internet of thing operations. Entity resolution (ER) is a problem of matching and resolving records that represent the same real world entity. This is a long-standing challenge in distributed databases and information retrieval as a statistic. In a centralized approach, the problem of ER has not been scaled well as large amount of data need to be sent to a central node. In this paper, we present an algorithm which deals with heterogeneous distributed probabilistic data (HDPD) and also reduces processing time in a distributed environment. We propose two different approaches. First, we explore this instance with a matching (identification) problem to integrate different data models with expectation–maximization (EM) algorithm. Second, we apply ER methodology for HDPD to achieve major performance in terms of response time to produce the outcome. We validate HDPD through experiments over a 100-node cluster that records significant performance improvements over naive approaches. This paper is expected to provide insights in to database organizations and new technological development for the growth of distributed environment.

The Key Technologies for Classification of Distributed Data Streams

With advances in data collection and generation technologies, environments that produce data streams is more and more. In recent years, the network application is further universal and the applications of a single data stream transfer toward a multi-node distributed data streams, such as sensor network, network monitoring, web log analysis and the credit card transaction data of multiple sites. These data is not only real-time, continuous and large scale, but also distributed. How to manage and analyze large dynamic datasets is an important subject that researchers are faced with. In view of the situation, it presented the formalization description of homogeneous and heterogeneous distributed data stream in this paper, analyzed advantages and disadvantages of the centralized stream processing architecture and distributed streaming processing architecture, discussed the recent progress in distributed data stream classification algorithm, summed up the problems and challenges faced by the distributed data stream mining, and possible future research directions.

Swept under the carpet: organic matter burial decreases global ocean biogeochemical model sensitivity to remineralization length scale

Abstract. Although of substantial importance for marine tracer distributions and eventually global carbon, oxygen, and nitrogen fluxes, the interaction between sinking and remineralization of organic matter, benthic fluxes and burial is not always represented consistently in global biogeochemical models. We here aim to investigate the relationships between these processes with a suite of global biogeochemical models, each simulated over millennia, and compared against observed distributions of pelagic tracers and benthic and pelagic fluxes. We concentrate on the representation of sediment–water interactions in common numerical models, and investigate their potential impact on simulated global sediment–water fluxes and nutrient and oxygen distributions. We find that model configurations with benthic burial simulate global oxygen well over a wide range of possible sinking flux parameterizations, making the model more robust with regard to uncertainties about the remineralization length scale. On a global scale, burial mostly affects oxygen in the meso- to bathypelagic zone. While all model types show an almost identical fit to observed pelagic particle flux, and the same sensitivity to particle sinking speed, comparison to observational estimates of benthic fluxes reveals a more complex pattern, but definite interpretation is not straightforward because of heterogeneous data distribution and methodology. Still, evaluating model results against observed pelagic and benthic fluxes of organic matter can complement model assessments based on more traditional tracers such as nutrients or oxygen. Based on a combined metric of dissolved tracers and biogeochemical fluxes, we here identify two model descriptions of burial as suitable candidates for further experiments and eventual model refinements.

A Novel Approach Towards Large Scale Cross-Media Retrieval

With the rapid development of Internet and multimedia technology, cross-media retrieval is concerned to retrieve all the related media objects with multi-modality by submitting a query media object. Unfortunately, the complexity and the heterogeneity of multi-modality have posed the following two major challenges for cross-media retrieval: 1) how to construct a unified and compact model for media objects with multi-modality, 2) how to improve the performance of retrieval for large scale cross-media database. In this paper, we propose a novel method which is dedicate to solving these issues to achieve effective and accurate cross-media retrieval. Firstly, a multi-modality semantic relationship graph (MSRG) is constructed using the semantic correlation amongst the media objects with multi-modality. Secondly, all the media objects in MSRG are mapped onto an isomorphic semantic space. Further, an efficient indexing MK-tree based on heterogeneous data distribution is proposed to manage the media objects within the semantic space and improve the performance of cross-media retrieval. Extensive experiments on real large scale cross-media datasets indicate that our proposal dramatically improves the accuracy and efficiency of cross-media retrieval, outperforming the existing methods significantly.

Integration of hydrological observations into a Spatial Data Infrastructure under a Sensor Web environment

Various sensors connected to the World Wide Web are used to obtain real-time hydrological observations. Thus, real-time management and utilization of such distributed in situ observations in the cyber-physical environment becomes possible. A Sensor Observation Service (SOS) chaining Web Feature Service (WFS) method is proposed to integrate geographical reference observation data collected by a hydrological Sensor Web into a virtual globe. This method hides the complexity of a series of information and service models in the Sensor Web realm to enable the integration of heterogeneous distributed hydrological data sources into a Spatial Data Infrastructure (SDI). The core components – a dynamic schema transformer and automatic information extractor – were designed and implemented. The SOS schema is matched to WFS schema that uses the schema transformer dynamically. The information extractor extracts and serves features automatically, conforming to standard SOS operations for observation retrieval and insertion. Feasibility experiments conducted on the Jinsha River tested this proposed method. Results show that the proposed approach allows the integration of SOS servers into legacy applications that have a higher degree of availability within many SDIs. However, this is accompanied with the drawback that only a limited part of the SOS functionality is available to clients.

Scrutiny New Framework in Integrated Distributed Reliable Systems

In this paper we represent a new framework for integrated distributed systems. In the proposed framework we have used three parts to increase Satisfaction and Performance of this framework. At first we analyse integrated systems and their evolution process and also ERPSD and ERPDRT framework briefly then we explain the new FDIRS framework. Finally we compare the results of simulation of the new framework with presented frameworks .Result showed In FIDRS framework, the technique of heterogeneous distributed data base is used to improve Performance and speed in responding to users. Finally by using FDIRS framework we succeeded to increase Efficiency, Performance and reliability of integrated systems and remove some of previous framework’s problems.

Learning latent variable models from distributed and abstracted data

Discovering global knowledge from distributed data sources is challenging, where the important issues include the ever-increasing data volume at the highly distributed sources and the general concern on data privacy. Properly abstracting the distributed data with a compact representation which can retain sufficient local details for global knowledge discovery in principle can address both the scalability and the data privacy challenges. This calls for the need to develop formal methodologies to support knowledge discovery on abstracted data. In this paper, we propose to abstract distributed data as Gaussian mixture models and learn a family of generative models from the abstracted data using a modified EM algorithm. To demonstrate the effectiveness of the proposed approach, we applied it to learn (a) data cluster models and (b) data manifold models, and evaluated their performance using both synthetic and benchmark data sets with promising results in terms of both effectiveness and scalability. Also, we have demonstrated that the proposed approach is robust against heterogeneous data distributions over the distributed sources.

An incident information management framework based on data integration, data mining, and multi-criteria decision making

An effective incident information management system needs to deal with several challenges. It must support heterogeneous distributed incident data, allow decision makers (DMs) to detect anomalies and extract useful knowledge, assist DMs in evaluating the risks and selecting an appropriate alternative during an incident, and provide differentiated services to satisfy the requirements of different incident management phases. To address these challenges, this paper proposes an incident information management framework that consists of three major components. The first component is a high-level data integration module in which heterogeneous data sources are integrated and presented in a uniform format. The second component is a data mining module that uses data mining methods to identify useful patterns and presents a process to provide differentiated services for pre-incident and post-incident information management. The third component is a multi-criteria decision-making (MCDM) module that utilizes MCDM methods to assess the current situation, find the satisfactory solutions, and take appropriate responses in a timely manner. To validate the proposed framework, this paper conducts a case study on agrometeorological disasters that occurred in China between 1997 and 2001. The case study demonstrates that the combination of data mining and MCDM methods can provide objective and comprehensive assessments of incident risks.

Design and Implementation of SOA-Based Professional Title Assessment System

In order to provide one-stop service of professional title assessment for the college staff, the business process based on SOA integrating personal qualification, automatic generation of vocational assessment materials, quantification of evaluation index, online expert assessment and vocational assessment information release is presented. Especially, the business process reengineering and business service lay-out is designed, heterogeneous distributed data sharing and exchange of the problem is solved, and the original isolation of the information system is integrated as a whole. The professional title assessment system implemented in China Three Gorges University digital campus is setup, which increased the objectivity, scientific and transparency of professional title assessment and improved the working efficiency.

Hierarchical distributed data classification in wireless sensor networks

Wireless sensor networks promise an unprecedented opportunity to monitor physical environments via inexpensive wireless embedded devices. Given the sheer amount of sensed data, efficient classification of them becomes a critical task in many sensor network applications. The large scale and the stringent energy constraints of such networks however challenge the conventional classification techniques that demand enormous storage space and centralized computation. In this paper, we propose a novel decision-tree-based hierarchical distributed classification approach, in which local classifiers are built by individual sensors and merged along the routing path forming a spanning tree. The classifiers are iteratively enhanced by combining strategically generated pseudo data and new local data, eventually converging to a global classifier for the whole network. We also introduce some control factors to facilitate the effectiveness of our approach. Through extensive simulations, we study the impact of the introduced control factors, and demonstrate that our approach maintains high classification accuracy with very low storage and communication overhead. The approach also addresses a critical issue of heterogeneous data distribution among the sensors.

Rule-Based Querying of Distributed, Heterogeneous Data

When searching for data, users tend to think in terms of the information they need to retrieve and not where and how it is stored. This is especially true in the highly complex domain of neurological research. The generic medical querying service described herein, aims to close the gap between users and data resources that are intricate, distributed and heterogeneous in nature. A theme that is common to both this domain and the emerging Internet of things is that users often need to query more than a single resource. This has come about with the large-scale fragmentation and distribution of data. Our work thus far has produced a prototype architecture for the querying of heterogeneous distributed data in the medical domain. A flexible querying mechanism that has the potential to support agent-based personalization has been developed to provide users with accurate and relevant results and some initial results are presented.

An XML Schema integration and query mechanism system

The availability of large amounts of heterogeneous distributed web data necessitates the integration of XML data from multiple XML sources for many reasons. For example, currently, there are many e-commerce companies, which offer similar products but use different XML Schemas with possibly different ontologies. When any two such companies merge, or make an effort to service customers in cooperation, there is a need for an integrated schema and query mechanism for the interoperability of applications. In applications like comparison-shopping, there is a need for an illusionary centralized homogeneous information system. In this paper, we propose XML Schema integration and querying methodology. We define an object-oriented data model called XSDM (XML Schema Data Model) and present a graphical representation of XML Schema for the purpose of schema integration. We use a three-layered architecture for XML Schema integration. The three layers included are namely pre-integration, comparison, and integration. The three layers can conceptually be regarded as three phases of the integration process. During pre-integration, the schemas present in XML Schema notation are read and converted into the XSDM notation. During the comparison phase of integration, correspondences as well as conflicts between elements are identified. During the integration phase, conflict resolution, restructuring and merging of the initial schemas takes place to obtain the global schema. We define integration policies for integrating element definitions as well as their datatypes and attributes. An integrated global schema forms the basis for querying a set of local XML documents. We discuss various strategies for rewriting the global query over the global schema into the sub-queries over local schemas. Their respective local schemas validate the sub-queries over the local XML documents. This requires the identification and use of mapping rules and relationships between the local schemas.

A Service-Oriented Grid Infrastructure for Biomedical Data and Compute Services

Service-oriented Grid technologies are increasingly utilized for the realization of future biomedical IT infrastructures since they offer unprecedented opportunities for the integration of advanced analysis and simulation applications as well as distributed heterogeneous data sources and information systems. The European Union's @neurIST project is developing a Grid-based IT infrastructure for the management of all processes linked to research, diagnosis, and treatment development for complex and multifactorial diseases encompassing data repositories, computational analysis services, and information systems handling multiscale, multimodal information at distributed sites. This paper provides an overview of the @neurIST Grid middleware and outlines the infrastructure offered for the provision of advanced compute and data services to support computationally demanding modeling and simulation tasks and to access heterogeneous distributed data sources through semantic integration.