Data Integration Techniques Research Articles

Data Integration Techniques for Cross-Platform Analytics

This paper highlights various data integration techniques that I applied when implementing cross Platform analysis including ETL integration techniques, API integration and real-time integration techniques. This looks at examples from Netflix, Amazon and Uber how these enable organisations to aggregate information from various sources for decisions, customisation and business operations. The consequence of concerning working discussed in this paper is an evaluation of the potentials of effective data integration and presenting it as a crucial factor towards the realization of competitiveness and innovation.

Single-cell transcriptional networks in differentiating preadipocytes suggest drivers associated with tissue heterogeneity

White adipose tissue plays an important role in physiological homeostasis and metabolic disease. Different fat depots have distinct metabolic and inflammatory profiles and are differentially associated with disease risk. It is unclear whether these differences are intrinsic to the pre-differentiated stage. Using single-cell RNA sequencing, a unique network methodology and a data integration technique, we predict metabolic phenotypes in differentiating cells. Single-cell RNA-seq profiles of human preadipocytes during adipogenesis in vitro identifies at least two distinct classes of subcutaneous white adipocytes. These differences in gene expression are separate from the process of browning and beiging. Using a systems biology approach, we identify a new network of zinc-finger proteins that are expressed in one class of preadipocytes and is potentially involved in regulating adipogenesis. Our findings gain a deeper understanding of both the heterogeneity of white adipocytes and their link to normal metabolism and disease.

Integration of multisource data to support the identification of lateritic regolith in Eastern - Bahia, northeastern Brazil

This work used multi-source data integration techniques (gamma-spectrometry, magnetometry, SRTM-Altimetry) to identify areas favorable for the occurrence of well-developed lateritic regolith in the eastern region of the state of Bahia. The variables observed to define target potential were the high eTh/K and eU/K ratios and altitudes (obtained from SRTM). Based on these parameters, Boolean and fuzzy logic were applied to produce favorability maps. The best results were obtained under the fuzzy model (γ= 0.7) with a hit accuracy in the areas for potential laterite occurrence of 97.4% and a kappa value of 0.5, consisting of 118 control points obtained through fieldwork. The cross-referencing of the fuzzy image (γ = 0.7) with the magnetometry, total gradient (ASA-Analytical Signal Amplitude) suggested the predominance of more felsic protoliths in the most favorable areas. The lateritic index [IL = (eTh*eU)/K2)] was also applied, demonstrating good correlation with the areas determined by the Boolean and Fuzzy models. Mineralogical associations (clay minerals and/or iron oxides) of the target areas were estimated by the Crosta Technique in OLI/Landsat-8 images. The processing results were integrated in GIS environment, together with control points and data found in the bibliography (e.g. geological map, vertical electrical profile, drill-holes, occurrence of Fe and Al, lateritic crusts, geochemical anomalies).The observed correlation between the generated models and the direct data validates the effectiveness of the techniques used.

Integrative analysis of DNA methylation and gene expression in papillary renal cell carcinoma.

Patterns of DNA methylation are significantly altered in cancers. Interpreting the functional consequences of DNA methylation requires the integration of multiple forms of data. The recent advancement in the next-generation sequencing can help to decode this relationship and in biomarker discovery. In this study, we investigated the methylation patterns of papillary renal cell carcinoma (PRCC) and its relationship with the gene expression using The Cancer Genome Atlas (TCGA) multi-omics data. We found that the promoter and body of tumor suppressor genes, microRNAs and gene clusters and families, including cadherins, protocadherins, claudins and collagens, are hypermethylated in PRCC. Hypomethylated genes in PRCC are associated with the immune function. The gene expression of several novel candidate genes, including interleukin receptor IL17RE and immune checkpoint genes HHLA2, SIRPA and HAVCR2, shows a significant correlation with DNA methylation. We also developed machine learning models using features extracted from single and multi-omics data to distinguish early and late stages of PRCC. A comparative study of different feature selection algorithms, predictive models, data integration techniques and representations of methylation data was performed. Integration of both gene expression and DNA methylation features improved the performance of models in distinguishing tumor stages. In summary, our study identifies PRCC driver genes and proposes predictive models based on both DNA methylation and gene expression. These results on PRCC will aid in targeted experiments and provide a strategy to improve the classification accuracy of tumor stages.

Key Issues in Healthcare Data Integrity: Analysis and Recommendations

Managing data integrity is a challenging task for any expert or a researcher. This study attempts to collate a Systematic Literature Review of the research efforts done in the domain of healthcare data integrity. The paper highlights the criticalness of data integrity issues in healthcare through attack statistics in the first section. The second section of the paper systematically reviews the previous studies discussing the healthcare related Systematic literature reviews and data integrity techniques in healthcare sector. The third section of this study examines the collated literature through various analysis methodologies and discusses about the most prioritized technique as well as its challenges in healthcare data security. The fourth section illustrates about the various challenges and future directions to take while constructing a roadmap for the future research endeavors in healthcare data integrity management techniques. The concluding segment of the paper presents an objective assessment and sensitivity analysis for finding the implications and difficulties in the studies while outlining feasible solutions. Furthermore, this research endeavour also conducts a Scientometric analysis of all the studies for better understanding of the literature reviewed. Ranking or the Priority analysis part of the paper is totally dedicated to the previously used techniques in healthcare. The paper also discusses about data integrity techniques and postulates that the most prioritized data integrity technique is the blockchain.

Hierarchical Canonical Correlation Analysis Reveals Phenotype, Genotype, and Geoclimate Associations in Plants.

The local environment of the geographical origin of plants shaped their genetic variations through environmental adaptation. While the characteristics of the local environment correlate with the genotypes and other genomic features of the plants, they can also be indicative of genotype-phenotype associations providing additional information relevant to environmental dependence. In this study, we investigate how the geoclimatic features from the geographical origin of the Arabidopsis thaliana accessions can be integrated with genomic features for phenotype prediction and association analysis using advanced canonical correlation analysis (CCA). In particular, we propose a novel method called hierarchical canonical correlation analysis (HCCA) to combine mutations, gene expressions, and DNA methylations with geoclimatic features for informative coprojections of the features. HCCA uses a condition number of the cross-covariance between pairs of datasets to infer a hierarchical structure for applying CCA to combine the data. In the experiments on Arabidopsis thaliana data from 1001 Genomes and 1001 Epigenomes projects and climatic, atmospheric, and soil environmental variables combined by CLIMtools, HCCA provided a joint representation of the genomic data and geoclimate data for better prediction of the special flowering time at 10°C (FT10) of Arabidopsis thaliana. We also extended HCCA with information from a protein-protein interaction (PPI) network to guide the feature learning by imposing network modules onto the genomic features, which are shown to be useful for identifying genes with more coherent functions correlated with the geoclimatic features. The findings in this study suggest that environmental data comprise an important component in plant phenotype analysis. HCCA is a useful data integration technique for phenotype prediction, and a better understanding of the interactions between gene functions and environment as more useful functional information is introduced by coprojections of multiple genomic datasets.

Data Integration Techniques for the Identification of Poverty Profiles

Economic connotation of poverty concerns mainly the level of household’s spending and income as privileged indicators. Sometimes, data sources have several lack of information, but such information is available in other data sources related to the same statistical units, or to other households. In order to identify records relating to the same or similar units belonging to two data archives, some methodologies (record linkage in the first case, statistical matching in the latter) are used for integrating data. In this study we illustrate a model of data integration between two surveys conducted by the Italian National Institute of Statistics (Statistics on Income and Living Conditions and Survey on Household Expenses). The construction of an integrated database starting from those two surveys is useful for studying consumer behaviours in relation to specific groups of goods, analysing decisions related to household savings, examining economic and social inequalities, such as studying the impact of public policies through simulations.

Variable Selection in the Regularized Simultaneous Component Analysis Method for Multi-Source Data Integration

Interdisciplinary research often involves analyzing data obtained from different data sources with respect to the same subjects, objects, or experimental units. For example, global positioning systems (GPS) data have been coupled with travel diary data, resulting in a better understanding of traveling behavior. The GPS data and the travel diary data are very different in nature, and, to analyze the two types of data jointly, one often uses data integration techniques, such as the regularized simultaneous component analysis (regularized SCA) method. Regularized SCA is an extension of the (sparse) principle component analysis model to the cases where at least two data blocks are jointly analyzed, which - in order to reveal the joint and unique sources of variation - heavily relies on proper selection of the set of variables (i.e., component loadings) in the components. Regularized SCA requires a proper variable selection method to either identify the optimal values for tuning parameters or stably select variables. By means of two simulation studies with various noise and sparseness levels in simulated data, we compare six variable selection methods, which are cross-validation (CV) with the “one-standard-error” rule, repeated double CV (rdCV), BIC, Bolasso with CV, stability selection, and index of sparseness (IS) - a lesser known (compared to the first five methods) but computationally efficient method. Results show that IS is the best-performing variable selection method.

Understanding dynamic data interaction between civil integrated management technologies: a review of use cases and enabling techniques

Civil integrated management (CIM) has emerged as an advanced paradigm to transform transportation project delivery and asset management from a paper-based to a digital data-based approach. CIM is associated with the use of computerized technologies (e.g. three-dimensional models, geographical information systems, and light detection and ranging) for data collection, organization, and utilization in across project stages from design, construction, to operation and maintenance. This change results in an increasing amount of digital data exchange between relevant tools. CIM is a new concept, and a comprehensive understanding of the data flow among CIM tools is still lacking. This article aims to investigate the data interaction between CIM technologies through an in-depth review of recently published academic articles. A critical review was conducted focusing on identifying use cases of data interaction and existing methods enabling the data communication between CIM technologies. The study found nine pairs of tools that have a strong data relationship. The study also revealed a shortcoming of efficient data extraction and integration techniques to support seamless data workflows. The research is original, being the first examination of the data interaction between CIM technologies. The findings from this research are expected to enhance awareness among researchers and professionals about the value of the data generated by each CIM technology and the state-of-the-art on supporting methods for connecting proprietary tools.

A survey on data aggregation techniques in IoT sensor networks

There is a growing interest in using wireless sensor technologies in various Internet of things scenarios. Considering the huge growth of smart objects and their applications, the need to collect and analyze their product data are becoming one of the main challenges. Sensor nodes are powered by batteries, efficient operations in term of energy are critical. Toward that end, it is desirable for a sensor node to eliminate redundancies in the received data from the neighboring nodes before transferring the final data to the central station. Data aggregation is one of the influential techniques in elimination of data redundancy and improvement of energy efficiency; also it increases the lifespan of Wireless Sensor Networks. In addition, the efficient data aggregation protocol can reduce network traffic. When a specific objective takes place in a specific area, it might be detected by more than one sensor. Considering the main challenges and aspects of data aggregation in wireless sensor networks, a review on different types of data aggregation techniques and protocols are presented in this paper. The ultimate objective of this study is to make the basic foundations to develop new advanced designs based on data integration techniques and clustering that have been proposed so far. Major techniques of data integration in wireless sensor networks covering ground, underground and underwater sensor networks are presented in this paper and the applications, advantages and disadvantages of using each technique are described.

A Mixed-Methods Evaluation of a Nurse-Led Community-Based Health Home for Ethnically Diverse Older Adults With Multimorbidity in the Adult Day Health Setting.

Multimorbidity affects 75% of older adults (aged 65 years and older) in the United States and increases risk of poor medical outcomes, especially among the poor and underserved. The creation of a Medicaid option allowing states to establish health homes under the Affordable Care Act was intended to enhance coordinated care for Medicaid beneficiaries with multimorbidity. The Community-Based Health Home (CBHH) model uses the infrastructure of the Adult Day Health Center (ADHC) to serve as a health home to improve outcomes for medically complex vulnerable adults. Between 2017 and 2018, we used a sequential explanatory mixed-methods approach to (a) quantitatively examine changes in depression, fall risk, loneliness, cognitive function, nutritional risk, pain classification, and health care utilization over the course of 12 months in the program and (b) qualitatively explore the perspectives of key stakeholders (registered nurse navigators, participants, ADHC administrators, and caregivers) to identify the most effective components of CBHH. Using data integration techniques, we identified components of CBHH that were most likely driving outcomes. After 12 months in CBHH, our racially diverse sample (N = 126), experienced statistically significant (p < .05) reductions in loneliness, depression, nutritional risk, poorly controlled pain, and emergency department utilization. Stakeholders who were interviewed (n = 40) attributed positive changes to early clinical intervention by the registered nurse navigators, communication with providers across settings, and a focus on social determinants of health, in conjunction with social stimulation and engagement provided by the ADHC. CBHH positions the ADHC as the locus of an effective health home site and is associated with favorable results. CBHH also demonstrates the unique capacity and skill of registered nurses in integrating health and social services across community settings. Continued exploration of CBHH among diverse populations with multimorbidity is warranted.

Integration of Metabolomic and Other Omics Data in Population-Based Study Designs: An Epidemiological Perspective.

It is not controversial that study design considerations and challenges must be addressed when investigating the linkage between single omic measurements and human phenotypes. It follows that such considerations are just as critical, if not more so, in the context of multi-omic studies. In this review, we discuss (1) epidemiologic principles of study design, including selection of biospecimen source(s) and the implications of the timing of sample collection, in the context of a multi-omic investigation, and (2) the strengths and limitations of various techniques of data integration across multi-omic data types that may arise in population-based studies utilizing metabolomic data.

MidSemI

The web diversification into the Web of Data and social media means that companies need to gather all the necessary data to help make the best-informed market decisions. However, data providers on the web publish data in various data models and may equip it with different search capabilities, thus requiring data integration techniques to access them. This work explores the current challenges in this area, discusses the limitations of some existing integration tools, and addresses them by proposing a semantic mediator-based approach to virtually integrate enterprise data with large-scale social and linked data. The implementation of the proposed approach is a configurable middleware application and a user-friendly keyword search interface that retrieves its input from internal enterprise data combined with various SPARQL endpoints and Web APIs. An evaluation study was conducted to compare its features with recent integration approaches. The results illustrate the added value and usability of the contributed approach.

Enhancing the Radiometric Map of Australia

The Radiometric Map of Australia shows the distribution of potassium (% K), uranium (ppm eU) and thorium (ppm eTh) over Australia. A suite of image enhancement and data integration techniques can be used to enhance the value of these data for both mineral exploration and environmental mapping. Gradient-enhanced ternary and pseudo-colour image enhancements are now routinely used for the presentation and interpretation of gridded radioelement data. Where digital elevation data are available, these colour representations can be draped over the elevation data to form 3D perspective views, or hill-shaded derivatives of the DEM can be embedded into the ternary imagery as an intensity component. These are useful because the radioelement response can then be interpreted within the context of the relative position of anomalous features in the landscape. However, subtle variations in the concentrations of K, U, and Th are best interpreted using the ratios of the radioelements. The U/Th and U2/Th ratios are important indicators of uranium mineralization. The Th/K ratio is widely used for the detection of several styles of mineralization associated with K alteration. For the interpretation of map data, classification and clustering methods can be used to assist pattern recognition and are useful for the rapid assessment of large multivariate datasets. Automatic edge detection procedures can be used to speed up the annotation of unit boundaries. Residual modelling techniques can be used to highlight potential anomalies in the data.

Adaptive, efficient and effective graph data integration and search framework

This paper proposes an idea for a new graph data integration and search system leveraging techniques of provenance, visualization and association rules. The graph data model, along with techniques of web data integration and database usability are central to the development of the web which provides motivation for this work. The overall proposed system consists of three components:a) Quality-aware graph visualization leveraging user feedback: we propose a solution to utilize user feedback on relevant portions of schema of graph database in graph sample selections so as to provide an adaptive way to choose graph samples via schemas in order to provide relevant quality-aware visualization later on and a solution based on statistical techniques of averages and heatmaps visualization in assigning color intensity to spatial positions of vertices and edges based on quality to allow the user to gauge the quality of graph datasets.b) Visualization-enabled graph alignment leveraging query logs: we propose a solution to utilize graph query logs along with run-time information of graph neighborhoods that led to graph alignment decisions in task of graph alignment so as to improve its effectiveness as well as a solution based on assigning spatial positions to graph vertices and edges based on graph similarity measures to allow the user to visually compare graph datasets during integration.c) Efficient batch generation of multiple graph query autocompletion suggestions: Techniques in literature typically make use of data clustering, data reduction, utilizing query logs to provide feedback and autocompletion suggestions to users during query formulation. As another solution with a focus on efficiency in context of graph datasets, we propose to leverage the fact that multiple queries may have many autocompletion suggestions in common and therefore propose an efficient algorithm to generate them together efficiently and show the performance improvement over the current sequential approach that involves repeated scans.

Verification of data integrity and co-operative loss recovery for secure data storage in cloud computing

In Cloud Computing, the data stored in the external servers may be tampered or deleted by unauthorized persons or selfish Cloud Service Providers (CSPs). Hence, the Cloud Data Owners (CDOs) have to provide assurance to the integrity and correctness of the stored data in the server. In this paper, a Verification of Data Integrity and Co-operative Loss Recovery technique for secure data storage in Cloud Computing is proposed. In this technique, a ciphertext file is split into various cipher blocks and distributed to randomly selected cloud service providers (CSPs) by the cloud data owner (CDO). If a cloud data user (CDU) wants to access any file, the corresponding ciphertext file is reconstructed from the blocks and downloaded by the user. The file can be decrypted if the attribute set of the user matches the access policy of the application. By simulation results, we show that the proposed technique enhances the data integrity and confidentiality.

An Evaluation Instrument (Q-U) for Measuring the usability of Business Intelligence Application

The main purpose of Business Intelligence (BI) applications is to focus on supporting organizations' strategic, operational and tactical decisions by providing comprehensive, accurate and vivid data to the decision makers. Usability testing is an important part of software development. It focuses on how well users can understand and utilize a product/software in fulfilling their intended goals. In the software development, most software tends to be complex. Hence, the way to ensure that such software will satisfy users will be done by measuring the usability of such software. Due to using many data integration techniques in developing BI application, thus, the different software have different instruments to measure the usability of BI software. In this paper, the new instrument (Q-U) for measuring BI software was developed. A systematic approach was adopted which suggested by many researchers in developing the intended Q-U instrument.The literature has been reviewed in a systematic manner to elicitation Q-U instrument attributes. Multi-methods were used in validating Q-U instrument such as face validity, a pilot study for testing the goodness of Q-U consistency, and factor analysis. Multi-methods were used in validating Q-U instrument such as face validity, a pilot study for testing the goodness of Q-U consistency, and factor analysis as well as Bartlett’s test for measuring the reliability of Q_U. The finding obtained indicates the workability of Q_U in measuring the usability OF BI applications. Ultimately, Q_U was used in measuring BI application in two different sectors.

Approximate top-K answering under uncertain schema mappings

Data integration techniques provide a communication bridge between isolated sources and offer a platform for information exchange. When the schemas of heterogeneous data sources map to the centralized schema in a mediated data integration system or a source schema maps to a target schema in a peer-to-peer system, multiple schema mappings may exist due to the ambiguities in the attribute matching. The obscure schema mappings lead to the uncertainty in query answering, and frequently people are only interested in retrieving the best k answers (top-k) with the biggest probabilities. Retrieving the top-k answers efficiently has become a research issue. For uncertain queries, two semantics, by-table and by-tuple, have been developed to capture top-k answers based on the schema mapping probabilities. However, although the existing algorithms support certain features to capture the accurate top-k answers and avoid accessing all data from sources, they cannot effectively reduce the number of processed tuples in most cases. In this paper, new algorithms based on the histogram approximation and heuristic are proposed to efficiently identify the top-k answers for the data integration systems under uncertain schema mappings. In the experiments, the Histogram algorithm in the by-table semantics and the expected approach in the by-tuple semantics are shown to significantly reduce the number of processed tuples while maintaining high accuracy with the estimated probabilistic confidence.

Introduction to this special section: Cross-disciplinary applications of geophysics

As the volume of seismic and other data types continues to increase, the use of such data sets has extended to different approaches and techniques of data integration and interpretation. The intent of this special section on cross-disciplinary applications of geophysics is to highlight such uses of multiple data types. Although not limited to any type or location of a given reservoir, the two articles in this section primarily focus on onshore unconventional reservoirs. Nonetheless, the techniques and approaches will also be of interest to readers and practitioners who deal with conventional reservoirs, both onshore and offshore.

Multi-omics analysis of ten carbon nanomaterials effects highlights cell type specific patterns of molecular regulation and adaptation

New strategies to characterize the effects of engineered nanomaterials (ENMs) based on omics technologies are emerging. However, given the intricate interplay of multiple regulatory layers, the study of a single molecular species in exposed biological systems might not allow the needed granularity to successfully identify the pathways of toxicity (PoT) and, hence, portraying adverse outcome pathways (AOPs). Moreover, the intrinsic diversity of different cell types composing the exposed organs and tissues in living organisms poses a problem when transferring in vivo experimentation into cell-based in vitro systems.To overcome these limitations, we have profiled genome-wide DNA methylation, mRNA and microRNA expression in three human cell lines representative of relevant cell types of the respiratory system, A549, BEAS-2B and THP-1, exposed to a low dose of ten carbon nanomaterials (CNMs) for 48 h. We applied advanced data integration and modelling techniques in order to build comprehensive regulatory and functional maps of the CNM effects in each cell type.We observed that different cell types respond differently to the same CNM exposure even at concentrations exerting similar phenotypic effects. Furthermore, we linked patterns of genomic and epigenomic regulation to intrinsic properties of CNM. Interestingly, DNA methylation and microRNA expression only partially explain the mechanism of action (MOA) of CNMs. Taken together, our results strongly support the implementation of approaches based on multi-omics screenings on multiple tissues/cell types, along with systems biology-based multi-variate data modelling, in order to build more accurate AOPs.