Data Integration Applications Research Articles

A Common Data Dictionary and Common Data Model for Additive Manufacturing

Additive manufacturing (AM) leverages emerging technologies and well-adopted processes to produce near-net-shape products. The advancement of AM technology requires data management tools to collect, store, and share information through the product development lifecycle and across the material and machine value chain. To address the need for sharing data among AM developers and practitioners, an AM common data dictionary (AM-CDD) was first developed based on community consensus to provide a common lexicon for AM, and later standardized by ASTM International. Following the AM-CDD work, the development of a common data model (AM-CDM) defining the structure and relationships of the key concepts, and terms in the AM-CDD is being developed. These efforts have greatly facilitated system integrations and AM data exchanges among various organizations. This work outlines the effort to create the AM-CDD and AM-CDM, with a focus on the design of the AM-CDM. Two use cases are provided to demonstrate the adoption of these efforts and the interoperability enabled by the AM-CDM for different engineering applications managed by different types of database technology. In these case studies, the AM-CDM is implemented in two distinct formats to curate AM data from NIST—the first in XML from their additive manufacturing material database and the second in OWL from their 2022 AM bench database. These use cases present the power of the AM-CDM for data representation, querying, and seamless data exchange. Our implementation experiences and some challenges are highlighted that can assist others in future adoptions of the AM-CDM for data integration and data exchange applications.

Data standards based Mine Side Data Integration Governance Platform Research and Applications

INTRODUCTION: With the advent of the mining digitalization era, the rapid growth and diversity of mine-side data make mining production management more complex. In this context, it becomes crucial to build a set of mine-side data integration and governance platforms based on data standards. The establishment of this platform aims to solve the problems of mining data fragmentation and inconsistent standards and to improve the efficiency of data management and utilization.OBJECTIVES: The purpose of this study is to construct an efficient mine-side data integration and governance platform by studying and applying data standards to address the challenges of data heterogeneity and fragmentation. Through the development of unified data standards, the Author can realize the standardized management of mining data, improve data quality, and provide more reliable support for mining production.METHODS: The study adopts a comprehensive approach. First, a comprehensive understanding of the problems and needs facing mining data management was obtained through a literature review and field research. Second, data standards applicable to mine-side data are developed to ensure that all types of data can be consistently understood and processed. Next, a data integration and governance platform based on these standards is established, and advanced data mining and artificial intelligence technologies are applied to realize automatic integration and cleaning of multi-source data.RESULTS: Through experiments, this study proved that a mine-side data integration governance platform based on data standards was successfully constructed. The platform realizes efficient integration and cleaning of data and improves the consistency and quality of data. In practical application, the platform provides more accurate data support for mining production management and achieves remarkable results.CONCLUSION: This study provides strong theoretical and practical support for the research and application of mine-side data integration and governance platforms based on data standards. Through the development of data standards, an efficient data management platform has been established, which provides practical solutions for solving mining data management challenges. In the future, the performance of the platform can be further optimized, and the updating and maintenance of data standards can be strengthened to adapt to the constant changes in the digital development of the mining industry. This research has a positive impact on promoting the construction of mining industry informatization and improving the level of production management.

Research on the Application of Digital Twin in Smart Cities

With the development of technology, digital twin has played a non-negligible positive impact on the development of smart cities. From the mutually influential nature of digital twin and smart city, this paper analyzes the progress of digital twin technology in smart city applications through theory and case studies. The author examines three key cases: city brain, IoT cloud platform, and smart hub, to explore the application of digital twin technology in smart cities. In addition, the paper presents the potential problems of digital twin in smart cities, such as insufficient infrastructure, insufficient systematic data integration and few application scenarios. In future research, the author suggests that digital twin technology holds promise for enabling innovative applications in areas such as environmental protection, security, energy, healthcare, and tourism within smart cities.

Implementation of Business Intelligence In Analyzing Data Using Tableau at PT Global Bintan Permata

PT Global Bintan Permata is member group Yafindo (Batam, Indonesia) which is engaged in the distribution and import goods consumption and goods consumption throughout Indonesia . one _ business being run is distributing sugar _ throughout modern market and delivery some are via transportation and some are done in a manner independent . The target market namely the big modern market like Alfamart , Alfamidi , Indomarco , Indogrosir , Tiptop, and there is also a small modern market like Totalfruits , Grandlucky , etc. _ Of the many transactions made by the company this company _ This Still Not yet own Dashboard application for monitoring activity operations and transactions . So from it is needed something deep data visualization dashboard shape . Method used to data warehouse design is use method kimball nine-step technology. For the Extract, Transform and Load (ETL) process for data warehouse design using tools in the Pentaho Data Integration application , meanwhile For dashboard visualization is done with use the Tableau Public app . Result of application Tableau Public app is form generated data visualization dashboard reports containing information in form the data visualization it consists of from transaction and marketing data in 2021. Dashboard provides information For know How development sale both product and sales are also developing transactions with customers in 2021 later will become base in taking decisions required by management within PT Global Bintan Permata for help in the process of taking decision.

DAIKON: A Data Acquisition, Integration, and Knowledge Capture Web Application for Target-Based Drug Discovery.

Primitive data organization practices struggle to deliver at the scale and consistency required to meet multidisciplinary collaborations in drug discovery. For effective data sharing and coordination, a unified platform that can collect and analyze scientific information is essential. We present DAIKON, an open-source framework that integrates targets, screens, hits, and manages projects within a target-based drug discovery portfolio. Its knowledge capture components enable teams to record subsequent molecules as their properties improve, facilitate team collaboration through discussion threads, and include modules that visually illustrate the progress of each target as it advances through the pipeline. It serves as a repository for scientists sourcing data from Mycobrowser, UniProt, PDB. The goal is to globalize several variations of the drug-discovery program without compromising local aspects of specific workflows. DAIKON is modularized by abstracting the database and creating separate layers for entities, business logic, infrastructure, APIs, and frontend, with each tier allowing for extensions. Using Docker, the framework is packaged into two solutions: daikon-server-core and daikon-client. Organizations may deploy the project to on-premises servers or VPC. Active-Directory/SSO is supported for user administration. End users can access the application with a web browser. Currently, DAIKON is implemented in the TB Drug Accelerator program (TBDA).

Study on behavioral risk for aging based on smart mattress monitoring data

With an aging population and a gap in demand for professional caregivers, China's elderly care system is facing severe challenges. Using data mining to expand the smart senior care scenario can effectively improve elderly care services. Based on smart mattress datasets, this paper uses machine learning classification and unsupervised anomaly detection models to analyze the possible risks in the daily behavior of older people from the perspectives of both sleep apnea problems and abnormal physiological information. The model results show that the Stacking algorithm based on data fusion can effectively identify the risk of sleep apnea. In contrast, the Prophet and DBSCAN models can carry out anomaly mining of physiological information of single and combined variables, respectively. Ultimately, based on the research, this paper provides targeted recommendations regarding data collection and integration applications.

Managing 3D GIS Data for Indoor Environment Using Property Graph Database

With the diversification of three-dimensional (3D) urban space data sources and data collection methods, the Open Geospatial Consortium published 3D spatial data standards in Geography Markup Language (GML). Among them, IndoorGML is a data model for expressing the topological relationship between indoor spaces with advantages in terms of semantic information storage and navigation applications. Conversion into a suitable binary file must be accomplished to use the IndoorGML documents through integration with other data. This study proposes a method of generating graph database objects from the documents of IndoorGML to promote usability and interoperability. The labeled property graph model is designed to reflect IndoorGML’s multilayered graph concept. Moreover, in the proposed graph model, the geometry of physical indoor spaces is segregated into separate surfaces. All elements from IndoorGML are managed with specific labels in Neo4j, the most dominant graph database. By applying the proposed method, the graph database is constructed using the officially distributed IndoorGML documents of three buildings. The number of generated graph objects is determined to confirm whether the IndoorGML features are accurately converted into graph database objects. Furthermore, the utility of the constructed graph database is verified through two scenario-based routing tests consisting of finding the optimal room with the lowest route cost and allocating indoor spaces for patrolling to outdoor security offices through indoor and outdoor data integration. In conclusion, this study confirms that the utility of IndoorGML can be promoted using a graph database system in data integration, data queries, and pathfinding applications while preserving data topology and interoperability.

TokenJoin

Set similarity join is an important problem with many applications in data discovery, cleaning and integration. To increase robustness, fuzzy set similarity join calculates the similarity of two sets based on maximum weighted bipartite matching instead of set overlap. This allows pairs of elements, represented as sets or strings, to also match approximately rather than exactly, e.g., based on Jaccard similarity or edit distance. However, this significantly increases the verification cost, making even more important the need for efficient and effective filtering techniques to reduce the number of candidate pairs. The current state-of-the-art algorithm relies on similarity computations between pairs of elements to filter candidates. In this paper, we propose token-based instead of element-based filtering, showing that it is significantly more lightweight, while offering similar or even better pruning effectiveness. Moreover, we address the top- k variant of the problem, alleviating the need for a user-specified similarity threshold. We also propose early termination to reduce the cost of verification. Our experimental results on six real-world datasets show that our approach always outperforms the state of the art, being an order of magnitude faster on average.

Integration Method for Web based Visualization Framework of Green Highway Index and Carbon Footprint Calculator

Global warming and the development of the green economy have imposed more pressure on government economies to de-carbonise their economic initiatives. This, along with the global financial crisis, has prompted a significant rethinking of digital solutions to monitor carbon emissions. Likewise, drawing investors’ investment to further development of green highways by digitised tools for infrastructure projects is crucial. However, there is a demand for integration and visualisation of green highways assessment tools, which highlights an important gap in developing data integration and web-based visualisation application. This paper presents a research proposal method to build an integrated web-based dashboard that monitors the green highway rating assessment and the Carbon Footprint. It emphasizes variables of Malaysia Green Highway Index (MyGHI) and Carbon Footprint Calculator (CFC). A framework for integrating and transforming towards visualisation of web-based dashboard is established. The integration of MyGHI-CFC to the digital level and create a web-based dashboard used different methodologies from mixed qualitative and quantitative research. The results of this integration and visualization foresee contribute to an effective example, paving researchers with effective tools and standards that will contribute to the development of the green highway and future sustainability projects.

Construction and application of an ontology-based domain-specific knowledge graph for petroleum exploration and development

The massive amount and multi-sourced, multi-structured data in the upstream petroleum industry impose great challenge on data integration and smart application. Knowledge graph, as an emerging technology, can potentially provide a way to tackle the challenges associated with oil and gas big data. This paper proposes an engineering-based method that can improve upon traditional natural language processing to construct the domain knowledge graph based on a petroleum exploration and development ontology. The exploration and development knowledge graph is constructed by assembling Sinopec’s multi-sourced heterogeneous database, and millions of nodes. The two applications based on the constructed knowledge graph are developed and validated for effectiveness and advantages in providing better knowledge services for the oil and gas industry.

Infrastructure for Rapid Open Knowledge Network Development

The past decade has witnessed a growth in the use of knowledge graph technologies for advanced data search, data integration, and query-answering applications. The leading example of a public, general-purpose open knowledge network (aka knowledge graph) is Wikidata, which has demonstrated remarkable advances in quality and coverage over this time. Proprietary knowledge graphs drive some of the leading applications of the day including, for example, Google Search, Alexa, Siri, and Cortana. Open Knowledge Networks are exciting: they promise the power of structured database-like queries with the potential for the wide coverage that is today only provided by the Web. With the current state of the art, building, using, and scaling large knowledge networks can still be frustratingly slow. This article describes a National Science Foundation Convergence Accelerator project to build a set of Knowledge Network Programming Infrastructure systems to address this issue.

Introduction of medical genomics and clinical informatics integration for p-Health care.

Achieving predictive, precise, participatory, preventive, and personalized health (abbreviated as p-Health) requires comprehensive evaluations of an individual's conditions captured by various measurement technologies. Since the 1950s, analysis of care providers' and physicians' notes and measurement data by computers to improve healthcare delivery has been termed clinical informatics. Since the 2010s, wide adoptions of Electronic Health Records (EHRs) have greatly improved clinical informatics development with fast growing pervasive wearable technologies that continuously capture the human physiological profile in-clinic (EHRs) and out-of-clinic (PHRs or Personal Health Records) to bolster mobile health (mHealth). In addition, after the Human Genome Project in the 1990s, medical genomics has emerged to capture the high-throughput molecular profile of a person. As a result, integrated data analytics is becoming one of the fast-growing areas under Biomedical Big Data to improve human healthcare outcomes. In this chapter, we first introduce the scope of data integration and review applications, data sources, and tools for clinical informatics and medical genomics. We then describe the data integration analytics at the raw data level, feature level, and decision level with case studies, and the opportunity for research and translation using advanced artificial intelligence (AI), such as deep learning. Lastly, we summarize the opportunities in biomedical big data integration that can reshape healthcare toward p-health.

A versatile and scalable single-cell data integration algorithm based on domain-adversarial and variational approximation

Single-cell technologies provide us new ways to profile transcriptomic landscape, chromatin accessibility, spatial expression patterns in heterogeneous tissues at the resolution of single cell. With enormous generated single-cell datasets, a key analytic challenge is to integrate these datasets to gain biological insights into cellular compositions. Here, we developed a domain-adversarial and variational approximation, DAVAE, which can integrate multiple single-cell datasets across samples, technologies and modalities with a single strategy. Besides, DAVAE can also integrate paired data of ATAC profile and transcriptome profile that are simultaneously measured from a same cell. With a mini-batch stochastic gradient descent strategy, it is scalable for large-scale data and can be accelerated by GPUs. Results on seven real data integration applications demonstrated the effectiveness and scalability of DAVAE in batch-effect removing, transfer learning and cell-type predictions for multiple single-cell datasets across samples, technologies and modalities. Availability: DAVAE has been implemented in a toolkit package "scbean" in the pypi repository, and the source code can be also freely accessible at https://github.com/jhu99/scbean. All our data and source code for reproducing the results of this paper can be accessible at https://github.com/jhu99/davae_paper.

View selection over knowledge graphs in triple stores

Knowledge Graphs (KGs) are collections of interconnected and annotated entities that have become powerful assets for data integration, search enhancement, and other industrial applications. Knowledge Graphs such as DBPEDIA may contain billion of triple relations and are intensively queried with millions of queries per day. A prominent approach to enhance query answering on Knowledge Graph databases is View Materialization, ie., the materialization of an appropriate set of computations that will improve query performance. We study the problem of view materialization and propose a view selection methodology for processing query workloads with more than a million queries. Our approach heavily relies on subgraph pattern mining techniques that allow to create efficient summarizations of massive query workloads while also identifying the candidate views for materialization. In the core of our work is the correspondence between the view selection problem to that of Maximizing a Nondecreasing Submodular Set Function Subject to a Knapsack Constraint . The latter leads to a tractable view-selection process for native triple stores that allows a (1 - e ---1 )-approximation of the optimal selection of views. Our experimental evaluation shows that all the steps of the view-selection process are completed in a few minutes, while the corresponding rewritings accelerate 67.68% of the queries in the DBPEDIA query workload. Those queries are executed in 2.19% of their initial time on average.

Spherical Regression Under Mismatch Corruption With Application to Automated Knowledge Translation

Motivated by a series of applications in data integration, language translation, bioinformatics, and computer vision, we consider spherical regression with two sets of unit-length vectors when the data are corrupted by a small fraction of mismatch in the response-predictor pairs. We propose a three-step algorithm in which we initialize the parameters by solving an orthogonal Procrustes problem to estimate a translation matrix ignoring the mismatch. We then estimate a mapping matrix aiming to correct the mismatch using hard-thresholding to induce sparsity, while incorporating potential group information. We eventually obtain a refined estimate for by removing the estimated mismatched pairs. We derive the error bound for the initial estimate of in both fixed and high-dimensional setting. We demonstrate that the refined estimate of achieves an error rate that is as good as if no mismatch is present. We show that our mapping recovery method not only correctly distinguishes one-to-one and one-to-many correspondences, but also consistently identifies the matched pairs and estimates the weight vector for combined correspondence. We examine the finite sample performance of the proposed method via extensive simulation studies, and with application to the unsupervised translation of medical codes using electronic health records data. Supplementary materials for this article are available online.

A Data Integration Application for Occupational Health and Safety Management in the Mining Industry

Madencilik endüstrisi, iş sağlığı ve güvenliği açısından değerlendirildiğinde, farklı mühendislik alanlarını ve bunlara bağlı riskleri kapsamaktadır. Günümüzde, teknoloji alanında yaşanan gelişmeler, madencilik endüstrisini mevcut donanım ve yazılımlar aracılığıyla, üretim, ekipmanların bakım/onarım durumu, maliyet, çevresel koşullar ve en önemlisi iş sağlığı ve güvenliği konusunda, veri toplayabilecek duruma getirmiştir. Bu çalışmada, iş sağlığı ve güvenliği ile ilgili farklı kaynaklardan elde edilen verilerin entegre edildiği bir uygulama sunulmaktadır. Veri entegrasyonu uygulamasında kullanılan, vaka çalışması sonucunda ekipman sağlığı ile ilgili güvensiz durum kayıtlarının, farklı ekipler ve taşıma işlemleri için incelenebileceği ortaya konmuştur. Özellikle vardiya başlangıçlarının, iş sağlığı ve güvenliği ile ilgili olayların yaşanabileceği bir zaman aralığı olduğu belirlenmiştir. Veri görselleştirmesi, analiz kabiliyeti açısından sınırlı olup, günümüzde kullanılan bilgi görselleri ve modern görselleştirme araçlarına rağmen, veri içerisindeki yapılar karmaşık hale gelebilmektedir. Belirli olayların sebepleri incelenirken, veri madenciliği gibi daha sistematik yöntemlerin kullanılması önerilmektedir.

Contextualizing condoms: a cross-sectional study mapping intersections of locations of sexual contact, partner type, and substance use as contexts for sexual risk behavior among MSM in Peru

BackgroundCondomless anal intercourse (CAI) appears to be increasing among men who have sex with men (MSM) globally, and is reported to be as high as 70% in recent studies in Peru. To improve understanding of the evolving context of CAI among MSM in Peru, we studied associations between partner type, substance use, and condomless anal intercourse (CAI) in locations where MSM commonly report having sexual encounters.MethodsIn a 2017 cross-sectional study of rectal STI screening and HIV prevention, a convenience sample of MSM recruited from community venues in Lima completed a survey of demographic characteristics and sexual risk behavior with their three most recent partners. Generalized estimating equations estimated correlations of CAI with location of last sexual contact, participant substance use prior to sex, and negotiation of condom use before or during sex. The network data integration application, Cytoscape, mapped intersections of partner type, sexual orientation, substance use, and CAI by four types of locations where sex occurred: 1) Home, 2) Hotel, 3) Sauna or Internet Cabin, and 4) Public Spaces.ResultsOf 447 MSM (median age 27 years), 76.9% reported CAI with ≥1 of their last three partners. Participants reported sex with casual partners most commonly in homes (64.6%) and hotels (60.4%), and with anonymous partners most often in saunas/Internet cabins (57.5%) and public spaces (52.6%). CAI was less commonly reported in hotels (aPR, 95% CI: 0.85, 0.75–0.97) compared to homes. Participants who used marijuana before sex at home were more likely to report CAI than MSM who did not use marijuana (1.36, 1.01–1.92). Partner alcohol use before sex was associated with CAI in saunas/Internet cabins (3.17, 1.45–6.91) and public spaces (2.65, 1.41–4.98). In the sexual network maps, almost all MSM who used drugs prior to their sexual encounters used drugs with more than one of their last three partners.ConclusionsCAI was common and associated with different risk factors, like partner type and substance use, based on location where sex occurred. Novel combination HIV, STI, and substance use prevention interventions must consider how the social environments of MSM influence condom use and other sexual risk behaviors.Trial registrationClinicalTrials.gov Identifier NCT03010020, January 4, 2017.

Entity Resolution for Big Data using Combination of Supervised Meta-Blocking and pay-as-you-go Configuration

Entity resolution refers to the method of identifying the same real world object from multiple data sets. In Data cleaning and data integration application, entity resolution is an important process. When data is large the task of entity resolution becomes complex and time consuming. End-to-end entity resolution proposal involves stages like blocking (efficiently identifies duplicates), detailed comparison (refines blocking output) and clustering (identifies the set of records which may refer to the same entity). In this paper, an approach for feedback based optimization of complete entity resolution is proposed in which supervised meta-blocking is used for blocking stage. This paper proposes a technique for entity resolution which does optimization of each phase of entity resolution with benefits of supervised Meta-blocking to improve performance of entity resolution for big data

Schema-Agnostic Progressive Entity Resolution

Entity Resolution (ER) is the task of finding entity profiles that correspond to the same real-world entity. Progressive ER aims to efficiently resolve large datasets when limited time and/or computational resources are available. In practice, its goal is to provide the best possible partial solution by approximating the optimal comparison order of the entity profiles. So far, Progressive ER has only been examined in the context of structured (relational) data sources, as the existing methods rely on schema knowledge to save unnecessary comparisons: they restrict their search space to similar entities with the help of schema-based blocking keys (i.e., signatures that represent the entity profiles). As a result, these solutions are not applicable in Big Data integration applications, which involve large and heterogeneous datasets, such as relational and RDF databases, JSON files, Web corpus etc. To cover this gap, we propose a family of schema-agnostic Progressive ER methods, which do not require schema information, thus applying to heterogeneous data sources of any schema variety. First, we introduce two naive schema-agnostic methods, showing that straightforward solutions exhibit a poor performance that does not scale well to large volumes of data. Then, we propose four different advanced methods. Through an extensive experimental evaluation over 7 real-world, established datasets, we show that all the advanced methods outperform to a significant extent both the na\"ive and the state-of-the-art schema-based ones. We also investigate the relative performance of the advanced methods, providing guidelines on the method selection.

Sequence Segmentation Using Semi-Markov Conditional Random Fields

Many applications in natural language, speech processing and data integration require model-based segmentation of sequences. Semi-Markov conditional random fields (semi-CRFs) are a generalization of CRFs and provide a full conditional distribution over all possible segmentation of a sequence. Semi-CRFs are particularly suitable for tasks that entail segment-level features such as match with existing dictionary of segments. Empirical results on real-life NER tasks show that they yield higher accuracy than CRFs, but the straightforward foreword–backward inference algorithm requires 3–10 times the computation cost of CRFs. This running time can be reduced significantly by exploiting overlapping features across segments. We present a succinct representation of overlapping features and an efficient training algorithm that can sum over all possible input segmentation in time that is sub-quadratic in the input length, even while imposing no bound on the maximum segment length. Consequently, the running time becomes comparable to CRFs even with the addition of useful entity-level features on large input segments.