Data Discovery Research Articles

MSTGI: a multi-scale spatio-temporal grid index model for remote-sensing big data retrieval

ABSTRACT To promote the transformation of remote sensing (RS) data into geoscience knowledge, it is necessary to provide better data discovery capabilities, especially when large amounts of RS data have been accumulated. Spatio-temporal range query is one of them to enable data discovery on RS images by the spatio-temporal range. Although existing RS image indexing methods are suitable for data discovery based on spatio-temporal range queries, they do not fully consider the problem of low data retrieval efficiency when the temporal and spatial range scales are unbalanced. To address this problem, we propose a multi-scale spatio-temporal grid index model (MSTGI). First, we divide the time dimension into three levels according to the granularity of year, month and day, and build an independent grid index structure for the object at different levels. Second, we design an adaptive hierarchical indexing strategy to perform parallel retrieval in an appropriate combination of partitions. MSTGI linearizes the grid obtained after global geospatial subdivision using Hilbert curves. Our experimental results, obtained on the Landsat series satellite dataset, reveal that the proposed method improves query efficiency levels by approximately 12.582% and 87.754% compared with GeoSOT-ST and AGMC at various spatial scales, respectively, and reaches 100% recall.

Advancements in condition monitoring and fault diagnosis of rotating machinery: A comprehensive review of image-based intelligent techniques for induction motors

Recently, condition monitoring (CM) and fault detection and diagnosis (FDD) techniques for rotating machinery (RM) have witnessed substantial advancements in recent decades, driven by the increasing demand for enhanced reliability, efficiency, and safety in industrial operations. CM of valuable and high-cost machinery is crucial for performance tracking, reducing maintenance costs, enhancing efficiency and reliability, and minimizing mechanical failures. While various FDD methods for RM have been developed, these predominantly focus on signal processing diagnostics techniques encompassing time, frequency, and time-frequency domains, intelligent diagnostics, image processing, data fusion, data mining, and expert systems. However, there is a noticeable knowledge gap regarding the specific review of image-based CM and FDD. The objective of this research is to address the aforementioned gap in the literature by conducting a comprehensive review of image-based intelligent techniques for CM and fault FDD specifically applied to induction motors (IMs). The focus of the study is to explore the utilization of image-based methods in the context of IMs, providing a thorough examination of the existing literature, methodologies, and applications. Furthermore, the integration of image-based techniques in CM and FDD holds promise for enhanced accuracy, as visual information can provide valuable insights into the physical condition and structural integrity of the IMs, thereby facilitating early FDD and proactive maintenance strategies. The review encompasses the three main faults associated with IMs, namely bearing faults, stator faults, and rotor faults. Furthermore, a thorough assessment is conducted to analyze the benefits and drawbacks associated with each approach, thereby enabling an evaluation of the efficacy of image-based intelligent techniques in the context of CM and FDD. Finally, the paper concludes by highlighting key issues and suggesting potential avenues for future research.

Approximate sorting and its applications in I/O model

Sorting is a fundamental component in many different applications including web indexing engines, geographic information systems, data mining and database systems. However, the existing optimal algorithms still cause huge costs when the input data becomes very large. Thus, the approximate sorting for big data is considered in this paper. The goal of approximate sorting for big data is to generate an approximate sorted result, but using less CPU and I/O cost. For big data, we consider the approximate sorting and applications for approximate sorted results in I/O model.The quality of approximate sorting results is usually measured by the distance metrics on permutation space. However, the existing metrics on permutation space are not available for external approximate sorting algorithms. Thus, we propose a new kind of metric named External metric, which ignores the errors and dislocations that happened in each I/O block. The External metric of Spearman's footrule metric is named as External Spearman's footrule metric (short as ESP metric), which is analyzed in this paper.Furthermore, to facilitate a better evaluation of the approximate sorted result, we propose a new metric, named as errors, which directly states the number of dislocation of the elements. The External errors are also considered in this paper.Then, according to the rate-distortion relationship under these two metrics, the lower bound of these two metrics on external approximate sorting problem with t I/O operations is proved. We propose a k-passes external approximate sorting algorithm, named as EASORT, and prove that EASORT is asymptotically optimal under ESP metric.Finally, we consider the applications on approximate sorting results. An index for the approximate sorted result is proposed. The single and range query on approximate sorted result using this index are also analyzed. Further, the sort-merge join on two relations, where one of the relations is approximate sorted or both relations are approximate sorted, are all discussed in this paper.

Ranking Opportunities for Autonomous Trucks Using Data Mining and GIS

The inefficiency of transporting goods contributes to reduced economic growth and environmental sustainability in a country. Autonomous trucks (ATs) are emerging as a solution, but the imbalance in the weight moved and ton-miles produced by long-haul and short-haul trucking creates a challenge in targeting initial deployments. This study offers a unique solution by presenting a robust method that combines data mining and geographic information systems (GISs) to identify the optimal routes for ATs based on a top-down approach to maximize business benefits. Demonstrated in a U.S. case study, this method revealed that despite accounting for only 16% of the weight moved, long-haul trucking produced 56% of the ton-miles, implying a high potential for ATs in this segment. The method identified eight key freight zones in five U.S. states that accounted for 27% of the long-haul weight and suggested optimal routes for initial AT deployment. Interstate 45 emerged as a pivotal route in the shortest paths among these freight zones. This suggests that stakeholders should seek to prioritize funding for infrastructure upgrades and maintenance along that route and the other routes identified. The findings will potentially benefit a broad range of stakeholders. Companies can strategically focus resources to achieve maximum market share, regulators can streamline policymaking to facilitate AT adoption while ensuring public safety, and transportation agencies can better plan infrastructure upgrades and maintenance. Users globally can apply the methodological framework as a reliable tool for decision-making about where to initially deploy ATs.

Sharing, curation and metadata as essential components of the data management plan

This paper stresses the critical importance that data management plans actually be data management and sharing plans, that encourage data-sharing at the early stages of research; that indicate that data must be made as open as possible (and as closed as necessary); and that specify a minimum level of curation and collection of essential metadata. The author stresses that fact that it is no longer sufficient to post material openly and hope that others will be able to make use of it. The use of metadata and careful data curation of the data are essential to facilitate the discovery and successful re-use of data.

Solo: Data Discovery Using Natural Language Questions Via A Self-Supervised Approach

Most deployed data discovery systems, such as Google Datasets, and open data portals only support keyword search. Keyword search is geared towards general audiences but limits the types of queries the systems can answer. We propose a new system that lets users write natural language questions directly. A major barrier to using this learned data discovery system is it needs expensive-to-collect training data, thus limiting its utility. In this paper, we introduce a self-supervised approach to assemble training datasets and train learned discovery systems without human intervention. It requires addressing several challenges, including the design of self-supervised strategies for data discovery, table representation strategies to feed to the models, and relevance models that work well with the synthetically generated questions. We combine all the above contributions into a system, Solo, that solves the problem end to end. The evaluation results demonstrate the new techniques outperform state-of-the-art approaches on well-known benchmarks. All in all, the technique is a stepping stone towards building learned discovery systems.

User Segmentation Based on Purchasing Habits and Preferences on the Amazon Platform Using K-Means Clustering

As a large company, Amazon operates an online marketplace with a diverse user base exhibiting varied purchasing habits. This diversity challenges Amazon to provide tailored services and marketing strategies for each user with distinct characteristics. Therefore, this research aims to assist Amazon in segmenting its users based on their characteristics, enabling the implementation of targeted marketing strategies and service provision for each user. The study employs the K-Means Clustering method to segment Amazon platform users based on their purchasing behavior, site feature interactions, and preferences. The research process involves Knowledge Data Discovery (KDD) stages, including data processing, attribute selection, and applying the K-Means Clustering algorithm. The analysis results reveal five distinct user clusters, each with unique characteristics reflecting user behavior and preferences. These clusters depict variations in purchasing frequency, interactions with site features, and responses to product recommendations. This user segmentation provides valuable insights for Amazon to develop more focused marketing strategies, enhance personalized services, and improve overall customer satisfaction.

A Quantum-Inspired Direct Learning Strategy for Positive and Unlabeled Data

Learning from only positive and unlabeled (PU) data has broad applications in fields such as web data mining, product recommendations and medical diagnosis, which aims to train a binary classifier in the absence of negative labeled data. However, due to the lack of negative label information, prevailing PU learning methods usually rely on prior knowledge of unknown class distributions heavily. In fact, without additional constraints imposed by the prior knowledge, a direct learning strategy to coordinate the underlying clustering information in unlabeled data with the label information from positive training data is often considered challenging. To tackle this challenge, we propose a direct PU learning strategy using quantum formalization. By employing neural networks as backends, the samples are mapped into two-qubit composite systems, which should be understood here as mathematical entities encapsulating various classical distributions of two classical bits. Subsequently, the two qubits within the systems are trained to be as independent as possible from each other, capturing patterns of different classes. At the same time, their measurement results serving as the model outputs are encouraged to be maximally dissimilar. These characteristics enable effective training of classifiers on PU data. After formulating an appropriate discriminant rule, we introduce a quantum-inspired PU method named qPU using the direct learning strategy. This method not only has the potential to alleviate parameter sensitivity issues caused by prior estimation in other methods but is also straightforward to implement. Finally, experiments conducted on 13 classical datasets validate the effectiveness of qPU.

The European Platform for Neurodegenerative Diseases (EPND)

AbstractBackgroundBiomarker discovery, development, and validation are reliant on large‐scale analyses of high‐quality samples and data. However, discovery, access, and sharing of data and samples are hindered both by silos that limit collaboration, and by complex requirements for secure, legal, and ethical sharing. The European Platform for Neurodegenerative Diseases (EPND) project aims to address these challenges and accelerate biomarker discovery, development and validation, by building a scalable, sustainable platform for collaboration, data and sample sharing. In this presentation we will give an overview of the design of EPND and the achievements in the first year of this 5‐year project.MethodEPND is a multidisciplinary consortium of 29 public and private partners with expertise on data discovery, ethical, legal and regulatory aspects of data sharing, biomarker studies, stakeholder engagement and sustainability. EPND will leverage existing data and research infrastructures, including the Alzheimer’s Disease Data Initiative (ADDI) AD Workbench, which will allow the platform to connect to a global research network. EPND was launched in November 2021 and is funded by the Innovative Medicines Initiative (IMI, grant agreement number 101034344, epnd.org).ResultIn this first year of the project, the design of the data platform has been finalised. EPND has launched its Cohort Catalogue, which features information on participants, samples and data from 60+ neurodegeneration cohorts, spanning 17 countries and 13 disease areas, and involving over 155,000 patients. In addition to standardised operational procedures on sample selection, biobanking, and data harmonisation, we have also developed a white paper on legal, ethical and regulatory aspects of data and sample sharing. Finally, we have performed case studies on ATN biomarkers in plasma and blood in 350 patients with Alzheimer’s disease, Parkinson’s disease and Lewy body dementia.ConclusionWe initiated a multidisciplinary consortium with the goal to accelerate collaborative research into the discovery and validation of biomarkers, to support development of diagnostics and disease‐modifying therapies for AD and other neurodegenerative diseases.

Dementias Platform UK: Bringing genetics into life

AbstractBackgroundDPUK is a data repository bringing together a wide range of population and clinical cohorts from the UK, Europe and South Korea. It enables data discovery, variable selection, and data access for multi‐modal, multi‐cohort analysis in a shared and secure environment. Neurodegenerative dementias are group of diseases with highly heterogeneous pathology and with an overlapping genetic component that is poorly understood. Combing genetic data from different studies is important due to the increase in power this provides, but this presents an additional challenge due to differences in genotyping arrays and lack of overlapping variants.MethodWe have curated and harmonized genome‐wide data from 6 studies (Brains for Dementia Research, EPIC Norfolk, Generation Scotland, Airwave‐chip 1, Airwave‐chip 2, MRC National Survey of Health Development (NSHD)) within the DPUK platform. We created a pipeline that performs rigorous quality‐control (QC) analysis and imputes genetic variance with the 1000 Genome reference panel using the minimac algorithm. Further, standard pruning and thresholding Polygenic Risk Score (PRS) have been generated with 5 summary statistics related to neurodegenerative diseases (clinical AD, clinical/proxy AD, FTD, ALS, PD) for all 6 studies separately and for the combined dataset.ResultResearch‐ready imputed genetic data that have undergone QC and the PRS, based upon the latest neurodegenerative GWAS, are available for the 6 cohorts separately, and for the combined dataset of 60,670 individuals.ConclusionPreparing multiple datasets to a common standard for research‐readiness increases scientific opportunity and allows the wider research community to access and process data at scale and pace.

Perceptions of Librarians toward the Adoption of Linked Open Data; a Case of Selected University Libraries in Uganda

The study aimed at exploring the perception of librarians toward adopting Linked Open Data in Uganda, a case of selected university libraries. An exploratory survey approach was employed for this study using a mixed methods approach of data collection. The data was analyzed using SPSS and Ms Excel. The targeted population was 78 professional librarians in seven selected public universities. The study was guided by the UTAUT model using the performance expectancy construct. The findings revealed that the respondents perceived that Linked Data would assist the patrons in discovering relevant information and data through links to other data repositories. They also believed that Linked Data interlinks would be used to describe the relationship between two Uniform Resource Identifiers (URIs). The respondents were convinced that Linked Data will enrich data discovery and improve the users’ overall search experiences. The study found that respondents perceived that Linked Data will assist the patrons to discover relevant data and benefit research, enrich discovery of data, hence improving their overall searches. Linked Open Data is very important to enable data sharing through the Semantic Web.

Plant Reactome Knowledgebase: empowering plant pathway exploration and OMICS data analysis.

Plant Reactome (https://plantreactome.gramene.org) is a freely accessible, comprehensive plant pathway knowledgebase. It providescurated referencepathways from rice (Oryza sativa) andgene-orthology-based pathway projections to 129 additional species, spanning single-cell photoautotrophs, non-vascular plants, and higher plants, thus encompassing a wide-ranging taxonomic diversity. Currently, Plant Reactome houses a collection of 339 reference pathways, covering metabolic andtransport pathways, hormone signaling, genetic regulations ofdevelopmental processes, and intricate transcriptional networks that orchestrate a plant's response to abiotic and biotic stimuli. Beyond being a mere repository, Plant Reactome serves as a dynamic data discovery platform. Users can analyze andvisualize omics data, such as gene expression, gene-gene interaction, proteome,and metabolome data, all within the rich context of plant pathways. Plant Reactome is dedicated to fostering data interoperability, upholding global data standards, and embracing the tenets of the Findable, Accessible, Interoperableand Re-usable (FAIR) data policy.

Mining toxicogenomic data for dose-responsive pathways: implications in advancing next-generation risk assessment.

Introduction: While targeted investigation of key toxicity pathways has been instrumental for biomarker discovery, unbiased and holistic analysis of transcriptomic data provides a complementary systems-level perspective. However, in a systematic context, this approach has yet to receive comprehensive and methodical implementation. Methods: Here, we took an integrated bioinformatic approach by re-analyzing publicly available MCF7 cell TempO-seq data for 44 ToxCast chemicals using an alternative pipeline to demonstrate the power of this approach. The original study has focused on analyzing the gene signature approach and comparing them to in vitro biological pathway altering concentrations determined from ToxCast HTS assays. Our workflow, in comparison, involves sequential differential expression, gene set enrichment, benchmark dose modeling, and identification of commonly perturbed pathways by network visualization. Results: Using this approach, we identified dose-responsive molecular changes, biological pathways, and points of departure in an untargeted manner. Critically, benchmark dose modeling based on pathways recapitulated points of departure for apical endpoints, while also revealing additional perturbed mechanisms missed by single endpoint analyses. Discussion: This systems-toxicology approach provides multifaceted insights into the complex effects of chemical exposures. Our work highlights the importance of unbiased data-driven techniques, alongside targeted methods, for comprehensively evaluating molecular initiating events, dose-response relationships, and toxicity pathways. Overall, integrating omics assays with robust bioinformatics holds promise for improving chemical risk assessment and advancing new approach methodologies (NAMs).

The dark side of KIBS agency, the creative economy, and regional sustainability

This contribution develops critique from analysing forms of misconduct by knowledge-intensive business services (KIBS) firms, especially the largest, globally located and client-interactive on all five continents. Management consultancy infractions range from supplying spurious advice, to advising clients on fraudulent practice, to acting complicitly in corrupt government practices, including engaging in ‘state capture’ by channelling internal state revenue into private holding bank accounts. Because the litany of misconduct is too massive for encompassing in a single contribution, and in the spirit of this task, the spotlight is only on a few cases that represent typical ‘creative economy’ companies contracted to ‘Big 3′ P-KIBS (Professional KIBS) corporations. Of importance to open innovation’s critical edge, the contribution articulates the puzzle of why academic KIBS research has presented the field without comment on its ‘dark’ side, questioning ‘leadership’ and ‘agency’ ‘observer’ and ‘confirmation bias’. For interpretation of data discovery, we utilise evolutionary ‘pattern recognition’ methodology set within an evolutionary theoretical framework.

Dataset Discovery and Exploration: A Survey

Data scientists are tasked with obtaining insights from data. However, suitable data is often not immediately at hand, and there may be many potentially relevant datasets in a data lake or in open data repositories. As a result, data discovery and exploration are necessary, but often time consuming, steps in a data analysis workflow. Data discovery is the process of identifying datasets that may meet an information need. Data exploration is the process of understanding the properties of candidate datasets and the relationships between them. Data discovery and data exploration often go hand in hand and benefit from tool support. This article surveys research areas that can contribute to data discovery and exploration, particularly considering dataset search, data navigation, data annotation and schema inference. For each of these areas, we identify key dimensions that can be used to characterize approaches and the values they can hold, and apply the dimensions to describe and compare prominent results. In addition, by surveying several adjacent areas that are often considered in isolation, we identify recurring techniques and alternative approaches to related challenges, thereby placing results within a wider context than is generally considered.

InterMat: A Blockchain-Based Materials Data Discovery and Sharing Infrastructure

Material research and development driven by data analysis necessitates a substantial volume of data. However, conventional material data sharing platforms encounter challenges in sharing and integrating data across multiple platforms. This article proposes a blockchain-based materials data discovery and sharing infrastructure—InterMat, which is a material big data management and sharing framework model integrating cloud platforms and blockchain. It could support the full lifecycle of materials data sharing, including data generation, management, discovery, sharing, traceability, and valuation. The architecture of the InterMat, its unique method of constructing a consortium chain, and the protocol for data discovery are presented in this paper. Additionally, the method for materials data identifier and blockchain certification is established, which allows for a unified identifier on the blockchain and cloud-based data addressing from various organizations. InterMat has data discovery algorithms for various materials to achieve the discovery of similar materials data from different nodes. Furthermore, we have designed some blockchain smart contracts for InterMat to encourage data sharing across nodes. These contracts include a proof smart contract that records data sharing activities, ensuring transparency and traceability in the materials data flow. The other contract is a value-estimating contract to encourage high-quality data sharing. Finally, this article introduces the application case of InterMat, using steel materials as an example to demonstrate its applications in data management, data discovery, data valuation, etc. This study successfully addresses various challenges associated with the cross-platform sharing of materials data, such as issues related to data discovery, data rights and control, and willingness to share. InterMat can assist material researchers in discovering and accessing more data, which would create a new ecology for sharing data.

Discovery of Novel miRNAs in Colorectal Cancer: Potential Biological Roles and Clinical Utility.

Deregulated miRNAs are associated with colorectal cancer (CRC), with alterations depending on the tumor location. Novel tissue-specific miRNAs have been identified in different tumors and are associated with cancer. We used miRMaster to identify novel miRNAs in CRC from the TCGA and GEO data (discovery and validation groups). We used TCGA data from five tissues to analyze miRNA tissue specificity. miRDB was used to predict miRNA targets, and the UCSC Xena Browser was used to evaluate target expression. After successive analyses, we identified 15 novel miRNAs with the same expression patterns in CRC in both the discovery and validation groups. Four molecules (nov-miR-13844-5p, nov-miR-7154-5p, nov-miR-5035-3p, and nov-miR-590-5p) were differentially expressed in proximal and distal CRC. The nov-miR-3345-5p and nov-miR-13172-3p, which are upregulated in tumors, are only expressed in colorectal tissues. These molecules have been linked to a worse prognosis in right-sided colon and rectal carcinomas. An analysis revealed an association between eight novel miRNAs and 81 targets, mostly cancer-related genes, with varying expression based on tumor location. These findings provide new miRNAs with potential biological relevance, molecular biomarkers, and therapeutic targets for CRC treatment.

Thematic Analysis of Big Data in Financial Institutions Using NLP Techniques with a Cloud Computing Perspective: A Systematic Literature Review

This literature review explores the existing work and practices in applying thematic analysis natural language processing techniques to financial data in cloud environments. This work aims to improve two of the five Vs of the big data system. We used the PRISMA approach (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) for the review. We analyzed the research papers published over the last 10 years about the topic in question using a keyword-based search and bibliometric analysis. The systematic literature review was conducted in multiple phases, and filters were applied to exclude papers based on the title and abstract initially, then based on the methodology/conclusion, and, finally, after reading the full text. The remaining papers were then considered and are discussed here. We found that automated data discovery methods can be augmented by applying an NLP-based thematic analysis on the financial data in cloud environments. This can help identify the correct classification/categorization and measure data quality for a sentiment analysis.

Exploring traditional Chinese medicine as a potential treatment for sarcopenia: A network pharmacology and data mining analysis of drug selection and efficacy.

Sarcopenia, as an increasingly pressing clinical issue, can be ameliorated through employment of traditional Chinese medicines. However, the current lack of specific pharmacological interventions for Sarcopenia necessitates further exploration of novel possibilities in traditional Chinese medicine for the treatment of this condition, utilizing advanced methodologies such as web pharmacology and data mining. Screening the essential targets of Sarcopenia, conducting matching between target and active molecules, as well as active molecules and herbs. Employing data mining techniques to analyze the screening outcomes, and molecular docking to compare the binding activities of active molecules with target proteins. The approach of using herbs for the treatment of Sarcopenia involves 13 targets, with 414 active compounds and 367 types of herbs. Data mining reveals that the herbs used in treating Sarcopenia are primarily characterized by their bitter taste, exerting their effects through dispelling dampness and promoting blood circulation. Moreover, 2 new formulas are postulated. Furthermore, molecular docking analysis indicates that the main active components of the herbs can be observed to tightly bind with the targets. Through network pharmacology and molecular docking, our findings reveal that herbs contain 15 key active components and 5 key targets, which correspond to 7 major herbs and 2 new formulas. Academically, these findings hold significant reference value for the development of novel drugs targeting Sarcopenia.

Self-Organizing Computational System for Network Anomaly Exploration using Learning Algorithms

The forum in the nation for reporting information security flaws had 14,871 reports by the end of 2021, a 46.6% increase from 2020. The total of 5,567 high risk vulnerabilities, an increase of nearly 1,400 over the previous year. Evidently, both the total number of vulnerabilities found annually, and the total number of high-risk vulnerabilities are rising. In order for data mining technology to play a wider part in the predictive investigation of network security models, it is advised that its capability have to be improved. This paper combines the concepts of data mining (DM) with machine learning (ML), which introduces similar technologies from DM technology and security establishing collection channel, thereby finally introduces the computer network security maintenance process based on data mining in order to improve the application effect of DM in the predictive analysis of network security models. In this paper, a self-organizing neural network technique that detects denial of service (DOS) in complicated networks quickly, effectively, and precisely is introduced. It also analyses a number of frequently employed computer data mining methods, including association, clustering, classification, neural networks, regression, and web data mining. Finally, it introduces a computer data mining method based on the self-organizing (SO) algorithm. In comparison to conventional techniques, the SO algorithm-based computer data mining technology is also used in defensive detection tests against Dos attacks. A detection average accuracy rate of more than 98.56% and a detection average efficiency gain of more than 20% are demonstrated by experimental data to demonstrate that tests based on the Data Mining connected SO algorithm have superior defensive detection effects than standard algorithms.