List Intersection Research Articles

Efficient List Intersection Algorithm for Short Documents by Document Reordering

List intersection plays a pivotal role in various domains such as search engines, database systems, and social networks. Efficient indexes and query strategies can significantly enhance the efficiency of list intersection. Existing inverted index-based algorithms fail to utilize the length information of documents and require excessive list intersections, resulting in lower efficiency. To address this issue, in this paper, we propose the LDRpV (Length-based Document Reordering plus Verification) algorithm. LDRpV filters out documents that are unlikely to satisfy the intersection results by reordering documents based on their length, thereby reducing the number of candidates. Additionally, to minimize the number of list intersection operations, an intersection and verification strategy is designed, where only the first m lists are intersected, and the resulting candidate set is directly verified. This approach effectively improves the efficiency of list intersection. Experimental results on four real datasets demonstrate that LDRpV can achieve a maximum efficiency improvement of 46.69% compared to the most competitive counterparts.

Preservation of 5-Hydroxymethylcytosine Levels in LRIG1 across Genomic DNA and Cell-Free DNA in Glioma Patients.

Cell-free DNA (cfDNA) has recently emerged as a promising minimally invasive diagnostic biomarker for various cancers. In this study, our aim was to identify cfDNA biomarkers by investigating genes that displayed significant differences between glioma patients and their corresponding controls. To accomplish this, we utilized publicly available data from the Gene Expression Omnibus, focusing on 5-hydroxymethylcytosine (5hmC) profiles in both cfDNA and genomic DNA (gDNA) from glioma patients and healthy individuals. The intersection of gene lists derived from these comparative analyses unveiled LRIG1 and ZNF703 as the two genes with elevated 5hmC levels in both the cfDNA of glioma patients and gDNA of glioma tissue compared to their respective controls. The gene expression data revealed both genes were upregulated in glioma tissue compared to normal brain tissue. Integration of 5hmC data revealed a strong positive correlation in the glioma tissue group between 5hmC and the gene expression of the LRIG1 gene. Furthermore, exploration using the AmiCa web tool indicated that LRIG1 gene expression was elevated compared to 17 other cancers included in the database, emphasizing its potential as a distinctive biomarker across multiple cancer types.

Intersections of twisted forms: New theories and double copies

Tree–level scattering amplitudes of particles have a geometrical description in terms of intersection numbers of pairs of twisted differential forms on the moduli space of Riemann spheres with punctures. We customize a catalog of twisted differential forms containing both already known and new differential forms. By pairing elements from this list intersection numbers of various theories can be furnished to compute their scattering amplitudes. Some of the latter are familiar through their CHY description, but others are unknown. Likewise, certain pairings give rise to various known and novel double–copy constructions of spin–two theories. This way we find double copy constructions for many theories, including higher derivative gravity, (partial massless) bimetric gravity and some more exotic theories. Furthermore, we present a derivation of amplitude relations in intersection theory.

A multithreaded hybrid framework for mining frequent itemsets

<p><span>Mining frequent itemsets is an area of data mining that has beguiled several researchers in recent years. Varied data structures such as Nodesets, DiffNodesets, NegNodesets, N-lists, and Diffsets are among a few that were employed to extract frequent items. However, most of these approaches fell short either in respect of run time or memory. Hybrid frameworks were formulated to repress these issues that encompass the deployment of two or more data structures to facilitate effective mining of frequent itemsets. Such an approach aims to exploit the advantages of either of the data structures while mitigating the problems of relying on either of them alone. However, limited efforts have been made to reinforce the efficiency of such frameworks. To address these issues this paper proposes a novel multithreaded hybrid framework comprising of NegNodesets and N-list structure that uses the multicore feature of today’s processors. While NegNodesets offer a concise representation of itemsets, N-lists rely on List intersection thereby speeding up the mining process. To optimize the extraction of frequent items a hash-based algorithm has been designed here to extract the resultant set of frequent items which further enhances the novelty of the framework.</span></p>

Join optimization for inverted index technique on relational database management systems

In relational database management systems (RDBMSs), an efficient join method for text retrieval using an inverted index has been developed and implemented. However, the existing crossing of the posting inverted list increases the keyword search time for large texts because of unnecessary comparisons. The relation-based search produces results by utilizing the posting list intersection. To reduce the search time for queries, a multi-way skip-merge join algorithm is proposed in this study. The proposed algorithm improves the execution speed by using a sorted inverted index posting list to minimize unnecessary comparison operations in the posting list intersection. The skip-merge join method, which minimizes unnecessary comparison operations using the aggregate function, is integrated with the multi-way join as a replacement for the existing two-way join method. The join algorithm combining skip-merge join and multi-way join shows good performance because the number of search keywords and the number of documents increase. The performance improvement of the keyword search is verified by implementing the multi-way skip-merge join algorithm in PostgreSQL, an RDBMS.

Trust: Triangle Counting Reloaded on GPUs

Triangle counting is a building block for a wide range of graph applications. Traditional wisdom suggests that i) hashing is not suitable for triangle counting, ii) edge-centric triangle counting beats vertex-centric design, and iii) communication-free and workload balanced graph partitioning is a grand challenge for triangle counting. On the contrary, we advocate that i) hashing can help the key operations for scalable triangle counting on Graphics Processing Units (GPUs), i.e., list intersection and graph partitioning, ii)vertex-centric design reduces both hash table construction cost and memory consumption, which is limited on GPUs. In addition, iii) we exploit graph and workload collaborative, and hashing-based 2D partitioning to scale vertex-centric triangle counting over 1,000 GPUswith sustained scalability. In this work, we present TRUST which performs triangle counting with the hash operation and vertex-centric mechanism at the core. To the best of our knowledge, TRUSTis the first work that achieves over one trillion Traversed Edges Per Second (TEPS) rate for triangle counting.

Internal and external memory set containment join

A set containment join operates on two set-valued attributes with a subset ( $$\subseteq $$ ) relationship as the join condition. It has many real-world applications, such as in publish/subscribe services and inclusion dependency discovery. Existing solutions can be broadly classified into union-oriented and intersection-oriented methods. Based on several recent studies, union-oriented methods are not competitive as they involve an expensive subset enumeration step. Intersection-oriented methods build an inverted index on one attribute and perform inverted list intersection on another attribute. Existing intersection-oriented methods intersect inverted lists one-by-one. In contrast, in this paper, we propose to intersect all the inverted lists simultaneously while skipping many irrelevant entries in the lists. To share computation, we utilize the prefix tree structure and extend our novel list intersection method to operate on the prefix tree. To further improve the efficiency, we propose to partition the data and process each partition separately. Each partition will be associated with a much smaller inverted index, and the set containment join cost can be significantly reduced. Moreover, to support large-scale datasets that are beyond the available memory space, we develop a novel adaptive data partition method that is designed to fully leverage the available memory and achieve high I/O efficiency, and thereby exhibiting outstanding performance for external memory set containment join. We evaluate our methods using both real-world and synthetic datasets. Experimental results demonstrate that our method outperforms state-of-the-art methods by up to 10 $$\times $$ when the dataset is completely resided in memory. Furthermore, our approach achieves up to two orders of magnitude improvement on I/O efficiency compared with a baseline method when the dataset size exceeds the main memory space.

On Efficient External-Memory Triangle Listing

Discovering triangles in large graphs is a well-studied area; however, both external-memory performance of existing methods and our understanding of the complexity involved leave much room for improvement. To shed light on this problem, we first generalize the existing in-memory algorithms into a single framework of 18 triangle-search techniques. We then develop a novel external-memory approach, which we call Pruned Companion Files (PCF), that supports operation of all 18 algorithms, while significantly reducing I/O compared to the common methods in this area. After finding the best node-traversal order, we build an implementation around it using SIMD instructions for list intersection and PCF for I/O. This method runs 5-10 times faster than the available implementations and exhibits orders of magnitude less I/O. In one of our graphs, the program finds 1 trillion triangles in 237 seconds using a desktop CPU.

Towards Predicting Graft Vs Host Disease from Predicted Tissue Specific Minor Histocompatibility Antigens

Graft versus host disease (GVHD) continues to be a life-threatening complication of stem cell transplantation for hematological malignancies. GVHD occurs in perfectly HLA-matched, related (MRD) transplants, and conversely can be absent in matched un-related (MUD) or haploidentical stem cell transplants (SCT). Minor histocompatibility antigens (mHA) have been investigated for their contribution to GVHD in SCT, which is mediated by differential presentation of these antigens in the recipient tissues to the donor derived T cells. Our group and others have shown that predicted mHA in bone marrow transplant donor-recipient pairs (DRP) range in the 1,000s, with MUD on the average having 2x as many mHA as MRD. However, repeated studies have shown that absolute number of mHA does not correlate with GVHD - that is more minor histocompatibility antigens do not lead to higher rates of GVHD. It is unknown if the tissue distribution of mHA in SCT differs significantly between patients, and what role this may have in the biology of GVHD. Therefore, our group investigated if the distribution of in silico derived mHA might correlate with GVHD occurrence. Whole exome sequencing (WES) of 77 HLA matched SCT DRP was performed and revealed thousands of nucleotide polymorphisms unique to the recipient when compared with the donor. When translated into peptide fragments and queried for binding to HLA-A, B, and C, in each DRP, these polymorphic peptides generated thousands of putative mHA/DRP in silico. Lists with the genes of origin for the mHA were then compiled for each DRP and sorted by the binding affinity of polymorphic recipient peptides to HLA; presented binders (IC50<500, termed “PB”) and strong binders (IC50<50, termed “SB”) numbered in the hundreds and were considered further.Normal gene expression data for 13 distinct tissues was obtained from the GTEx database v6 consisting of 54,354 transcripts averaged over hundreds of samples. The R package “pSI” was utilized to generate lists of genes that were disproportionately expressed in various tissues. A “Specificity index” (SI) was calculated for each gene in specific tissues by comparing its expression level in that tissue, against expression level in all other tissues, with cutoffs given by P value (Wells et al., 2015 Nucleic Acids Research). Using these SI gene lists as reference, SB mHA in the patient data set showed significant differences in predicted tissue expression of the polymorphic genes between the 77 patients. The probabilities of mHA being presented by specific tissues was seen to follow a normal distribution across patient samples, allowing for further statistical analyses that assume normal distributions. The intersection of the mHA lists (PB or SB) and the SI lists (normal tissue specific expression) were calculated (Figure 1). 16 models were then analyzed for ability to predict GVHD, and best fit was discovered using C-statistics. The SI list at a specificity of P=0.001 together with the proportion analysis had the best C-statistic (0.831) and was used for further investigation. Univariate and multivariate analysis revealed that higher proportions, but not absolute number, of antigens expressed in colonic tissue was statistically correlated with GVHD in this pilot study (Table 1). In conclusion, this pilot study reveals that the tissue distribution of WES derived mHA are not uniform across DRP. It also suggests that differences in mHA tissue distribution may contribute to GVHD. Specifically, a heavier burden of polymorphisms in the genes differentially expressed in the colon compared to other tissues may make recipients more likely to experience GVHD. [Display omitted] DisclosuresNo relevant conflicts of interest to declare.

IntegRATE: a desirability-based data integration framework for the prioritization of candidate genes across heterogeneous omics and its application to preterm birth

BackgroundThe integration of high-quality, genome-wide analyses offers a robust approach to elucidating genetic factors involved in complex human diseases. Even though several methods exist to integrate heterogeneous omics data, most biologists still manually select candidate genes by examining the intersection of lists of candidates stemming from analyses of different types of omics data that have been generated by imposing hard (strict) thresholds on quantitative variables, such as P-values and fold changes, increasing the chance of missing potentially important candidates.MethodsTo better facilitate the unbiased integration of heterogeneous omics data collected from diverse platforms and samples, we propose a desirability function framework for identifying candidate genes with strong evidence across data types as targets for follow-up functional analysis. Our approach is targeted towards disease systems with sparse, heterogeneous omics data, so we tested it on one such pathology: spontaneous preterm birth (sPTB).ResultsWe developed the software integRATE, which uses desirability functions to rank genes both within and across studies, identifying well-supported candidate genes according to the cumulative weight of biological evidence rather than based on imposition of hard thresholds of key variables. Integrating 10 sPTB omics studies identified both genes in pathways previously suspected to be involved in sPTB as well as novel genes never before linked to this syndrome. integRATE is available as an R package on GitHub (https://github.com/haleyeidem/integRATE).ConclusionsDesirability-based data integration is a solution most applicable in biological research areas where omics data is especially heterogeneous and sparse, allowing for the prioritization of candidate genes that can be used to inform more targeted downstream functional analyses.

List intersection for web search

This paper studies the optimization of list intersection, especially in the context of the matching phase of search engines. Given a user query, we intersect the postings lists corresponding to the query keywords to generate the list of documents matching all keywords. Since the speed of list intersection depends the algorithm, hardware, and list lengths and their correlations, none the existing intersection algorithms outperforms the others in every scenario. Therefore, we develop a cost-based approach in which we identify a search space, spanning existing algorithms and their combinations. We propose a cost model to estimate the cost of the algorithms with their combinations, and use the cost model to search for the lowest-cost algorithm. The resulting plan is usually a combination of 2-way algorithms, outperforming conventional 2-way and k -way algorithms. The proposed approach is more general than designing a specific algorithm, as the cost models can be adapted to different hardware. We validate the cost model experimentally on two different CPUs, and show that the cost model closely estimates the actual cost. Using both real and synthetic datasets, we show that the proposed cost-based optimizer outperforms the state-of-the-art alternatives.

Efficient set containment join

In this paper, we study the problem of set containment join. Given two collections \(\mathcal {R}\) and \(\mathcal {S}\) of records, the set containment join \(\mathcal {R} \bowtie _\subseteq \mathcal {S}\) retrieves all record pairs \(\{(r,s)\} \in \mathcal {R}\times \mathcal {S}\) such that \(r \subseteq s\). This problem has been extensively studied in the literature and has many important applications in commercial and scientific fields. Recent research focuses on the in-memory set containment join algorithms, and several techniques have been developed following intersection-oriented or union-oriented computing paradigms. Nevertheless, we observe that two computing paradigms have their limits due to the nature of the intersection and union operators. Particularly, intersection-oriented method relies on the intersection of the relevant inverted lists built on the elements of \(\mathcal {S}\). A nice property of the intersection-oriented method is that the join computation is verification free. However, the number of records explored during the join process may be large because there are multiple replicas for each record in \(\mathcal {S}\). On the other hand, the union-oriented method generates a signature for each record in \(\mathcal {R}\) and the candidate pairs are obtained by the union of the inverted lists of the relevant signatures. The candidate size of the union-oriented method is usually small because each record contributes only one replica in the index. Unfortunately, union-oriented method needs to verify the candidate pairs, which may be cost expensive especially when the join result size is large. As a matter of fact, the state-of-the-art union-oriented solution is not competitive compared to the intersection-oriented ones. In this paper, we propose a new union-oriented method, namely TT-Join, which not only enhances the advantage of the previous union-oriented methods but also integrates the goodness of intersection-oriented methods by imposing a variant of prefix tree structure. We conduct extensive experiments on 20 real-life datasets and synthetic datasets by comparing our method with 7 existing methods. The experiment results demonstrate that TT-Join significantly outperforms the existing algorithms on most of the datasets and can achieve up to two orders of magnitude speedup. Furthermore, to support large scale of datasets, we extend our techniques to distributed systems on top of MapReduce framework. With the help of careful designed load-aware distribution mechanisms, our distributed join algorithm can achieve up to an order of magnitude speedup than the baselines methods.

Choosability with union separation

List coloring generalizes graph coloring by requiring the color of a vertex to be selected from a list of colors specific to that vertex. One refinement of list coloring, called choosability with separation, requires that the intersection of adjacent lists is sufficiently small. We introduce a new refinement, called choosability with union separation, where we require that the union of adjacent lists is sufficiently large. For t≥k, a (k,t)-list assignment is a list assignment L where |L(v)|≥k for all vertices v and |L(u)∪L(v)|≥t for all edges uv. A graph is (k,t)-choosable if there is a proper coloring for every (k,t)-list assignment. We explore this concept through examples of graphs that are not (k,t)-choosable, demonstrating sparsity conditions that imply a graph is (k,t)-choosable, and proving that all planar graphs are (3,11)-choosable and (4,9)-choosable.

SILVERBACK+: scalable association mining via fast list intersection for columnar social data

We present Silverback+, a scalable probabilistic framework for accurate association rule and frequent item-set mining of large-scale social behavioral data. Silverback+ tackles the problem of efficient storage utilization and management via: (1) probabilistic columnar infrastructure and (2) using Bloom filters and sampling techniques. In addition, probabilistic pruning techniques based on Apriori method are developed, for accelerating the mining of frequent item-sets. The proposed target-driven techniques yield a significant reduction of the size of the frequent item-set candidates, as well as the required number of repetitive membership checks through a novel list intersection algorithm. Extensive experimental evaluations demonstrate the benefits of this context-aware consideration and incorporation of the infrastructure limitations when utilizing the corresponding research techniques. When compared to the traditional Hadoop-based approach for improving scalability by straightforwardly adding more hosts, Silverback+ exhibits a much better runtime performance, with negligible loss of accuracy.

Sebaceous Gland Atrophy in Psoriasis: An Explanation for Psoriatic Alopecia?

In a transcriptome study of lesional psoriatic skin (PP) versus normal skin, we found a coexpressed gene module (N5) enriched 11.5-fold for lipid biosynthetic genes. We also observed fewer visible hairs in PP skin, compared with uninvolved nonlesional psoriatic skin or normal skin (P < 0.0001). To ask whether these findings might be due to abnormalities of the pilosebaceous unit, we carried out three-dimensional morphometric analysis of paired PP and nonlesional psoriatic skin biopsies. Sebaceous glands were markedly atrophic in PP versus nonlesional psoriatic skin (91% average reduction in volume, P= 0.031). Module N5 genes were strongly downregulated in PP versus normal skin (fold change < 0.25, 44.4-fold) and strongly upregulated in sebaceous hyperplasia (fold change > 4, 54.1-fold). The intersection of PP-downregulated and sebaceous hyperplasia-upregulated gene lists generated a gene expression signature consisting solely of module N5 genes, whose expression in PP versus normal skin was inversely correlated with the signature of IL17-stimulated keratinocytes. Despite loss of visible hairs, morphometry identified elongated follicles in PP versus nonlesional psoriatic skin (average 1.7 vs. 1.2 μm, P= 0.020). These results document sebaceous gland atrophy in nonscalp psoriasis, identify a cytokine-regulated set of sebaceous gland signature genes, and suggest that loss of visible hair in PP skin may result from abnormal sebaceous gland function.

Meta-Analysis of Microarray Data of Rainbow Trout Fry Gonad Differentiation Modulated by Ethynylestradiol.

Sex differentiation in fish is a highly labile process easily reversed by the use of exogenous hormonal treatment and has led to environmental concerns since low doses of estrogenic molecules can adversely impact fish reproduction. The goal of this study was to identify pathways altered by treatment with ethynylestradiol (EE2) in developing fish and to find new target genes to be tested further for their possible role in male-to-female sex transdifferentiation. To this end, we have successfully adapted a previously developed bioinformatics workflow to a meta-analysis of two datasets studying sex reversal following exposure to EE2 in juvenile rainbow trout. The meta-analysis consisted of retrieving the intersection of the top gene lists generated for both datasets, performed at different levels of stringency. The intersecting gene lists, enriched in true positive differentially expressed genes (DEGs), were subjected to over-representation analysis (ORA) which allowed identifying several statistically significant enriched pathways altered by EE2 treatment and several new candidate pathways, such as progesterone-mediated oocyte maturation and PPAR signalling. Moreover, several relevant key genes potentially implicated in the early transdifferentiation process were selected. Altogether, the results show that EE2 has a great effect on gene expression in juvenile rainbow trout. The feminization process seems to result from the altered transcription of genes implicated in normal female gonad differentiation, resulting in expression similar to that observed in normal females (i.e. the repression of key testicular markers cyp17a1, cyp11b, tbx1), as well as from other genes (including transcription factors) that respond specifically to the EE2 treatment. The results also showed that the bioinformatics workflow can be applied to different types of microarray platforms and could be generalized to (eco)toxicogenomics studies for environmental risk assessment purposes.

Real-time twitter recommendation

We describe a production Twitter system for generating relevant, personalized, and timely recommendations based on observing the temporally-correlated actions of each user's followings. The system currently serves millions of recommendations daily to tens of millions of mobile users. The approach can be viewed as a specific instance of the novel problem of online motif detection in large dynamic graphs. Our current solution partitions the graph across a number of machines, and with the construction of appropriate data structures, motif detection can be translated into the lookup and intersection of adjacency lists in each partition. We conclude by discussing a generalization of the problem that perhaps represents a new class of data management systems.

Design and Implementation of a Cache System in Web Search Engines

Web search engine caches the results which is frequently queried by users. It is an effective approach to improve the efficiency of Web search engines. In this paper, we give some valuable experience in our design and implementation of a Web search engine cache system. We present there design principles: logical layer processing, event-based communication architecture and avoiding frequent data copy. We also introduce the architecture presented in practice, including connection processor, application processor, query results caching processor, inverted list caching processor and list intersection caching processor. Experiments are conducted in our cache system using a real Web search engine query log.

Effects of Sirt1 on DNA methylation and expression of genes affected by dietary restriction

Changes in DNA methylation across the life course may contribute to the ageing process. We hypothesised that some effects of dietary restriction to extend lifespan and/or mitigate against features of ageing result from changes in DNA methylation, so we determined if genes that respond to dietary restriction also show age-related changes in DNA methylation. In support of our hypothesis, the intersection of lists of genes compiled from published sources that (1) were differentially expressed in response to dietary restriction and (2) showed altered methylation with increased age was greater than expected. We also hypothesised that some effects of Sirt1, which may play a pivotal role in beneficial effects of dietary restriction, are mediated through DNA methylation. We thus measured effects of Sirt1 overexpression and knockdown in a human cell line on DNA methylation and expression of a panel of eight genes that respond to dietary restriction and show altered methylation with age. Six genes were affected at the level of DNA methylation, and for six expressions were affected. In further support of our hypothesis, we observed by DNA microarray analysis that genes showing differential expression in response to Sirt1 knockdown were over-represented in the complied list of genes that respond to dietary restriction. The findings reveal that Sirt1 has effects on DNA methylation across the genome and affects, in particular, the expression of genes that respond to dietary restriction. Sirt1-mediated effects on DNA methylation and, consequently, gene expression may thus be one of the mechanisms underlying the response to dietary restriction.

Statistical tests for the intersection of independent lists of genes: Sensitivity, FDR, and type I error control

Public data repositories have enabled researchers to compare results across multiple genomic studies in order to replicate findings. A common approach is to first rank genes according to an hypothesis of interest within each study. Then, lists of the top-ranked genes within each study are compared across studies. Genes recaptured as highly ranked (usually above some threshold) in multiple studies are considered to be significant. However, this comparison strategy often remains informal, in that type I error and false discovery rate (FDR) are usually uncontrolled. In this paper, we formalize an inferential strategy for this kind of list-intersection discovery test. We show how to compute a $p$-value associated with a "recaptured" set of genes, using a closed-form Poisson approximation to the distribution of the size of the recaptured set. We investigate operating characteristics of the test as a function of the total number of studies considered, the rank threshold within each study, and the number of studies within which a gene must be recaptured to be declared significant. We investigate the trade off between FDR control and expected sensitivity (the expected proportion of true-positive genes identified as significant). We give practical guidance on how to design a bioinformatic list-intersection study with maximal expected sensitivity and prespecified control of type I error (at the set level) and false discovery rate (at the gene level). We show how optimal choice of parameters may depend on particular alternative hypothesis which might hold. We illustrate our methods using prostate cancer gene-expression datasets from the curated Oncomine database, and discuss the effects of dependence between genes on the test.