Similarity Search Research Articles

New Library-Based Methods for Nontargeted Compound Identification by GC-EI-MS.

While gas chromatography mass spectrometry (GC-MS) has long been used to identify compounds in complex mixtures, this process is often subjective and time-consuming and leaves a large fraction of seemingly good-quality spectra unidentified. In this work, we describe a set of new mass spectral library-based methods to assist compound identification in complex mixtures. These methods employ mass spectral uniqueness and compound ubiquity of library entries alongside noise reduction and automated comparison of retention indices to library compounds. As a test data set, we used a publicly available electron ionization mass spectrometry data set consisting of 4833 spectra of particulate organic compounds emitted by combustion of wildland fuels. In the present work, spectra in this data set were first identified using the NIST 2023 EI-MS Library and associated batch process identification software (NIST MS PepSearch) using retention-index corrected Identity Search scoring. Resulting identifications and related information were then employed to parametrize other factors that correlate with identification. A method for identifying compounds absent from but related to those present in mass spectral libraries using the Hybrid Similarity Search is illustrated. Nevertheless, some 90% of the spectra remain unidentified. Through comparison of unidentified to identified mass spectra in this data set, a new simple measure, namely median relative abundance, was developed for evaluating the likelihood of identification.

Identification and Evaluation of Neuroprotective Effect of a Butanol Extract from Centella asiatica (L.) URB

In recent years, herbal plants have been widely used in the treatment of neurodegenerative diseases. Centella asiatica (l.) urb is a significant medicinal plant renowned for its diverse ethnomedicinal applications. The present study evaluated biological activities for neuroprotective effect of C. asiatica. In summary, Centella asiatica (l.) urb was identified using DNA barcoding such as matK and rbcL genes. The sequences obtained were then compared through a similarity search using the NCBI GenBank database. Notably, the extracts of Centella asiatica exhibited high acetylcholinesterase inhibitory activity. Furthermore, the butanol extract of C. asiatica demonstrated the most significant neurotrophic effects in preliminary cell-based screenings, particularly in promoting neurite outgrowth in C6 cells. These findings suggest that C. asiatica holds considerable promise as a potential candidate for the development of pharmacological treatments for neurodegenerative diseases.

Assessing the geographical diversity of climate change risks in Japan by overlaying climatic impacts with exposure and vulnerability indicators.

Understanding multifaceted climate change risks and their interconnections is essential for effective adaptation strategies, which require comprehensive assessments of both climatic impact variations and social-environmental exposures/vulnerabilities. This study examines these interconnections and creates multitier delineations of future climate risks across Japan by overlaying homogeneous impact zones (HIZs) with exposure-vulnerability complexes (EVCs). We delineated eight EVC regions, each exhibiting similar patterns of exposure and vulnerability, via multivariate clustering and similarity search on the basis of future population and land cover/use data. Under the Shared Socioeconomic Pathway 5 scenario, 8.07% of Japan's area was identified as urban, 66.18% as forest, and 16.66% as agricultural land, depending on vulnerability and exposure characteristics. We then spatially intersected these EVCs with six HIZs, which represent common patterns of multisectoral climatic impact variations. This intersection revealed low geographical correspondence between climatic impact variations and exposure-vulnerability factors, indicating high spatial variability in climate risk across Japan. The use of EVCs helps identify areas with anticipated climate change risks. The spatial nexus between HIZs and EVCs underscores the geographical complexity and diversity of climate risks, revealing regions with high-impact variations alongside significant exposures and vulnerabilities. For example, urban EVCs highlight critical issues such as heat-related mortality and flood damage. By mapping these impact variations while focusing on exposure and vulnerability disparities, insights from HIZs and EVCs can inform future climate risk management and effective adaptation strategies at the national and regional levels.

Building knowledge graph for relevant degree recommendations using semantic similarity search and named entity recognition

Career guidance is a critical and often daunting process, particularly during the transition from high school to higher education within the Moroccan education system. Faced with a vast array of university programs and career options, students frequently struggle to make informed decisions that align with their aspirations and skills. To address this challenge, our research introduces an innovative system that combines semantic similarity search with knowledge graph (KG) construction to enhance the precision and personalization of academic recommendations. By utilizing Sentence-BERT (SBERT) for semantic similarity, we generate embedding vectors that capture nuanced relationships between student profiles and degree descriptions. Subsequently, named entity recognition (NER) is applied to extract essential information such as skills, fields of study, and career opportunities from these profiles and descriptions. The extracted entities and their interrelationships are then structured into a coherent KG, stored in a Neo4j database, enabling efficient querying and visualization of complex data connections. This approach provides a transparent and explainable framework, ultimately delivering tailored advice that aligns with students’ individual needs and educational goals.

DIMS: Distributed Index for Similarity Search in Metric Spaces

DIMS: Distributed Index for Similarity Search in Metric Spaces

AI Powered Legal Querying System using NLP

Abstract: The RAG-Based Legal Assistant Chatbot is a cutting-edge AI-powered solution designed to streamline legal information retrieval and document analysis. By leveraging Retrieval-Augmented Generation (RAG) and integrating technologies such as LangChain and FAISS (Facebook AI Similarity Search), the chatbot processes and indexes large volumes of legal documents, delivering precise and contextually relevant insights to user queries. This project addresses the challenge of navigating extensive legal literature by transforming legal PDFs into searchable embeddings stored in FAISS. Users interact with the system through a conversational interface built with Streamlit, which maintains query context to ensure accurate and accessible responses for legal professionals and non-specialists alike.The system’s backend infrastructure is implemented in Python, utilizing LangChain for efficient language model operations and FAISS for semantic search. Key functionalities include document ingestion, intelligent retrieval, and conversational interaction. The design prioritizes scalability and adaptability, enabling future integrations with diverse legal knowledge bases and supporting additional languages or jurisdictions.The chatbot’s performance has been validated through extensive testing, showcasing high retrieval precision and low response times. These features significantly enhance productivity by automating time-consuming tasks such as document search and legal analysis, allowing users to focus on critical decision-making.This project highlights the transformative potential of AI in the legal domain, bridging the gap between complex legal information and user accessibility. Future developments will aim to enhance the system's natural language processing capabilities, incorporate real-time data updates, and integrate advanced security measures to safeguard sensitive legal information. In conclusion, the RAG-Based Legal Assistant Chatbot is an intelligent and robust tool that simplifies legal information access, demonstrating how AI can revolutionize traditional industries through precision, scalability, and innovation

DPAM-AI: a domain parser for AlphaFold models powered by artificial intelligence.

Due to the breakthrough in protein structure prediction by AlphaFold, the scientific community has access to 200 million predicted protein structures with near-atomic accuracy from the AlphaFold protein structure DataBase (AFDB), covering nearly the entire protein universe. Segmenting these models into domains and classifying them into an evolutionary hierarchy hold tremendous potential for unraveling essential insights into protein function. We introduce DPAM-AI, a Domain Parser for AlphaFold Models based on Artificial Intelligence. DPAM-AI utilizes a convolutional neural network trained with previously classified domains in the Evolutionary Classification Of protein Domains (ECOD) database. DPAM-AI integrates inter-residue distances, predicted aligned errors, and sequence and structural alignments to previously classified domains detected via sequence (HHsuite) and structural (Dali) similarity searches. DPAM-AI has demonstrated its power through rigorous tests, excelling in several benchmark sets compared to its predecessor, DPAM, and other recently published domain parsers, Merizo and Chainsaw. We applied DPAM-AI to representative AFDB models for proteins classified in Pfam. We obtained representative 3D structures for 18487 (89%) of the 20795 Pfam families. The remaining families either (i) belong to viral proteins that were excluded from AFDB or (ii) do not adopt globular 3D structures. Our structure-aware domain delineation uncovered a considerable fraction (15%) of Pfam domains containing multiple structural and evolutionary units and refined the boundaries for over half. Pfam and corresponding DPAM-AI domains are at http://prodata.swmed.edu/DPAM-pfam/. Our code is deposited at https://github.com/Jsauce5p/DPAM/tree/dpam_ai, and updates will be released through https://github.com/CongLabCode/DPAM.

Research on flood peak prediction in the three gorges region based on similarity search with multisource information fusion

Research on flood peak prediction in the three gorges region based on similarity search with multisource information fusion

Search anything: segmentation-based similarity search via region prompts

AbstractSearch Anything is presented, a novel approach to perform similarity search in images. In contrast to other approaches to image similarity search, Search Anything enables users to utilize point, box, and text prompts to search for similar regions in a set of images. The region selected by a prompt is automatically segmented, and a binary feature vector is extracted. This feature vector is then used as a query for an image region index, and the images that contain the corresponding regions are returned. Search Anything is trained in a self-supervised manner on mask features extracted by the FastSAM foundation model and semantic features for masked image regions extracted by the CLIP foundation model to learn binary hash code representations for image regions. By coupling these two foundation models, images can be indexed and searched at a more fine-grained level than finding only entire similar images. Experiments on several datasets from different domains in a zero-shot setting demonstrate the benefits of Search Anything as a versatile region-based similarity search approach for images. The efficacy of the approach is further supported by qualitative results. Ablation studies are performed to evaluate how the proposed combination of semantic features and segmentation features together with masking improves the performance of Search Anything over the baseline using CLIP features alone. For large regions, relative improvements of up to 9.87% in mean average precision are achieved. Furthermore, considering context is beneficial for searching small image regions; a context of 3 times an object’s bounding box gives the best results. Finally, we measure computation time and determine storage requirements.

From Fact Drafts to Operational Systems: Semantic Search in Legal Decisions Using Fact Drafts

This research paper presents findings from an investigation in the semantic similarity search task within the legal domain, using a corpus of 1172 Hungarian court decisions. The study establishes the groundwork for an operational semantic similarity search system designed to identify cases with comparable facts using preliminary legal fact drafts. Evaluating such systems often poses significant challenges, given the need for thorough document checks, which can be costly and limit evaluation reusability. To address this, the study employs manually created fact drafts for legal cases, enabling reliable ranking of original cases within retrieved documents and quantitative comparison of various vectorization methods. The study compares twelve different text embedding solutions (the most recent became available just a few weeks before the manuscript was written) identifying Cohere’s embed-multilingual-v3.0, Beijing Academy of Artificial Intelligence’s bge-m3, Jina AI’s jina-embeddings-v3, OpenAI’s text-embedding-3-large, and Microsoft’s multilingual-e5-large models as top performers. To overcome the transformer-based models’ context window limitation, we investigated chunking, striding, and last chunk scaling techniques, with last chunk scaling significantly improving embedding quality. The results suggest that the effectiveness of striding varies based on token count. Notably, employing striding with 16 tokens yielded optimal results, representing 3.125% of the context window size for the best-performing models. Results also suggested that from the models having 8192 token long context window the bge-m3 model is superior compared to jina-embeddings-v3 and text-embedding-3-large models in capturing the relevant parts of a document if the text contains significant amount of noise. The validity of the approach was evaluated and confirmed by legal experts. These insights led to an operational semantic search system for a prominent legal content provider.

P111 An AI-assisted similarity search engine for enhanced psoriasis case identification and patient engagement

Abstract Each psoriasis patient's clinical features follow a unique pattern that does not normally fit perfectly within classical subtypes of psoriasis as described in textbooks. The objective of this study is to develop an AI-assisted similarity search engine for psoriasis patients. This enables users to upload images and retrieve similar cases from both publicly available data, namely peer-reviewed literature and public datasets, and a private collection. With a match, a patient may glean additional information about his or her personal subtype of psoriasis including potential treatment strategies, and even get in touch with other patients affected with a similar clinical pattern. The system is designed to be self-hosted, thereby enabling patients to share their own images through the completion of an informed consent form. The types of psoriasis covered are plaque-type, guttate, inverse, erythrodermic, pustular and mixed. The system offers the option of linking to similar images of lesions by manually selecting an area of interest. Furthermore, the system is capable of suggesting similarities to differential diagnoses, i.e. eczema or pityriasis rubra pilaris. The tool is intended for use by patients, general physicians and dermatologists, with the objective of enhancing patients' understanding of their conditions and facilitating discussion and exchange through an integrated forum. The system employs visual transformers as feature extractors to vectorize the uploaded images and utilizes cosine similarity to compare them with images in the database. Furthermore, supplementary data, such as sparse matching, is presented to the user in order to facilitate a deeper comprehension of the similarities between the images in question. Furthermore, the system is capable of displaying supplementary data regarding the cases stored in the database, as well as links to pertinent literature. Should they so desire, patients are afforded the opportunity to interact with other patients who have been diagnosed with similar conditions, thereby facilitating discussion of their respective cases. The preliminary findings suggest that this tool has the potential to markedly enhance patients' comprehension of their conditions, while also enabling them to utilize the tool for diagnostic assistance. Furthermore, patients can gain deeper insights into their condition, which may facilitate informed discussions and community support through the forum. The system has the potential to transform psoriasis management by providing a reliable, user-friendly platform for case comparison and patient engagement. Further research will focus on expanding the database and refining the algorithm to enhance its diagnostic capabilities and user experience.

A BLAST from the past: revisiting blastp's E-value.

The Basic Local Alignment Search Tool, BLAST, is an indispensable tool for genomic research. BLAST established itself as the canonical tool for sequence similarity search in large part thanks to its meaningful statistical analysis. Specifically, BLAST reports the E-value of each reported alignment, which is defined as the expected number of optimal local alignments that will score at least as high as the observed alignment score, assuming that the query and the database sequences are randomly generated. Here we critically evaluate the E-values provided by the standard protein BLAST (blastp), showing that they can be at times significantly conservative while at others too liberal. We offer an alternative approach based on generating a small sample from the null distribution of random optimal alignments, and testing whether the observed alignment score is consistent with it. In contrast with blastp, our significance analysis seems valid, in the sense that it did not deliver inflated significance estimates in any of our extensive experiments. Moreover, although our method is slightly conservative, it is often significantly less so than the blastp E-value. Indeed, in cases where blastp's analysis is valid (ie, not too liberal), our approach seems to deliver a greater number of correct alignments. One advantage of our approach is that it works with any reasonable choice of substitution matrix and gap penalties, avoiding blastp's limited options of matrices and penalties. In addition, we can formulate the problem using a canonical family-wise error rate control setup, thereby dispensing with E-values, which can at times be difficult to interpret. The Apache licensed source code is available at (https://github.com/batmen-lab/SGPvalue). Supplementary data are available at Bioinformatics online.

Supervised Screening of EGFR Inhibitors Validated through Computational Structural Biology Approaches.

One of the prominent challenges in breast cancer (BC) treatment is human epidermal growth factor receptor (EGFR) overexpression, which facilitates tumor proliferation and presents a viable target for anticancer therapies. This study integrates multiomics data to pinpoint promising therapeutic compounds and employs a machine learning (ML)-based similarity search to identify effective alternatives. We used BC cell line data from the Cancer Cell Line Encyclopedia (CCLE) and Genomics of Drug Sensitivity in Cancer (GDSC) databases and single-cell RNA sequencing (scRNA-seq) information that established afatinib as an efficacious candidate demonstrating superior IC50 values. Next, ML models, including support vector machine (SVM), artificial neural networks (ANN), and random forest (RF), were trained on ChEMBL data to classify compounds with similar activity to the reference drug as active or inactive. The promising candidates underwent computational structural biology assessments for their molecular interactions and conformational dynamics. Our findings indicate that compounds ChEMBL233324, ChEMBL233325, ChEMBL234580, and ChEMBL372692 exhibit potent repressive action against EGFR, underscoring their potential as active antibreast cancer agents.

Searching for similarities in the accretion flow of Seyfert 1 galaxies and cataclysmic variables based on the flare profiles of IRAS 13224–3809, 1H 0707–495, Mrk 766, and MV Lyr

Aims. We studied the fast variability of three selected active galactic nuclei (AGNs), IRAS 13224−3809, 1H 0707−495, and Mrk 766, and the cataclysmic variable MV Lyr, which were observed by the XMM-Newton and Kepler spacecrafts, respectively. Our goal is to search for the common origin of the variability and to test the so-called sandwich model, in which a geometrically thick corona surrounds a geometrically thin disc. Context. We studied the substructures of the averaged flare profiles. The flare profile method identifies individual flares in the light curve, and averages them. The direct fitting of the profile substructures identified individual characteristic frequencies that are seen in standard power density spectra (PDS) as a break frequency or quasi-periodic oscillation. The credibility of the flare profile substructures is demonstrated by comparison with the autocorrelation function. Results. We found that the flare profiles of AGNs are similar to those of a cataclysmic variable in the low state. We explain this as a consequence of a truncated inner disc in a sandwich model. The same scenario is also able to explain the characteristic break frequencies in X-ray PDS, which are not seen in the optical. We also searched for substructures in the flare profile of IRAS 13224−3809. In addition to a permanently present main flare, we found that a transient side-lobe appears before the main flare and is only seen in a high-flux period. The complex flare profile of this AGN suggests that an additional source of X-rays appears during the high-flux period. We propose a scenario in which an accretion flow fluctuation enters the sandwich corona and propagates further to some very central part of the accretion disc.

Billion-Scale Similarity Search Using a Hybrid Indexing Approach with Advanced Filtering

Abstract This paper presents a novel approach for similarity search with complex filtering capabilities on billion-scale datasets, optimized for CPU inference. Our method extends the classical IVF-Flat index structure to integrate multi-dimensional filters. The proposed algorithm combines dense embeddings with discrete filtering attributes, enabling fast retrieval in high-dimensional spaces. Designed specifically for CPU-based systems, our disk-based approach offers a cost-effective solution for large-scale similarity search. We demonstrate the effectiveness of our method through a case study, showcasing its potential for various practical uses.

Computer-Aided formulation design for pharmaceutical drug product development, part 01: Materials exploration through a visualization tool

An interactive tool has been developed to help design oral solid dosage form formulations. The tool enables quantitative explorations and comparisons of physical, bulk, and mechanical properties, and takes into account functional characteristics as well. In this manner, comparisons and clustering of both excipients and APIs can be carried out. These comparisons enable the generation of alternatives as well as surrogate identification, so as to spare resources and material.Multiple data sources were merged to create a “joint” data table with all relevant properties. Four main workflow activities are supported: Explore Materials, Search Similar APIs, Search Similar Excipients and Search Material Clusters.Multi-dimensional filtering can be superimposed to each functionality. Suggested visualizations are made particularly accessible by providing them as “standard plots”.The underlying philosophy is to empower formulation scientists to explore options, rather than prescribe decisions on exclusively mathematical grounds. The tool described here is the first step towards a holistic optimization incorporating predictions of mixture properties. Methodology of use is illustrated through three material selection application examples.

A semantic similarity search engine for movies

A semantic similarity search engine for movies

ResJobFit - end-to-end artificial neural networks based technology for job-resume matching

With the ever-growing expansion of online recruitment, a reliable person-job matching has become increasingly crucial. Due to different experience, education and specialization requirements, as well as location considerations, specified in the job advertisement, various aspects should be taken into account for reliable matching and ranking of candidates. It has been shown that matching of resumes and vacancies can be approached as either pair classification or semantic similarity search based on embeddings. While classification approaches process each vacancy-resume pair sequentially, thus resulting in quadratic time complexity, independent text embeddings and ranking is a much more efficient and scalable solution, since it has linear time complexity. In this article semantic similarity search to rank suitability of candidates with regards to vacancies has been used. ResJobFit - an end-to-end Artificial Neural Networks based technology for job-resume matching is proposed. ResJobFit technology consists of Segmentation, Parsing, Summarization and HR Embedding Module models, and their outputs (vector and attributes defining each resume or job advertisement), as well as a Vector Database in which the records are stored. Unsupervised text embeddings training for HR domain encapsulating two novel training objectives - intra- and cross-section contrastive alignment is introduced. Pretrained BERT-base model is adapted by teaching it to match summary-last employment sections of the resume with parts of the same vacancy or employment section. As baselines TFIDF, BERT, E5 and GTE have been used. The proposed unsupervised training strategy was compared against SimCSE, DeCLUTR and ConFit approaches. NDCG, MAP and MRR are used as metrics for measuring accuracy of the designed algorithm. It has been shown that the novel training objective lets it achieve significant improvement in comparison to other unsupervised training approaches. Improvement of 11% in NDCG was achieved by adapting the DeCLUTR training strategy for the HR domain based on exploiting the structure of resumes over the classical DeCLUTR training strategy on the task of ranking summaries of vacancies and resumes generated by large language models. 2% and 6% have been achieved using ResJobFit and ResJobFit with requirements matching over state-of-the-art ConFit model on the task of ranking full-text vacancies and resumes.

Federated learning-driven collaborative recommendation system for multi-modal art analysis and enhanced recommendations.

With the rapid development of artificial intelligence technology, recommendation systems have been widely applied in various fields. However, in the art field, art similarity search and recommendation systems face unique challenges, namely data privacy and copyright protection issues. To address these problems, this article proposes a cross-institutional artwork similarity search and recommendation system (AI-based Collaborative Recommendation System (AICRS) framework) that combines multimodal data fusion and federated learning. This system uses pre-trained convolutional neural networks (CNN) and Bidirectional Encoder Representation from Transformers (BERT) models to extract features from image and text data. It then uses a federated learning framework to train models locally at each participating institution and aggregate parameters to optimize the global model. Experimental results show that the AICRS framework achieves a final accuracy of 92.02% on the SemArt dataset, compared to 81.52% and 83.44% for traditional CNN and Long Short-Term Memory (LSTM) models, respectively. The final loss value of the AICRS framework is 0.1284, which is better than the 0.248 and 0.188 of CNN and LSTM models. The research results of this article not only provide an effective technical solution but also offer strong support for the recommendation and protection of artworks in practice.

COCONUT 2.0: a comprehensive overhaul and curation of the collection of open natural products database.

The COCONUT (COlleCtion of Open Natural prodUcTs) database was launched in 2021 as an aggregation of openly available natural product datasets and has been one of the biggest open natural product databases since. Apart from the chemical structures of natural products, COCONUT contains information about names and synonyms, species and organism parts in which the natural product has been found, geographic information about where the respective sample has been collected and literature references, where available. COCONUT is openly accessible at https://coconut.naturalproducts.net. Users can search textual information and perform structure, substructure, and similarity searches. The data in COCONUT areavailable for bulk download as SDF, CSV and a database dump. The web application for accessing the data is open-source. Here, we describe COCONUT 2.0, for which the web application has been completely rewritten, and the data have been newly assembled and extensively curated. New features include data submissions by users and community curation facilitated in various ways.