Storage and Retrieval of Large Data Sets: Dimensionality Reduction and Nearest Neighbour Search

Abstract

Storing and querying are two important issues that need to be addressed while designing an information retrieval system for a large and high-dimensional data set. In this work, we discuss about tackling such data, specifically about the nearest neighbour search and the efficient storage layout to store such data. The data set used in the current work has been taken from an online source called ZINC, a repository for drug like chemical structures. Processing a high dimensional data is a tough task hence dimensionality reduction should be employed. Here for dimensionality reduction is achieved through a filter-based feature selection method, based on correlation fractal dimension (CFD) discrimination measure, is used. The number of dimensions using the correlation fractal dimension are reduced from 58 to 7. To identify the nearest neighbours for a given chemical structure Tanimoto similarity coefficient is used with these reduced set of features. The nearest neighbours identified using the Tanimoto measure are stored in a storage layout known as modified inverted file. Nearest neighbours for a query can be retrieved back from the storage layout, with just one read operation from the data file thereby reducing the time for retrieval.