Billions Of Entries Research Articles

Effective Protein-Ligand Docking Strategy via Fragment Reuse and a Proof-of-Concept Implementation.

Virtual screening is a commonly used process to search for feasible drug candidates from a huge number of compounds during the early stages of drug design. As the compound database continues to expand to billions of entries or more, there remains an urgent need to accelerate the process of docking calculations. Reuse of calculation results is a possible way to accelerate the process. In this study, we first propose yet another virtual screening-oriented docking strategy by combining three factors, namely, compound decomposition, simplified fragment grid storing k-best scores, and flexibility consideration with pregenerated conformers. Candidate compounds contain many common fragments (chemical substructures). Thus, the calculation results of these common fragments can be reused among them. As a proof-of-concept of the aforementioned strategies, we also conducted the development of REstretto, a tool that implements the three factors to enable the reuse of calculation results. We demonstrated that the speed and accuracy of REstretto were comparable to those of AutoDock Vina, a well-known free docking tool. The implementation of REstretto has much room for further performance improvement, and therefore, the results show the feasibility of the strategy. The code is available under an MIT license at https://github.com/akiyamalab/restretto.

MIN: Co-Governing Multi-Identifier Network Architecture and Its Prototype on Operator’s Network

IP protocol is the core of TCP/IP network layer. However, since IP address and its Domain Name are allocated and managed by a single agency, there are risks of centralization. The semantic overload of IP address also reduces its scalability and mobility, which further hinders the security. This paper proposes a co-governing Multi-Identifier Network (MIN) architecture that constructs a network layer with parallel coexistence of multiple identifiers, including identity, content, geographic information, and IP address. On the management plane, we develop an efficient management system using consortium blockchain with voting consensus, so the network can simultaneously manage and support by hundreds or thousands of nodes with high throughput. On the data plane, we propose an algorithm merging hash table and prefix tree (HTP) for FIB, which avoids the false-negative error and can inter-translate different identifiers with tens of billions of entries. Further, we propose a scheme to transport IP packets using CCN as a tunnel for supporting progressive deployment. We deployed the prototype of MIN to the largest operators' network in Mainland China, Hongkong and Macao, and demonstrated that the network can register identifier under co-governing consensus algorithm, support VoD service very well.

Fast and scalable method for distributed Boolean tensor factorization

How can we analyze tensors that are composed of 0’s and 1’s? How can we efficiently analyze such Boolean tensors with millions or even billions of entries? Boolean tensors often represent relationship, membership, or occurrences of events such as subject–relation–object tuples in knowledge base data (e.g., ‘Seoul’-‘is the capital of’-‘South Korea’). Boolean tensor factorization (BTF) is a useful tool for analyzing binary tensors to discover latent factors from them. Furthermore, BTF is known to produce more interpretable and sparser results than normal factorization methods. Although several BTF algorithms exist, they do not scale up for large-scale Boolean tensors. In this paper, we propose DBTF, a distributed method for Boolean CP (DBTF-CP) and Tucker (DBTF-TK) factorizations running on the Apache Spark framework. By distributed data generation with minimal network transfer, exploiting the characteristics of Boolean operations, and with careful partitioning, DBTF successfully tackles the high computational costs and minimizes the intermediate data. Experimental results show that DBTF-CP decomposes up to 163–323 $$\times $$ larger tensors than existing methods in 82–180 $$\times $$ less time, and DBTF-TK decomposes up to 83–163 $$\times $$ larger tensors than existing methods in 86–129 $$\times $$ less time. Furthermore, both DBTF-CP and DBTF-TK exhibit near-linear scalability in terms of tensor dimensionality, density, rank, and machines.

Distributed stochastic gradient descent for link prediction in signed social networks

This paper considers the link prediction problem defined over a signed social network, where the relationship between any two network users can be either positive (friends) or negative (foes). Given a portion of the relationships, the goal of link prediction is to identify the rest unknown ones. This task resorts to completing the adjacency matrix of the signed social network, which is low rank or approximately low rank. Considering the large scale of the adjacency matrix, in this paper, we adopt low-rank matrix factorization models for the link prediction problem and solve them through asynchronous distributed stochastic gradient descent algorithms. The low-rank matrix factorization models effectively reduce the size of the parameter space, while the asynchronous distributed stochastic gradient descent algorithms enable fast completion of the adjacency matrix. We validate the proposed algorithms using two real-world datasets on a distributed shared-memory computation platform. Numerical results demonstrate that the asynchronous distributed stochastic gradient descent algorithms achieve nearly linear computional speedups with respect to the number of computational threads, and are able to complete an adjacency matrix of ten billions of entries within 10 s.

On the Nyström and Column-Sampling Methods for the Approximate Principal Components Analysis of Large Datasets

In this article, we analyze approximate methods for undertaking a principal components analysis (PCA) on large datasets. PCA is a classical dimension reduction method that involves the projection of the data onto the subspace spanned by the leading eigenvectors of the covariance matrix. This projection can be used either for exploratory purposes or as an input for further analysis, for example, regression. If the data have billions of entries or more, the computational and storage requirements for saving and manipulating the design matrix in fast memory are prohibitive. Recently, the Nyström and column-sampling methods have appeared in the numerical linear algebra community for the randomized approximation of the singular value decomposition of large matrices. However, their utility for statistical applications remains unclear. We compare these approximations theoretically by bounding the distance between the induced subspaces and the desired, but computationally infeasible, PCA subspace. Additionally we show empirically, through simulations and a real data example involving a corpus of emails, the trade-off of approximation accuracy and computational complexity.

Nanocubes for Real-Time Exploration of Spatiotemporal Datasets

Consider real-time exploration of large multidimensional spatiotemporal datasets with billions of entries, each defined by a location, a time, and other attributes. Are certain attributes correlated spatially or temporally? Are there trends or outliers in the data? Answering these questions requires aggregation over arbitrary regions of the domain and attributes of the data. Many relational databases implement the well-known data cube aggregation operation, which in a sense precomputes every possible aggregate query over the database. Data cubes are sometimes assumed to take a prohibitively large amount of space, and to consequently require disk storage. In contrast, we show how to construct a data cube that fits in a modern laptop's main memory, even for billions of entries; we call this data structure a nanocube. We present algorithms to compute and query a nanocube, and show how it can be used to generate well-known visual encodings such as heatmaps, histograms, and parallel coordinate plots. When compared to exact visualizations created by scanning an entire dataset, nanocube plots have bounded screen error across a variety of scales, thanks to a hierarchical structure in space and time. We demonstrate the effectiveness of our technique on a variety of real-world datasets, and present memory, timing, and network bandwidth measurements. We find that the timings for the queries in our examples are dominated by network and user-interaction latencies.

Database Access Pattern Protection Without Full-Shuffles

Privacy protection is one of the fundamental security requirements for database outsourcing. A major threat is information leakage from database access patterns generated by query executions. The standard private information retrieval (PIR) schemes, which are widely regarded as theoretical solutions, entail <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> ) computational overhead per query for a database with <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> items. Recent works propose to protect access patterns by introducing a trusted component with constant storage size. The resulting privacy assurance is as strong as PIR, though with <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i> (1) online computation cost, they still have <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> ) amortized cost per query due to periodically full database shuffles. In this paper, we design a novel scheme in the same model with provable security, which only shuffles a portion of the database. The amortized server computational complexity is reduced to <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i> (√{ <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> log <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> / <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> }). With a secure storage storing thousands of items, our scheme can protect the access pattern privacy of databases of billions of entries, at a lower cost than those using ORAM-based poly-logarithm algorithms.

PARAFAC algorithms for large-scale problems

Parallel factor analysis (PARAFAC) is a tensor (multiway array) factorization method which allows to find hidden factors (component matrices) from a multidimensional data. Most of the existing algorithms for the PARAFAC, especially the alternating least squares (ALS) algorithm need to compute Khatri–Rao products of tall factors and multiplication of large matrices, and due to this require high computational cost and large memory and are not suitable for very large-scale-problems. Hence, PARAFAC for large-scale data tensors is still a challenging problem. In this paper, we propose a new approach based on a modified ALS algorithm which computes Hadamard products, instead Khatri–Rao products, and employs relatively small matrices. The new algorithms are able to process extremely large-scale tensors with billions of entries. Extensive experiments confirm the validity and high performance of the developed algorithm in comparison with other well-known algorithms.

Simulation of Huge Data Processing via Large Matrix Manipulation Using Hilbert Engine

Modeling multi-physics structure is always accompanied by a huge complex data that need to be analyzed. For example, modeling of an explosion interacting with a solid object inherently consists of multiple phenomena including shock propagation, solid interaction, response of the solid, damage, debris movement, and thermal effects. Thus, the data processing speed always plays an important role. However, currently available computational software takes long time to reach the processing goals. The main goal of this presentation is to demonstrate faster processing using Hilbert Engine in huge matrix (billions of entries) manipulation in a single work-station. Indeed, Hilbert Engine is extremely efficient, fast and versatile in computation based on its abilities to bypass any operating system. Besides, it can operate as a stand-alone data analysis system or in a distributed, asynchronous processing network of cooperating system.