Query Time Research Articles

Hierarchical Cut Labelling - Scaling Up Distance Queries on Road Networks

Answering the shortest-path distance between two arbitrary locations is a fundamental problem in road networks. Labelling-based solutions are the current state-of-the-arts to render fast response time, which can generally be categorised into hub-based labellings, highway-based labellings, and tree decomposition labellings. Hub-based and highway-based labellings exploit hierarchical structures of road networks with the aim to reduce labelling size for improving query efficiency. However, these solutions still result in large search spaces on distance labels at query time, particularly when road networks are large. Tree decomposition labellings leverage a hierarchy of vertices to reduce search spaces over distance labels at query time, but such a hierarchy is generated using tree decomposition techniques, which may yield very large labelling sizes and slow querying. In this paper, we propose a novel solution hierarchical cut 2-hop labelling (HC2L) to address the drawbacks of the existing works. Our solution combines the benefits of hierarchical structures from both perspectives - reduce the size of a distance labelling at preprocessing time and further reduce the search space on a distance labelling at query time. At its core, we propose a new hierarchy, balanced tree hierarchy, which enables a fast, efficient data structure to reduce the size of distance labelling and to select a very small subset of labels to compute the shortest-path distance at query time. To speed up the construction process of HC2L, we further propose a parallel variant of our method, namely HC2L^p. We have evaluated our solution on 10 large real-world road networks through extensive experiments. The results show that our method is 1.5-4 times faster in terms of query processing while being comparable in terms of labelling construction time and achieving up to 60% smaller labelling size compared to the state-of-the-art approaches.

Equitable Top-k Results for Long Tail Data

For datasets exhibiting long tail phenomenon, we identify a fairness concern in existing top-k algorithms, that return a "fixed" set of k results for a given query. This causes a handful of popular records (products, items, etc) getting overexposed and always be returned to the user query, whereas, there exists a long tail of niche records that may be equally desirable (have similar utility). To alleviate this, we propose θ-Equiv-top-k-MMSP inside existing top-k algorithms - instead of returning a fixed top-k set, it generates all (or many) top-k sets that are equivalent in utility and creates a probability distribution over those sets. The end user will be returned one of these sets during the query time proportional to its associated probability, such that, after many draws from many end users, each record will have as equal exposure as possible (governed by uniform selection probability). θ-Equiv-top-k-MMSP is formalized with two sub-problems. (a) θ-Equiv-top-k-Sets to produce a set S of sets, each set has k records, where the sets are equivalent in utility with the top-k set; (b) MaxMinFair to produce a probability distribution over S, that is, PDF(S), such that the records in S have uniform selection probability. We formally study the hardness of θ-Equiv-top-k-MMSP. We present multiple algorithmic results - (a) An exact solution for θ-Equiv-top-k-Sets, and MaxMinFair. (b) We design highly scalable algorithms that solve θ-Equiv-top-k-Sets through a random walk and is backed by probability theory, as well as a greedy solution designed for MaxMinFair. (c) We finally present an adaptive random walk based algorithm that solves θ-Equiv-top-k-Sets and MaxMinFair at the same time. We empirically study how θ-Equiv-top-k-MMSP can alleviate a equitable exposure concerns that group fairness suffers from. We run extensive experiments using 6 datasets and design intuitive baseline algorithms that corroborate our theoretical analysis.

Intelligent design of network multimedia using big data and virtual Artificial Intelligence technology

AbstractIn recent years, with the rapid development of the Internet of Things and communication technology, the application of network multimedia has become increasingly popular. At present, network multimedia has become a new type of mass media. Due to its interchangeability and extensive dissemination, it creates favourable conditions for information dissemination and acceptance in terms of vision, hearing, touch etc. However, at the same time, there are also some problems. The network multimedia does not have enough original information and the information retrieval is not intelligent, so this has caused great trouble to users. In order to solve these problems, the authors attempted to use the combination of Big Data (BD) and virtual Artificial Intelligence (AI) technology to realise the intelligent design of network multimedia. Therefore, based on the theoretical analysis, the authors also verified the effect of BD and virtual AI technology on the intelligent design of network multimedia. It was found that the intelligent design method proposed by the authors had good application effect in network multimedia, and could effectively improve the information retrieval speed of network multimedia. The query time of this method was 1.87 s less than that of the traditional method when the amount of information was 1000 and 18.16 s less than that of the traditional method when the amount of information was 40,000. At the same time, this method could also improve the accuracy of network multimedia information recommendation, and make network multimedia have better sharing, so as to provide more services for users. In addition, the discussion of BD and virtual AI technology in network multimedia intelligent design could also broaden the application scope of the Internet of Things and promote its better development.

A One-Pass Clustering Based Sketch Method for Network Monitoring

Network monitoring solutions need to cope with increasing network traffic volumes, as a result, sketch-based monitoring methods have been extensively studied to trade accuracy for memory scalability and storage reduction. However, sketches are sensitive to skewness in network flow distributions due to hash collisions, and need complicated performance optimization to adapt to line-rate packet streams. We provide Jellyfish, an efficient sketch method that performs one-pass clustering over the network stream. One-pass clustering is realized by adapting the monitoring granularity from the whole network flow to fragments called subflows, which not only reduces the ingestion rate but also provides an efficient intermediate representation for the input to the sketch. Jellyfish provides the network-flow level query interface by reconstructing the network-flow level counters by merging subflow records from the same network flow. We provide probabilistic analysis of the expected accuracy of both existing sketch methods and Jellyfish. Real-world trace-driven experiments show that Jellyfish reduces the average estimation errors by up to six orders of magnitude for per-flow queries, by six orders of magnitude for entropy queries, and up to ten times for heavy-hitter queries.

CoCo-trie: Data-aware compression and indexing of strings

We address the problem of compressing and indexing a sorted dictionary of strings to support efficient lookups and more sophisticated operations, such as prefix, predecessor, and range searches. This problem occurs as a key task in a plethora of applications, and thus it has been deeply investigated in the literature since the introduction of tries in the ’60s.We introduce a new data structure, called the COmpressed COllapsed Trie (CoCo-trie), that hinges on a pool of techniques to compress subtries (of arbitrary depth) into succinctly-encoded and efficiently-searchable trie macro-nodes with a possibly large fan-out. Then, we observe that the choice of the subtries to compress depends on the trie structure and its edge labels. Hence, we develop a data-aware optimisation approach that selects the best subtries to compress via the above pool of succinct encodings, with the overall goal of minimising the total space occupancy and still achieving efficient query time. We also investigate some variants of this approach that induce interesting space–time trade-offs in the CoCo-trie design.Our experimental evaluation on six diverse and large datasets (representing URLs, XML data, DNA and protein sequences, database records, and search-engine dictionaries) shows that the space–time performance of well-established and highly-engineered data structures solving this problem is very input-sensitive. Conversely, our CoCo-trie provides a robust and uniform improvement over all competitors for half of the datasets, and it results on the Pareto space–time frontier for the others, thus offering new competitive trade-offs.

Application of EFL pedagogy based on cloud computing platform in online interactive teaching of foreign languages

Abstract Internet information technology has greatly aided the improvement of EFL teaching. Aiming at the weaknesses of traditional EFL teaching, this paper proposes a hybrid EFL teaching model based on the cloud computing platform, which is designed in terms of both object model and platform architecture. In the task assignment of EFL teaching, the CRP algorithm is improved based on fuzzy control, and the FCRP algorithm for teaching task assignments is constructed. Meanwhile, from the needs of professional training, the recommendation optimization of teaching resources was realized through keyword parsing, and the resource recommendation effect and learning input were analyzed based on the performance test. The final results of the platform test in this paper show that the average response time for login and query is 1.621 seconds and 0.935 seconds, respectively, and both meet the EFL interactive teaching requirements.

OmniSketch: Efficient Multi-Dimensional High-Velocity Stream Analytics with Arbitrary Predicates

A key need in different disciplines is to perform analytics over fast-paced data streams, similar in nature to the traditional OLAP analytics in relational databases - i.e., with filters and aggregates. Storing unbounded streams, however, is not a realistic, or desired approach due to the high storage requirements, and the delays introduced when storing massive data. Accordingly, many synopses/sketches have been proposed that can summarize the stream in small memory (usually sufficiently small to be stored in RAM), such that aggregate queries can be efficiently approximated, without storing the full stream. However, past synopses predominantly focus on summarizing single-attribute streams, and cannot handle filters and constraints on arbitrary subsets of multiple attributes efficiently. In this work, we propose OmniSketch, the first sketch that scales to fast-paced and complex data streams (with many attributes), and supports count aggregates with filters on multiple attributes, dynamically chosen at query time. The sketch offers probabilistic guarantees, a favorable space-accuracy tradeoff, and a worst-case logarithmic complexity for updating and for query execution. We demonstrate experimentally with both real and synthetic data that the sketch outperforms the state-of-the-art, and that it can approximate complex ad-hoc queries within the configured accuracy guarantees, with small memory requirements.

Similarity Discriminating Algorithm for Scientific Research Projects

An enormous challenge for project management is to identify similar research projects accurately and efficiently among numerous proposals. To address this challenge, this paper proposes an algorithm to calculate the similarity between research projects using an improved generating method for fused word order sentence vectors based on USIF (unsupervised random walk sentence embeddings). The experimental results show that the proposed algorithm is about 15.8% more accurate than the existing approaches. The authors also propose a pre-checking algorithm by introducing a complex research cooperation graph to enhance query efficiency. The results show the pre-checking method reduces the query time cost by 96% on average.

The Examination of SQL Queries Efficiency in Chosen IT System

Optimization of SQL queries is an important part of any system using a database. Several different optimization methods have been selected and presented in this article, The research was conducted on the MS SQL database engine. The execution times of SQL queries were carefully examined according to defined criteria, then performance optimization was done, followed by repeated tests to obtain results after the optimization. Finally, the results were compared through which conclusions were drawn.

Performance analysis of databases created in virtualized and containerized environment

Database systems are currently implemented on different environments - containerized and virtualized. This paper aimed to compare the performance of relational databases running on these two environments. Therefore, two research hypotheses were formulated. The first hypothesis assumed that databases running on Docker containers are more efficient than those on virtual machines. The second one assumed that the Oracle is the most efficient database regardless of which environment it was run on. MySQL, PostgreSQL, Microsoft SQL Server and Oracle databases were tested. The study measured the execution times of INSERT, UPDATE, DELETE and SELECT queries. Each test was repeated 100 times. It was stated that databases running on Docker containers outperform instances running on virtual machines. Furthermore, it was found that the PostgreSQL database have a definite advantage in performance over the rest, analysed databases, which allowed the second hypothesis to be rejected.

Analyze the effectiveness of ETL processes implemented using SQL and Apache HiveQL languages

In the era of digitization, where data is collected in ever-increasing quantities, efficient processing is required. The article analyzes the performance of SQL and HiveQL, for scenarios of varying complexity, focusing on the execution time of individual queries. The tools used in the study are also discussed. The results of the study for each language are summarized and compared, highlighting their strengths and weaknesses, as well as identifying their possible areas of application.

A comparative analysis of the performance of the relational database and the Hadoop environment in the context of analytical data processing

The article presents a detailed comparative analysis of the performance of a Microsoft SQL Server relational database and an Apache Hadoop environment in the context of analytical data processing. The study was carried out by execut-ing more than a dozen research scenarios with different queries on datasets of varying sizes. For each research scenario, the average query execution time on different datasets was compared. Based on the results, it was found that the average execution time of queries from the presented scenarios is significantly shorter in MS SQL Server than in Apache Ha-doop.

String Indexing with Compressed Patterns

Given a string S of length n , the classic string indexing problem is to preprocess S into a compact data structure that supports efficient subsequent pattern queries. In this article, we consider the basic variant where the pattern is given in compressed form and the goal is to achieve query time that is fast in terms of the compressed size of the pattern. This captures the common client-server scenario, where a client submits a query and communicates it in compressed form to a server. Instead of the server decompressing the query before processing it, we consider how to efficiently process the compressed query directly. Our main result is a novel linear space data structure that achieves near-optimal query time for patterns compressed with the classic Lempel-Ziv 1977 (LZ77) compression scheme. Along the way, we develop several data structural techniques of independent interest, including a novel data structure that compactly encodes all LZ77 compressed suffixes of a string in linear space and a general decomposition of tries that reduces the search time from logarithmic in the size of the trie to logarithmic in the length of the pattern.

Searching natural neighbors in an accelerated way

Natural neighbor (NaN) is an adaptive neighbor concept for automatically determining k value for k-nearest neighbors, which has been proved effective in clustering analysis, classification, and outlier detection. However, the existing algorithms for searching NaN employ the global search strategy, which increases unnecessary consumption of time on non-critical points. To solve this problem, we propose a novel accelerated algorithm by utilizing a local search strategy, called ASNN. ASNN is based on the fact that if remote points have reverse nearest neighbors, others are likely to have reverse nearest neighbors. Firstly, it extracts the remote points by defining remote radius. Secondly, only the neighbors of remote points are searched, so that ASNN can quickly obtain the natural neighbor eigenvalue (NaNE). Compared to existing algorithms for searching NaNs, ASNN drastically reduces the number of query points and time consumption. The efficiency and effectiveness of ASNN are proved on synthetic and real datasets.

Selecting cardiac magnetic resonance images suitable for annotation of pulmonary arteries using an active-learning based deep learning model

An increasing and aging patient population poses a growing burden on healthcare professionals. Automation of medical imaging diagnostics holds promise for enhancing patient care and reducing manpower required to accommodate an increasing patient-population. Deep learning, a subset of machine learning, has the potential to facilitate automated diagnostics, but commonly requires large-scaled labeled datasets. In medical domains, data is often abundant but labeling is a laborious and costly task. Active learning provides a method to optimize the selection of unlabeled samples that are most suitable for improvement of the model and incorporate them into the model training process. This approach proves beneficial when only a small number of labeled samples are available. Various selection methods currently exist, but most of them employ fixed querying schedules. There is limited research on how the timing of a query can impact performance in relation to the number of queried samples. This paper proposes a novel approach called dynamic querying, which aims to optimize the timing of queries to enhance model development while utilizing as few labeled images as possible. The performance of the proposed model is compared to a model trained utilizing a fully-supervised training method, and its effectiveness is assessed based on dataset size requirements and loss rates. Dynamic querying demonstrates a considerably faster learning curve in relation to the number of labeled samples used, achieving an accuracy of 70% using only 24 samples, compared to 82% for a fully-supervised model trained on the complete training dataset of 1017 images.

Transferable adversarial distribution learning: Query-efficient adversarial attack against large language models

It is a challenging task to fool a text classifier based on deep neural networks under the black-box setting where the target model can only be queried. Among the existing black-box attacks, decision-based methods have a large query cost due to exponential perturbation space and greedy search strategy. Transfer-based methods, on the other hand, tend to overfit the surrogate model and thus fail when applied to unknown target models. In this paper, we propose a straightforward yet highly effective adversarial attack framework for black-box transformer-based models, thereby exposing vulnerabilities within large language models. Specifically, we leverage a fine-tuned large language model as a white-box surrogate model and optimize a distribution of adversarial text. This distribution is parameterized by a continuous-valued matrix based upon the surrogate model. To avoid overfitting of the distribution and improve its adversarial transferability, we incorporate an additional causal language model into our framework as a constraint model. Based on this constraint model, we add language model perplexity and semantic consistency as regularization terms during the distribution training process. To further reduce the number of queries to the target model, i.e., improve the threat level of examples drawn from our distribution, we employ a geometric loss strategy to ensure that the distribution training process learns the optimal perturbation. Extensive experimental studies have been carried out on benchmark datasets and the results demonstrate significant improvement on the performance and query efficiency under black-box setting in comparison with well-established approaches. Our approach achieves an 80.98% reduction in BERT model accuracy while consuming only 21.86% of the query times required by prior attacks.

Efficient Complex Aggregate Queries with Accuracy Guarantee Based on Execution Cost Model over Knowledge Graphs

Knowledge graphs (KGs) have gained prominence for representing real-world facts, with queries of KGs being crucial for their application. Aggregate queries, as one of the most important parts of KG queries (e.g., “ What is the average price of cars produced in Germany?”), can provide users with valuable statistical insights. An efficient solution for KG aggregate queries is approximate aggregate queries with semantic-aware sampling (AQS). This balances the query time and result accuracy by estimating an approximate aggregate result based on random samples collected from a KG, ensuring that the relative error of the approximate aggregate result is bounded by a predefined error. However, AQS is tailored for simple aggregate queries and exhibits varying performance for complex aggregate queries. This is because a complex aggregate query usually consists of multiple simple aggregate queries, and each sub-query influences the overall processing time and result quality. Setting a large error bound for each sub-query yields quick results but with a lower quality, while aiming for high-quality results demands a smaller predefined error bound for each sub-query, leading to a longer processing time. Hence, devising efficient and effective methods for executing complex aggregate queries has emerged as a significant research challenge within contemporary KG querying. To tackle this challenge, we first introduced an execution cost model tailored for original AQS (i.e., supporting simple queries) and founded on Taylor’s theorem. This model aids in identifying the initial parameters that play a pivotal role in the efficiency and efficacy of AQS. Subsequently, we conducted an in-depth exploration of the intrinsic relationship of the error bounds between a complex aggregate query and its constituent simple queries (i.e., sub-queries), and then we formalized an execution cost model for complex aggregate queries, given the accuracy constraints on the error bounds of all sub-queries. Harnessing the multi-objective optimization genetic algorithm, we refined the error bounds of all sub-queries with moderate values, to achieve a balance of query time and result accuracy for the complex aggregate query. An extensive experimental study on real-world datasets demonstrated our solution’s superiority in effectiveness and efficiency.

An Efficient Keywords Search in Temporal Social Networks

With the increasing of requirements from many aspects, various queries and analyses arise focusing on social network. Time is a common and necessary dimension in various types of social networks. Social networks with time information are called temporal social networks, in which time information can be the time when a user sends message to another user. Keywords search in temporal social networks consists of finding relationships between a group users that has a set of query labels and is valid within the query time interval. It provides assistance in social network analysis, classification of social network users, community detection, etc. However, the existing methods have limitations in solving temporal social network keyword search problems. We propose a basic algorithm, the discrete timestamp algorithm, with the intention of turning the problem into a traditional keyword search on social networks. We also propose an approximative algorithm based on the discrete timestamp algorithm, but it still suffers from the traditional algorithms’ low efficiency. To further improve the performance, we propose a new algorithm based on dynamic programming to solve the keyword search in temporal social network. The main idea is to extend a vertex into a solution by edge-growth operation and tree-merger operation. We also propose two powerful pruning techniques to reduce the intermediate results during the extension. Additionally, all of the algorithms we proposed are capable of handling a variety of ranking functions, and all of them can be made to conform to top-N keyword querying. The efficiency and effectiveness of the proposed algorithms are verified through extensive empirical studies.

BioThings Explorer: a query engine for a federated knowledge graph of biomedical APIs.

Knowledge graphs are an increasingly common data structure for representing biomedical information. These knowledge graphs can easily represent heterogeneous types of information, and many algorithms and tools exist for querying and analyzing graphs. Biomedical knowledge graphs have been used in a variety of applications, including drug repurposing, identification of drug targets, prediction of drug side effects, and clinical decision support. Typically, knowledge graphs are constructed by centralization and integration of data from multiple disparate sources. Here, we describe BioThings Explorer, an application that can query a virtual, federated knowledge graph derived from the aggregated information in a network of biomedical web services. BioThings Explorer leverages semantically precise annotations of the inputs and outputs for each resource, and automates the chaining of web service calls to execute multi-step graph queries. Because there is no large, centralized knowledge graph to maintain, BioThings Explorer is distributed as a lightweight application that dynamically retrieves information at query time. More information can be found at https://explorer.biothings.io and code is available at https://github.com/biothings/biothings_explorer.

Efficient Dynamic Weighted Set Sampling and Its Extension

Given a weighted set S of n elements, weighted set sampling (WSS) samples an element in S so that each element a i ; is sampled with a probability proportional to its weight w ( a i ). The classic alias method pre-processes an index in O ( n ) time with O ( n ) space and handles WSS with O (1) time. Yet, the alias method does not support dynamic updates. By minor modifications of existing dynamic WSS schemes, it is possible to achieve an expected O (1) update time and draw t independent samples in expected O ( t ) time with linear space, which is theoretically optimal. But such a method is impractical and even slower than a binary search tree-based solution. How to support both efficient sampling and updates in practice is still challenging. Motivated by this, we design BUS , an efficient scheme that handles an update in O (1) amortized time and draws t independent samples in O (log n + t) time with linear space. A natural extension of WSS is the weighted independent range sampling (WIRS) , where each element in S is a data point from R. Given an arbitrary range Q = [ℓ, r ] at query time, WIRS aims to do weighted set sampling on the set S Q of data points falling into range Q. We show that by integrating the theoretically optimal dynamic WSS scheme mentioned above, it can handle an update in O (log n ) time and can draw t independent samples for WIRS in O (log n + t ) time, the same as the state-of-the-art static algorithm. Again, such a solution by integrating the optimal dynamic WSS scheme is still impractical to handle WIRS queries. We further propose WIRS-BUS to integrate BUS to handle WIRS queries, which handles each update in O (log n ) time and draws t independent samples in O (log 2 n + t ) time with linear space. Extensive experiments show that our BUS and WIRS-BUS are efficient for both sampling and updates.