Query Capabilities Research Articles

Leveraging Graph Analytics for Energy Efficiency Certificates

As energy efficiency is becoming a subject of utter importance in today’s societies, the European Union and a vast number of organizations have put a lot of focus on it. As a result, huge amounts of data are generated at an unprecedented rate. After thorough analysis and exploration, these data could provide a variety of solutions and optimizations regarding the energy efficiency subject. However, all the potential solutions that could derive from the aforementioned procedures still remain untapped due to the fact that these data are yet fragmented and highly sophisticated. In this paper, we propose an architecture for a Reasoning Engine, a mechanism that provides intelligent querying, insights and search capabilities, by leveraging technologies that will be described below. The proposed architecture has been developed in the context of the H2020 project called MATRYCS. In this paper, the reasons that resulted from the need of efficient ways of querying and analyzing the large amounts of data are firstly explained. Subsequently, several use cases, where related technologies were used to address real-world challenges, are presented. The main focus, however, is put in the detailed presentation of our Reasoning Engine’s implementation steps. Lastly, the outcome of our work is demonstrated, showcasing the derived results and the optimizations that have been implemented.

A Framework for Interactive Visual Interpretation of Remote Sensing Data

Machine learning methods have shown tremendous success in understanding earth observation data; however, recently, there is a rising claim toward explainable machine learning approaches. Concerned researchers found interpretable visualizations to be greatly helpful in understanding how a model works. In this research, we propose a framework for interactive and interpretable visualization of remote sensing data using two machine learning models and an Elasticsearch (ES) database. Two explainable machine learning models, namely, bag-of-visual-words (BoVWs) and latent Dirichlet allocation (LDA) are chosen to model the data in an unsupervised manner and give a textual representation. The textualized remote sensing data are stored in an ES database. This framework offers several fast content-based search functionalities exploiting the full-text query capabilities of ES based on the respective representations and also offers an efficient storage mechanism for the data.

Fission stories: using PomBase to understand Schizosaccharomyces pombe biology.

PomBase (www.pombase.org), the model organism database (MOD) for the fission yeast Schizosaccharomyces pombe, supports research within and beyond the S. pombe community by integrating and presenting genetic, molecular, and cell biological knowledge into intuitive displays and comprehensive data collections. With new content, novel query capabilities, and biologist-friendly data summaries and visualization, PomBase also drives innovation in the MOD community.

Query Processing in Blockchain Systems: Current State and Future Challenges

When, in 2008, Satoshi Nakamoto envisioned the first distributed database management system that relied on cryptographically secured chain of blocks to store data in an immutable and tamper-resistant manner, his primary use case was the introduction of a digital currency. Owing to this use case, the blockchain system was geared towards efficient storage of data, whereas the processing of complex queries, such as provenance analyses of data history, is out of focus. The increasing use of Internet of Things technologies and the resulting digitization in many domains, however, have led to a plethora of novel use cases for a secure digital ledger. For instance, in the healthcare sector, blockchain systems are used for the secure storage and sharing of electronic health records, while the food industry applies such systems to enable a reliable food-chain traceability, e.g., to prove compliance with cold chains. In these application domains, however, querying the current state is not sufficient—comprehensive history queries are required instead. Due to these altered usage modes involving more complex query types, it is questionable whether today’s blockchain systems are prepared for this type of usage and whether such queries can be processed efficiently by them. In our paper, we therefore investigate novel use cases for blockchain systems and elicit their requirements towards a data store in terms of query capabilities. We reflect the state of the art in terms of query support in blockchain systems and assess whether it is capable of meeting the requirements of such more sophisticated use cases. As a result, we identify future research challenges with regard to query processing in blockchain systems.

SamQL: a structured query language and filtering tool for the SAM/BAM file format

BackgroundThe Sequence Alignment/Map Format Specification (SAM) is one of the most widely adopted file formats in bioinformatics and many researchers use it daily. Several tools, including most high-throughput sequencing read aligners, use it as their primary output and many more tools have been developed to process it. However, despite its flexibility, SAM encoded files can often be difficult to query and understand even for experienced bioinformaticians. As genomic data are rapidly growing, structured, and efficient queries on data that are encoded in SAM/BAM files are becoming increasingly important. Existing tools are very limited in their query capabilities or are not efficient. Critically, new tools that address these shortcomings, should not be able to support existing large datasets but should also do so without requiring massive data transformations and file infrastructure reorganizations.ResultsHere we introduce SamQL, an SQL-like query language for the SAM format with intuitive syntax that supports complex and efficient queries on top of SAM/BAM files and that can replace commonly used Bash one-liners employed by many bioinformaticians. SamQL has high expressive power with no upper limit on query size and when parallelized, outperforms other substantially less expressive software.ConclusionsSamQL is a complete query language that we envision as a step to a structured database engine for genomics. SamQL is written in Go, and is freely available as standalone program and as an open-source library under an MIT license, https://github.com/maragkakislab/samql/.

Ontology-based modelling of lifecycle underground utility information to support operation and maintenance

Operation and maintenance (O&M) of underground utilities is crucial to ensure normal functionality of utilities by performing different inspection and maintenance activities. A variety of data are required by maintenance staff to evaluate utility condition and make decisions in planning maintenance works. Nevertheless, data required for utility O&M are usually originated from different sources, with various formats or representations. Interpreting and integrating such heterogenous data for utility O&M remains a significant challenge. Therefore, this paper proposes an ontology-based framework for modelling lifecycle utility information such as to facilitate activities and support decision making during utility O&M. The framework consists of (1) a data layer to collect various types of utility lifecycle data, (2) a knowledge layer to extract common mechanisms and methods for utility O&M, (3) an ontology layer where a domain ontology, Utility Operation and Maintenance Ontology (UOMO), is developed with entities and semantic rules to represent different categories of information related to utility projects, (4) an application layer to automate manual tasks and support decision making by leveraging the query and reasoning capability of ontology. To validate the framework, a case study is performed using real-world inspection data where a user interface is developed to allow users to implement different functions without technical knowledge. The case study demonstrates that the framework can, efficiently and automatically, extract and integrate various information to assist different activities, e.g. defect rating, condition assessment and validation, as well as to support decision making in prioritizing maintenance works, and selecting appropriate maintenance methods. This study contributes by proposing the first ontology-based framework for improving efficiency in utility O&M, where the query and semantic reasoning capabilities of the ontology are fully utilized. The developed ontology with lifecycle utility information is expected to facilitate data interoperability in the utility domain and can be extended with other knowledge for large-scale infrastructure management.

Unleashing the power of querying streaming data in a temporal database world: A relational algebra approach

Modern data-intensive applications have to manage huge quantities of streaming/relational data and need advanced query capabilities involving combinations of continuous queries (CQs) and one-time queries (OTQs) also requiring the verification of complex temporal conditions.In this paper, we go beyond the disjointed panorama of current approaches and adopt a new holistic approach to the integration of stream processing capabilities into the temporal database world based on the streaming table concept. To this end, we propose a full-fledged query interface composed of a TSQL2-like query language with an underlying algebraic framework. The algebraic framework, which is aimed at implementing the query interface on top of a working DBMS, is made up of: (a) the extended temporal algebra TA⋆ supporting OTQs with an hybrid temporal semantics (sequenced and non-sequenced); (b) the continuous temporal algebra CTA that extends TA⋆ with window expressions for CQ specification; (c) the translation of CTA expressions into TA⋆ ones that can be executed by a traditional DBMS with an extended kernel.

Research on Classroom Teaching Evaluation and Instruction System Based on GIS Mobile Terminal

With the development of mobile terminal technology, GIS has begun to be applied in all walks of life. With its powerful spatial indexing, positioning, query, analysis, and graphics processing capabilities, it has become the evaluation and guidance analysis of domestic and foreign scholars. As an essential tool to improve the quality of teaching in our country’s primary and secondary schools and universities and to understand the status quo of education and teaching levels, appropriate teaching evaluation and guidance are essential. In order to adopt a better method to carry out teaching work, this research uses GIS mobile terminal to analyze the classroom teaching evaluation and guidance system, and through the research on teaching evaluation, the teaching work is made more effective. This paper uses questionnaire survey method, index weighted evaluation method, and fuzzy analytic hierarchy process as research methods. In the experimental part, a classroom teaching evaluation and guidance system based on GIS mobile terminal is designed. In the analysis part, a comprehensive analysis of teaching evaluation and guidance is carried out, including the analysis of teacher teaching evaluation, the analysis of teaching evaluation system, and the analysis of teachers and students’ understanding of teaching evaluation. Among them, after adopting the teaching evaluation system of GIS mobile terminal, the number of students who thought the teaching evaluation system was useless dropped from 4.6% to 1.8%. Students tend to emphasize the teacher’s effectiveness in teaching (57.2%) and interesting teaching content in the classroom (56.9%). This analysis has a certain significance for teaching evaluation, and it occupies a relatively important proportion in the improvement of the evaluation system. In general, the teaching evaluation system using GIS mobile terminals can improve the communication and teaching diagnosis between teachers and students and help further improve teaching.

Research on Engineering Project Management Method Based on BIM Technology

Construction projects require a significant amount of money and other resources in order to carry them out in an effective way. Cost control is a critical step in ensuring the success of the project and increasing its value. Nowadays, the large-scale application of BIM technology has ushered a technical change in the development of construction projects, which has greatly increased the level and efficiency of project management. Due to BIM technology, the construction engineering quality is significantly improved which really helped in obtaining the social and economic benefits. However, the BIM technology in our country is started relatively late and the technical force is not strong enough; then correspondingly, we must deal with many problems in the process of developing and using BIM technology based projects. In view of the fact and to address this issue, this paper establishes a three-dimensional architectural model based on the relevant information and data of the construction project using the BIM technology. The proposed model optimizes the clustering of construction project information data and adaptively configures the whole life cycle process of the construction projects. In addition, the performance of the proposed system has been compared and tested with the other systems as well which shows how good the proposed system is, as compared to the other systems. Further, the proposed model makes the artificial intelligence efficiency of the project management better. The simulation results show that the model not only has good access and query capabilities but also greatly improves the intelligence level of the project management.

A Spatio-Temporal Local Association Query Algorithm for Multi-Source Remote Sensing Big Data

It is extremely important to extract valuable information and achieve efficient integration of remote sensing data. The multi-source and heterogeneous nature of remote sensing data leads to the increasing complexity of these relationships, and means that the processing mode based on data ontology cannot meet requirements any more. On the other hand, the multi-dimensional features of remote sensing data bring more difficulties in data query and analysis, especially for datasets with a lot of noise. Therefore, data quality has become the bottleneck of data value discovery, and a single batch query is not enough to support the optimal combination of global data resources. In this paper, we propose a spatio-temporal local association query algorithm for remote sensing data (STLAQ). Firstly, we design a spatio-temporal data model and a bottom-up spatio-temporal correlation network. Then, we use the method of partition-based clustering and the method of spectral clustering to measure the correlation between spatio-temporal correlation networks. Finally, we construct a spatio-temporal index to provide joint query capabilities. We carry out local association query efficiency experiments to verify the feasibility of STLAQ on multi-scale datasets. The results show that the STLAQ weakens the barriers between remote sensing data, and improves their application value effectively.

DisMaNET: A network-based tool to cross map disease vocabularies

Background and ObjectivesThe growing integration of healthcare sources is improving our understanding of diseases. Cross-mapping resources such as UMLS play a very important role in this area, but their coverage is still incomplete. With the aim to facilitate the integration and interoperability of biological, clinical and literary sources in studies of diseases, we built DisMaNET, a system to cross-map terms from disease vocabularies by leveraging the power and interpretability of network analysis. MethodsFirst, we collected and normalized data from 8 disease vocabularies and mapping sources to generate our datasets. Next, we built DisMaNET by integrating the generated datasets into a Neo4j graph database. Then we exploited the query mechanisms of Neo4j to cross-map disease terms of different vocabularies with a relevance score metric and contrasted the results with some state-of-the-art solutions. Finally, we made our system publicly available for its exploitation and evaluation both through a graphical user interface and REST APIs. ResultsDisMaNET contains almost half a million nodes and near nine hundred thousand edges, including hierarchical and mapping relationships. Its query capabilities enabled the detection of connections between disease vocabularies that are not present in major mapping sources such as UMLS and the Disease Ontology, even for rare diseases. Furthermore, DisMaNET was capable of obtaining more than 80% of the mappings with UMLS reported in MonDO and DisGeNET, and it was successfully exploited to resolve the missing mappings in the DISNET project. ConclusionsDisMaNET is a powerful, intuitive and publicly available system to cross-map terms from different disease vocabularies. Our study proves that it is a competitive alternative to existing mapping systems, incorporating the potential of network analysis and the interpretability of the results through a visual interface as its main advantages. Expansion with new sources, versioning and the improvement of the search and scoring algorithms are envisioned as future lines of work.

Exploiting Real-time Search Engine Queries for Earthquake Detection: A Summary of Results

Online search engine has been widely regarded as the most convenient approach for information acquisition. Indeed, the intensive information-seeking behaviors of search engine users make it possible to exploit search engine queries as effective “crowd sensors” for event monitoring. While some researchers have investigated the feasibility of using search engine queries for coarse-grained event analysis, the capability of search engine queries for real-time event detection has been largely neglected. To this end, in this article, we introduce a large-scale and systematic study on exploiting real-time search engine queries for outbreak event detection, with a focus on earthquake rapid reporting. In particular, we propose a realistic system of real-time earthquake detection through monitoring millions of queries related to earthquakes from a dominant online search engine in China. Specifically, we first investigate a large set of queries for selecting the representative queries that are highly correlated with the outbreak of earthquakes. Then, based on the real-time streams of selected queries, we design a novel machine learning–enhanced two-stage burst detection approach for detecting earthquake events. Meanwhile, the location of an earthquake epicenter can be accurately estimated based on the spatial-temporal distribution of search engine queries. Finally, through the extensive comparison with earthquake catalogs from China Earthquake Networks Center, 2015, the detection precision of our system can achieve 87.9%, and the accuracy of location estimation (province level) is 95.7%. In particular, 50% of successfully detected results can be found within 62 s after earthquake, and 50% of successful locations can be found within 25.5 km of seismic epicenter. Our system also found more than 23.3% extra earthquakes that were felt by people but not publicly released, 12.1% earthquake-like special outbreaks, and meanwhile, revealed many interesting findings, such as the typical query patterns of earthquake rumor and regular memorial events. Based on these results, our system can timely feed back information to the search engine users according to various cases and accelerate the information release of felt earthquakes.

A cost model for random access queries in document stores

Document stores have become one of the key NoSQL storage solutions. They have been widely adopted in different domains due to their ability to store semi-structured data and expressive query capabilities. However, implementations differ in terms of concrete data storage and retrieval. Unfortunately, a standard framework for data and query optimization for document stores is nonexistent, and only implementation-specific design and query guidelines are used. Hence, the goal of this work is to aid automating the data design for document stores based on query costs instead of generic design rules. For this, we define a generic storage and query cost model based on disk access and memory allocation that allows estimating the impact of design decisions. Since all document stores carry out data operations in memory, we first estimate the memory usage by considering characteristics of the stored documents, their access patterns, and memory management algorithms. Then, using this estimation and metadata storage size, we introduce a cost model for random access queries. We validate our work on two well-known document store implementations: MongoDB and Couchbase. The results show that the memory usage estimates have the average precision of 91% and predicted costs are highly correlated to the actual execution times. During this work, we have managed to suggest several improvements to document storage systems. Thus, this cost model also contributes to identifying discordance between document store implementations and their theoretical expectations.

Implementing a high-efficiency similarity analysis approach for firmware code.

The rapid expansion of the open-source community has shortened the software development cycle, but the spread of vulnerabilities has been accelerated, especially in the field of the Internet of Things. In recent years, the frequency of attacks against connected devices is increasing exponentially; thus, the vulnerabilities are more serious in nature. The state-of-the-art firmware security inspection technologies, such as methods based on machine learning and graph theory, find similar applications depending on the known vulnerabilities but cannot do anything without detailed information about the vulnerabilities. Moreover, model training, which is necessary for the machine learning technologies, requires a significant amount of time and data, resulting in low efficiency and poor extensibility. Aiming at the above shortcomings, a high-efficiency similarity analysis approach for firmware code is proposed in this study. First, the function control flow features and data flow features are extracted from the functions of the firmware and of the vulnerabilities, and the features are used to calculate the SimHash of the functions. The mass storage and fast query capabilities of the SimHash are implemented by the pigeonhole principle. Second, the similarity function pairs are analyzed in detail within and among the basic blocks. Within the basic blocks, the symbolic execution is used to generate the basic block semantic information, and the constraint solver is used to determine the semantic equivalence. Among the basic blocks, the local control flow graphs are analyzed to obtain their similarity. Then, we implemented a prototype and present the evaluation. The evaluation results demonstrate that the proposed approach can implement large-scale firmware function similarity analysis. It can also get the location of the real-world firmware patch without vulnerability function information. Finally, we compare our method with existing methods. The comparison results demonstrate that our method is more efficient and accurate than the Gemini and StagedMethod. More than 90% of the firmware functions can be indexed within 0.1 s, while the search time of 100,000 firmware functions is less than 2 s.

Privacy‐Guarding Optimal Route Finding with Support for Semantic Search on Encrypted Graph in Cloud Computing Scenario

The arrival of cloud computing age makes data outsourcing an important and convenient application. More and more individuals and organizations outsource large amounts of graph data to the cloud computing platform (CCP) for the sake of saving cost. As the server on CCP is not completely honest and trustworthy, the outsourcing graph data are usually encrypted before they are sent to CCP. The optimal route finding on graph data is a popular operation which is frequently used in many fields. The optimal route finding with support for semantic search has stronger query capabilities, and a consumer can use similar words of graph vertices as query terms to implement optimal route finding. Due to encrypting the outsourcing graph data before they are sent to CCP, it is not easy for data customers to manipulate and further use the encrypted graph data. In this paper, we present a solution to execute privacy‐guarding optimal route finding with support for semantic search on the encrypted graph in the cloud computing scenario (PORF). We designed a scheme by building secure query index to implement optimal route finding with support for semantic search based on searchable encryption idea and stemmer mechanism. We give formal security analysis for our scheme. We also analyze the efficiency of our scheme through the experimental evaluation.

Enabling Massive XML-Based Biological Data Management in HBase.

Publishing biological data in XML formats is attractive for organizations who would like to provide their bioinformatics resources in an extensible and machine-readable format. In the era of big data, massive XML-based biological data management is emerged as a challengeable issue. With the continuous growth of the XML-based biological data sets, it is usually frustrating to use traditional declarative query languages to provide efficient query capabilities in terms of processing speed and scale. In this study, we report a novel platform to store and query massive XML-based biological data collections. A prototype tool for constructing HBase tables from XML-based biological data collections is first developed, and then a formal approach to transform the XML query model into the MapReduce query model is proposed. Finally, an evaluation of the query performance of the proposed approach on the existing XML-based biological databases is presented, showing that the performance advantages of the proposed solution. The source code of the massive XML-based biological data management platform is freely available at https://github.com/lyotvincent/X2H.

Data Lab—A community science platform

Abstract Data Lab is an open-access science platform developed and operated by the Community and Science Data Center (CSDC) at NSF’s National Optical-Infrared Astronomy Research Laboratory (NOIRLab). It serves public photometric survey datasets, provides interactive and programmatic data access, and SQL/ADQL query capabilities via TAP. Users also receive generous storage allocations with VOSpace and MyDB, co-located with our data holdings. A host of services such as cross-matching, image cutouts via SIA, file services for survey data, and a Jupyter notebook interface for analysis close to the data complement the mission statement. Launched in 2017 at the National Optical Astronomy Observatory, Data Lab supports a base of over 1,300 registered users, processes on average 15,000 queries daily, serves over 50 TB of photometric catalogs, and provides access to over 2 PB of survey image products at NOIRLab’s Science Data Archive. Future development will include support for massive spectroscopic datasets and for processing of alert streams generated by e.g. ZTF and LSST. Users will also be able to create and administrate ad hoc user groups for shared data access and scientific analysis, and will enjoy containerized services and notebook spaces.

GTFS-Madrid-Bench: A benchmark for virtual knowledge graph access in the transport domain

A large number of datasets are being made available on the Web using a variety of formats and according to diverse data models. Ontology Based Data Integration (OBDI) has been traditionally proposed as a mechanism to facilitate access to such heterogeneous datasets, providing a unified view over their data by means of ontologies. Recently, the term “Virtual Knowledge Graph Access” has begun to be used to refer to the mechanisms that provide query-based access to knowledge graphs virtually generated from heterogeneous data sources. Several OBDI engines exist in the state of the art, with overlapping capabilities but also clear differences among them (in terms of the data formats that they can deal with, mapping languages that they support, query expressivity that they allow, etc.). These engines have been evaluated with different testbeds and benchmarks. However, their heterogeneity has made it difficult to come up with a common comprehensive benchmark that allows for comparisons among them to facilitate their selection by practitioners, and more importantly, for their continuous improvement by the teams that maintain them. In this paper we present GTFS-Madrid-Bench, a benchmark to evaluate OBDI engines that can be used for the provision of access mechanisms to virtual knowledge graphs. Our proposal introduces several scenarios that aim at measuring the query capabilities, performance and scalability of all these engines, considering their heterogeneity. The data sources used in our benchmark are derived from the GTFS data files of the subway network of Madrid. They have been transformed into several formats (CSV, JSON, SQL and XML) and scaled up. The query set aims at addressing a representative number of SPARQL 1.1 features while covering usual queries that data consumers may be interested in.

ST Sequence Miner: visualization and mining of spatio-temporal event sequences

As a promising field of research, event sequence analysis seems to assist in facilitating clear reasoning behind human decisions by mining reality behind the sequential actions. Mining frequent patterns from event sequences has proved to be promising in extracting actionable insights, which plays an important role in many application domains. Much of the related work challenges the problem solely from the temporal perspective omitting the information that could be gained from the spatial part. This could be in part due to the fact that analysis of event sequences with references to both time and space is attributed as a challenging task due to the additional variance in the data introduced by the spatial aspect. We propose a visual analytics approach that incorporates spatio-temporal pattern extraction leveraging an extended sequential pattern mining algorithm and a pattern discovery guidance mechanism operating on geographic query and selection capabilities. As an implementation of our approach, we introduce a visual analytics tool, namely ST Sequence Miner, enabling event pattern exploration in time-location space. We evaluate our approach over a credit card transaction dataset by adopting case study methodology. Our study unveils that patterns mined from event sequences can better explain possible relationships with proper visualization of time-location data.

BAD to the bone: Big Active Data at its core

Virtually all of today's Big Data systems are passive in nature, responding to queries posted by their users. Instead, we are working to shift Big Data platforms from passive to active. In our view, a Big Active Data (BAD) system should continuously and reliably capture Big Data while enabling timely and automatic delivery of relevant information to a large pool of interested users, as well as supporting retrospective analyses of historical information. While various scalable streaming query engines have been created, their active behavior is limited to a (relatively) small window of the incoming data. To this end we have created a BAD platform that combines ideas and capabilities from both Big Data and Active Data (e.g., Publish/Subscribe, Streaming Engines). It supports complex subscriptions that consider not only newly arrived items but also their relationships to past, stored data. Further, it can provide actionable notifications by enriching the subscription results with other useful data. Our platform extends an existing open-source Big Data Management System, Apache AsterixDB, with an active toolkit. The toolkit contains features to rapidly ingest semistructured data, share execution pipelines among users, manage scaled user data subscriptions, and actively monitor the state of the data to produce individualized information for each user. This paper describes the features and design of our current BAD data platform and demonstrates its ability to scale without sacrificing query capabilities or result individualization.