MongoDB Database Research Articles

Incremental Rapidly Grouping Aggregation Method for Similar Web News Headline

With the popularity of the Internet, real-time web news has been extensively used in human daily life. Because of competition and repetitive reporting between the media, there may be multiple similar news reporting the same searing event, which may waste a lot of time for users to obtain and read the similar news. Hence, how to rapid aggregating similar news to help users obtain relevant news information accurately is a very meaningful and essential study. However, the accuracy of the existing aggregation method needs to be strengthened, and it not meet real-time incremental processing requirements.In this paper, rapid incremental aggregating a lot of similar news headlines are our main study targets. We proposed a novel Rapidly Grouping Aggregation (RGA) method. It carries out similar news headline aggregation via two core aspects --- Improved Jaro-Winkler (IJW) similarity calculation, normalized mapping and rapidly grouping aggregation. The IJW using hierarchical matching windows to enhance the performance of similarity calculation, and using the MD5 normalization strategy and the Group By function of the MongoDB database to carry out the similarity news headline rapidly grouping. Compared with the state-of-the-art method, the actual system application results show that the RGA method achieves higher performance on similar news headline aggregation tasks.

A computational EXFOR database

The EXFOR library is a useful resource for many people in the field of nuclear physics. In particular, the experimental data in the EXFOR library serves as a starting point for nuclear data evaluations. There is an ongoing discussion about how to make evaluations more transparent and reproducible. One important ingredient may be convenient programmatic access to the data in the EXFOR library from high-level languages. To this end, the complete EXFOR library can be converted to a MongoDB database. This database can be conveniently searched and accessed from a wide variety of programming languages, such as C++, Python, Java, Matlab, and R. This contribution provides some details about the successful conversion of the EXFOR library to a MongoDB database and shows simple usage examples to underline its merits. All codes required for the conversion have been made available online and are open-source. In addition, a Dockerfile has been created to facilitate the installation process.

A Comparative Study of Unstructured Data with SQL and NO-SQL Database Management Systems

This paper aims to establish a relative study between a relational Microsoft SQL Server database and a non-relational MongoDB database within the unstructured representation of data in JSON format. There is a great amount of work done regarding comparison of multiple database management applications on the basis of their performances, security etc., but we have limited information available where these databases are assessed on the basis of provided data. This study will mainly focus on looking at all the possibilities that both these database types offer us when handling data in JSON. We will accomplish this by implementing a series of experiments while taking into consideration that the subjected data does not require to be normalized; and therefore, evaluate the outcome to conclude the result.

Analysis of performance of selected geospatial analyses implemented on the basis of relational and NoSQL databases

Abstract Databases are a basic component of every GIS system and many geoinformation applications. They also hold a prominent place in the tool kit of any cartographer. Solutions based on the relational model have been the standard for a long time, but there is a new increasingly popular technological trend – solutions based on the NoSQL database which have many advantages in the context of processing of large data sets. This paper compares the performance of selected spatial relational and NoSQL databases executing queries with selected spatial operators. It has been hypothesised that a non-relational solution will prove to be more effective, which was confirmed by the results of the study. The same spatial data set was loaded into PostGIS and MongoDB databases, which ensured standardisation of data for comparison purposes. Then, SQL queries and JavaScript commands were used to perform specific spatial analyses. The parameters necessary to compare the performance were measured at the same time. The study’s results have revealed which approach is faster and utilises less computer resources. However, it is difficult to clearly identify which technology is better because of a number of other factors which have to be considered when choosing the right tool.

A Comparative Study of NOSQL System Vulnerabilities with Big Data

With the emerging third wave in the development of the Internet, the past year has witnessed huge data exposure resulting in cyber-attacks that have increased four times as that of the previous year’s record. In this digital era, businesses are making use of NoSQL technologies for managing such Big Data. However, the NoSQL database systems come with inherent security issues, which pose a major challenge to many organisations worldwide. There is a paucity of research studies for exposing the security threats and vulnerabilities of NoSQL technologies comprehensively. This paper presents an in-depth study of NoSQL security issues by performing a detailed comparative study of the security vulnerabilities identified in NoSQL database systems. A set of key security features offered by the four commonly used NoSQL database systems, namely Redis, Cassandra, MongoDB and Neo4j are analysed with an aim to identify their strengths and weaknesses. The vulnerabilities associated with built-in security, encryption, authentication/authorization and auditing that impact Big Data management are compared among these popular NoSQL database systems and the risk levels are identified. In addition, illustrations of possible injection attacks experimented with these NoSQL systems are provided. Finally, a high-level framework is proposed for NoSQL databases with considerations for security measures in Big Data deployments. The discussion forms a significant technical contribution for learners, application developers and Big Data deployers paving way for a better awareness and management of the NoSQL systems in an organization.

A Distributed Storage and Access Approach for Massive Remote Sensing Data in MongoDB

With the rapid development of earth-observation technology, the amount of remote sensing data has increased exponentially, and traditional relational databases cannot satisfy the requirements of managing large-scale remote sensing data. To address this problem, this paper undertakes intensive research of the NoSQL (Not Only SQL) data management model, especially the MongoDB database, and proposes a new approach to managing large-scale remote sensing data. Firstly, based on the sharding technology of MongoDB, a distributed cluster architecture was designed and established for massive remote sensing data. Secondly, for the convenience in the unified management of remote sensing data, an archiving model was constructed, and remote sensing data, including structured metadata and unstructured image data, were stored in the above cluster separately, with the metadata stored in the form of a document, and image data stored with the GridFS mechanism. Finally, by designing different shard strategies and comparing MongoDB cluster with a typical relational database, several groups of experiments were conducted to verify the storage performance and access performance of the cluster. The experimental results show that the proposed method can overcome the deficiencies of traditional methods, as well as scale out the database, which is more suitable for managing massive remote sensing data and can provide technical support for the management of massive remote sensing data.

Jointly Integrating VCF-Based Variants and OWL-Based Biomedical Ontologies in MongoDB.

The development of the next-generation sequencing (NGS) technologies has led to massive amounts of VCF (Variant Call Format) files, which have been the standard formats developed with 1000 Genomes Project. At the same time, with the widespread use of biomedical ontologies in the biomedical community, more and more applications have accepted the Web Ontology Language (OWL) as the dominant data format for the specifications of biomedical ontology descriptions, leading to the rapid growth of OWL-based biomedical ontology scale. In this paper, we seek to explore an effective method for the management of VCF-based genetic variants and OWL-based biological ontologies using the MongoDB database. Considering many current applications (such as the short genetic variations database dbSNP, etc.) are transitioning to the new design by using JSON (JavaScript Object Notation) to support future massive data expansion and interchanges. We firstly propose a series of rules for the mapping from VCF and OWL files to JSON files, and then present rule-based algorithms for transforming VCF-based genetic variants and OWL-based biological ontologies into JSON objects. On this basis, we introduce effective approaches of integrating the mapped JSON files in MongoDB. Finally, we complement this work with a set of experiments to show the performance of our proposed approaches. The source code of the proposed approaches could be freely available at https://github.com/lyotvincent/AJIA.

Comparison of MongoDB and Cassandra Databases for Spectrum Monitoring As-a-Service

Due to the growing number of devices accessing the Internet through wireless networks, the radio spectrum has become a highly contended resource. The availability of low cost radio spectrum monitoring sensors enables a geographically distributed, real-time observation of the spectrum to spot inefficiencies and to develop new strategies for its utilization. The potentially large number of sensors to be deployed and the intrinsic nature of data make this task a Big Data problem. In this work we design, implement, and validate a hardware and software architecture for wideband radio spectrum monitoring inspired to the Lambda architecture. This system offers Spectrum Sensing as a Service to let end users easily access and process radio spectrum data. To minimize the latency of services offered by the platform, we fine tune the data processing chain. From the analysis of sensor data characteristics, we design the data models for MongoDB and Cassandra, two popular NoSQL databases. A MapReduce job for spectrum visualization has been developed to show the potential of our approach and to identify the challenges in processing spectrum sensor data. We experimentally evaluate and compare the performance of the two databases in terms of application processing time for different types of queries applied on data streams with heterogeneous generation rate. Our experiments show that Cassandra outperforms MongoDB in most cases, with some exceptions depending on data stream rate.

Blind Queries Applied to JSON Document Stores

Social Media, Web Portals and, in general, information systems offer their own Application Programming Interfaces (APIs), used to provide large data sets concerning every aspect of day-by-day life. APIs usually provide data sets as collections of JSON documents. The heterogeneous structure of JSON documents returned by different APIs constitutes a barrier to effectively query and analyze these data sets. The adoption of NoSQL document stores, such as MongoDB, is useful for gathering these data sets, but does not solve the problem of querying the final heterogeneous repository. The aim of this paper is to provide analysts with a tool, named HammerJDB, that allows for blind querying collections of JSON documents within a NoSQL document database. The idea below is that users may know the application domain but it may be that they are not aware of the real structures of the documents stored in the database—the tool for blind querying tries to bridge the gap, by adopting a query rewriting mechanism. This paper is an evolution of a technique for blind querying Open Data portals and of its implementation within the Hammer framework, presented in some previous work. In this paper, we evolve that approach in order to query a NoSQL document database by evolving the Hammer framework into the HammerJDB framework, which is able to work on MongoDB databases. The effectiveness of the new approach is evaluated on a data set (derived from a real-life one), containing job-vacancy ads collected from European job portals.

Relational into Non-Relational Database Migration with Multiple-Nested Schema Methods on Academic Data

The rapid development of internet technology has increased the need of data storage and processing technology application. One application is to manage academic data records at educational institutions. Along with massive growth of information, decrement in the traditional database performance is inevitable. Hence, there are many companies choose to migrate to NoSQL, a technology that is able to overcome the traditional database shortcomings. However, the existing SQL to NoSQL migration tools have not been able to represent SQL data relations in NoSQL without limiting query performance. In this paper, a relational database transformation system transforming MySQL into non-relational database MongoDB was developed, using the Multiple Nested Schema method for academic databases. The development began with a transformation scheme design. The transformation scheme was then implemented in the migration process, using PDI/Kettle. The testing was carried out on three aspects, namely query response time, data integrity, and storage requirements. The test results showed that the developed system successfully represented the relationship of SQL data in NoSQL, provided complex query performance 13.32 times faster in the migration database, basic query performance involving SQL transaction tables 28.6 times faster on migration results, and basic performance Queries without involving SQL transaction tables were 3.91 times faster in the migration source. This shows that the theory of the Multiple Nested Schema method, aiming to overcome the poor performance of queries involving many JOIN operations, is proved. In addition, the system is also proven to be able to maintain data integrity in all tested queries. The space performance test results indicated that the migrated database transformed using the Multiple Nested Schema method showed a storage requirement of 10.53 times larger than the migration source database. This is due to the large amount of data redundancy resulting from the transformation process. However, at present, storage performance is not a top priority in data processing technology, so large storage requirements are a consequence of obtaining efficient query performance, which is still considered as the first priority in data processing technology.

Benchmarking MySQL and NoSQL Databases on Egovbench Application

E-Government is a result of technological advances in the government field. E-government assessment is needed to encourage the development of e-government in a better direction. The e-Government assessment can be measured using the Egovbench application. Egovbench performs crawling to obtain information from related websites or social media. The process of crawling done by Egovbench produces extensive data, which reduced performance in data processing. Therefore, there is a need for a database solution that has the best performance such as high processing speed and small database size. This study examined the comparison between relational databases and non-relational databases based on selected metrics to obtain the most suitable database solution for Egovbench. The results show that the MySQL database has the advantage of complex query processing and the use of the database with the smallest storage space. MongoDB database has the advantage of low data transfer volumes. Couchbase database has the advantage of short and straightforward query processing with a high number. The evaluation results show that MySQL is more suitable for Egovbench needs, which is the best response time and query per second. MySQL outperformed the other two databases on backup and storage file sizes testing.

The XENON1T data acquisition system

The XENON1T liquid xenon time projection chamber is the most sensitive detector built to date for the measurement of direct interactions of weakly interacting massive particles with normal matter. The data acquisition system (DAQ) is constructed from commercial, open source, and custom components to digitize signals from the detector and store them for later analysis. The system achieves an extremely low signal threshold by triggering each channel independently, achieving a single photoelectron acceptance of (93 ± 3)%, and deferring the global trigger to a later, software stage. The event identification is based on MongoDB database queries and has over 98% efficiency at recognizing interactions at the analysis threshold in the center of the target. A readout bandwidth over 300 MB/s is reached in calibration modes and is further expandable via parallelization. This DAQ system was successfully used during three years of operation of XENON1T.

Prototype of the 3D Cadastral System Based on a NoSQL Database and a JavaScript Visualization Application

3D cadastral systems are more complex than traditional cadastral systems and they require more complex technical solutions and innovative use of developing technologies. Regarding data integrity and data consistency, 3D cadastral data should be maintained by a Database Management System (DBMS). Furthermore, there are still challenges regarding visualization of 3D cadastral data. A prototype of the 3D cadastral system based on a NoSQL database and a JavaScript application for 3D visualization is designed and tested in order to investigate the possibilities of using new technical solutions. It is assumed that this approach, with further development, could be a good basis for the development of a modern 3D cadastral system. MongoDB database is used for storing data and Cesium JavaScript library is used for 3D visualization. The system uses an LADM (Land Administration Domain Model) based data model. Additionally, script languages, libraries, application programming interfaces (APIs), software and data formats are used for the system development. The case study is based on the real cadastral data. The underground object and building units located below and above the ground level are used to test the proposed data model and the system’s functionality. The proposed system needs further development in order to provide full support to a modern 3D cadastral system. However, it allows maintenance of 3D cadastral data and basic 3D visualization with the interactive approach.

Transplant2Mongo: Python Scripts that Insert Organ Procurement and Transplantation Network (OPTN) Data in MongoDB

transplant2mongo allows users to transform Standard Transplant Analysis and Research (STAR) ASCII data files from the Organ Procurement and Transplantation Network (OPTN) into a MongoDB database [1, 2]. The STAR data are a complex collection of tab-separated files with inter-related records that are not amenable to complex queries. A researcher planning to use OPTN STAR data can use transplant2mongo to convert the data into a MongoDB database and then use open-source tool software for analysis. The source code for transplant2mongo is available on GitHub at https://github.com/ceharvs/transplant2mongo and includes sample data files for initial testing and queries. Funding Statement: The software referenced herein is copyright of The MITRE Corporation and the result of MITRE’s Early Career Research program and work done in the Computational Science and Informatics program at George Mason University. Approved for Public Release; Distribution Unlimited. Case Number 18-0298. The author’s affiliation with The MITRE Corporation is provided for identification purposes only and is not intended to convey or imply MITRE’s concurrence with, or support for, the positions, opinions, or viewpoints expressed by the author.

Design and Evaluation of IoT-Enabled Instrumentation for a Soil-Bentonite Slurry Trench Cutoff Wall

In this work, we describe our approach and experiences bringing an instrumented soil-bentonite slurry trench cutoff wall into a modern IoT data collection and visualization pipeline. Soil-bentonite slurry trench cutoff walls have long been used to control ground water flow and contaminant transport. A Raspberry Pi computer on site periodically downloads the sensor data over a serial interface from an industrial datalogger and transmits the data wirelessly to a gateway computer located 1.3 km away using a reliable transmission protocol. The resulting time-series data is stored in a MongoDB database and data is visualized in real-time by a custom web application. The system has been in operation for over two years achieving 99.42% reliability and no data loss from the collection, transport, or storage of data. This project demonstrates the successful bridging of legacy scientific instrumentation with modern IoT technologies and approaches to gain timely web-based data visualization facilitating rapid data analysis without negatively impacting data integrity or reliability. The instrumentation system has proven extremely useful in understanding the changes in the stress state over time and could be deployed elsewhere as a means of on-demand slurry trench cutoff wall structural health monitoring for real-time stress detection linked to hydraulic conductivity or adapted for other infrastructure monitoring applications.

Exploiting Information-Centric Networking to Federate Spatial Databases

This paper explores the methodologies, challenges, and expected advantages related to the use of the information-centric network (ICN) technology for federating spatial databases. ICN services allow simplifying the design of federation procedures, improving their performance, and providing so-called data-centric security. In this work, we present an architecture that is able to federate spatial databases and evaluate its performance using a real data set coming from OpenStreetMap within a heterogeneous federation formed by MongoDB and CouchBase spatial database systems.

Automated data collection and real-time data analysis suite for serial synchrotron crystallography.

At the Swiss Light Source macromolecular crystallography (MX) beamlines the collection of serial synchrotron crystallography (SSX) diffraction data is facilitated by the recent DA+ data acquisition and analysis software developments. The SSX suite allows easy, efficient and high-throughput measurements on a large number of crystals. The fast continuous diffraction-based two-dimensional grid scan method allows initial location of microcrystals. The CY+ GUI utility enables efficient assessment of a grid scan's analysis output and subsequent collection of multiple wedges of data (so-called minisets) from automatically selected positions in a serial and automated way. The automated data processing (adp) routines adapted to the SSX data collection mode provide near real time analysis for data in both CBF and HDF5 formats. The automatic data merging (adm) is the latest extension of the DA+ data analysis software routines. It utilizes the sxdm (SSX data merging) package, which provides automatic online scaling and merging of minisets and allows identification of a minisets subset resulting in the best quality of the final merged data. The results of both adp and adm are sent to the MX MongoDB database and displayed in the web-based tracker, which provides the user with on-the-fly feedback about the experiment.

Database Systems Performance Evaluation for IoT Applications

The amount of data stored in IoT databases increases as the IoT applications extend throughout smart city appliances, industry and agriculture. Contemporary database systems must process huge amounts of sensory and actuator data in real-time or interactively. Facing this first wave of IoT revolution, database vendors struggle day-by-day in order to gain more market share, develop new capabilities and attempt to overcome the disadvantages of previous releases, while providing features for the IoT. There are two popular database types: The Relational Database Management Systems and No SQL databases, with No SQL gaining ground on IoT data storage.In the context of this paper these two types are examined. Focusing on open source databases, the authors experiment on IoT data sets and pose an answer to the question which one performs better than the other. It is a comparative study on the performance of the commonly market used open source databases, presenting results for the No SQL MongoDB database and SQL databases of MySQL and PostgreSQL.

Comparative Analysis for Web Applications Based on REST Services: MEAN Stack and Java EE Stack

.

A Magnetoencephalographic/Encephalographic (MEG/EEG) Brain-Computer Interface Driver for Interactive iOS Mobile Videogame Applications Utilizing the Hadoop Ecosystem, MongoDB, and Cassandra NoSQL Databases.

In Phase I, we collected data on five subjects yielding over 90% positive performance in Magnetoencephalographic (MEG) mid-and post-movement activity. In addition, a driver was developed that substituted the actions of the Brain Computer Interface (BCI) as mouse button presses for real-time use in visual simulations. The process was interfaced to a flight visualization demonstration utilizing left or right brainwave thought movement, the user experiences, the aircraft turning in the chosen direction, or on iOS Mobile Warfighter Videogame application. The BCI’s data analytics of a subject’s MEG brain waves and flight visualization performance videogame analytics were stored and analyzed using the Hadoop Ecosystem as a quick retrieval data warehouse. In Phase II portion of the project involves the Emotiv Encephalographic (EEG) Wireless Brain–Computer interfaces (BCIs) allow for people to establish a novel communication channel between the human brain and a machine, in this case, an iOS Mobile Application(s). The EEG BCI utilizes advanced and novel machine learning algorithms, as well as the Spark Directed Acyclic Graph (DAG), Cassandra NoSQL database environment, and also the competitor NoSQL MongoDB database for housing BCI analytics of subject’s response and users’ intent illustrated for both MEG/EEG brainwave signal acquisition. The wireless EEG signals that were acquired from the OpenVibe and the Emotiv EPOC headset can be connected via Bluetooth to an iPhone utilizing a thin Client architecture. The use of NoSQL databases were chosen because of its schema-less architecture and Map Reduce computational paradigm algorithm for housing a user’s brain signals from each referencing sensor. Thus, in the near future, if multiple users are playing on an online network connection and an MEG/EEG sensor fails, or if the connection is lost from the smartphone and the webserver due to low battery power or failed data transmission, it will not nullify the NoSQL document-oriented (MongoDB) or column-oriented Cassandra databases. Additionally, NoSQL databases have fast querying and indexing methodologies, which are perfect for online game analytics and technology. In Phase II, we collected data on five MEG subjects, yielding over 90% positive performance on iOS Mobile Applications with Objective-C and C++, however on EEG signals utilized on three subjects with the Emotiv wireless headsets and (n < 10) subjects from the OpenVibe EEG database the Variational Bayesian Factor Analysis Algorithm (VBFA) yielded below 60% performance and we are currently pursuing extending the VBFA algorithm to work in the time-frequency domain referred to as VBFA-TF to enhance EEG performance in the near future. The novel usage of NoSQL databases, Cassandra and MongoDB, were the primary main enhancements of the BCI Phase II MEG/EEG brain signal data acquisition, queries, and rapid analytics, with MapReduce and Spark DAG demonstrating future implications for next generation biometric MEG/EEG NoSQL databases.