Horizontal Scaling Research Articles

RESEARCH OF ACID TRANSACTION IMPLEMENTATION METHODS FOR DISTRIBUTED DATABASES USING REPLICATION TECHNOLOGY

Today, databases are an integral part of most modern applications designed to store large amounts of data and to request from many users. To solve business problems in such conditions, databases are scaled, often horizontally on several physical servers using replication technology. At the same time, many business operations require the implementation of transactional compliance with ACID properties. For relational databases that traditionally support ACID transactions, horizontal scaling is not always effective due to the limitations of the relational model itself. Therefore, there is an applied problem of efficient implementation of ACID transactions for horizontally distributed databases. The subject matter of the study is the methods of implementing ACID transactions in distributed databases, created by replication technology. The goal of the work is to increase the efficiency of ACID transaction implementation for horizontally distributed databases. The work is devoted to solving the following tasks: analysis and selection of the most relevant methods of implementation of distributed ACID transactions; planning and experimental research of methods for implementing ACID transactions by using of NoSQL DBMS MongoDB and NewSQL DBMS VoltDB as an example; measurements of metrics of productivity of use of these methods and formation of the recommendation concerning their effective use. The following methods are used: system analysis; relational databases design; methods for evaluating database performance. The following results were obtained: experimental measurements of the execution time of typical distributed transactions for the subject area of e-commerce, as well as measurements of the number of resources required for their execution; revealed trends in the performance of such transactions, formed recommendations for the methods studied. The obtained results allowed to make functions of dependence of the considered metrics on loading parameters. Conclusions: the strengths and weaknesses of the implementation of distributed ACID transactions using MongoDB and VoltDB were identified. Practical recommendations for the effective use of these systems for different types of applications, taking into account the resources consumed and the types of requests.

ОСОБЛИВОСТІ МАСШТАБУВАННЯ КОНТЕЙНЕРНОГО НАВАНТАЖЕННЯ НА БАЗІ СИСТЕМИ KUBERNETES

Today, the field of software development using container technology is becoming widespread in many areas of business, and the problem of scaling is one of the most important to achieve the expansion of the volume of tasks performed over time, so it is important to use the right scaling schemes. Kubernetes requires relatively easy-to-adapt container management mechanisms, but the problem of scaling them remains poorly described and researched, which increases the complexity of the business transition to container technologies and, in particular, the Kubernetes system. Thanks to the separation of infrastructure and payload concepts in Kubernetes, the classic scaling methods – horizontal and vertical – are revealed in each component separately. Although the Kubernetes system is quite flexible, it has only recently been developed, and the features of scaling with concept separation have not yet been described. Actual scientific researches and issues analysis showed that there is a general trend towards cloud migration recently. In this context, the architectural style of microservices and the use of container load has attracted considerable attention, especially based on the Kubernetes system. It can be concluded that Kubernetes has significant advantages in the speed of development and deployment of container-based applications, but the problem of scaling and features of horizontal and vertical scaling in Kubernetes has not received much attention from academia so far. The aim of the work is to study the features of scaling the container load based on the Kubernetes system. To explain the basic operations to achieve horizontal and vertical scaling. To describe the scalability property of the Kubernetes system. This work reveals the main architectural features of Kubernetes, the principles of working with applications in container virtualization. The main methods of regulating the resource capacity of containers are shown. The concepts of horizontal and vertical scaling in the Kubernetes system with an indication of scalability properties are revealed. The main operations for scaling in the Kubernetes system in the context of separation of payload and infrastructure concepts are given. In this article the features of container load scaling based on the Kubernetes system with a separation of the concepts of payload and infrastructure scaling were investigated and described. This article is a review.

MODELS AND METHODS FOR DIAGNOSING ZERO-DAYTHREATS IN CYBERSPACE

The article is devoted to the development of models and methods for detectingZero-Daythreats in cyberspace to improve the efficiency of detecting high-level malicious complexes that are using polymorphic mutators. Themethod for detecting samplesby antivirus solutions using a public and local multiscanneris proposed. Themethod for diagnosing polymorphic malware using Yara rules is being developed. The multicomponent service that allows organizing a free malware analysis solution with a hybrid deploy-ment architecture in public and private clouds is described. The cloud service for detecting malware based on open-source sandboxes and MAS, allowing horizontal scalability in hybrid clouds, and showing high capacity during malicious and non-maliciousobjectprocessing is designed. The main task of the service is to collect artifacts after dynamic and static object analysis to detect zero-daythreats. The effectiveness of the proposed solutions is shown.Scientific novelty and originality consist in the creation of the follow-ing methods:1) detecting the sample by preinstalled antivirus solutions that allow static scanning in separate threads without requests restrictions for increasing the malware processing speed and restrict public access to confidential files;2) diagnosing polymorphic malware using Yara rules, that allows detecting new modifications that are not detected by available solutions.The proposed hybrid system architecture allows to perform a retrospective search by families, tracking changes in destructive components, collect the malicious URLs database to block traffic to C&C servers, collect dropped and downloaded files, analyze phishing emails attach-ments, integrate with SIEM, IDS, IPS, antiphishing and Honeypot systems, improve the quality of the SOC analyst, decrease the incidents response times and block new threats that are not detected by available antivirus solutions. The practical significance of the results is in the cloud service development that combines MAS Sandbox and a modified distributed Cuckoo sandbox, which allows to respond to Zero-Day threats quickly, store a knowledge base for artifacts correlation between polymorphic malware samples, actively search for new malware samples and integrate with cyber protection hardwareand software systems that support the Cuckoo API.

Studi Komparasi Performa NGINX dan HAPROXY Sebagai Load Balancer di Cloud Menggunakan Teknologi Kontainer

High availability and scalability is a must have feature for server that owned by industrial sector. One concept that used by technology practitioner ishorizontal scaling. Horizontal scaling can be achieved by using load balancer. Even though those technology practitioner started to adopt containerization technology, those technology practitioner still using load balancer. The purpose of this research is to compare NGINX and HAPROXY performance as load balancer based on response time and error rate. Both NGINX and HAPROXY will run on docker that installed on various virtual machine type in both AWS and GCP. The result shows that based on writer’s configuration, NGINX could handle medium and heavy load better than HAPROXY. Another result shows that AWS could handle medium and heavy load better than GCP.

TECHNOLOGIES OF SOFTWARE DEVELOPMENT BASED ON NON-RELATIVE DATABASES

The paper presents an overview of some technologies of software development based on non-relational databases. It is emphasized that nowadays more and more data are generated to be stored and processed in real time. In addition, the nature of these data is often increasingly unstructured or poorly structured. This causes a niche of problems when applying traditional approaches to building databases that are integrated into software systems. At the same time, NoSQL databases provide a mechanism for storing and retrieving data that are organized in a different way than the usual relational approach. The motivation for this approach is the simplicity of architecture, horizontal scaling, and accessibility control. The growing interest to non-relational databases is represented by a review of some publications.

Design and evaluation of a scalable Internet of Things backend for smart ports

AbstractInternet of Things (IoT) technologies, when adequately integrated, cater for logistics optimisation and operations' environmental impact monitoring, both key aspects for today's EU ports management. This article presents Obelisk, a scalable and multi‐tenant cloud‐based IoT integration platform used in the EU H2020 PortForward project. The landscape of IoT protocols being particularly fragmented, the first role of Obelisk is to provide uniform access to data originating from a myriad of devices and protocols. Interoperability is achieved through adapters that provide flexibility and evolvability in protocol and format mapping. Additionally, due to ports operating in a hub model with various interacting actors, a second role of Obelisk is to secure access to data. This is achieved through encryption and isolation for data transport and processing, respectively, while user access control is ensured through authentication and authorisation standards. Finally, as ports IoTisation will further evolve, a third need for Obelisk is to scale with the data volumes it must ingest and process. Platform scalability is achieved by means of a reactive micro‐services based design. Those three essential characteristics are detailed in this article with a specific focus on how to achieve IoT data platform scalability. By means of an air quality monitoring use‐case deployed in the city of Antwerp, the scalability of the platform is evaluated. The evaluation shows that the proposed reactive micro‐service based design allows for horizontal scaling of the platform as well as for logarithmic time complexity of its service time.

HANSEL: Adaptive horizontal scaling of microservices using Bi-LSTM

With the rapid development of 5G network, business scenarios such as intelligent service and new retail are becoming more and more popular. The demand for more flexible and scalable real-time data processing, in particular, the AI-related data processing has also increased in edge computing. Therefore, how to meet such business development has become a major challenge. Focusing on this requirement, microservice architecture, proposed and developed by some big cloud computing companies’ platform, such as Google Kubernetes platform, has gradually become a mainstream technology solution in edge computing.However, many microservices used in edge computing cannot achieve an even time distribution, which is random or sudden. Kubernetes built-in Horizontal POD Autoscaling (HPA) is unable to well handle the change of microservice load, which inevitably leads to the waste of system resources and affects the SLA of microservice.To solve this issue, this paper proposes a HANSEL system based on Kubernetes platform, which can optimize the horizontal elastic scaling policy of Kubernetes by accurately predicting the load of microservices based on the Bi-LSTM load prediction algorithm with attention mechanism. Furthermore, active elastic scaling is realized through reinforcement learning method, and we design a hybrid elastic scaling mechanism through combining reactive and active methods, so as to construct an elastic scaling system for automatic scheduling of working nodes. Our experimental results show that HANSEL system can improve the system resource utilization by about 20% when meeting the microservice SLA of edge computing.

Protected Network Architecture for Ensuring Consistency of Medical Data through Validation of User Behavior and DICOM Archive Integrity

The problem of consistency of medical data in Hospital Data Management Systems is considered in the context of correctness of medical images stored in a PACS (Picture Archiving and Communication System) and legality of actions authorized users perform when accessing MIS (Medical Information System) facilities via web interfaces. The purpose of the study is to develop a SIEM-like (Security Information and Event Management) architecture for offline analysis of DICOM (Digital Imaging and Communications in Medicine) archive integrity and users’ activity. To achieve amenable accuracy when validating DICOM archive integrity, two aspects are taken into account: correctness of periodicity of the incoming data stream and correctness of the image data (time series) itself for the considered modality. Validation of users’ activity assumes application of model-driven approaches using state-of-the-art machine learning methods. This paper proposes a network architecture with guard clusters to protect sensitive components like the DICOM archive and application server of the MIS. New server roles were designed to perform traffic interception, data analysis and alert management without reconfiguration of production software components. The cluster architecture allows the analysis of incoming big data streams with high availability, providing horizontal scalability and fault tolerance. To minimize possible harm from spurious DICOM files the approach should be considered as an addition to other securing techniques like watermarking, encrypting and testing data conformance with a standard.

Knowledge Graph Question and Answer System for Mechanical Intelligent Manufacturing Based on Deep Learning

With the development of deep learning and its wide application in the field of natural language, the question and answer research of knowledge graph based on deep learning has gradually become the focus of attention. After that, the natural language query is converted into a structured query sentence to identify the entities and attributes in the user’s natural language query and the specified entities and attributes are used to retrieve answers to the knowledge graph. Using the advantage of deep learning in capturing sentence information, it incorporates the attention mechanism to obtain the semantic vector of the relevant attributes in the query and uses the parameter sharing mechanism to insert candidate attributes into the triple in the same model to obtain the semantic vector of typical candidates. The experiment measured that under the 100,000 RDF dataset, the single entity query of the MIQE model does not exceed 3 seconds, and the connection query does not exceed 5 seconds. Under the one-million RDF dataset, the single entity query of the MIQE model does not exceed 8 seconds, and the connection query will not be more than 10 seconds. Experimental data show that the system of knowledge-answering questions of engineering of intelligent construction based on deep learning has good horizontal scalability.

Video image scaling technology based on adaptive interpolation algorithm and TTS FPGA implementation

With the rapid development of digital multimedia terminals, the scaling of video images has solved the needs of high-end displays for low-resolution signals and high-definition signals displayed by low-end displays. The main purpose of this article is to convert the input video image signal into a video image signal with the required resolution through video image scaling. This article mainly studies various image scaling algorithms, analyzes their advantages and disadvantages, and adopts a top-down design method to implement FPGA video scaling design and verify the correctness of the results. This article introduces various traditional image interpolation algorithms. On the basis of comparing the subjective and objective quality of the image after the summation of various algorithms, the fuzzy interpolation algorithm is used to zoom the video image. In FPGA design, horizontal scaling and vertical scaling are handled separately, which reduces the computational complexity of image upgrades and facilitates control logic and specific implementation. The experimental results of this paper show that the maximum working frequency that the designed image scaling unit can reach is 153.12 MHz, which can process 1080p video signals in real time. Video image scaling technology has broad market prospects.

SDPSO: Spark Distributed PSO-based approach for feature selection and cancer disease prognosis

The Dimensionality Curse is one of the most critical issues that are hindering faster evolution in several fields broadly, and in bioinformatics distinctively. To counter this curse, a conglomerate solution is needed. Among the renowned techniques that proved efficacy, the scaling-based dimensionality reduction techniques are the most prevalent. To insure improved performance and productivity, horizontal scaling functions are combined with Particle Swarm Optimization (PSO) based computational techniques. Optimization algorithms are an interesting substitute to traditional feature selection methods that are both efficient and relatively easier to scale. Particle Swarm Optimization (PSO) is an iterative search algorithm that has proved to achieve excellent results for feature selection problems. In this paper, a composite Spark Distributed approach to feature selection that combines an integrative feature selection algorithm using Binary Particle Swarm Optimization (BPSO) with Particle Swarm Optimization (PSO) algorithm for cancer prognosis is proposed; hence Spark Distributed Particle Swarm Optimization (SDPSO) approach. The effectiveness of the proposed approach is demonstrated using five benchmark genomic datasets as well as a comparative study with four state of the art methods. Compared with the four methods, the proposed approach yields the best in average of purity ranging from 0.78 to 0.97 and F-measure ranging from 0.75 to 0.96.

Scalable Pattern Matching in Metadata Graphs via Constraint Checking

Pattern matching is a fundamental tool for answering complex graph queries. Unfortunately, existing solutions have limited capabilities: They do not scale to process large graphs and/or support only a restricted set of search templates or usage scenarios. Moreover, the algorithms at the core of the existing techniques are not suitable for today’s graph processing infrastructures relying on horizontal scalability and shared-nothing clusters, as most of these algorithms are inherently sequential and difficult to parallelize. We present an algorithmic pipeline that bases pattern matching on constraint checking . The key intuition is that each vertex and edge participating in a match has to meet a set of constraints implicitly specified by the search template. These constraints can be verified independently and typically are less expensive to compute than searching the full template. The pipeline we propose generates these constraints and iterates over them to eliminate all the vertices and edges that do not participate in any match, thus reducing the background graph to a subgraph that is the union of all template matches—the complete set of all vertices and edges that participate in at least one match. Additional analysis can be performed on this annotated, reduced graph, such as full match enumeration, match counting, or computing vertex/edge centrality. Furthermore, a vertex-centric formulation for constraint checking algorithms exists, and this makes it possible to harness existing high-performance, vertex-centric graph processing frameworks. This technique (i) enables highly scalable pattern matching in metadata (labeled) graphs; (ii) supports arbitrary patterns with 100% precision; (iii) enables tradeoffs between precision and time-to-solution, while always selects all vertices and edges that participate in matches, thus offering 100% recall; and (iv) supports a set of popular data analytics scenarios. We implement our approach on top of HavoqGT, an open-source asynchronous graph processing framework, and demonstrate its advantages through strong and weak scaling experiments on massive scale real-world (up to 257 billion edges) and synthetic (up to 4.4 trillion edges) labeled graphs, respectively, and at scales (1,024 nodes / 36,864 cores), orders of magnitude larger than used in the past for similar problems. This article serves two purposes: First, it synthesises the knowledge accumulated during a long-term project [Reza et al. 2017, 2018; Tripoul et al. 2018]. Second, it presents new system features, usage scenarios, optimizations, and comparisons with related work that strengthen the confidence that pattern matching based on iterative pruning via constraint checking is an effective and scalable approach in practice. The new contributions include the following: (i) We demonstrate the ability of the constraint checking approach to efficiently support two additional search scenarios that often emerge in practice, interactive incremental search and exploratory search . (ii) We empirically compare our solution with two additional state-of-the-art systems, Arabsque [Teixeira et al. 2015] and TriAD [Gurajada et al. 2014]. (iii) We show the ability of our solution to accommodate a more diverse range of datasets with varying properties, e.g., scale, skewness, label distribution, and match frequency. (iv) We introduce or extend a number of system features (e.g., work aggregation, load balancing, and the ability to cap the generated traffic) and design optimizations and demonstrate their advantages with respect to improving performance and scalability. (v) We present bottleneck analysis and insights into artifacts that influence performance. (vi) We present a theoretical complexity argument that motivates the performance gains we observe.

Digital Transformation Enables Automated Real-Time Torque-and-Drag Modeling

This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 199670, “Digital Transformation Strategy Enables Automated Real-Time Torque-and-Drag Modeling,” by Dingzhou Cao, Occidental Petroleum; Don Hender, SPE, IPCOS; and Sam Ariabod, Apex Systems, et al., prepared for the 2020 IADC/SPE International Drilling Conference, Galveston, Texas, 3-5 March. The paper has not been peer reviewed. Automated real-time torque-and-drag (RT-T&D) analysis compares real-time measurements with evergreen models to monitor and manage downhole wellbore friction, improving drilling performance and safety. Enabling RT-T&D modeling with contextual well data, rig-state detection, and RT-interval event filters poses significant challenges. The complete paper presents a solution that integrates a physics-based T&D stiff/soft string model with a real-time drilling (RTD) analytics system using a custom-built extract, transform, and load (ETL) translator and digital-transformation applications to automate the T&D modeling work flow. Methodology A T&D representational state transfer (REST) application program interface (API) was integrated with an RTD analytics system capable of receiving and processing both real-time (hookload, torque, and rig-state) and digitized (drillstring and casing components, trajectory profiles, and mud-property) well data across multiple platforms. This strategy consists of four parts: Digital transformation apps, ETL, and translator Physics-based stiff/soft string T&D model API Pre-existing data infrastructure RTD analytics system The data-flow architecture reveals a flexible design in the sense that it can accommodate different types of T&D models or any other physics-based REST API models (e.g., drillstring buckling or drilling hydraulics) and can be accessed offline for prejob/post-job planning. Drilling engineers can also leverage the RTD systems’ historical database to perform recalculations, comparative analysis, and friction calibrations. The RT-T&D model also can be deployed in a cloud environment to ensure that horizontal scalability is achieved.

Hybrid Auto-Scaled Service-Cloud-Based Predictive Workload Modeling and Analysis for Smart Campus System

The Internet of Things is an emerging technology used in cloud computing and provides many services of the cloud. The cloud services users mostly suffer from service delays and disruptions due to service cloud resource management based on vertical and horizontal scalable systems. Adding more resources to a single cloud server is called vertical scaling, and an increasing number of servers is known as horizontal scaling. The service-bursts significantly impact the vertical scaled environment where the scale-up degrades the service quality and users' trust after reaching the server's maximum capacity. Besides, the horizontally scaled environment, though being resilient, is cost-inefficient. It is also hard to detect and manage bursts online to sustain application efficiency for complex workloads. Burst detection in real-time workloads is a complicated issue because even in the presence of auto-scaling methods, it can dramatically degrade the application's efficiency. This research study presents a new bursts-aware auto-scaling approach that detects bursts in dynamic workloads using resource estimation, decision-making scaling, and workload forecasting while reducing response time. This study proposes a hybrid auto-scaled service cloud model that ensures the best approximation of vertical and horizontal scalable systems to ensure Quality of Service (QoS) for smart campus-based applications. This study carries out the workload prediction and auto-scaling employing an ensemble algorithm. The model pre-scales the scalable vertical system by leveraging the service-load predictive modeling using an ensemble classification of defined workload estimation. The prediction of the upcoming workload helped scale-up the system, and auto-scaling dynamically scaled the assigned resources to many users' service requests. The proposed model efficiently managed service-bursts by addressing load balancing challenges through horizontal auto-scaling to ensure application consistency and service availability. The study simulated the smart campus environment model to monitor the time-stamped diverse service-requests appearing with different workloads.

Dynamic VM Scaling: Provisioning and Pricing through an Online Auction

Today's IaaS clouds allow dynamic scaling of VMs allocated to a user, according to real-time demand of the user. There are two types of scaling: horizontal scaling (scale-out) by allocating more VM instances to the user, and vertical scaling (scale-up) by boosting resources of VMs owned by the user. It has been a daunting issue how to efficiently allocate the resources on physical servers to meet the scaling demand of users on the go, which achieves the best server utilization and user utility. An accompanying critical challenge is how to effectively charge the incremental resources, such that the economic benefits of both the cloud provider and cloud users are guaranteed. There has been online auction design dealing with dynamic VM provisioning, where the resource bids are not related to each other, failing to handle VM scaling where later bids may rely on earlier bids of the same user. As the first in the literature, this paper designs an efficient, truthful online auction for resource provisioning and pricing in the practical cases of dynamic VM scaling, where: (i) users bid for customized VMs to use in future durations, and can bid again in the following time to increase resources, indicating both scale-up and scale-out options; (ii) the cloud provider packs the demanded VMs on heterogeneous servers for energy cost minimization on the go. We carefully design resource prices maintained for each type of resource on each server to achieve threshold-based online allocation and charging, as well as a novel competitive analysis technique based on submodularity of the offline objective, to show a good competitive ratio is achieved. The efficacy of the online auction is validated through solid theoretical analysis and trace-driven simulations.

A system for off-line validation of medical data in DICOM archive

The problem of consistency of medical data in Hospital Data Management Systems is considered in the context of correctness of medical images itself to minimize possible harm from spurious DICOM files. The approach should be considered as an addition to other securing techniques like watermarking, encrypting, testing conformance with the standard. To achieve amenable accuracy in practice two aspects are taken into account: correctness of periodicity and correctness of image data (time series) itself for considered modality. This paper proposes an architecture of an information system and network filter integrated to it to provide facilities for analysis and alert management. The architecture is designed to perform the analysis of incoming data streams within components working in a clustered manner, providing horizontal scalability and fault tolerance to be BigData-ready.

Method for constructing distributed system components for calculating the stationary mode of an electronic schematic representation of microservices with a full-duplex communication channel

The joint use of a microservice architecture and a full-duplex WebSocket data transfer protocol for building components of a circuit design automation system is considered. Comparison of the organization of API based on the REST approach and using the WebSocket technology is made. The technique of horizontal scaling in systems with microservice architecture and full-duplex communication channels is determined. The advantages of using the WebSocket protocol for the interaction of distributed system components, in particular, with client PWA (Progressive Web Apps) applications, are given. The technique of creating a software implementation of the client and server components based on the full-duplex WebSocket protocol is considered. The technique of mathematical description of multi-pole components is considered. An algorithm for solving the problem of modeling a stationary mode of nonlinear circuits is described.

ASML: Algorithm-Agnostic Architecture for Scalable Machine Learning

Machine Learning (ML) applications are growing in an unprecedented scale. The development of easy-to-use machine-learning application frameworks has enabled the development of advanced artificial intelligence (AI) applications with only a few lines of self-explanatory code. As a result, ML-based AI is becoming approachable by mainstream developers and small businesses. However, the deployment of ML algorithms for remote high throughput ML task execution, involving complex data-processing pipelines can still be challenging, especially with respect to production ML use cases. To cope with this issue, in this paper we propose a novel system architecture that enables Algorithm-agnostic, Scalable ML (ASML) task execution for high throughput applications. It aims to provide an answer to the research question of how to design and implement an abstraction framework, suitable for the deployment of end-to-end ML pipelines in a generic and standard way. The proposed ASML architecture manages horizontal scaling, task scheduling, reporting, monitoring and execution of multi-client ML tasks using modular, extensible components that abstract the execution details of the underlying algorithms. Experiments in the context of obstacle detection and recognition, as well as in the context of abnormality detection in medical image streams, demonstrate its capacity for parallel, mission critical, task execution.

GRAPH-BASED FOG COMPUTING NETWORK MODEL

IoT networks generate numerous amounts of data that is then transferred to the cloud for processing. Transferring data cleansing and parts of calculations towards these edge-level networks improves system’s, latency, energy consumption, network bandwidth and computational resources utilization, fault tolerance and thus operational costs. On the other hand, these fog nodes are resource-constrained, have extremely distributed and heterogeneous nature, lack horizontal scalability, and, thus, the vanilla SOA approach is not applicable to them. Utilization of Software Defined Network (SDN) with task distribution capabilities advocated in this paper addresses these issues. Suggested framework may utilize various routing and data distribution algorithms allowing to build flexible system most relevant for particular use-case. Advocated architecture was evaluated in agent-based simulation environment and proved its’ feasibility and performance gains compared to conventional event-stream approach.

Modified Data Storage and Replication Mechanism with Frequent Use-Case Based Indexing

The real time applications demands high speed and reliable data access from the remote database. An effective logical data management strategy that handles simultaneous connections with better performance negotiation is inevitable. This work considers an e-health care application that proposes MongoDB based modified indexing and performance tuning methods. To cope with certain high frequency use case and its performance mandates, a flexible and efficient logical data management may be preferred. By analysing the data dependency, data decomposition concerns and the performance requirements of the specific use case of the medical application, a logical schema may be customized on an ala-carte basis. This work focused on the flexible logical data modeling schemes and its performance factors of the NoSql DB. The efficiency of unstructured data base management in storing and retrieving the e-health care data was analysed with a web based tool. To enable faster data retrieval and query processing over the distributed nodes, a Spark based storage engine was built on top of the MongoDB based data storage management. With Spark tool, the database has been made distributed as master–slave structures with suitable data replication mechanisms. In such distributed database the fail-over also implemented with the suitable replication mechanism. This work considered MongoDB based flexible schema modeling and Spark based distributed computation with multiple chunks of data. The flexible data modeling scheme with MongoDB with the on-demand Spark based computation framework was proposed. To facilitate the eventual consistency, scalability aspects of the e-health care applications, use case based indexing was proposed. With the effective data management, faster query processing the horizontal scalability has been increased. The overall efficiency and scalability of the proposed logical data management approach was analysed. Through the simulation studies, the proposed approach has been claimed to boost the performance of the bigdata based application to a considerable extent.