Azure SQL Research Articles

METHOD OF DETERMINING THE PARAMETER OF QUALITATIVE EVALUATION OF A WEB FORUM

Context. The development of new types of virtual environments is an urgent task of informatisation of modern education, since such services allow enhancing the quality of educational services and contribute to a deeper assimilation of new knowledge by students. A web application proposed in this paper has been built using modern approaches to creating web pages using the .NET programming language, Bootstrap and ASP.NET MVC frameworks, Azure cloud solutions and Azure SQL databases, which has enabled the simplification of software development by distributing functions between the application modules and provided the flexibility, performance, and security necessary to work with relational data. The effectiveness of the application in the educational process has been experimentally tested using the method of determining the qualitative evaluation of the web forum usefulness parameter, which was developed by introducing an informative parameter of the discussion quality based on the h-index (sometimes called the Hirsch index or Hirsch number). Objective. To build a mathematical model of a web forum and develop a method of determining the qualitative evaluation of the parameter of usefulness of discussions in the created web application, which would allow improving the quality of educational and scientific activities in a higher education institution. Method. A method of determining the parameter of qualitative evaluation of a web forum using the h-index has been developed, which enabled analysing the interest in covering the trends of discussion on the forum pages and planning on its basis further work of the forum as a tool of a virtual learning environment. Results. Based on the analysis of the results of the implementation of the web application in the educational process of the Department of Information Technologies Vasyl Stefanyk Precarpathian National University, the user activity of posts has been analysed and the effectiveness of discussions of the proposed topics on the forum pages has been determined using the introduced activity parameter. Conclusions. A mathematical model of a web forum has been built, and the application has been implemented using modern approaches to software development using an optimised MVC architecture, which enabled simplification of creating a service by distributing responsibilities between the application modules and facilitating testing and technical support of the service. The scientific novelty of the study is the development of a method of evaluating the usefulness of discussions in a web forum by introducing a new informative quality parameter, the use of which allowed broadening the scope of existing limitations in quantitative analytics of discussions and feedbacks in popular services. Experimental studies carried out on the basis of a higher education institution have confirmed the effectiveness of the method application to improve the quality of educational services. The practical significance of the obtained results is the development of a software product as a tool of the virtual learning environment of a higher education institution.

Web Development for Bank Statement Analysis: Technologies, Challenges, and Innovations

The purpose of this project is to fulfill the requirements of a bachelor’s degree in Cloud Technology and Information Security. The project aims to design and develop a Bank Statement Analysis system that provides a more secure approach to managing bank customers' statements, strengthening the relationship between banks and their customers by offering solutions with multi-level security to improve customer satisfaction. Methodology The project utilizes a range of technologies, including: · Python Flask Framework · Computer Vision (Google's Tesseract OCR) · PDF to Image Python package (pdf2image) · Machine Learning (Scikit-Learn) to predict transaction categories · Azure SQL Database to store transaction data and user login details · Power-BI for data visualization · Docker and AWS EC2 for service hosting. Findings The Bank Statement Analysis system developed in this project offers a secure and efficient solution for managing bank customers' statements. The use of multi-level security measures, such as Azure SQL Database and Power-BI visualizations, enhances the overall security and user experience, strengthening the relationship between banks and their customers. Conclusion This project demonstrates the successful design and development of a Bank Statement Analysis system that leverages various technologies, including cloud computing, computer vision, and machine learning, to provide a more secure and user-friendly solution for managing bank customers' statements. Keywords Azure SQL Database, Power-BI Tool, AWS EC2 Service

Validation of a novel web-based application for hepatic arterial infusion floxuridine dosing.

15 Background: Hepatic arterial infusion (HAI) floxuridine (FUDR) dosing is a complex, multifactorial process with weight-based dosing dependent on the fold change of aspartate aminotransferase, alkaline phosphatase, and total bilirubin relative to reference values from preceding cycles and compounded over time. Given the complexity of dosing calculations and lack of experience with this treatment in community centers, HAI is typically administered only at tertiary oncology centers with limited access for rural disadvantaged patients. We developed an online tool to automate the dosing of FUDR to broaden safe and effective administration of HAI. Methods: The FUDR dosing algorithm developed at Memorial Sloan Kettering Cancer Center was digitalized as eDoseHAI, a novel multi-layer architecture web application tool featuring JavaScript (front-end), C# RESTful API (back-end) and Azure SQL server (data storage with Splunk data lake event registration). To validate this tool, manual dosing from patients who received ≥1 dose of HAI FUDR at our institution 6/2020-8/2023 was retrospectively compared to app-based dosing. Mismatch between app-based dosing and historical manual dosing was designated as dosing errors, excluding intentional adjustments based on physician discretion. Results: Internal validity was tested by entering records into eDoseHAI from 54 patients who received a total of 370 FUDR cycles (median 6; range 1-19). Ten patients (18.5%) were found to have unintentional dosing errors (e.g., delayed dose reductions) during treatment, impacting 52 cycles (14.1%). The median cycle at first error was 3 (range 2-6), impacting a median of 5 cycles (range 3-12). Dosing errors occurred in only 11.1% of the 45 patients supervised by a senior HAI physician, compared to 55.6% of the 9 patients supervised by three less experienced HAI physicians. Dosing accuracy improved over time; among the 14 patients on active HAI therapy, only 1 had an identified dosing error. eDoseHAI was able to reliably and accurately calculate dosing for all patients. Conclusions: HAI FUDR dosing is nuanced and complex, such that dosing errors occurred even with senior HAI oncologists. eDoseHAI is a web-based application tool designed to improve accuracy and thus safety of HAI dosing. We plan to further validate eDoseHAI using external data from an international HAI consortium, and to conduct a phase II clinical trial in which our team supervises FUDR dosing by community oncologists using eDoseHAI's supervisory mode with a goal to disseminate HAI therapy to disadvantaged rural patients who lack access.

Flexible Resource Allocation for Relational Database-as-a-Service

Oversubscription is an essential cost management strategy for cloud database providers, and its importance is magnified by the emerging paradigm of serverless databases. In contrast to general purpose techniques used for oversubscription in hypervisors, operating systems and cluster managers, we develop techniques that leverage our understanding of how DBMSs use resources and how resource allocations impact database performance. Our techniques are designed to flexibly redistribute resources across database tenants at the node and cluster levels with low overhead. We have implemented our techniques in a commercial cloud database service: Azure SQL Database. Experiments using microbenchmarks, industry-standard benchmarks and real-world resource usage traces show that using our approach, it is possible to tightly control the impact on database performance even with a relatively high degree of oversubscription.

Microsoft Purview: A System for Central Governance of Data

Modern data estates are spread across data located on premises, on the edge and in one or more public clouds, spread across various sources like multiple relational databases, file and storage systems, and no-SQL systems, both operational and analytic; this phenomenon is referred to as data sprawl. Data administrators who wish to enforce compliance across the entire organization have to inventory their data, identify what parts of it are sensitive, and govern the sensitive data appropriately --- across the entirety of their sprawling data estate. Today, governance of data is completely siloed; each of the data subsystems has its own (and varied) governance features. Policies applied to sensitive data are applied piece-meal by iterating over all the data sources in a custom language specific to each source. This makes data governance cumbersome, error-prone (because a given policy must be manually enforced across different subsystems, inconsistencies can easily arise), and expensive. This paper presents Microsoft Purview , a service for unified governance of the entire data estate of an organization from a single central pane of glass. The Purview service consists of three parts: (1) a Data Map or metadata catalog that is populated by automated scanning of data sources in the organization, (2) a system to store and manage sensitivity classification of data, and (3) a policy system that enables data security officers to author and implement policies that span the entire organization, e.g., a policy that says, "Non-full-time employees should be denied access to data classified as PII (Personally Identifiable Information.") Purview transforms data governance across a complex data estate by offering the ability to govern centrally and automating data discovery, classification and policy enforcement. While other commercial catalog systems also build a global catalog, Purview is unique in its support for policies. It is also distinguished by covering both structured and unstructured data, thanks to its deep integration with Office 365 and its governance framework; indeed, "Microsoft Purview" represents a new unified offering that combines Office 365 governance and what was formerly a service for governing structured data called "Azure Purview". By integrating with Office 365's Rights Management Service, Purview offers central governance over structured data stored in databases and stores, reports in systems such as Power BI, as well as document data stored in Office 365. The Purview vision is to make the metadata in the Data Map increasingly richer through further automation and curation support and to use this 360 degree view of the data estate to support a wide range of governance policies, ranging from access control to lifecycle management (e.g., retention, deletion, restricting data movement). This paper covers the design and implementation challenges in building the Purview service for Attribute-Based Access Control (ABAC) policies, focusing specifically on a detailed description of its integration with Azure SQL Database. We illustrate the power of unifying Office 365 governance with structured data governance through Purview policies that enforce consistent access control even as data flows between Office 365 and structured data engines like Azure SQL Database. We also describe the results of our empirical evaluation of the performance overheads imposed by Purview.

ПОРІВНЯЛЬНИЙ АНАЛІЗ ХМАРНИХ СЕРВІСІВ ДЛЯ ОБРОБКИ ГЕОІНФОРМАЦІЙНИХ ДАНИХ

The article is devoted to a comparative analysis of cloud services for processing geographic data. It describes in detail the services - Google Cloud, Amazon Web Services and Microsoft Azure - that provide tools for storing, processing and analyzing large amounts of geographic data. The article also describes the parameters of geoinformation services, the access algorithm, and examples of program code for processing satellite data. The article describes such opportunities and limitations of using cloud services as automation, security and scalability. The conclusions and recommendations for further development of geographic information systems based on cloud services are provided. Services. Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) offer a variety of geodata storage solutions. These solutions include object storage, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage, as well as geospatial databases, such as Amazon RDS, Azure Cosmos DB, and Google Cloud Firestore. In addition, each of these services provides a set of services for analyzing and processing geographic information data. For example, AWS offers services such as Amazon Athena, Amazon Redshift, and AWS Glue, which allow you to run SQL queries, conduct analytics, and integrate geodata with other services. Azure offers services such as Azure SQL Database, Azure Databricks, and HDInsight, which provide capabilities for processing and analyzing geographic data. GCP also provides services such as BigQuery, Dataflow, and Dataproc, which allow you to perform analytical operations and process large amounts of geodata. Support for integration with various geo-tools is important for analysis, such as AWS, Amazon Location Service, Amazon Ground Truth, and Amazon Rekognition, which allow you to work with geodata at different levels of complexity. Azure has Azure Maps, which provides services for geocoding, routing, and visualization of geodata. GCP also offers Google Maps Platform, which provides extensive integration with geographic technologies such as routing, geocoding, and map visualization. All these processes will allow for more efficient data processing. Keywords: cloud technologies,

Doppler

Selecting the optimal cloud target to migrate SQL estates from on-premises to the cloud remains a challenge. Current solutions are not only time-consuming and error-prone, requiring significant user input, but also fail to provide appropriate recommendations. We present Doppler, a scalable recommendation engine that provides right-sized Azure SQL Platform-as-a-Service (PaaS) recommendations without requiring access to sensitive customer data and queries. Doppler introduces a novel price-performance methodology that allows customers to get a personalized rank of relevant cloud targets solely based on low-level resource statistics, such as latency and memory usage. Doppler supplements this rank with internal knowledge of Azure customer behavior to help guide new migration customers towards one optimal target. Experimental results over a 9-month period from prospective and existing customers indicate that Doppler can identify optimal targets and adapt to changes in customer workloads. It has also found cost-saving opportunities among over-provisioned cloud customers, without compromising on capacity or other requirements. Doppler has been integrated and released in the Azure Data Migration Assistant v5.5, which receives hundreds of assessment requests daily.

Tenant placement in over-subscribed database-as-a-service clusters

Relational cloud Database-as-a-Service offerings run on multi-tenant infrastructure consisting of clusters of nodes, with each node hosting multiple tenant databases. Such clusters may be over-subscribed to increase resource utilization and improve operational efficiency. When resources are over-subscribed, it is possible that anode has insufficient resources to satisfy the resource demands of all databases on it, making it necessary to move databases to other nodes. Such moves can significantly impact database performance and availability. Therefore, it is important to reduce the likelihood of such resource shortages through judicious placement of databases in the cluster. We propose a novel tenant placement approach that leverages historical traces of tenant resource demands to estimate the probability of resource shortages and leverages these estimates in placement. We have prototyped our techniques in the Service Fabric cluster manager. Experiments using production resource traces from Azure SQL DB and an evaluation on a real cluster deployment show significant improvements over the state-of-the-art.

PERFORMANCE STUDY OF THE DTU MODEL FOR RELATIONAL DATABASES ON THE AZURE PLATFORM

When solving problems of working with relational databases on cloud platforms, the problem arises of choosing a specific model to ensure the performance of executing queries of varying complexity. The object of research is the processes of implementing various types of queries to relational databases within the framework of the DTU purchase model of the MS Azure platform. The subject is methods for evaluating the performance of work with relational databases based on the timing of query execution and indicators of the load on the resources of the cloud platform. The aim of the study is to develop a system of indicators for monitoring the current state of work with the database for reasonable decision-making on the choice of a certain price category of the DTU model of the MS Azure cloud service, which will optimize the results of working with the database. platforms Achieving the set goals involves the following tasks: to analyze modern tools and services for working with databases, in particular relational databases, on Azure and AWS cloud platforms, the features of their application and implementation; develop software for generating test relational databases of different sizes; test the generated databases on a local resource; taking into account the characteristics of the levels of the Azure DTU model, develop a new system of performance indicators, which includes 2 groups - time indicators and indicators of the load on existing platform resources; develop and implement queries of varying complexity for the generated test database for different levels of the DTU model and analyze the results. Methods. The following methods were used in the research: methods of relational database design; methods of creating queries in SQL-oriented databases with any number of tables; methods of creating and migrating data to cloud platforms; methods of monitoring the results of queries based on time and resource indicators; methods of generating test data for relational databases; system approach for complex assessment and analysis of productivity of work with relational databases. Results. On the basis of the developed scorecard used for the current analysis of the processes of working with relational databases of the MS Azure platform, numerous experiments were carried out for different levels of the model for simple and complex queries to a database with a total volume of 20 GB: loading of DTU model levels when executing various queries, the influence of model levels DTU Azure SQL database on the performance of simple and complex queries, the dependence of the execution time of various queries on the load of the CPU and the speed of write/read operations for different levels of the model. Conclusions. The results of the experiments allow us to conclude that the levels of the DTU model - S3 and S7 - are used to generate test data of various sizes (up to 20 GB) and execute database queries. The practical use of the proposed indicators to evaluate the results of applying the DTU model will improve the efficiency of decision-making on choosing the model level when implementing various queries and generating test data on the Azure cloud platform. The developed set of indicators for working with relational databases on the Azure cloud platform expands the basis of the methodological framework for evaluating the performance of working with relational databases on cloud platforms by analyzing the results of executing the simple and complex database queries on the resources involved.

Moneyball

Microsoft Azure SQL Database is among the leading relational database service providers in the cloud. Serverless compute automatically scales resources based on workload demand. When a database becomes idle its resources are reclaimed. When activity returns, resources are resumed. Customers pay only for resources they used. However, scaling is currently merely reactive, not proactive, according to customers' workloads. Therefore, resources may not be immediately available when a customer comes back online after a prolonged idle period. In this work, we focus on reducing this delay in resource availability by predicting the pause/resume patterns and proactively resuming resources for each database. Furthermore, we avoid taking away resources for short idle periods to relieve the back-end from ineffective pause/resume workflows. Results of this study are currently being used worldwide to find the middle ground between quality of service and cost of operation.

Comparative Case Study Difference Between Azure Cloud SQL and Mongo Atlas MongoDB NoSQL Database

Azure SQL and Atlas Mongodb NoSQL(Azure instance) databases are the most popular, systematic process to database solutions. Which Azure SQL database is also referred to as RDBMS (Relational Database Management Systems). The data are structured into tables or associations. The Atlas Mongodb NoSQL database is called a non-relational database management systems. The data are included in unstructured tables or associations. In this research, evaluate both the Azure SQL and Atlas Mongodb NoSQL databases. During the experiment compare the loading time, response time, and retrieval time of both Azure SQL and Atlas Mongodb NoSQL databases, and justify which one is fast, efficient and better performance.

Mobile lung cancer early warning based on Windows Azure cloud computing

In view of the repeated investment in the current medical lung cancer early warning system, 'information island' and the low rate of lung cancer early warning, this paper proposes a mobile lung cancer early warning system based on Windows Azure cloud computing with the application requirements of medical information system. The system is applied to the smart phones of Windows system, and the Windows Azure Microsoft cloud platform is accessed through wireless network, using SQL Azure to store medical data (such as lung CT image, ECG curve and high-resolution colour ultrasound image), embed the medical lung cancer early warning model based on deep learning in the cloud computing platform to process and analyse the patient's medical data, and realise the real-time synchronous update of doctors, patients and cloud platform data. The accuracy of lung cancer early-warning system on LIDC-IDRI is high, and the detection results are uploaded to doctors and patients' smart phones in real time. The results show that the mobile lung cancer early warning system based on Windows Azure cloud computing not only effectively integrates medical resources, but also provides patients with medical services and consultation at home, which integrates the advantages of all aspects and adapts to the development of the times.

Mobile lung cancer early warning based on Windows Azure cloud computing

In view of the repeated investment in the current medical lung cancer early warning system, 'information island' and the low rate of lung cancer early warning, this paper proposes a mobile lung cancer early warning system based on Windows Azure cloud computing with the application requirements of medical information system. The system is applied to the smart phones of Windows system, and the Windows Azure Microsoft cloud platform is accessed through wireless network, using SQL Azure to store medical data (such as lung CT image, ECG curve and high-resolution colour ultrasound image), embed the medical lung cancer early warning model based on deep learning in the cloud computing platform to process and analyse the patient's medical data, and realise the real-time synchronous update of doctors, patients and cloud platform data. The accuracy of lung cancer early-warning system on LIDC-IDRI is high, and the detection results are uploaded to doctors and patients' smart phones in real time. The results show that the mobile lung cancer early warning system based on Windows Azure cloud computing not only effectively integrates medical resources, but also provides patients with medical services and consultation at home, which integrates the advantages of all aspects and adapts to the development of the times.

Automatic Database Troubleshooting of Azure SQL Databases

Exponentially growing number of workloads has been transferred from on-premises to the cloud environment during last decade. It incurs a constantly increasing load on the monitoring and troubleshooting capabilities of the platforms that host those applications nowadays. Data mining techniques and use of telemetry data have recently become an unavoidable means for tracking the behavior of cloud services. The research effort elaborated in this paper is focused on exploiting real-world data to build an automatic database troubleshooting system that exploits the combination of the more comprehensive statistical data science models and an expert system to perform the root cause analysis. An extensive evaluation study was conducted during the eight-month period with a plethora of Azure SQL production workloads for a varying number of databases. The obtained results confirmed the viability and cost-effectiveness of such an approach at the scale of the cloud.

Improving the Content of a Client-Server Technology Training Course: Set up and Collaborative Implementation of Local and Cloud-Based Remote Servers

The integration of software solutions in the content of the training courses as a result of the socio-economic demand has a positive effect on the improvement of training the students for future professional activities. However, the latest achievements of hardware and software are not included in the content of the high education in a timely manner. Therefore, in our research we set the task to improve the content of the training course on the client-server technology using modern software capabilities. The content of the article consists of the conceptual structure of the training course, settings and integration of local and public cloud-based servers, describes the approaches of its using in the learning process. Microsoft SQL Server 2016, Microsoft Azure SQL Database, ASP.NET MVC 5 Framework were used as teaching environments. Training course was held for the students in the specialty of Computer Science of L.N.Gumilyov Eurasian National University and Kh.Dosmukhamedov Atyrau State University in Kazakhstan.

Constant time recovery in Azure SQL database

Azure SQL Database and the upcoming release of SQL Server introduce a novel database recovery mechanism that combines traditional ARIES recovery with multi-version concurrency control to achieve database recovery in constant time, regardless of the size of user transactions. Additionally, our algorithm enables continuous transaction log truncation, even in the presence of long running transactions, thereby allowing large data modifications using only a small, constant amount of log space. These capabilities are particularly important for any Cloud database service given a) the constantly increasing database sizes, b) the frequent failures of commodity hardware, c) the strict availability requirements of modern, global applications and d) the fact that software upgrades and other maintenance tasks are managed by the Cloud platform, introducing unexpected failures for the users. This paper describes the design of our recovery algorithm and demonstrates how it allowed us to improve the availability of Azure SQL Database by guaranteeing consistent recovery times of under 3 minutes for 99.999% of recovery cases in production.

ДОБІР ХМАРО ОРІЄНТОВАНИХ ЗАСОБІВ НАВЧАННЯ БАЗ ДАНИХ МАЙБУТНІХ ФАХІВЦІВ З ІНФОРМАЦІЙНИХ ТЕХНОЛОГІЙ

The article describes the selection of cloud-oriented learning tools for educational process of the students of higher education establishments by the expert appraisal method. The criteria and appropriate indicators for learning tool selection have been defined. Three learning tools to study databases were selected out of the list of offered tools, such as: Amazon RDS, Google Cloud SQL, Heroku PostgreSQL, Microsoft SQL Azure, Oracle Database Cloud Service, Rackspace Cloud Databases, SQLite Viewer. The expert appraisal method has been used with involvement of teachers, lecturers, departments’ heads and deans that have experience in teaching databases at the Ukrainian higher education establishments. The comparing analysis of the cloud-oriented database learning tools, such as Google Cloud SQL, Microsoft SQL Azure, SQLite Viewer with Google Drive, has been conducted. The functionally-didactical and organizational criteria were defined. The possibility to use such cloud-oriented learning tools in the higher education establishments were examined. Appropriate indicators for each criterion were defined. The functionally-didactical criterion has the next indicators: possibility to create, to edit and drop tables in the databases; finding the primary and external keys for the tables; creation the connections for the tables in the databases; data modification in the tables of databases; functionality to analyze the results and errors of the queries execution in the databases. The organizational criterion has the following indicators: affordability (presence of the free of charge version; validity term for the free of charge version); need in the financial ability of the customer prove; customer interface usability. As the result of the expert appraisal selection method, has been revealed that SQLite Viewer with Google Drive is the most reasonable learning tool, by the manifestation of all the criteria, for the students of higher education establishments.

Analysing Signal Strength and Connection Speed in Cloud Networks for Enterprise Business Intelligence

Signal strength and speed connection data which are collected and available in a company have not been optimally and beneficially processed and stored for more added value business purposes. Therefore, the collected data need to be utilized in more strategic way for business intelligent in the company that enables management to conduct better and smarter decision making. This research is aimed to develop a business intelligence system based on cloud computing platform which is more flexible and manageable in terms of cost and resources. The developed system adopts the three tier architectures of data warehouse that provides data extraction, transform, and load (ETL) functions and the creation of dimensional models and visualization in dashboard forms. Business intelligence solutions have been created based on cloud computing using Microsoft Azure SQL Database as database storage of data warehouse and Power BI as a dimensional model and dashboard visualization. The developed system prototype has been implemented and tested for its functionalities and capabilities in a web platform.

An extended evaluation of schema issues advisor in the azure SQL database

An extended evaluation of schema issues advisor in the azure SQL database

Resumable online index rebuild in SQL server

Azure SQL Database and the upcoming release of SQL Server enhance Online Index Rebuild to provide fault-tolerance and allow index rebuild operations to resume after a system failure or a user-initiated pause. SQL Server is the first commercial DBMS to support pause and resume functionality for index rebuilds. This is achieved by splitting the operation into incremental units of work and persisting the required state so that it can be resumed later with minimal loss of progress. At the same time, the proposed technology minimizes the log space required for the operation to succeed, making it possible to rebuild large indexes using only a small, constant amount of log space. These capabilities are critical to guarantee the reliability of these operations in an environment where a) the database sizes are increasing at a much faster pace compared to the available hardware, b) system failures are frequent in Cloud architectures using commodity hardware, c) software upgrades and other maintenance tasks are automatically handled by the Cloud platforms, introducing further unexpected failures for the users and d) most modern applications need to be available 24/7 and have very tight maintenance windows. This paper describes the design of "Resumable Online Index Rebuild" and discusses how this technology can be extended to cover more schema management operations in the future.