Scalable Monitoring Tool Research Articles

Leveraging Django: A Web Framework for Efficient Infrastructure Monitoring Tools

Abstract: Infrastructure monitoring is crucial for ensuring the reliability and performance of modern software systems. As these systems grow in complexity and scale, the demand for efficient and scalable monitoring tools becomes increasingly important. This review paper examines the utilization of Django, a powerful web framework, for constructing infrastructure monitoring tools and provides an in-depth review of Django's capabilities, focusing on its robustness, scalability, and extensibility, as well as its compatibility with various databases and integration with other technologies. It explores the different components and modules within Django that can be leveraged to develop efficient infrastructure monitoring tools. The review delves into the Django ORM (Object-Relational Mapping) system, highlighting its role in effective database management for monitoring data. It also evaluates Django's authentication and authorization system, emphasizing its contribution to secure access control in the monitoring environment. The review further examines the Django admin interface, which simplifies monitoring configuration management and provide a comprehensive understanding of the benefits and considerations of utilizing Django for developing efficient and scalable infrastructure monitoring tools. It serves as a valuable resource for researchers and practitioners looking to construct robust monitoring systems using Django and contributes to the ongoing discourse on effective infrastructure monitoring in modern software ecosystems. In conclusion, the paper summarizes the advantages and challenges associated with leveraging Django for infrastructure monitoring tools. It highlights Django's versatility, productivity, and strong community support, while also addressing potential areas for improvement and future research directions.

Dynamic β-VAEs for quantifying biodiversity by clustering optically recorded insect signals

While insects are the largest and most diverse group of terrestrial animals, constituting ca. 80% of all known species, they are difficult to study due to their small size and similarity between species. Conventional monitoring techniques depend on time consuming trapping methods and tedious microscope-based work by skilled experts in order to identify the caught insect specimen at species, or even family level. Researchers and policy makers are in urgent need of a scalable monitoring tool in order to conserve biodiversity and secure human food production due to the rapid decline in insect numbers.Novel automated optical monitoring equipment can record tens of thousands of insect observations in a single day and the ability to identify key targets at species level can be a vital tool for entomologists, biologists and agronomists. Recent work has aimed for a broader analysis using unsupervised clustering as a proxy for conventional biodiversity measures, such as species richness and species evenness, without actually identifying the species of the detected target.In order to improve upon existing insect clustering methods, we propose an adaptive variant of the variational autoencoder (VAE) which is capable of clustering data by phylogenetic groups. The proposed dynamic β-VAE dynamically adapts the scaling of the reconstruction and regularization loss terms (β value) yielding useful latent representations of the input data. We demonstrate the usefulness of the dynamic β-VAE on optically recorded insect signals from regions of southern Scandinavia to cluster unlabelled targets into possible species. We also demonstrate improved clustering performance in a semi-supervised setting using a small subset of labelled data. These experimental results, in both unsupervised- and semi-supervised settings, with the dynamic β-VAE are promising and, in the near future, can be deployed to monitor insects and conserve the rapidly declining insect biodiversity.

Machine learning approach to dynamic risk modeling of mortality in COVID-19: a UK Biobank study

The COVID-19 pandemic has created an urgent need for robust, scalable monitoring tools supporting stratification of high-risk patients. This research aims to develop and validate prediction models, using the UK Biobank, to estimate COVID-19 mortality risk in confirmed cases. From the 11,245 participants testing positive for COVID-19, we develop a data-driven random forest classification model with excellent performance (AUC: 0.91), using baseline characteristics, pre-existing conditions, symptoms, and vital signs, such that the score could dynamically assess mortality risk with disease deterioration. We also identify several significant novel predictors of COVID-19 mortality with equivalent or greater predictive value than established high-risk comorbidities, such as detailed anthropometrics and prior acute kidney failure, urinary tract infection, and pneumonias. The model design and feature selection enables utility in outpatient settings. Possible applications include supporting individual-level risk profiling and monitoring disease progression across patients with COVID-19 at-scale, especially in hospital-at-home settings.

Performance-Aware Scheduling of Parallel Applications on Non-Dedicated Clusters

This work presents a HPC framework that provides new strategies for resource management and job scheduling, based on executing different applications in shared compute nodes, maximizing platform utilization. The framework includes a scalable monitoring tool that is able to analyze the platform’s compute node utilization. We also introduce an extension of CLARISSE, a middleware for data-staging coordination and control on large-scale HPC platforms that uses the information provided by the monitor in combination with application-level analysis to detect performance degradation in the running applications. This degradation, caused by the fact that the applications share the compute nodes and may compete for their resources, is avoided by means of dynamic application migration. A description of the architecture, as well as a practical evaluation of the proposal, shows significant performance improvements up to 20% in the makespan and 10% in energy consumption compared to a non-optimized execution.

병행 Java 프로그램의 확장적 경합탐지를 위한 JDI 기반의 투명한 감시도구

병행 Java 프로그램의 경합은 프로그램의 비결정성을 초래하므로 반드시 탐지되어야 한다. 이러한 경합을 수행 중에 탐지하기 위해서는 스레드에 대한 수행양상과 모든 접근사건들을 감시할 수 있어야 한다. 기존의 경합탐지 기법들은 프로그램의 수행중에 기록된 파일들을 분석하거나 대상 프로그램을 수정하여 감시하므로 스레드나 모든 접근사건들에 대한 감시가 현실적으로 어렵다. 본 연구에서는 JDI(Java Debug Interface)를 이용하여 스레드에 대한 수행양상과 모든 접근사건을 감시하여 확장적 경합탐지를 할 수 있는 투명한 감시도구를 제안한다. 여기서 JDI는 JDPA(Java Platform Debugger Architecture)에서 제공하는 상위 레벨의 100% 순수 자바 인터페이스로써 자바프로그램의 수행중에 특정 정보를 제공할 수 있다. 그리고 제안된 도구의 투명성을 입증하기 위해서 벤치마크 프로그램으로 실험한 결과, 모든 스레드와 접근사건들을 프로그램 수정없이 감시할 수 있었고 프로그램의 감시시간이 20배 이상 증가되었다. Race conditions in current Java programs must be detected because it may cause unexpected result by non-deterministic executions. For detecting such races during program execution, execution flows of all threads and all access events can be monitored. It is difficult for previous race detection techniques to monitor all threads and access events in actuality because these techniques analyze the files traced during program execution or modify original source programs and then monitor these programs. This paper presents a transparent scalable monitoring tool to detect races using JDI(Java Debug Interface) where JDI is 100% pure java interface to provide in JDPA(Java Platform Debugger Architecture) and is able to provide information corresponding to events occurred in run-time of programs. This tool thus can monitor execution flows of all threads and all access events without program modification. We prove transparency of the presented tool and grasp the efficiency of it using a set of published benchmark programs. As a result of this, the suggested tool can monitor all threads and accesses of these programs without their modification, and their monitoring time is increased to more than 20 times.