Scalable Software Research Articles

From DevOps to MLOps: Overview and Application to Electricity Market Forecasting

In the Software Development Life Cycle (SDLC), Development and Operations (DevOps) has been proven to deliver reliable, scalable software within a shorter time. Due to the explosion of Machine Learning (ML) applications, the term Machine Learning Operations (MLOps) has gained significant interest among ML practitioners. This paper explains the DevOps and MLOps processes relevant to the implementation of MLOps. The contribution of this paper towards the MLOps framework is threefold: First, we review the state of the art in MLOps by analyzing the related work in MLOps. Second, we present an overview of the leading DevOps principles relevant to MLOps. Third, we derive an MLOps framework from the MLOps theory and apply it to a time-series forecasting application in the hourly day-ahead electricity market. The paper concludes with how MLOps could be generalized and applied to two more use cases with minor changes.

Design and Deployment of a Generic Software for Managing Industrial Vision Systems

Computer vision systems have become a key element of modern companies. However, its implantation not only requires specialized hardware components but also software to control the correct performance of all these components. The purpose of this article is to present the design and deployment of a software created to manage the different computer vision systems of any company. First, using our experience in mechanization and industrialization of company processes, a general company specification will be proposed as well as the main software requirements that have to be satisfied. Then, a three-level modular design composed of the core, a configuration module, and user interfaces with functionalities capable of satisfying the defined requirements will be presented. Special attention will be given to time restrictions and components’ synchronization. In addition, the different tests that have been carried out to control the correct performance of the software will be shown. The development process will end with a generic, modular, and scalable software able to fit different industrial scenarios by simply modifying a set of input parameters. To illustrate the correct performance of the proposal, the details of its installation in four real companies with different needs will be presented. The proposed work has a practical use in industry and it also provides a thorough description of the main components involved in computer vision systems of real company environments and how to manage them. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The aim of the article is to provide a software solution to manage the computer vision systems of a company. The proposed software is simple enough to be controlled by company operators with no need for great expertise on the technical aspects, but rather on the industrial workflow to be controlled. To develop such a software, it has been necessary to know about all the elements involved in the industrial scenarios controlled by computer vision systems as well as their relationships. This article presents all these elements providing a global and complete view of the problem with special attention to synchronization, which is fundamental to satisfy industrial time restriction requirements. The software is presented from a technical point of view, but also from a practical one, since different examples of its application in real cases are given. These examples illustrate the adaptability of the proposal to different scenarios and also exemplify how the different elements are modified to fit the different situations. In addition, this article provides a comprehensive overview of real industrial situations, focusing on the most important issues faced during real installations.

Teknik Elisitasi Kebutuhan Perangkat Lunak: Literatur Review

The selection of software requirements elicitation techniques is very important for organizations as the first step in the software development process. This study is motivated by the existence of problems, where the organization does not understand the type of software being developed, so it is difficult to determine the right need elicitation technique. In this study, a literature review was used by collecting existing studies. Used 17 studies related to the discussion of elicitation techniques used. This study aims to review previous studies so as to produce recommendations for software requirements elicitation techniques. Based on the study, it can be concluded that interview, prototyping and JAD are most often used in software projects. In addition, for the small-medium scale software projects, more use of interviews, JAD, and ERD-based specifications. Meanwhile, the large-scale software projects involve more stakeholders with elicitation techniques such as group discussions, interviews, and parallel discussions to find joint solutions.

An efficient and scalable data analysis solution for automated electrophysiology platforms.

Ion channels are drug targets for neurologic, cardiac, and immunologic diseases. Many disease-associated mutations and drugs modulate voltage-gated ion channel activation and inactivation, suggesting that characterizing state-dependent effects of test compounds at an early stage of drug development can be of great benefit. Historically, the effects of compounds on ion channel biophysical properties and voltage-dependent activation/inactivation could only be assessed by using low-throughput, manual patch clamp recording techniques. In recent years, automated patch clamp (APC) platforms have drastically increased in throughput. In contrast to their broad utilization in compound screening, APC platforms have rarely been used for mechanism of action studies, in large part due to the lack of sophisticated, scalable analysis methods for processing the large amount of data generated by APC platforms. In the current study, we developed a highly efficient and scalable software workflow to overcome this challenge. This method, to our knowledge the first of its kind, enables automated curve fitting and complex analysis of compound effects. Using voltage-gated sodium channels as an example, we were able to immediately assess the effects of test compounds on a spectrum of biophysical properties, including peak current, voltage-dependent steady state activation/inactivation, and time constants of activation and fast inactivation. Overall, this automated data analysis method provides a novel solution for in-depth analysis of large-scale APC data, and thus will significantly impact ion channel research and drug discovery.

PARCSIM: a parallel computing simulator for scalable software optimization

PARCSIM is a parallel software simulator that allows a user to capture, through a graphical interface, matrix algorithm schemes that solve scientific problems. With this tool, the user can analyse the execution times that would be obtained by using different spatio-temporal mapping of computational tasks on available computational units, parallelism parameters and computational libraries. Furthermore, for complex problem models, the self-optimization engine incorporated in this tool analyses the huge tree of possible calculations grouping and mapping strategies in search of the choice that makes the best use of the available hardware resources. This tool also offers polyalgorithmic resolution by making automatically the best decision between different software approaches to solve a given problem on the hardware system available. This work shows the usefulness of this simulator to efficiently solve hierarchical problems constructed from previously modelled subproblems. This task is performed by reusing, in a scalable way, the optimization information of these subproblems to establish the best execution configuration for the composite problem.

A hybrid approach based on k-nearest neighbors and decision tree for software fault prediction

Software testing is a very important part of the software development life cycle to develop reliable and bug-free software but it consumes a lot of resources like development time, cost, and effort. Researchers have developed many techniques to get prior knowledge of fault-prone modules so that testing time and cost can be reduced. In this research article, a hybrid approach of distance-based pruned classification and regression tree (CART) and k- nearest neighbors is proposed to improve the performance of software fault prediction. The proposed technique is tested on eleven medium to large scale software fault prediction datasets and performance is compared with decision tree classifier, SVM and its three variations, random forest, KNN, and classification and regression tree. Four performance metrics are used for comparison purposes that are accuracy, precision, recall, and f1-score. Results show that our proposed approach gives better performance for accuracy, precision, and f1-score performance metrics. The second experiment shows a significant amount of running time improvement over the standard k-nearest neighbor algorithm.

CaliPER: A software for blood-free parametric Patlak mapping using PET/MRI input function

Routine clinical use of absolute PET quantification techniques is limited by the need for serial arterial blood sampling for input function and more importantly by the lack of automated pharmacokinetic analysis tools that can be readily implemented in clinic with minimal effort. PET/MRI provides the ability for absolute quantification of PET probes without the need for serial arterial blood sampling using image-derived input functions (IDIFs). Here we introduce caliPER, a modular and scalable software for simplified pharmacokinetic modeling of PET probes with irreversible uptake or binding based on PET/MR IDIFs and Patlak Plot analysis. caliPER generates regional values or parametric maps of net influx rate (Ki) using reconstructed dynamic PET images and anatomical MRI aligned to PET for IDIF vessel delineation. We evaluated the performance of caliPER for blood-free region-based and pixel-wise Patlak analyses of [18F] FDG by comparing caliPER IDIF to serial arterial blood input functions and its application in imaging brain glucose hypometabolism in Frontotemporal dementia. IDIFs corrected for partial volume errors including spill-out and spill-in effects were similar to arterial blood input functions with a general bias of around 6–8%, even for arteries <5 mm. The Ki and cerebral metabolic rate of glucose estimated using caliPER IDIF were similar to estimates using arterial blood sampling (<2%) and within limits of whole brain values reported in literature. Overall, caliPER is a promising tool for irreversible PET tracer quantification and can simplify the ability to perform parametric analysis in clinical settings without the need for blood sampling.

Biological Breakdown of Sports in Athletics Based on Multimedia Image Acquisition Techniques.

This paper provides an in-depth analysis and study of the decomposition of sports biology of athletics events through multimedia image acquisition techniques. The proposed design uses SOPC hardware-software collaboration technology, which makes full use of the parallelism of hardware as well as the flexibility of software to essentially improve the speed of image processing and greatly improve the efficiency of image processing. The high speed and parallelism of hardware are used to realize the image acquisition part with high timing requirements and the preprocessing algorithm with a large number of operations and strong repetition, in addition to the difficulties of many initialization configuration parameters of each peripheral. Additionally, the need for frequent adjustment of peripheral working parameters is solved by making full use of the scalability and flexibility of software, and the control signals output by software can coordinate the work of each hardware module to ensure that each module of the system cooperates. It is also possible to coordinate the work of each hardware module through the control signal output from the software to ensure that each module of the system cooperates and operates in an orderly and efficient manner and provides the possibility of realizing a higher level of the image processing algorithm. The system includes four main factors: swinging action, supporting action, vacating action, and speed rhythm. Based on this system, 14 key kinematic indicators were selected to reflect the technical status of the youth walkers. The landing angle decreased with the increase of speed, and the landing technique was insufficient; it had better control of the large and small arm angle, and their stride width was open, while the other four athletes showed increased tension in the hip joint and upper limb with the increase of walking speed, as well as the wrong action of forward and internal rotation of the large and small arms.

PUGH Decision Trapezoidal Fuzzy and Gradient Reinforce Deep Learning for Large Scale Requirement Prioritization

Objective: To prioritize requirements for large scale software projects within time involving uncertainty in the opinions among different stakeholders. Methods: We propose Pugh Trapezoidal Fuzzy and Gradient Reinforce Learning (PTF-GRL) methods for large scale software requirement prioritization. A Pugh Decision-based Trapezoidal Fuzzy Requirement Selection model is designed, inputting the functional and non-functional requirements of the corresponding stakeholders. With the assistance of Trapezoidal Fuzzy Inference, the qualitative factors are mapped with the corresponding numeric factors, which increases the computational efficiency. Findings: Performance is analyzed based on four parameters: The first parameter is accuracy and our method showed improvement of 4%, 7% and 3% compared to JRD-SCRUM, IFS and SRPTackle respectively. The second parameter is prioritization time and found that our method had reduced time of 30%, 37% and 39% compared with existing methods. The third parameter is precision and it was found that our method improves precision by 6%, 10% and 5% compared with the other two methods. The final parameter we consider is the test suite execution and our method showed improvement of 12%, 19% and 5% compared with the existing two methods. Novelty/Applications: The originality of this work indicates the better performance along with the optimal test suite execution even considering the uncertainty factor in the proposed method compared with existing similar methods. Keywords: Software Project; Pugh Decision Matrix; Trapezoidal Fuzzy Inference; Gradient Orientation; Reinforce Learning; Requirement Prioritization

METABOLIC: high-throughput profiling of microbial genomes for functional traits, metabolism, biogeochemistry, and community-scale functional networks

BackgroundAdvances in microbiome science are being driven in large part due to our ability to study and infer microbial ecology from genomes reconstructed from mixed microbial communities using metagenomics and single-cell genomics. Such omics-based techniques allow us to read genomic blueprints of microorganisms, decipher their functional capacities and activities, and reconstruct their roles in biogeochemical processes. Currently available tools for analyses of genomic data can annotate and depict metabolic functions to some extent; however, no standardized approaches are currently available for the comprehensive characterization of metabolic predictions, metabolite exchanges, microbial interactions, and microbial contributions to biogeochemical cycling.ResultsWe present METABOLIC (METabolic And BiogeOchemistry anaLyses In miCrobes), a scalable software to advance microbial ecology and biogeochemistry studies using genomes at the resolution of individual organisms and/or microbial communities. The genome-scale workflow includes annotation of microbial genomes, motif validation of biochemically validated conserved protein residues, metabolic pathway analyses, and calculation of contributions to individual biogeochemical transformations and cycles. The community-scale workflow supplements genome-scale analyses with determination of genome abundance in the microbiome, potential microbial metabolic handoffs and metabolite exchange, reconstruction of functional networks, and determination of microbial contributions to biogeochemical cycles. METABOLIC can take input genomes from isolates, metagenome-assembled genomes, or single-cell genomes. Results are presented in the form of tables for metabolism and a variety of visualizations including biogeochemical cycling potential, representation of sequential metabolic transformations, community-scale microbial functional networks using a newly defined metric “MW-score” (metabolic weight score), and metabolic Sankey diagrams. METABOLIC takes ~ 3 h with 40 CPU threads to process ~ 100 genomes and corresponding metagenomic reads within which the most compute-demanding part of hmmsearch takes ~ 45 min, while it takes ~ 5 h to complete hmmsearch for ~ 3600 genomes. Tests of accuracy, robustness, and consistency suggest METABOLIC provides better performance compared to other software and online servers. To highlight the utility and versatility of METABOLIC, we demonstrate its capabilities on diverse metagenomic datasets from the marine subsurface, terrestrial subsurface, meadow soil, deep sea, freshwater lakes, wastewater, and the human gut.ConclusionMETABOLIC enables the consistent and reproducible study of microbial community ecology and biogeochemistry using a foundation of genome-informed microbial metabolism, and will advance the integration of uncultivated organisms into metabolic and biogeochemical models. METABOLIC is written in Perl and R and is freely available under GPLv3 at https://github.com/AnantharamanLab/METABOLIC.8JzsRQQL6mihmS_qxcrFZsVideo abstract

Accelerating covering array generation by combinatorial join for industry scale software testing.

Combinatorial interaction testing, which is a technique to verify a system with numerous input parameters, employs a mathematical object called a covering array as a test input. This technique generates a limited number of test cases while guaranteeing a given combinatorial coverage. Although this area has been studied extensively, handling constraints among input parameters remains a major challenge, which may significantly increase the cost to generate covering arrays. In this work, we propose a mathematical operation, called “weaken-product based combinatorial join”, which constructs a new covering array from two existing covering arrays. The operation reuses existing covering arrays to save computational resource by increasing parallelism during generation without losing combinatorial coverage of the original arrays. Our proposed method significantly reduce the covering array generation time by 13–96% depending on use case scenarios.

Interdependency Aware Qubit and Brownboost Rank Requirement Learning for Large Scale Software Requirement Prioritization

Interdependency Aware Qubit and Brownboost Rank Requirement Learning for Large Scale Software Requirement Prioritization

A Fog and Blockchain Software Architecture for a Global Scale Vaccination Strategy

Nowadays, there are many fragmented records of patient’s health data in different locations like hospitals, clinics, and organizations all around the world. With the arrival of the COVID-19 pandemic, several governments and institutions struggled to have satisfactory, fast, and accurate decision-making in a wide, dispersed, and global environment. In the current literature, we found that the most common related challenges include delay (network latency), software scalability, health data privacy, and global patient identification. We propose to design, implement and evaluate a healthcare software architecture focused on a global vaccination strategy, considering healthcare privacy issues, latency mitigation, support of scalability, and the use of a global identification. We have designed and implemented a prototype of a healthcare software called Fog-Care, evaluating performance metrics like latency, throughput and send rate of a hypothetical scenario where a global integrated vaccination campaign is adopted in wide dispensed locations (Brazil, USA, and United Kingdom), with an approach based on blockchain, unique identity, and fog computing technologies. The evaluation results demonstrate that the minimum latency spends less than 1 second to run, and the average of this metric grows in a linear progression, showing that a decentralized infrastructure integrating blockchain, global unique identification, and fog computing are feasible to make a scalable solution for a global vaccination campaign within other hospitals, clinics, and research institutions around the world and its data-sharing issues of privacy, and identification.

Scope2Screen: Focus+Context Techniques for Pathology Tumor Assessment in Multivariate Image Data.

Inspection of tissues using a light microscope is the primary method of diagnosing many diseases, notably cancer. Highly multiplexed tissue imaging builds on this foundation, enabling the collection of up to 60 channels of molecular information plus cell and tissue morphology using antibody staining. This provides unique insight into disease biology and promises to help with the design of patient-specific therapies. However, a substantial gap remains with respect to visualizing the resulting multivariate image data and effectively supporting pathology workflows in digital environments on screen. We, therefore, developed Scope2Screen, a scalable software system for focus+context exploration and annotation of whole-slide, high-plex, tissue images. Our approach scales to analyzing 100GB images of 109 or more pixels per channel, containing millions of individual cells. A multidisciplinary team of visualization experts, microscopists, and pathologists identified key image exploration and annotation tasks involving finding, magnifying, quantifying, and organizing regions of interest (ROIs) in an intuitive and cohesive manner. Building on a scope-to-screen metaphor, we present interactive lensing techniques that operate at single-cell and tissue levels. Lenses are equipped with task-specific functionality and descriptive statistics, making it possible to analyze image features, cell types, and spatial arrangements (neighborhoods) across image channels and scales. A fast sliding-window search guides users to regions similar to those under the lens; these regions can be analyzed and considered either separately or as part of a larger image collection. A novel snapshot method enables linked lens configurations and image statistics to be saved, restored, and shared with these regions. We validate our designs with domain experts and apply Scope2Screen in two case studies involving lung and colorectal cancers to discover cancer-relevant image features.

IoT-based agriculture monitoring platform architecture

The research is aimed at solving one of the most urgent tasks of our time: the development of reliable and scalable software for monitoring the climate of agricultural fields. The goal is to create an architecture of simple and universal systems for collecting and storing data from various measuring devices located on agricultural land. The methodological basis for the monitoring system is the “Internet of Things” (IoT) approach. The agro-industrial complex of the Tomsk region (Russia) was chosen as a model object. About a hundred measuring devices have been installed in agricultural fields, which provide continuous delivery of data series in real time. Our developed platform includes both the server part of the application that processes and stores these weather data, and a convenient user interface for visualizing data to agronomists. The results of the research can be used in designing the architecture of similar IoT-based monitoring systems.

Microservices for Data Analytics in IoT Applications: Current Solutions, Open Challenges, and Future Research Directions

The synergy between the Internet of Things (IoT) and big data technologies has resulted in the great development of multiple smart applications in varied fields such as energy management, environmental monitoring, elderly healthcare, etc. Due to the increasing demand for smart applications, opting for a flexible and scalable software architecture that supports and accelerates the development of these applications is in dire need nowadays. As an effective solution to continuously maintain, upgrade, and scale IoT-based applications, the microservices paradigm has been adopted as an architectural style allowing to provide several enhancements in terms of independent deployment, modularity, containerization, loose coupling, etc. These advantages provided by microservices impact also the efficiency of the analytics conducted by the IoT-based systems. In this view, the current research aims to address a survey about the adoption of the microservices paradigm for supporting data analytics in IoT applications. For this purpose, first of all, the theoretical concepts related to IoT, data analytics, and microservices are briefly presented. Second, relevant microservices-based solutions for data analytics in IoT applications are reviewed and discussed. Third, the challenges and opportunities offered by the microservices paradigm in the IoT context are outlined and explained. The present study will pave the way for promising future research on the issues of integrating the emerging microservices technology in data analytics and IoT applications.

Origins and Consequences of Misunderstood/Slipped Requirements on Large Scale Software Projects

Origins and Consequences of Misunderstood/Slipped Requirements on Large Scale Software Projects

Robotic Automation of Software Testing From a Machine Learning Viewpoint

The need to scale software test automation while managing the test automation process within a reasonable time frame remains a crucial challenge for software development teams (DevOps). Unlike hardware, the software cannot wear out but can fail to satisfy the functional requirements it is supposed to meet due to the defects observed during system operation. In this era of big data, DevOps teams can deliver better and efficient code by utilizing machine learning (ML) to scan their new codes and identify test coverage gaps. While still in its infancy, the inclusion of ML in software testing is a reality and requirement for coming industry demands. This study introduces the prospects of robot testing and machine learning to manage the test automation process to guarantee software reliability and quality within a reasonable timeframe. Although this paper does not provide any particular demonstration of ML-based technique and numerical results from ML-based algorithms, it describes the motivation, possibilities, tools, components, and examples required for understanding and implementing the robot test automation process approach.

Design Analysis of Vertical Axis Wind Turbine Blade Using Biomimicry

Performance and efficiency of a Vertical Axis Wind Turbine improved by using a modified turbine blade, derived from biological features, by harness more wind energy. The detailed simulations were carried in Q-blade’s X-foil, Java foil and Sim Scale software at low wind speed with bio-mimicable cambered foil add-ons to different biostructure blades such as Maple seed leaf, Eagle wing. Among these simulations, the Corrugated Dragonfly vein FX 63-137 foil shows improved performance over cambered foils and FX 63-37 itself. The Maple-Wing combined blade structure showed an improved lift-drag ratio with a high coefficient of power.

Korali: Efficient and scalable software framework for Bayesian uncertainty quantification and stochastic optimization

We present Korali, an open-source framework for large-scale Bayesian uncertainty quantification and stochastic optimization. The framework relies on non-intrusive sampling of complex multiphysics models and enables their exploitation for optimization and decision-making. In addition, its distributed sampling engine makes efficient use of massively-parallel architectures while introducing novel fault tolerance and load balancing mechanisms. We demonstrate these features by interfacing Korali with existing high-performance software such as Aphros, LAMMPS (CPU-based), and Mirheo (GPU-based) and show efficient scaling for up to 512 nodes of the CSCS Piz Daint supercomputer. Finally, we present benchmarks demonstrating that Korali outperforms related state-of-the-art software frameworks.