Chaos Engineering Research Articles

Enhancing DevSecOps practice with Large Language Models and Security Chaos Engineering

Recently, the DevSecOps practice has improved companies’ agile production of secure software, reducing problems and improving return on investment. However, overreliance on security tools and traditional security techniques can facilitate the implementation of vulnerabilities in different stages of the software lifecycle.. Thus, this paper proposes the integration of a Large Language Model to help automate threat discovery at the design stage and Security Chaos Engineering to support the identification of security flaws that may be undetected by security tools. A specific use case is described to demonstrate how our proposal can be applied to a retail company that has the business need to produce rapidly secure software.

Harnessing Chaos: The Role of Chaos Engineering in Cloud Applications and Impacts on Site Reliability Engineering

Harnessing Chaos: The Role of Chaos Engineering in Cloud Applications and Impacts on Site Reliability Engineering

Towards antifragility of cloud systems: An adaptive chaos driven framework

ContextUnlike resilience, antifragility describes systems that get stronger rather than weaker under stress and chaos. Antifragile systems have the capacity to overcome stressors and come out stronger, whereas resilient systems are focused on their capacity to return to their previous state following a failure. As technology environments become increasingly complex, there is a great need for developing software systems that can benefit from failures while continuously improving. Most applications nowadays operate in cloud environments. Thus, with this increasing adoption of Cloud-Native Systems they require antifragility due to their distributed nature. ObjectiveThe paper proposes UNFRAGILE framework, which facilitates the transformation of existing systems into antifragile systems. The framework employs chaos engineering to introduce failures incrementally and assess the system's response under such perturbation and improves the quality of system response by removing fragilities and introducing adaptive fault tolerance strategies. MethodThe UNFRAGILE framework's feasibility has been validated by applying it to a cloud-native using a real-world architecture to enhance its antifragility towards long outbound service latencies. The empirical investigation of fragility is undertaken, and the results show how chaos affects application performance metrics and causes disturbances in them. To deal with chaotic network latency, an adaptation phase is put into effect. ResultsThe findings indicate that the steady stage's behaviour is like the antifragile stage's behaviour. This suggests that the system could self-stabilise during the chaos without the need to define a static configuration after determining from the context of the environment that the dependent system was experiencing difficulties. ConclusionOverall, this paper contributes to ongoing efforts to develop antifragile software capable of adapting to the rapidly changing complex environment. Overall, the research provides an operational framework for engineering software systems that learn and improve through exposure to failures rather than just surviving them.

Exploring the impact of chaos engineering with various user loads on cloud native applications: an exploratory empirical study

One of the most popular models that provide computer resources today is cloud computing. Today’s dynamic and successful platforms are created to take advantage of various resources available from service providers. Ensuring the performance and availability of such resources and services is a crucial problem. Any software system may be subject to faults that might propagate to cause failures. Such faults with the potential of contributing to failures are critical because they impair performance and result in a delayed reaction, which is regarded as a dependability problem. To ensure that critical faults can be discovered as soon as possible, the impact of such faults on the system must be tested. The performance and dependability of cloud-native systems are examined in this empirical study using fault injection, one of the chaos engineering techniques. The study explores the impacts and results of injecting various delay times into two cloud-native applications with diverse user numbers. The performance of the applications with various numbers of users is measured in relation to these delays, which accordingly reflects measuring the dependability of those systems. Firstly, the systems’ architecture were identified, and serverless with two Lambda functions and containerised microservices applications were chosen, which depend on utilising and incorporating cloud-native services. Secondly, faults are injected in order to quantify performance attributes such as throughput and latency. The results of several controlled experiments carried out in real-world cloud environments provide exploratory empirical data, which promoted comparisons and statistical analysis that we utilised to identify the behaviour of the application while experiencing stress. Typical results from this investigation include an overall reduction in performance that is embodied in an increase in latency with injecting delays. However, a remarkable result is noticed at a particular delay in which defects and availability problems appear out of nowhere. These findings assist in highlighting the value of using chaos engineering in general and fault injection in particular to assess the dependability of cloud-native applications and to find unpredicted failures that could arise quickly from defects that aren’t supposed to spread and result in dependability issues.

Chaos Engineering for Resilience Assessment of Digital Twins

Within the Industry 4.0 vision, Digital Twins (DTs) have gained great attention as a promising approach to improve remote monitoring and control by means of virtual representations of physical objects. However, while DTs are becoming more and more sophisticated and even adopted for mission-critical applications, their resilience assessment has not received the required consideration yet. This paper originally proposes Chaos Engineering to assess and improve the resilience of DTs by testing multiple aspects of industrial environments in a coordinated, automated, and replicable manner. First, the paper discusses why and how Chaos Engineering is promising to improve the resilience of DTs. Then, it identifies and introduces a set of Chaos Engineering profiles specifically designed to take into account the many aspects an industrial environment is composed of. Finally, it shows the feasibility of assessing the resilience of a proof-of-concept DT through a testbed based on widely-adopted, open-source tools.

Enhancing the Security of Cloud using Fault Resolving Script (FRS)

Abstract: Cloud computing has made individual users lives and the work of businesses much easierby providing data storage services at very low costs. Individual users can store and access their data via a shared cloud storage service from any location at any time. A cloud attack is a cyber attack that targets cloud-based service platforms such as computing, storage, or hosted application ina Platform as a Service (PaaS) or Softwareas a Service (SaaS) model. A majority of breaches and cyber attacks in cloudinfrastructure are the result of human error and misconfiguration vulnerabilities customer-centric tools are crucial to mitigating these threats. Yet existing cloud security models are largely unable to address these issues. We propose Fault ResolvingScript (FRS) techniques to address the challenges. FRS applies the principle of chaos engineering to cloud security.

An Anomaly-based Detection System for Monitoring Kubernetes Infrastructures

Network monitoring is crucial to analyze infrastructure baselines and alert whenever an abnormal behavior is observed. However, human effort is limited in time and scope since many variables must be considered in real-time. In addition, infrastructures such as Kubernetes are complex by nature since they do not consider fixed equipment from which to gather data; instead, these infrastructures consider distributed, event-driven, and ephemeral containers that make it complicated to capture and track metrics. Artificial Intelligence models have demonstrated high detection rates for anomaly detection; therefore, there is a need to design and implement a global solution to collect complex data and orchestrate the whole Machine Learning Operations workflow. This document shares the findings and learnings from defining a cloud-native Artificial Intelligence infrastructure at Aligo to develop an anomaly-based detection system for monitoring on-premise Kubernetes infrastructures. After Chaos Engineering experiments, it is shown that the resulting deployed system is strong when alerting outliers and that an end-to-end infrastructure has been developed for conducting future Artificial Intelligence projects at the company.

Enhancing Operational Resilience of Critical Infrastructure Processes Through Chaos Engineering

Enhancing Operational Resilience of Critical Infrastructure Processes Through Chaos Engineering

On the Way to Automatic Exploitation of Vulnerabilities and Validation of Systems Security through Security Chaos Engineering

Software is behind the technological solutions that deliver many services to our society, which means that software security should not be considered a desirable feature anymore but more of a necessity. Protection of software is an endless labor that includes the improvement of security controls but also the understanding of the sources that induce incidents, which in many cases are due to bad implementation or assumptions of controls. As traditional methods may not be efficient in detecting those security assumptions, novel alternatives must be attempted. In this sense, Security Chaos Engineering (SCE) becomes an innovative methodology based on the definition of a steady state, a hypothesis, experiments, and metrics, which allow to identify failing components and ultimately protect assets under cyber risk scenarios. As an extension of a previous work, this paper presents ChaosXploit, an SCE-powered framework that employs a knowledge database, composed of attack trees, to expose vulnerabilities that exist in a software solution that has been previously defined as a target. The use of ChaosXploit may be part of a defensive security strategy to detect and correct software misconfigurations at an early stage. Finally, different experiments are described and executed to validate the feasibility of ChaosXploit in terms of auditing the security of cloud-managed services, i.e., Amazon buckets, which may be prone to misconfigurations and, consequently, targeted by potential cyberattacks.

Building Reliable Cloud Systems through Chaos Engineering

Cloud computing systems need to be reliable so that they can be accessed and used for computing at any given point in time. The complex nature of cloud systems is the motivation to conduct research in novel ways of ensuring that cloud systems are built with reliability in mind. In building cloud systems, it is expected that the cloud system will be able to deal with high demands and unexpected events that affect the reliability and performance of the system. In this paper, chaos engineering is considered a heuristic method that can be used to build reliable cloud systems. Chaos engineering is aimed at exposing weaknesses in systems that are in production. Chaos engineering will help identify system weaknesses and strengths when a system is exposed to unexpected knocks and shocks while it is in production. Chaos engineering allows system developers and administrators to get insights into how the cloud system will behave when it is exposed to unexpected occurrences.

Light chaotic dynamics in the transformation from curved to flat surfaces

Light propagation on a two-dimensional curved surface embedded in a three-dimensional space has attracted increasing attention as an analog model of four-dimensional curved spacetime in the laboratory. Despite recent developments in modern cosmology on the dynamics and evolution of the universe, investigation of nonlinear dynamics of light on non-Euclidean geometry is still scarce, with fundamental questions, such as the effect of curvature on deterministic chaos, challenging to address. Here, we study classical and wave chaotic dynamics on a family of surfaces of revolution by considering its equivalent conformally transformed flat billiard, with nonuniform distribution of the refractive index. We prove rigorously that these two systems share the same dynamics. By exploring the Poincaré surface of section, the Lyapunov exponent, and the statistics of eigenmodes and eigenfrequency spectrum in the transformed inhomogeneous table billiard, we find that the degree of chaos is fully controlled by a single, curvature-related geometric parameter of the curved surface. A simple interpretation of our findings in transformed billiards, the “fictitious force,” allows us to extend our prediction to other classes of curved surfaces. This powerful analogy between two a priori unrelated systems not only brings forward an approach to control the degree of chaos, but also provides potentialities for further studies and applications in various fields, such as billiards design, optical fibers, or laser microcavities.

A Mirror Environment to Produce Artificial Intelligence Training Data

With the increasing maturity of artificial intelligence (AI) technology, business automation technology has also become a trend. Particularly, network operation and maintenance (O<monospace>&#x0026;</monospace>M) is expected to soon become automated and more efficient. However, the automation of O<monospace>&#x0026;</monospace>M is hindered by the lack of network failure data and the cost of collecting data. We thus propose an approach to build a low-cost environment that can produce the same data as the actual production environment and use tools such as chaos engineering to generate training models for fault data. This paper attempts to build the underlying physical network layer using a low-cost single-board computer Raspberry Pi instead of an expensive PC server, while keeping the virtual network layer the same and performing fault simulation, data collection, and AI model training on the constructed virtual network layer. A comparison of the accuracy of the trained AI models verifies the feasibility of replacing the traditional PC server with an inexpensive Raspberry Pi device while keeping the structure and services of the virtual network layer unchanged. Also, a brief comparison with existing techniques is discussed. Our proposed approach solves the problem of insufficient data for AI training while reducing cost and risk.

Observability and chaos engineering on system calls for containerized applications in Docker

In this paper, we present a novel fault injection system called ChaosOrca for system calls in containerized applications. ChaosOrca aims at evaluating a given application’s self-protection capability with respect to system call errors. The unique feature of ChaosOrca is that it conducts experiments under production-like workload without instrumenting the application. We exhaustively analyze all kinds of system calls and utilize different levels of monitoring techniques to reason about the behavior under perturbation. We evaluate ChaosOrca on three real-world applications: a file transfer client, a reverse proxy server and a micro-service oriented web application. Our results show that it is promising to detect weaknesses of resilience mechanisms related to system calls issues.

Maximizing Error Injection Realism for Chaos Engineering With System Calls

In this paper, we present a novel fault injection framework for system call invocation errors, called Phoebe. Phoebe is unique as follows. First, Phoebe enables developers to have full observability of system call invocations. Second, Phoebe generates error models that are realistic in the sense that they mimic errors that naturally happen in production. Third, Phoebe is able to automatically conduct experiments to systematically assess the reliability of applications with respect to system call invocation errors in production. We evaluate the effectiveness and runtime overhead of Phoebe on two real-world applications in a production environment. The results show that Phoebe successfully generates realistic error models and is able to detect important reliability weaknesses with respect to system call invocation errors. To our knowledge, this novel concept of "realistic error injection", which consists of grounding fault injection on production errors, has never been studied before.

Metrological support integration in conditions of digital transformation

The article formulates the problems of the development of measurement control regulators in digital platforms. Attention is paid to the role of the digital revolution and metrology on the economy of enterprises and the transformation of the service sector model as a whole. Attention is paid to the impact of the digital revolution and metrology on the economy. The article presents and substantiates the policy of introduction of metrological support for innovative development in the context of the digital cybernetic revolution. The proposed policy pays attention to the consideration of the fundamental model of cyber-physical systems, as a technological basis for the implementation of a new industrial evolution in metrological context. The importance of organizing of measurement principles in network infrastructure and communication protocols in the context of economic transformation is emphasized. A model of the use and implementation of chaos engineering and information technologies in metrological apparatus is considered.

Continuous auditing and threat detection in multi-cloud infrastructure

Efficient change control and configuration management is imperative for addressing the emerging security threats in cloud infrastructure. These threats majorly exploit misconfiguration vulnerabilities e.g. excessive permissions, disabled logging features and publicly accessible cloud storage buckets. Traditional security tools and mechanisms are unable to effectively and continuously track changes in cloud infrastructure owing to transience and unpredictability of cloud events. Therefore, novel tools that are proactive, agile and continuous are imperative. This article proposes CSBAuditor, a novel cloud security system that continuously monitors cloud infrastructure, to detect malicious activities and unauthorized changes. CSBAuditor leverages two concepts: state transition analysis and reconciler pattern to overcome the aforementioned security issues. Furthermore, security metrics are used to compute severity scores for detected vulnerabilities using a novel scoring system: Cloud Security Scoring System. CSBAuditor has been evaluated using various strategies including security chaos engineering (fault injection) strategies on Amazon Web Services and Google Cloud Platform. CSBAuditor effectively detects misconfigurations in real-time with a detection rate of over 98%. Also, the performance overhead is within acceptable limits.

Chaos Engineering (Principles of Chaos Engineering) As the Pathway to Excellence and Relevance in Engineering Education in Africa

A study on engineering in sub-Saharan Africa revealed that engineering is pivotal for economic and social development of any country. This is profound as it underscores the potentials embedded in engineering education for excellence and relevance in Africa. This has not been the case in Africa, as the region has not developed evenly with other countries from the Global South. Hence, the impetus for chaos engineering as a panacea to excellence and relevance in engineering education in Africa. Chaos engineering has been defined by various authors and one of the profound definitions is that chaos engineering is the discipline of experimenting on a distributed system with the intent to build confidence in the system`s capability to withstand turbulent conditions during production. This study therefore looked at chaos engineering, its history and applicability and conceptualize it as a pathway for excellence and relevance in engineering education in Africa. Findings from the that engineering is pivotal for economic and social development of any country but it has not resulted to such in Africa which necessitates chaos principles. It was found out that experimentation is a basic principle of chaos engineering while the advanced principles are hypothesizing about steady state, vary real-world events, run experiments in production, automate experiments to run continuously, minimize blast radius. These all were conceptualized as the pathway to excellence and relevance in engineering education in Africa. The study recommended that there is a need to intensify effort on researching more into chaos engineering in Africa.

Chaos Engineering for Building Resilient Distributed Systems

Chaos Engineering for Building Resilient Distributed Systems

CloudStrike: Chaos Engineering for Security and Resiliency in Cloud Infrastructure

Most cyber-attacks and data breaches in cloud infrastructure are due to human errors and misconfiguration vulnerabilities. Cloud customer-centric tools are imperative for mitigating these issues, however existing cloud security models are largely unable to tackle these security challenges. Therefore, novel security mechanisms are imperative, we propose Risk-driven Fault Injection (RDFI) techniques to address these challenges. RDFI applies the principles of chaos engineering to cloud security and leverages feedback loops to execute, monitor, analyze and plan security fault injection campaigns, based on a knowledge-base. The knowledge-base consists of fault models designed from secure baselines, cloud security best practices and observations derived during iterative fault injection campaigns. These observations are helpful for identifying vulnerabilities while verifying the correctness of security attributes (integrity, confidentiality and availability). Furthermore, RDFI proactively supports risk analysis and security hardening efforts by sharing security information with security mechanisms. We have designed and implemented the RDFI strategies including various chaos engineering algorithms as a software tool: CloudStrike. Several evaluations have been conducted with CloudStrike against infrastructure deployed on two major public cloud infrastructure: Amazon Web Services and Google Cloud Platform. The time performance linearly increases, proportional to increasing attack rates. Also, the analysis of vulnerabilities detected via security fault injection has been used to harden the security of cloud resources to demonstrate the effectiveness of the security information provided by CloudStrike. Therefore, we opine that our approaches are suitable for overcoming contemporary cloud security issues.

A Framework of Virtual War Room and Matrix Sketch-Based Streaming Anomaly Detection for Microservice Systems

Recently, microservice has been a popular architecture to construct cloud-native systems. This novel architecture brings agility and accelerates the software development process significantly. However, it is not easy to manage and operate microservice systems due to their scale and complexity. Many approaches are proposed to automatically operate microservice systems such as anomaly detection. Nevertheless, those methods cannot be sufficiently validated and compared due to a lack of real microservice systems, which leads to the slow process of intelligent operation. These challenges inspire us to build a system named “VWR”, a framework of Virtual War Room for operating microservice applications which allows users to simulate their microservice architectures with low overhead and inject multiple types of faults into the microservice system with chaos engineering. VWR can mimic user requests and record the end-to-end tracing data (i.e., service call chains) for each request in a way consistent with OpenTracing. With easily designed tests and the produced streaming tracing data, the users can validate the performance of their intelligent operation algorithms and improve the algorithms as needed. Besides, based on the streaming tracing data generated by VWR, we introduce a novel unsupervised anomaly detection algorithm based on Matrix Sketch and set it as a default intelligent operation algorithm in VWR. This algorithm can detect anomalies by analyzing high-dimensional performance data collected from a microservice system in a streaming manner. The experimental result in VWR shows that the matrix sketch based method can precisely detect anomalies in microservice systems and outperform some widely used anomaly detection methods such as isolation forest in some scenario. We believe more approaches on the intelligent operation of microservice systems can be constructed based on VWR.