Robustness Evaluation Research Articles

A CARDIAC MOTION MODEL TO EVALUATE INTRA-FRACTION DOSIMETRIC VARIATIONS IN RADIOTHERAPY TREATMENTS

Purpose: A cardiac cycle model was implemented to simulate cardiac motion during radiotherapy to evaluate the intra-fraction dosimetric impact on cardiac sub-structures comparing different planning techniques. Methods: Cardiac sub-structures were automatically contoured in 10 CTs acquired in deep inspiration breath hold (DIBH) by using a recently developed hierarchical-clustering atlas-based algorithm. A deformable image registration algorithm was used to simulate the cardiac motion cycle based on volume variations available in the literature. Two synthetic CTs were created and contoured simulating contraction and expansion during the cardiac cycle. Ninety radiotherapy plans were calculated using three radiotherapy paradigms: tangential fields planned as Linac-3D-CRT with a steep linear dose gradient toward the heart-modulated therapy with an intermediately steep concave gradient of intermediate-to-high doses toward the heart, planned as Linac-VMAT; modulated therapy with a steep concave gradient of intermediate-to-high doses toward the heart, planned as helical tomotherapy. Python scripts were developed for autocontouring, automatic creation of synthetic CTs and data extraction. Results: Comparison between paradigms shows that different constraints (maximal gradient toward heart/lung versus maximal sparing of contralateral breast/axilla) do not necessarily result in preferred or reduced heart sparing, but this depends more on individual anatomy. A planning paradigm with an intermediate-steepness dose gradient showed the best robustness against intra-fraction organ motion. Conclusions: Patient-specific organ motion models may reduce differences between planned and delivered RT and may thus help to refine dose–volume–toxicity models for cardiac sub-structures and, as a consequence, clinical constraints. Automatized plan recalculation on synthetic image sets might be used for robustness optimization and evaluation.

A domain-region based evaluation of ML performance robustness to covariate shift

Most machine learning methods assume that the input data distribution is the same in the training and testing phases. However, in practice, this stationarity is usually not met and the distribution of inputs differs, leading to unexpected performance of the learned model in deployment. The issue in which the training and test data inputs follow different probability distributions while the input–output relationship remains unchanged is referred to as covariate shift. In this paper, the performance of conventional machine learning models was experimentally evaluated in the presence of covariate shift. Furthermore, a region-based evaluation was performed by decomposing the domain of probability density function of the input data to assess the classifier’s performance per domain region. Distributional changes were simulated in a two-dimensional classification problem. Subsequently, a higher four-dimensional experiments were conducted. Based on the experimental analysis, the Random Forests algorithm is the most robust classifier in the two-dimensional case, showing the lowest degradation rate for accuracy and F1-score metrics, with a range between 0.1% and 2.08%. Moreover, the results reveal that in higher-dimensional experiments, the performance of the models is predominantly influenced by the complexity of the classification function, leading to degradation rates exceeding 25% in most cases. It is also concluded that the models exhibit high bias toward the region with high density in the input space domain of the training samples.

The role of metrology in the cyber-security of embedded devices

The cyber-security of an embedded device is a crucial issue especially in the Internet of Things (IoT) paradigm, since the physical accessibility to the smart transducers eases an attacker to eavesdrop the exchanged messages. In this manuscript, the role of metrology in improving the characterization and security testing of embedded devices is discussed in terms of vulnerability testing and robustness evaluation. The presented methods ensure an accurate assessment of the device’s security by relying on statistical analysis and design of experiments. A particular focus is given on power analysis by means of a scatter attack. In this context, the metrological approach contributes to guaranteeing the confidentiality and integrity of the data exchanged by IoT transducers.

Robustness evaluation of trust and reputation systems using a deep reinforcement learning approach

One of the biggest challenges facing trust and reputation systems (TRSs) is evaluating their ability to withstand various attacks, as these systems are vulnerable to multiple types of attacks. While simulation methods have been used to evaluate TRSs, they are limited because they cannot detect new attacks and do not ensure the system’s overall robustness. To address this limitation, verification methods have been proposed that can examine the entire state space and detect all possible attacks. However, these methods are not always practical for large models and real environments because they suffer from the state space explosion problem. To tackle this issue, we propose a deep reinforcement learning approach for evaluating the robustness of TRSs. In this approach, an agent can learn how to attack a system and find the best attack plan without prior knowledge. Additionally, our method avoids the state space explosion problem because it uses a deep Q-network instead of storing and examining the entire state space. We tested our proposed method on five well-known reputation models, assessing various attack goals such as selfishness, maliciousness, competition, and slandering. The results showed that our method was successful in identifying the best attack plan and executing it successfully in the system, demonstrating its effectiveness in evaluating the robustness of TRSs.

PO-1851 Radiobiological optimization and robustness evaluation for radiotherapy combined with hyperthermia

PO-1851 Radiobiological optimization and robustness evaluation for radiotherapy combined with hyperthermia

Ecological network identification and connectivity robustness evaluation in the Yellow River Basin under a multi-scenario simulation

The deterioration of the environment has been aggravated by the rapid growth of cities and economies, and inter-regional political, economic, scientific, and technological frictions caused by ecological problems are increasing. To safeguard the ecological environment and design healthy urban development, it is crucial to assess the structure and quality of ecological networks (ENs), which serve as links between landscape structure and ecological processes. This paper simulates land use in 2035 under three development scenarios for the Yellow River Basin (YRB), identifying ENs, and assessing the robustness of network connectivity for each scenario. We found that in the ecological protection scenario (ES), forest and water Land expanded the most, increasing by 4.16% and 6.98%, respectively. The ES also produced the most ecological patches (53) and important ecological corridors (141), with the important corridors forming an east–west spatial pattern. In the baseline scenario (BS), important corridors formed a north–south pattern, and in the urban development scenario (US), the important corridors were characterised by small-scale aggregation. The initial connectivity robustness of ENs was 0.5, 0.46, and 0.45 in the ES, BS, and US, respectively. Whether using deliberate or random attack models, the robustness of network connectivity was strongest in the ES. This study provides a theoretical reference for the future planning of eco-friendly development and ecological security patterns in the YRB.

A deep learning-based approach for statistical robustness evaluation in proton therapy treatment planning: a feasibility study

Objective. Robustness evaluation is critical in particle radiotherapy due to its susceptibility to uncertainties. However, the customary method for robustness evaluation only considers a few uncertainty scenarios, which are insufficient to provide a consistent statistical interpretation. We propose an artificial intelligence–based approach that overcomes this limitation by predicting a set of percentile dose values at every voxel and allows for the evaluation of planning objectives at specific confidence levels. Approach. We built and trained a deep learning (DL) model to predict the 5th and 95th percentile dose distributions, which corresponds to the lower and upper bounds of a two-tailed 90% confidence interval (CI), respectively. Predictions were made directly from the nominal dose distribution and planning computed tomography scan. The data used to train and test the model consisted of proton plans from 543 prostate cancer patients. The ground truth percentile values were estimated for each patient using 600 dose recalculations representing randomly sampled uncertainty scenarios. For comparison, we also tested whether a common worst-case scenario (WCS) robustness evaluation (voxel-wise minimum and maximum) corresponding to a 90% CI could reproduce the ground truth 5th and 95th percentile doses. Main results. The percentile dose distributions predicted by DL yielded excellent agreements with the ground truth dose distributions, with mean dose errors below 0.15 Gy and average gamma passing rates (GPR) at 1 mm/1% above 93.9, which were substantially better than the WCS dose distributions (mean dose error above 2.2 Gy and GPR at 1 mm1% below 54). We observed similar outcomes in a dose-volume histogram error analysis, where the DL predictions generally yielded smaller mean errors and standard deviations than the WCS evaluation doses. Significance. The proposed method produces accurate and fast predictions (∼2.5 s for one percentile dose distribution) for a given confidence level. Thus, the method has the potential to improve robustness evaluation.

ECAsT: a large dataset for conversational search and an evaluation of metric robustness.

The Text REtrieval Conference Conversational assistance track (CAsT) is an annual conversational passage retrieval challenge to create a large-scale open-domain conversational search benchmarking. However, as of yet, the datasets used are small, with just more than 1,000 turns and 100 conversation topics. In the first part of this research, we address the dataset limitation by building a much larger novel multi-turn conversation dataset for conversation search benchmarking called Expanded-CAsT (ECAsT). ECAsT is built using a multi-stage solution that uses a combination of conversational query reformulation and neural paraphrasing and also includes a new model to create multi-turn paraphrases. The meaning and diversity of paraphrases are evaluated with human and automatic evaluation. Using this methodology, we produce and release to the research community a conversational search dataset that is 665% more extensive in terms of size and language diversity than is available at the time of this study, with more than 9,200 turns. The augmented dataset not only provides more data but also more language diversity to improve conversational search neural model training and testing. In the second part of the research, we use ECAsT to assess the robustness of traditional metrics for conversational evaluation used in CAsT and identify its bias toward language diversity. Results show the benefits of adding language diversity for improving the collection of pooled passages and reducing evaluation bias. We found that introducing language diversity via paraphrases returned up to 24% new passages compared to only 2% using CAsT baseline.

A probabilistic deep learning model of inter-fraction anatomical variations in radiotherapy

Objective. In radiotherapy, the internal movement of organs between treatment sessions causes errors in the final radiation dose delivery. To assess the need for adaptation, motion models can be used to simulate dominant motion patterns and assess anatomical robustness before delivery. Traditionally, such models are based on principal component analysis (PCA) and are either patient-specific (requiring several scans per patient) or population-based, applying the same set of deformations to all patients. We present a hybrid approach which, based on population data, allows to predict patient-specific inter-fraction variations for an individual patient. Approach. We propose a deep learning probabilistic framework that generates deformation vector fields warping a patient's planning computed tomography (CT) into possible patient-specific anatomies. This daily anatomy model (DAM) uses few random variables capturing groups of correlated movements. Given a new planning CT, DAM estimates the joint distribution over the variables, with each sample from the distribution corresponding to a different deformation. We train our model using dataset of 312 CT pairs with prostate, bladder, and rectum delineations from 38 prostate cancer patients. For 2 additional patients (22 CTs), we compute the contour overlap between real and generated images, and compare the sampled and ‘ground truth’ distributions of volume and center of mass changes. Results. With a DICE score of 0.86 ± 0.05 and a distance between prostate contours of 1.09 ± 0.93 mm, DAM matches and improves upon previously published PCA-based models, using as few as 8 latent variables. The overlap between distributions further indicates that DAM’s sampled movements match the range and frequency of clinically observed daily changes on repeat CTs. Significance. Conditioned only on planning CT values and organ contours of a new patient without any pre-processing, DAM can accurately deformations seen during following treatment sessions, enabling anatomically robust treatment planning and robustness evaluation against inter-fraction anatomical changes.

State of Charge Estimation of Lithium-ion Battery in Electric Vehicles Using the Smooth Variable Structure Filter: Robustness Evaluation against Noise and Parameters Uncertainties

State of Charge (SoC) of Lithium-ion battery is a key parameter in battery management systems for electric vehicles. This paper uses the fundamental theory of the smooth variable structure filter (SVSF) and proposes a SoC estimation algorithm for a Manganese Cobalt (NMC) cell with a nominal capacity of 20 Ah. Several tests are conducted considering different types of noise and parameters variation. A nonrandom Gaussian noise is first added to the battery voltage. The maximum root mean square error (RMSE) of the estimated SoC is about for a standard deviation of the noise set to P.U. The same noise is applied to the battery current and the maximum RMSE of the SoC is obtained as 1.36%. Moreover, an EMI noise is added to the battery voltage and the obtained RMSE of the SoC is about 1.73% for a peak amplitude of the noise set to 0.07 P.U. The convergence of the algorithm is also confirmed under battery parameters variation due to the temperature change. However, its accuracy degrades considerably. Finally, a comparative study is carried out with the extended Kalman filter and shows the superiority of SVSF in terms of accuracy and robustness against measurement noise.

Prediction of Post-Operative Clinical Parameters in Posterior Scoliosis Surgery Through An Adaptive Neuro-Fuzzy Interface System

Background and Objective: Postoperative clinical indices should be estimated accurately in scoliosis correction surgeries, which have been analyzed in various studies such as experimental (in vitro or in vivo) trials through different modeling methods (finite element or multibody analysis). These costly and time-consuming methods can only be conducted on a large number of scoliotic patients. An adaptive neuro-fuzzy interface system (ANFIS) is used in this study to estimate the postoperative cobb and thoracic kyphosis angles in adolescent idiopathic scoliosis patients undergoing posterior scoliosis correction surgeries. Methods: Four groups of 55 patients with distinct preoperative clinical indices (thoracic cobb and pelvic incidence) were considered the ANFIS inputs, whereas postoperative thoracic cobb and kyphosis angles were used as the outputs. For robustness evaluation, the predicted values of postoperative angles were compared with measurements by calculating the root mean square errors and clinical correction deviation indices (the relative deviation of postoperative predicted angles from the real angles). Results: The least root mean square errors (3.0º and 6.3° for the main thoracic cobb and thoracic kyphosis estimations, respectively) were recorded in the group with the main thoracic cobb, pelvic incidence, thoracic kyphosis, and T1 spinopelvic inclination used as inputs. The clinical correction deviation indices were calculated 0.0086 and 0.0641 for cobb angles in two cases and 0.0534 and 0.2879 for thoracic kyphosis in two other cases. Conclusion: Greater differences between preoperative and postoperative cobb angles compared with those of thoracic kyphosis decreased the root-mean-square errors and clinical deviation indices but improved accuracy

Optimizing Membership Function Tuning for Fuzzy Control of Robotic Manipulators Using PID-Driven Data Techniques

In this study, a method for optimizing membership function tuning for fuzzy control of robotic manipulators using PID-driven data techniques is presented. Traditional approaches for designing membership functions in fuzzy control systems often rely on the experience and knowledge of the system designer, which can lead to suboptimal performance. By utilizing data collected from a PID control system, the proposed method aims to enhance the precision and controllability of robotic manipulators through improved fuzzy logic control. A Mamdani-type fuzzy logic controller was developed and its performance was simulated in Simulink, demonstrating the effectiveness of the proposed optimization technique. The results indicate that the method can outperform conventional P control systems in terms of overshoot reduction while maintaining comparable transient response specifications. This research highlights the potential of the PID-driven data-based approach for optimizing membership function tuning in fuzzy control systems and offers valuable insights for the development and evaluation of fuzzy logic control in robotic manipulators. Future work may focus on further optimization of the tuning process, evaluation of system robustness under various operating conditions, and exploring the integration of other artificial intelligence techniques for improved control performance.

The impact of board-level female directors on firm performance: evidence from India

PurposeThis study investigates the impact of female directors on firms' financial performance by scrutinizing the different roles they are empowered to fulfill.Design/methodology/approachThis study examines the impact of the roles performed by female directors on firms' financial performance using a panel dataset of the top 100 listed Indian firms over a period of 5 years. The study uses an appropriate panel data model for empirical analysis. For the robustness evaluation, a two-stage least square (2SLS) with the instrumental variable model were used.FindingsThe findings reveal a significantly positive impact of the total percentage of female directors on firms' financial performance. Further, by disentangling the impact of the total percentage of female directors between independent directors and executive directors, the study shows that independent female directors make a significant positive contribution to their firms' financial performance. By contrast, the performance impact of female executive directors was insignificant. In addition, the findings reveal that firms with a higher proportion of independent female directors outperform firms with a higher percentage of female executive directors.Originality/valueThis study is the first of its kind to unravel the performance impact of female directors and distinguish between the roles of independent directors and executive directors in the context of the emerging market of India, after the imposition of a gender quota for corporate boards.

Exploring misclassifications of robust neural networks to enhance adversarial attacks

Progress in making neural networks more robust against adversarial attacks is mostly marginal, despite the great efforts of the research community. Moreover, the robustness evaluation is often imprecise, making it challenging to identify promising approaches. We do an observational study on the classification decisions of 19 different state-of-the-art neural networks trained to be robust against adversarial attacks. This analysis gives a new indication of the limits of the robustness of current models on a common benchmark. In addition, our findings suggest that current untargeted adversarial attacks induce misclassification toward only a limited amount of different classes. Similarly, we find that previous attacks under-explore the perturbation space during optimization. This leads to unsuccessful attacks for samples where the initial gradient direction is not a good approximation of the final adversarial perturbation direction. Additionally, we observe that both over- and under-confidence in model predictions result in an inaccurate assessment of model robustness. Based on these observations, we propose a novel loss function for adversarial attacks that consistently improves their efficiency and success rate compared to prior attacks for all 30 analyzed models.

Evaluation of Robustness and Effectiveness of the BV-STOP, A Novel Antiseptic Compound with Specific and Anti-Microbial Activity

Evaluation of Robustness and Effectiveness of the BV-STOP, A Novel Antiseptic Compound with Specific and Anti-Microbial Activity

Analytical Purity Determinations of Universal Food-Spice Curcuma longa through a QbD Validated HPLC Approach with Critical Parametric Predictors and Operable-Design’s Monte Carlo Simulations: Analysis of Extracts, Forced-Degradants, and Capsules and Tablets-Based Pharmaceutical Dosage Forms

Applications of analytical quality by design (QbD) approach for developing HPLC (High Performance Liquid Chromatography) methods for food components assays, and separations of complex natural product mixtures, are still limited. The current study developed and validated, for the first time, a stability-indicating HPLC method for simultaneous determinations of curcuminoids in Curcuma longa extracts, tablets, capsules, and curcuminoids' forced degradants under different experimental conditions. Towards separation strategy, critical method parameters (CMPs) were defined as the mobile phase solvents' percent-ratio, the pH of the mobile phase, and the stationary-phase column temperature, while the peaks resolution, retention time, and the number of theoretical plates were recognized as the critical method attributes (CMAs). Factorial experimental designs were used for method development, validation, and robustness evaluation of the procedure. The Monte Carlo simulation evaluated the developing method's operability, and that ensured the concurrent detections of curcuminoids in natural extracts, commercial-grade pharmaceutical dosage-forms, and the forced degradants of the curcuminoids in a single mixture. The optimum separations were accomplished using the mobile phase, consisting of an acetonitrile-phosphate buffer (54:46 v/v, 0.1 mM) with 1.0 mL/min flow rate, 33 °C column temperature, and 385 nm wavelength for UV (Ultra Violet) spectral detections. The method is specific, linear (R2 ≥ 0.999), precise (% RSD < 1.67%), and accurate (% recovery 98.76-99.89%), with LOD (Limit of Detection) and LOQ (Limit of Quantitation) at 0.024 and 0.075 µg/mL for the curcumin, 0.0105 µg/mL and 0.319 µg/mL for demethoxycurcumin, and 0.335 µg/mL and 1.015 µg/mL for the bisdemethoxycurcumin, respectively. The method is compatible, robust, precise, reproducible, and accurately quantifies the composition of the analyte mixture. It exemplifies the use of the QbD approach in acquiring design details for developing an improved analytical detection and quantification method.

Energy grid management system with anomaly detection and Q-learning decision modules

Stability and security issues in energy management have become widespread research topics, in which artificial intelligence techniques are often embedded in management systems to efficiently manage the smart grid. In this paper, we propose an energy grid management system with anomaly detection and Q-learning decision modules (EMSAD). The anomaly detection module is a multitask learning network that simultaneously classifies suppliers and predicts actual supply quantities. The Q-learning decision module then determines the operating reserve and subsidies to manage the energy grid. Experimental results show that the proposed anomaly detection module has an excellent performance in classifying malicious suppliers with F1-scores from 73.3% to 100.0%. The robustness evaluation also shows that EMSAD maintains high performance even in unseen environments without fine-tuning. Thus, the simulation results demonstrate the security, efficiency, transferability, and robustness of the proposed EMSAD in smart grid energy management.

Clinical 3D/4D cumulative proton dose assessment methods for thoracic tumours with large motion

PurposeDespite the anticipated clinical benefits of intensity-modulated proton therapy (IMPT), plan robustness may be compromised due to its sensitivity to patient treatment uncertainties, especially for tumours with large motion. In this study, we investigated treatment course-wise plan robustness for intra-thoracic tumours with large motion comparing a 4D pre-clinical evaluation method (4DREM) to our clinical 3D/4D dose reconstruction and accumulation methods. Materials and methodsTwenty patients with large target motion (>10 mm) were treated with five times layered rescanned IMPT. The 3D-robust optimised plans were generated on the averaged planning 4DCT. Using multiple 4DCTs, treatment plan robustness was assessed on a weekly and treatment course-wise basis through the 3D robustness evaluation method (3DREM, based on averaged 4DCTs), the 4D robustness evaluation method (4DREM, including the time structure of treatment delivery and 4DCT phases) and 4D dose reconstruction and accumulation (4DREAL, based on fraction-wise information). ResultsBaseline target motion for all patients ranged from 11–17 mm. For the offline adapted course-wise dose assessment, adequate target dose coverage was found for all patients. The target volume receiving 95% of the prescription dose was consistent between methods with 16/20 patients showing differences < 1%. 4DREAL showed the highest target coverage (99.8 ± 0.6%, p < 0.001), while no differences were observed between 3DREM and 4DREM (99.3 ± 1.3% and 99.4 ± 1.1%, respectively). ConclusionOur results show that intra-thoracic tumours can be adequately treated with IMPT in free breathing for target motion amplitudes up to 17 mm employing any of the accumulation methods. Anatomical changes, setup and range errors demonstrated a more severe impact on target coverage than motion in these patients treated with fractionated proton radiotherapy.

CoRE: Constrained Robustness Evaluation of Machine Learning-Based Stability Assessment for Power Systems

Dear Editor, Machine learning (ML) approaches have been widely employed to enable real-time ML-based stability assessment (MLSA) of large-scale automated electricity grids. However, the vulnerability of MLSA to malicious cyber-attacks may lead to wrong decisions in operating the physical grid if its resilience properties are not well understood before deployment. Unlike adversarial ML in prior domains such as image processing, specific constraints of power systems that the attacker must obey in constructing adversarial samples require new research on MLSA vulnerability analysis for power systems. In this letter, we propose a novel evaluation framework to analyze the robustness of MLSA against adversarial samples with key considerations for damage (i.e., the ability of the adversarial data to cause ML misclassification), bad data detection, physical consistency, and limited attacker's capacity to corrupt data. Extensive experiments are conducted to evaluate the robustness of MLSA under different settings.

ERMDS: A obfuscation dataset for evaluating robustness of learning-based malware detection system

Learning-based malware detection systems (LB-MDS) play a crucial role in defending computer systems from malicious attacks. Nevertheless, these systems can be vulnerable to various attacks, which can have significant consequences. Software obfuscation techniques can be used to modify the features of malware, thereby avoiding its classification as malicious by LB-MDS. However, existing portable executable (PE) malware datasets primarily use a single obfuscation technique, which LB-MDS has already learned, leading to a loss of their robustness evaluation ability. Therefore, creating a dataset with diverse features that were not observed during LB-MDS training has become the main challenge in evaluating the robustness of LB-MDS.We propose a obfuscation dataset ERMDS that solves the problem of evaluating the robustness of LB-MDS by generating malwares with diverse features. When designing this dataset, we created three types of obfuscation spaces, corresponding to binary obfuscation, source code obfuscation, and packing obfuscation. Each obfuscation space has multiple obfuscation techniques, each with different parameters. The obfuscation techniques in these three obfuscation spaces can be used in combination and can be reused. This enables us to theoretically obtain an infinite number of obfuscation combinations, thereby creating malwares with a diverse range of features that have not been captured by LB-MDS.To assess the effectiveness of the ERMDS obfuscation dataset, we create an instance of the obfuscation dataset called ERMDS-X. By utilizing this dataset, we conducted an evaluation of the robustness of two LB-MDS models, namely MalConv and EMBER, as well as six commercial antivirus software products, which are anonymized as AV1-AV6. The results of our experiments showed that ERMDS-X effectively reveals the limitations in the robustness of existing LB-MDS models, leading to an average accuracy reduction of 20% in LB-MDS and 32% in commercial antivirus software. We conducted a comprehensive analysis of the factors that contributed to the observed accuracy decline in both LB-MDS and commercial antivirus software. We have released the ERMDS-X dataset as an open-source resource, available on GitHub at https://github.com/lcjia94/ERMDS.