Test Scenarios Research Articles

The evaluation study for social cognition measures in Japan: Psychometric properties, relationships with social function, and recommendations

AimPatients with schizophrenia can have significant subjective difficulties in social cognition, but few undergo testing or treatment for social cognition. The Social Cognition Psychometric Evaluation (SCOPE) study recommends six social cognitive measures; however, the reliability and validity of these measures in different cultural and linguistic areas has not been adequately examined. We examined the psychometric properties of nine social cognitive measures and the relationship to social function, with the aim of determining the level of recommendation for social cognitive measures in clinical practice in Japan. MethodsFor our test battery, an expert panel previously selected nine measures: the Bell Lysaker Emotion Recognition Task (BLERT); Facial Emotion Selection Test (FEST); Hinting Task (Hinting); Metaphor and Sarcasm Scenario Test (MSST); Intentionality Bias Task (IBT); Ambiguous Intentions and Hostility Questionnaire (AIHQ); Social Attribution Task-Multiple Choice (SAT-MC); SAT-MCII; and Biological Motion (BM) task. In total, 121 outpatients with schizophrenia and 70 healthy controls were included in the analysis, and the results were provided to an expert panel to determine the recommendations for each measure. The quantitative psychological indices of each measure were evaluated for practicality, tolerability, test-retest reliability, correlation with social function, and the incremental validity of social function. ResultsHinting and FEST received the highest recommendations for use in screening, severity assessment, and longitudinal assessment, followed by BLERT, MSST AIHQ, SAT-MC, and SAT-MCII, with IBT and BM receiving the lowest recommendations. ConclusionThis study provides a uniform assessment tool that can be used in future international clinical trials for social cognitive impairment.

Improving Performance Convolutional Neural Networks Using Modified Pooling Function

The Visual Geometry Group-16 (VGG16) network architecture, as part of the development of convolutional neural networks, has been popular among researchers in solving classification tasks, so in this paper, we investigated the number of layers to find better performance. In addition, we also proposed two pooling function techniques inspired by existing research on mixed pooling functions, namely Qmax and Qavg. The purpose of the research was to see the advantages of our method; we conducted several test scenarios, including comparing several modified network configurations based on VGG16 as a baseline and involving our pooling technique and existing pooling functions. Then, the results of the first scenario, we selected a network that can adapt well to our pooling technique, whichwas then carried out several tests involving the Cifar10, Cifar100, TinyImageNet, and Street View House Numbers (SVHN) datasets as benchmarks. In addition, we were also involved in several existing methods. The experiment results showed that Net-E has the highest performance, with 93.90% accuracy for Cifar10, 71.17% for Cifar100, and 52.84% for TinyImageNet. Still, the accuracy was low when the SVHN dataset was used. In addition, in comparison tests with several optimization algorithms using the Qavg pooling function, it can be seen that the best accuracy results lie in the SGD optimization algorithm, with 89.76% for Cifar10 and 89.06% for Cifar100.

Incorporating Land Use and Transport Interaction Models to Evaluate Active Mobility Measures and Interventions in Urban Areas: A case study in Southampton, UK

New transport strategies, which aspire to create sustainable and resilient cities, prioritize active and public transport modes. This paper investigates the differences in accessibility, mobility and activity patterns as a result of radical road space allocation changes. To do so, the TRANUS Land Use and Transport Interaction model is utilized. Also, the city of Southampton is selected as a case study and four scenarios inspired by the city's ongoing or planned active travel initiatives are the “Business as Usual” (BAU) scenario, the “Southampton Cycling Network” (SCN) scenario, the “Traffic Calming” (TC) scenario, and the “Combined” scenario, which involves the simultaneous deployment of both test scenarios. In the SCN scenario, the proportion of bicycle trips has noticeably risen, whereas in the TC scenario, public transport usage is maintained. Meanwhile, the Combined scenario significantly diminishes the appeal of private motorized transport over the long term. Regarding the impacts on activities, urban sprawl and the creation of new urban centres are observed in all scenarios. Interestingly, the SCN and combined scenarios have similarities in land use effects, but this happens because accessibility increases in peripheral zones in the former, whereas it reduces in central zones in the latter.

Scalable evaluation methods for autonomous vehicles

Effective intelligent driving test and evaluation methods can improve the development and deployment process of autonomous vehicles (AVs). However, due to the extreme complexity and high dimensionality of driving behavior, how to objectively and efficiently evaluate the multi-dimensional performance of AVs in simulation and real-world environments is a long-standing problem. This paper proposes an objective multi-dimensional comprehensive evaluation (OMDCE) method that divides the intelligent driving test into four modules: test scenarios, scenario complexity model, simulation test platform, and automated evaluation system. The scenario complexity model is proposed to bridge the gap between the test scenarios and the automated evaluation system, thus enabling adaptive scaling of the evaluation scale for test scenarios with different difficulty levels. Besides, based on the existing four evaluation metrics for ego performance, altruism performance evaluation is first proposed to comprehensively portray AVs’ intelligence degree. The OMDCE method was evaluated by testing two different intelligent driving algorithms and conducting real vehicle testing. The experimental results demonstrated that the OMDCE method can effectively evaluate AVs in various scenarios and quantitatively measure their multi-dimensional performance. The proposed method reduces the subjectivity of manual evaluation and speeds up the evaluation. Moreover, it improves the universality and scalability of the evaluation metrics system.

Multi-criteria structural optimization of a hood inner structure using machine learning technique

This article presents an efficient multi-criteria structural optimization process based on machine learning for shape and size optimization of the hood inner. For this purpose, a low dimensional parameterization based on two-dimensional mapping and dimensionality reduction is used to decrease the number of design variables. There are many design parameters to be considered for designing this structure. In this study, the effect of three important criteria consisting of head injury, eigenfrequency, and static load were analyzed. A full vehicle finite element model is prepared for a newly developed product because head injury analysis requires a full vehicle model due to contact with the inner components of the engine compartment. Then, an experimental study is performed based on the real test scenario for model verification. A new rating method for analyzing the head injury criteria evaluated in different points is proposed and different machine learning techniques are compared in the approximation of the objective function. Finally, the problem is solved as both single and multi-objective optimization problems. Results present considerable improvement in accuracy and efficiency in the identified structure while the method has a low computational cost and the implementation is not difficult.

Adaptive attention fusion network for cross-device GUI element re-identification in crowdsourced testing

The rapid growth of mobile devices has ushered in an era of different device platforms. Different devices require a consistent user experience, especially with similar graphical user interfaces (GUIs). However, the different code bases of the various operating systems as well as the different GUI layouts and resolutions of the various devices pose a challenge for automated software testing. Crowdsourced software testing (CST) has emerged as a viable solution where crowdsourced workers perform tests on their own devices and provide detailed bug reports. Although CST is cost-effective, it is not very efficient and requires a large number of workers for manual testing. The potential of optimizing CST reproduction testing through computer vision remains largely untapped, especially when considering the uniformity of GUI elements on different devices. In this study, we present a novel deep learning model specifically designed to re-identify GUI elements in CST reproduction test scenarios, regardless of the underlying code changes on different devices. The model features a robust backbone network for feature extraction, an innovative attention mechanism with learnable factors to enhance the features of GUI elements and minimize interference from their backgrounds, and a classifier to determine matching labels for these elements. Our approach was validated on a large GUI element dataset containing 31,098 element images for training, 115,704 element images from real apps for testing, and 67 different background images. The results of our experiments underline the excellent accuracy of the model and the importance of each component. This work is a major step forward in improving the efficiency of reproduction testing in CST. The innovative solutions we propose could further reduce labor costs for CST platforms.

Enhanced cholera surveillance to improve vaccination campaign efficiency

Systematic testing for Vibrio cholerae O1 is rare, which means that the world’s limited supply of oral cholera vaccines (OCVs) may not be delivered to areas with the highest true cholera burden. Here we used a phenomenological model with subnational geographic targeting and fine-scale vaccine effects to model how expanding V. cholerae testing affected impact and cost-effectiveness for preventive vaccination campaigns across different bacteriological confirmation and vaccine targeting assumptions in 35 African countries. Systematic testing followed by OCV targeting based on confirmed cholera yielded higher efficiency and cost-effectiveness and slightly fewer averted cases than status quo scenarios targeting suspected cholera. Targeting vaccine to populations with an annual incidence rate greater than 10 per 10,000, the testing scenario averted 10.8 (95% prediction interval (PI) 9.4–12.6) cases per 1,000 fully vaccinated persons while the status quo scenario averted 6.9 (95% PI 6.0–7.8) cases per 1,000 fully vaccinated persons. In the testing scenario, testing costs increased by US$31 (95% PI 25–39) while vaccination costs reduced by US$248 (95% PI 176–326) per averted case compared to the status quo. Introduction of systematic testing into cholera surveillance could improve efficiency and reach of global OCV supply for preventive vaccination.

Behaviour-driven development and metrics framework for enhanced agile practices in scrum teams

ContextAgile methodologies highlight collaborative efforts among software engineering groups for iterative, high-quality product delivery within short timeframes. However, Scrum teams face persistent challenges in achieving these objectives, stemming from difficulties in seamless collaboration and effective communication among various roles, such as developers and testers. To address these issues, Scrum teams are increasingly adopting Behaviour-Driven Development (BDD), a testing technique fostering collaboration and shared understanding through test scenarios. ObjectivesThis research investigates the adoption of BDD practices in Scrum teams and the formulation of a metrics framework tailored for optimizing Scrum practices and product quality. MethodsEmploying action research, this study extends over two and half years, actively engaging Scrum team members and stakeholders to encompass their collaborative contributions, insights, and perspectives. It commences with defining a metrics framework through exploration within agile teams to measure and evaluate Scrum team performance. Subsequently, the focus shifts to implementing BDD practices systematically, employing training sessions, workshops, and iterative refinements. ResultsThe results of the study emphasize the substantial role of Behaviour-Driven Development (BDD) in improving collaboration, communication, and the comprehension of requirements within the Scrum team. Concurrently, the tailored metrics framework bolsters quality assurance practices, enhancing software quality and customer satisfaction. BDD adoption expedites automation and product delivery, while the metrics framework enables informed decision-making. ConclusionsCombining BDD practices with a custom metrics framework offers a holistic strategy for addressing Scrum challenges. Enhanced collaboration, communication, and requirements comprehension, resulting from BDD, synergize with the metrics framework to elevate Scrum teams' performance, software quality, and customer value. This research underlines the importance of adopting BDD as a testing methodology to achieve these improvements in Scrum teams.

Underwater glider 3D path planning with adaptive segments and optimal motion parameters based on improved JADE algorithm

This paper presents a novel 3D path planning method for the underwater glider (UG) that incorporates adaptive segment strategy, motion parameters optimization, and an improved JADE algorithm. The method aims to generate an energy-efficient path by adapting to ocean currents and seabed topography and selecting favorable motion parameters. We establish an energy consumption model for a blended-wing-body UG, examining the influence of motion parameters and ocean currents on its performance. The proposed method encodes an energy-optimal gliding path through multiple path segments, each defined by a set of path points, pitch angles, and diving depths. The fitness function, based on energy consumption, guides the optimization process. To enhance the optimization, we present an improved JADE algorithm with multi-mutation strategies, which adaptively updates the mutation operation and mean crossover probability. Our method was assessed and compared with a classical UG path planning method on 6 test scenarios. Simulation results confirm that adaptive segments and motion parameter optimization contribute to better adaptation to ocean environments and reduced energy consumption.

Homomorphic encrypted Yara rules evaluation

Malware signatures represent a powerful tool for malware detection and classification, widely used by security researchers and security solution providers. Yara rules describe malware based on string patterns that are evaluated on targeted files. Generally, the security provider sends signatures to the client endpoints and the rule evaluation is performed locally, such that the scanned files do not leave the client machines. However, if a zero-day vulnerability is discovered and the security provider exposes the corresponding signature, there is a considerable risk to also disclose the unpatched vulnerability. A solution is represented by homomorphic encrypted Yara rules, which can be evaluated on targeted files without being decrypted. In this article, we propose a homomorphic Yara rules evaluation method and do a comparative analysis with the HENFA method (Homomorphic Encryption for Finite Automata (Genise et al., 2019)). For our method, we propose a fully homomorphic exact string matching algorithm based on the TFHE scheme (Fully Homomorphic Encryption over the Torus (Chillotti et al., 2020)). In order to determine how suitable is the exact string matching approach in practice, we analyze a public Yara rules repository and generate various statistics about the patterns used in the Yara rules. For the two homomorphic Yara rules matching methods, both theoretical and experimental comparative evaluations are presented. Our proposed method implies ciphertexts with smaller sizes and has better efficiency in the average testing scenarios compared to the HENFA method.

AutoEval: Are Labels Always Necessary for Classifier Accuracy Evaluation?

Understanding model decision under novel test scenarios is central to the community. A common practice is evaluating models on labeled test sets. However, many real-world scenarios see unlabeled test data, rendering the common supervised evaluation protocols infeasible. In this paper, we investigate such an important but under-explored problem, named Automatic model Evaluation (AutoEval). Specifically, given a trained classifier, we aim to estimate its accuracy on various unlabeled test datasets. We construct a meta-dataset: a dataset comprised of datasets (sample sets) created from original images via various transformations such as rotation and background substitution. Correlation studies on the meta-dataset show that classifier accuracy exhibits a strong negative linear relationship with distribution shift (Pearson's Correlation ). This new finding inspires us to formulate AutoEval as a dataset-level regression problem. Specifically, we learn regression models (e.g., a regression neural network) to estimate classifier accuracy from overall feature statistics of a test set. In the experiment, we show that the meta-dataset contains sufficient and diverse sample sets, allowing us to train robust regression models and report reasonable and promising predictions of the classifier accuracy on various test sets. We also provide insights into application scopes, limitations, and potential future directions of AutoEval.

Non-photorealistic rendering as a feedback strategy in virtual reality for rehabilitation

Virtual reality (VR) rehabilitation has been proven to be a very promising method to increase the focus and attention of patients by immersing them in a virtual world, and through that, improve the effectiveness of the rehabilitation. One of the biggest challenges in designing VR Rehabilitation exercises is in choosing feedback strategies that guide the patient and give the appropriate success/failure indicators, without breaking their sense of immersion. A new strategy for feedback is proposed, using non-photorealistic rendering (NPR) to highlight important parts of the exercise the patient needs to focus on and fade out parts of the scene that are not relevant. This strategy is implemented into an authoring tool that allows rehabilitators specifying feedback strategies while creating exercise profiles. The NPR feedback can be configured in many ways, using different NPR schemes for different layers of the exercise environment such as the background environment, the non-interactive exercise objects, and the interactive exercise objects. The main features of the system including the support for universal render pipeline, camera stacking, and stereoscopic rendering are evaluated in a testing scenario. Performance tests regarding memory usage and supported frames per second are also considered. In addition, a group of rehabilitators evaluated the system usability. The proposed system meets all the requirements to apply NPR effect in VR scenarios and solves all the limitations with regard to technical function and image quality. In addition, the system performance has been shown to meet the targets for low-cost hardware. Regarding authoring tool usability rehabilitators agree that is easy to use and a valuable tool for rehabilitation scenarios. NPR schemes can be integrated into VR rehabilitation scenarios achieving the same image quality as non-VR visualizations with only a small impact on the frame rate. NPR schemes are a good visual feedback alternative.

Distributed Cooperative Control Strategy for Stable Voltage Restoration and Optimal Power Sharing in Islanded DC Microgrids

Conventionally, voltage regulation and economic dispatch in DC microgrid are performed at different time scales, which cannot keep the distributed generators (DGs) at the optimal operating point in real time. In this paper, a novel distributed cooperative control strategy is proposed to synchronously achieve stable bus voltage restoration and optimal power sharing in islanded DC microgrids. The proposed strategy breaks the constraints of hierarchical control strategies, and incorporates the objectives of voltage regulation and economic dispatch into a distributed cooperative control framework to eliminate average bus voltage deviation and minimize total power generation costs in real time. The stability of the proposed strategy is verified by eigenvalue analysis and steady-state analysis. Furthermore, the concept of “incremental cost (IC) margin” is introduced in cooperative control considering capacity constraints to avoid the converter output power violation and performance degradation. Finally, the MATLAB/Simulink simulation and StarSim HIL experimental results verify the effectiveness and robustness of the proposed strategy in an islanded DC microgrid under various testing scenarios.

An Improved Particle Swarm Optimization Algorithm for Automated Test Resource Allocation

With the rapid increase of software complexity, automated testing has become the mainstream of software testing. However, the current automated test tools adopt a large number of common test equipment resource. At the beginning of the test, for specific test scenarios, the testers need to plan the test equipment resources according to the requirements of the software under test. An automatic test equipment resource planning and optimization algorithm based on group optimization is proposed IPSO (Improved Particle Swarm Optimization) to solve this problem. This method maps the requirement description to the equipment description to obtain the matching matrix, and then establishes a multi-objective constrained optimization model for test equipment resource planning. The solution is obtained through improved particle swarm optimization algorithm to achieve the optimal planning of test equipment resources. Compared with traditional manual planning methods, this algorithm reduces the workload of testers, and improves the automation and efficiency of testing. Experimental results demonstrate that the optimal solutions found by the algorithm proposed in this paper is superior to those of the PSO (Particle Swarm Optimization)) and Linearly Decreasing Inertia Weight Particle Swarm Optimization (LWPSO) algorithms.

Uncertainty Quantification in CO2 Trapping Mechanisms: A Case Study of PUNQ-S3 Reservoir Model Using Representative Geological Realizations and Unsupervised Machine Learning

Evaluating uncertainty in CO2 injection projections often requires numerous high-resolution geological realizations (GRs) which, although effective, are computationally demanding. This study proposes the use of representative geological realizations (RGRs) as an efficient approach to capture the uncertainty range of the full set while reducing computational costs. A predetermined number of RGRs is selected using an integrated unsupervised machine learning (UML) framework, which includes Euclidean distance measurement, multidimensional scaling (MDS), and a deterministic K-means (DK-means) clustering algorithm. In the context of the intricate 3D aquifer CO2 storage model, PUNQ-S3, these algorithms are utilized. The UML methodology selects five RGRs from a pool of 25 possibilities (20% of the total), taking into account the reservoir quality index (RQI) as a static parameter of the reservoir. To determine the credibility of these RGRs, their simulation results are scrutinized through the application of the Kolmogorov–Smirnov (KS) test, which analyzes the distribution of the output. In this assessment, 40 CO2 injection wells cover the entire reservoir alongside the full set. The end-point simulation results indicate that the CO2 structural, residual, and solubility trapping within the RGRs and full set follow the same distribution. Simulating five RGRs alongside the full set of 25 GRs over 200 years, involving 10 years of CO2 injection, reveals consistently similar trapping distribution patterns, with an average value of Dmax of 0.21 remaining lower than Dcritical (0.66). Using this methodology, computational expenses related to scenario testing and development planning for CO2 storage reservoirs in the presence of geological uncertainties can be substantially reduced.

GD3N: Adaptive clustering-based detection of selective forwarding attacks in WSNs under variable harsh environments

Wireless sensor networks (WSNs) are susceptible to numerous security threats due to their reliance on open environments and broadcast communication methods. Among these, the selective forwarding attack is notably challenging to detect. This difficulty arises from the ability of malicious nodes to imitate the behavior of normal nodes, and selectively drop data packets, which makes them virtually indistinguishable from normal ones, particularly under conditions of poor channel quality. To address this challenge with harsh environments, we introduce a novel methodology termed GD3N. This approach is underpinned by the design of a unique type of data point that encapsulates both short-term and long-term forwarding behaviors of nodes. It combines a refined version of the Gradient Diffusion Density-based Spatial Clustering of Applications with Noise (GD-DBSCAN) algorithm, with a novel Double-Parameter Neighbor Voting (DP-NV) method based on the data set. These innovations contribute to a significant enhancement in detection accuracy and a reduction in computational complexity when compared to traditional DBSCAN and NV methods. Simulation results show that our GD3N achieves a false detection rate (FDR) of less than 2%, a missed detection rate (MDR) of below 10%, and an overall detection accuracy rate (DAR) of over 95% across various testing scenarios.

Application of computer vision technology in surface damage detection and analysis of shedthin tiles in China: a case study of the classical gardens of Suzhou

In computer artificial intelligence, there is great potential in research on the protection of Suzhou's traditional gardens, a world cultural heritage site. As a special material in Suzhou's traditional garden architecture, shedthin tile is widely used in roof base laying and is one of the important materials for building roofs. However, professionals need to reach the roof and spend much time and effort assessing the damage before repairing it. Therefore, the main goals of this study are to investigate a machine learning-based method for finding targets and determining the type of surface damage on a shedthin tile using the YOLOv4 model trained in this study. Using 500 shedthin tile on-site photos as training samples, the model was trained for 750 epochs. The main results of this study are as follows: (1) An object detection method based on machine learning can efficiently and accurately identify damage content, overcoming the manpower and time–cost limitations of traditional assessment methods. (2) The detection model in this study has an accuracy of 85.89% for water stain recognition of shedthin tiles, 93.29% for surface scaling, 87.37% for color aberration, and 96.15% for too wide a gap. The comprehensive accuracy is 90.20%, which meets the basic testing requirements. (3) The model demonstrated its robustness and reliability in complex environments in application tests in actual scenarios, providing a methodological reference for computer vision and target detection technology in cultural heritage protection.

Anomaly Sewing Pattern Detection for AIoT System using Deep Learning and Decision Tree

Artificial Intelligence of Things (AIoT), which combines AI and the Internet of Things (IoT), has recently gained popularity. Deep neural networks (DNNs) have achieved great success in many applications. Deploying complex AI models on embedded boards, nevertheless, may be challenging due to computational limitations or intelligent model complexity. This paper focuses on an AIoT-based system for smart sewing automation using edge devices. Our technique included developing a detection model and a decision tree for a sufficient testing scenario. YOLOv5 set the stage for our defective sewing stitches detection model, to detect anomalies and classify the sewing patterns. According to the experimental testing, the proposed approach achieved a perfect score with accuracy and F1score of 1.0, False Positive Rate (FPR), False Negative Rate (FNR) of 0, and a speed of 0.07 seconds with file size 2.43MB.

A Framework of Test Scenario Development for Issuance of Conditional Driver’s Licenses for Elderly Drivers

A Framework of Test Scenario Development for Issuance of Conditional Driver’s Licenses for Elderly Drivers

Using problem-based exploratory training to improve pilot understanding of autopilot functions

Previous research indicated a need to improve pilot training with regard to understanding of autopilot logic and behavior, especially in non-routine situations. Therefore, we tested the effect of problem-based exploratory training on pilots’ understanding of autopilot functions. Using a moving-base flight simulator, general aviation pilots (n = 45) were trained to diagnose failures either without foreknowledge and guidance (exploratory group), without foreknowledge but with some guidance (exploratory-guidance group) or with foreknowledge and full guidance (control group). They subsequently performed six test scenarios in which their understanding of the effects of failures was tested by requiring them to deduce the failures and select autopilot modes that were still functioning. Those who received exploratory training with guidance were significantly more likely than the other groups to diagnose failures correctly. The exploratory training group also selected the most appropriate functioning autopilot modes significantly faster than the control group. The results suggest that exploratory training with an appropriate level of guidance is useful for gaining a practical understanding of autopilot logic and behavior. Exploratory training may help to improve transfer of training to operational practice, and prevent automation surprises and accidents.