Simulated Task Research Articles

Neural-network-based parameter tuning for multi-agent simulation using deep reinforcement learning

This study proposes a new efficient parameter tuning method for multi-agent simulation (MAS) using deep reinforcement learning. MAS is currently a useful tool for social sciences, but is hard to realize realistic simulations due to its computational burden for parameter tuning. This study proposes efficient parameter tuning to address this issue using deep reinforcement learning methods. To improve compatibility with the tuning task, our proposed method employs actor-critic-based deep reinforcement learning, such as deep deterministic policy gradient (DDPG) and soft actor-critic (SAC). In addition to the customized version of DDPG and SAC for our task, we also propose three additional components to stabilize the learning: an action converter (DDPG only), a redundant full neural network actor, and a seed fixer. For experimental verification, we employ a parameter tuning task in an artificial financial market simulation, comparing our proposed model, its ablations, and the Bayesian estimation-based baseline. The results demonstrate that our model outperforms the baseline in terms of tuning performance, indicating that the additional components of the proposed method are essential. Moreover, the critic of our model works effectively as a surrogate model, that is, as an approximate function of the simulation, which allows the actor to tune the parameters appropriately. We have also found that the SAC-based method exhibits the best and fastest convergence, which we assume is achieved by the high exploration capability of SAC.

Binocular microscopes versus exoscopes: Experiences and performance in simulated otologic surgery.

Exoscopes represent a promising alternative to conventional binocular microscopes (OM) in otology offering potential advantages such as enhanced ergonomics and a more compact device design. While previous research has demonstrated the effectiveness of exoscopes in various surgical specialties, their objective assessment in the field of otology remains limited. Therefore, this investigation aims to assess task-based efficiency associated with exoscopes in the field of otology by use of simulated surgical models. A prospective cross-over study design was used to compare an OM to an exoscope in otolaryngology residents and medical students. Participants performed five tasks on 3D-printed ear models using both the exoscope and OM. Data collection included completion time, frequency of predefined errors, mental effort, and user experience. Subgroup analysis was performed based on level of experience. Fourteen students and fifteen residents participated. Participants completed four of five tasks faster with the OM and there was no difference in number of errors committed. When separated by surgical experience, residents performed four of five tasks faster using the OM while students completed one of five tasks faster with the OM. Students committed more errors with the exoscope for one task with no difference in errors for residents. There was no difference in perceived difficulty performing tasks with either visualization system. Exit survey results showed more favorable opinions of the OM among residents and more favorable opinions of the exoscope among students. The exoscope permits successful performance in simulated otologic tasks. Task performance and user experience between operative microscopes and exoscopes differ based on level of surgical experience. 2.

Be careful what you explain: Benefits and costs of explainable AI in a simulated medical task

We investigated the impact of explainability instructions with respect to system limitations on trust behavior and trust attitude when using an artificial intelligence (AI) support agent to perform a simulated medical task. In an online experiment (N = 128), participants performed a visual estimation task in a simulated medical setting (i.e., estimate the percentage of bacteria in a visual stimulus). All participants were supported by an AI that gave perfect recommendations for all but one color of bacteria (i.e., error-prone color with 50% reliability). We manipulated between-subjects whether participants knew about the error-prone color (XAI condition) or not (nonXAI condition). The analyses revealed that participants showed higher trust behavior (i.e., lower deviation from the AI recommendation) for the non-error-prone trials in the XAI condition. Moreover, participants showed lower trust behavior for the error-prone color in the XAI condition than in the nonXAI condition. However, this behavioral adaptation only applied to the subset of error-prone trials in which the AI gave correct recommendations, and not to the actual erroneous trials. Thus, designing explainable AI systems can also come with inadequate behavioral adaptations, as explainability was associated with benefits (i.e., more adequate behavior in non-error-prone trials), but also costs (stronger changes to the AI recommendations in correct error-prone trials).

Optimizing human-AI collaboration: Effects of motivation and accuracy information in AI-supported decision-making

Artificial intelligence (AI) systems increasingly support human decision-making in fields like medicine, management, and finance. However, such human-AI (HAI) collaboration is often less effective than AI systems alone. Moreover, efforts to make AI recommendations more transparent have failed to improve the decision quality of HAI collaborations. Based on dual process theories of cognition, we hypothesized that suboptimal HAI collaboration is partly due to heuristic information processing of humans, creating a trust imbalance towards the AI system. In an online experiment with 337 participants, we investigated motivation and accuracy information as potential factors to induce more deliberate elaboration of AI recommendations, and thus improve HAI collaboration. Participants worked on a simulated personnel selection task and received recommendations from a simulated AI system. Participants' motivation was varied through gamification, and accuracy information through additional information from the AI system. Results indicate that both motivation and accuracy information positively influenced HAI performance, but in different ways. While high motivation primarily increased humans’ use in high-quality recommendations only, accuracy information improved both the use of low- and high-quality recommendations. However, a combination of high motivation and accuracy information did not yield additional improvement of HAI performance.

Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning.

Efficient exploration remains a challenging problem in reinforcement learning, especially for tasks where extrinsic rewards from environments are sparse or even totally disregarded. Significant advances based on intrinsic motivation show promising results in simple environments but often get stuck in environments with multimodal and stochastic dynamics. In this work, we propose a variational dynamic model based on the conditional variational inference to model the multimodality and stochasticity. We consider the environmental state-action transition as a conditional generative process by generating the next-state prediction under the condition of the current state, action, and latent variable, which provides a better understanding of the dynamics and leads to a better performance in exploration. We derive an upper bound of the negative log likelihood of the environmental transition and use such an upper bound as the intrinsic reward for exploration, which allows the agent to learn skills by self-supervised exploration without observing extrinsic rewards. We evaluate the proposed method on several image-based simulation tasks and a real robotic manipulating task. Our method outperforms several state-of-the-art environment model-based exploration approaches.

Goal-Conditioned Reinforcement Learning With Disentanglement-Based Reachability Planning

Goal-Conditioned Reinforcement Learning (GCRL) can enable agents to spontaneously set diverse goals to learn a set of skills. Despite the excellent works proposed in various fields, reaching distant goals in temporally extended tasks remains a challenge for GCRL. Current works tackled this problem by leveraging planning algorithms to plan intermediate subgoals to augment GCRL. Their methods need two crucial requirements: (i) a state representation space to search valid subgoals, and (ii) a distance function to measure the reachability of subgoals. However, they struggle to scale to high-dimensional state space due to their non-compact representations. Moreover, they cannot collect high-quality training data through standard GC policies, which results in an inaccurate distance function. Both affect the efficiency and performance of planning and policy learning. In the paper, we propose a goal-conditioned RL algorithm combined with Disentanglement-based Reachability Planning (REPlan) to solve temporally extended tasks. In REPlan, a Disentangled Representation Module (DRM) is proposed to learn compact representations which disentangle robot poses and object positions from high-dimensional observations in a self-supervised manner. A simple REachability discrimination Module (REM) is also designed to determine the temporal distance of subgoals. Moreover, REM computes intrinsic bonuses to encourage the collection of novel states for training. We evaluate our REPlan in three vision-based simulation tasks and one real-world task. The experiments demonstrate that our REPlan significantly outperforms the prior state-of-the-art methods in solving temporally extended tasks.

Learning from experimental data to simulate pedestrian dynamics

Accuracy is a critical concern for those engaged in the simulation of pedestrian dynamics. Although a range of pedestrian models has been developed, the performance of these models is heavily limited by simplifications inherent in the usual knowledge-based approach. In this study, we propose data-driven models of pedestrian dynamics based on the machine learning approaches. The models are constructed in an iterative manner; thus, the long-term future trajectories are predicted using the current output as the input at the next timestep. The performance of our models is evaluated by comparison with an well-researched knowledge-based pedestrian model. The evaluation included two metrics: the ADE/FDE evaluation metric and another metric that measures whether the simulation corresponded to ground truths from the collective dynamics perspective. Experiments on two experimental datasets show that our models outperform the knowledge-based model. Our findings highlight the considerable potential of the data-driven approach for handling pedestrian simulation tasks.

Policy Gradient From Demonstration and Curiosity.

With reinforcement learning, an agent can learn complex behaviors from high-level abstractions of the task. However, exploration and reward shaping remain challenging for existing methods, especially in scenarios where extrinsic feedback is sparse. Expert demonstrations have been investigated to solve these difficulties, but a tremendous number of high-quality demonstrations are usually required. In this work, an integrated policy gradient algorithm is proposed to boost exploration and facilitate intrinsic reward learning from only a limited number of demonstrations. We achieved this by reformulating the original reward function with two additional terms, where the first term measured the Jensen-Shannon divergence between current policy and the expert's demonstrations, and the second term estimated the agent's uncertainty about the environment. The presented algorithm was evaluated by a range of simulated tasks with sparse extrinsic reward signals, where only limited demonstrated trajectories were provided to each task. Superior exploration efficiency and high average return were demonstrated in all tasks. Furthermore, it was found that the agent could imitate the expert's behavior and meanwhile sustain high return.

Flexible unmanned surface vehicles control using probabilistic model-based reinforcement learning with hierarchical Gaussian distribution

This paper focuses on improving the flexibility of probabilistic model-based reinforcement learning (MBRL) in unmanned surface vehicles (USV) against complicated ocean disturbances. Filtered probabilistic model predictive control using hierarchical Gaussian distribution (FPMPC-HG) is proposed to provide a more flexible representation of environmental uncertainties and better control capability against real-time disturbances by integrating hierarchical Gaussian distribution into existing approaches specific to USVs. The proposed approach was evaluated through position-keeping and targets-tracking tasks in a real boat data-driven simulation. The experimental results demonstrated significant improvement in control performance, generalization capability, and task completion compared to the baseline approaches without employing hierarchical Gaussian distribution. The improved expressivity and flexibility of the probabilistic model and the corresponding policy of USVs against unknown ocean disturbances indicate the potential of hierarchical Gaussian distribution in probabilistic MBRL as a growing trend in the USV domain.

Variational quasi-harmonic maps for computing diffeomorphisms

Computation of injective (or inversion-free) maps is a key task in geometry processing, physical simulation, and shape optimization. Despite being a longstanding problem, it remains challenging due to its highly nonconvex and combinatoric nature. We propose computation of variational quasi-harmonic maps to obtain smooth inversion-free maps. Our work is built on a key observation about inversion-free maps: A planar map is a diffeomorphism if and only if it is quasi-harmonic and satisfies a special Cauchy boundary condition. We hence equate the inversion-free mapping problem to an optimal control problem derived from our theoretical result, in which we search in the space of parameters that define an elliptic PDE. We show that this problem can be solved by minimizing within a family of functionals. Similarly, our discretized functionals admit exactly injective maps as the minimizers, empirically producing inversion-free discrete maps of triangle meshes. We design efficient numerical procedures for our problem that prioritize robust convergence paths. Experiments show that on challenging examples our methods can achieve up to orders of magnitude improvement over state-of-the-art, in terms of speed or quality. Moreover, we demonstrate how to optimize a generic energy in our framework while restricting to quasi-harmonic maps.

Anatomically Detailed Simulation of Human Torso

Many existing digital human models approximate the human skeletal system using rigid bodies connected by rotational joints. While the simplification is considered acceptable for legs and arms, it significantly lacks fidelity to model rich torso movements in common activities such as dancing, Yoga, and various sports. Research from biomechanics provides more detailed modeling for parts of the torso, but their models often operate in isolation and are not fast and robust enough to support computationally heavy applications and large-scale data generation for full-body digital humans. This paper proposes a new torso model that aims to achieve high fidelity both in perception and in functionality, while being computationally feasible for simulation and optimal control tasks. We build a detailed human torso model consisting of various anatomical components, including facets, ligaments, and intervertebral discs, by coupling efficient finite-element and rigid-body simulations. Given an existing motion capture sequence without dense markers placed on the torso, our new model is able to recover the underlying torso bone movements. Our method is remarkably robust that it can be used to automatically "retrofit" the entire Mixamo motion database of highly diverse human motions without user intervention. We also show that our model is computationally efficient for solving trajectory optimization of highly dynamic full-body movements, without relying any reference motion. Physiological validity of the model is validated against established literature.

Stabilizing deep Q-learning with Q-graph-based bounds

State-of-the art deep reinforcement learning has enabled autonomous agents to learn complex strategies from scratch on many problems including continuous control tasks. Deep Q-networks (DQN) and deep deterministic policy gradients (DDPGs) are two such algorithms which are both based on Q-learning. They therefore all share function approximation, off-policy behavior, and bootstrapping—the constituents of the so-called deadly triad that is known for its convergence issues. We suggest to take a graph perspective on the data an agent has collected and show that the structure of this data graph is linked to the degree of divergence that can be expected. We further demonstrate that a subset of states and actions from the data graph can be selected such that the resulting finite graph can be interpreted as a simplified Markov decision process (MDP) for which the Q-values can be computed analytically. These Q-values are lower bounds for the Q-values in the original problem, and enforcing these bounds in temporal difference learning can help to prevent soft divergence. We show further effects on a simulated continuous control task, including improved sample efficiency, increased robustness toward hyperparameters as well as a better ability to cope with limited replay memory. Finally, we demonstrate the benefits of our method on a large robotic benchmark with an industrial assembly task and approximately 60 h of real-world interaction.

Artificial intelligence based real-time video ergonomic assessment and training improves resident ergonomics

BackgroundSurgery demands long hours and intense exertion raising ergonomic concerns. We piloted a sensorless artificial intelligence (AI)-assisted ergonomics analysis app to determine its feasibility for use with residents. MethodsSurgery residents performed simulated laparoscopic tasks before and after a review of the SCORE ergonomics curriculum while filmed with a sensorless app from Kinetica Labs that calculates joint angles as a metric of ergonomics. A survey was completed before the session and a focus group was conducted after. ResultsThirteen surgical residents participated in the study. The brief intervention took little time and residents improved their ergonomic scores in neck and right shoulder angles. Residents expressed increased awareness of ergonomics based on the session content and AI information. All trainees desired more training in ergonomics. ConclusionsErgonomic assessment AI software can provide immediate feedback to surgical trainees to improve ergonomics. Additional studies using sensorless AI technology are needed.

Are humorous or distractor images more effective than self-compassion messages for combatting the negative body image consequences of social media? An experimental test of possible micro-intervention stimuli

Appearance-focused images on social media are thought to be particularly detrimental for body image. However, social media sites can also be used to encourage positive health behaviours. Three linked experiments with 620 Instagram users explored the protective capabilities of appearance-related self-compassion and appearance-related humorous messages for women’s body image during Instagram use. Using simulated Instagram browsing tasks, participants were exposed to a set of fitspiration Instagram posts mixed with either self-compassion or humorous body image messages, or appearance-neutral images. Results indicated that appearance-related self-compassion and humorous messages were not more effective at protecting against negative appearance and life satisfaction outcomes than appearance-neutral images, and did not influence appearance comparison (Experiment 1), even when the personal relevance to participants’ health was reinforced through experimental manipulation (Experiment 2). Rather, the presence of any image which did not contain pictures of women, regardless of image content, led to improved body image outcomes compared to exposure to fitspiration images alone (Experiment 3). Interpersonal factors such as the similarity of a female target’s appearance also influenced the nature of comparisons made. The study highlights the importance of diluting appearance-focused content with other social media images in ongoing research practice and for user well-being.

Evaluation of driving fitness using driving simulators in patients with right-hemisphere damage: an unmatched case-control study

ABSTRACT Background Patients with right hemisphere damage (RHD) may exhibit mild unilateral spatial neglect (USN), which is difficult to detect in general assessments performed during driving rehabilitation. Objectives We compared driving simulator performance, practical driving performance, and neuropsychological test results between patients with RHD who were able and unable to return to driving to predict driving fitness based on driving simulator performance. Methods This unmatched case-control study included 29 patients with RHD who were able (return-to-driving group, n = 16) and unable (non-return-to-driving group, n = 13) to return to driving. Patient demographics, motor function, attention, driving simulator performance (participants’ reaction time and rate to green lamps appearing in any of the three displays and average lane position), and practical driving performance were compared between the groups. Receiver operating characteristic (ROC) analysis was performed to examine the predictive performance of driving fitness in reaction rate and paper-and-pencil tests. Results The non-return-to-driving group had a significantly lower reaction rate than the return-to-driving group (p = 0.027; 95% confidence interval [CI], 0–7; r = 0.407). No significant difference in reaction time or lane position in either the left or right lane during driving simulation was observed. ROC analysis of the reaction rate in the driving simulator task showed sensitivity of 0.692, specificity of 0.812, and area under the curve of 0.738 [95% CI, 0.541–0.935]. Conclusions Decreases in reaction rates during simulated driving assessments are associated with an inability to resume driving in patients with RHD. Such assessments may aid in predicting fitness for driving in these patients.

Effect of muscular fatigue on the cumulative lumbar damage during repetitive lifting task: a comparative study of damage calculation methods

Several methods have been put forward to quantify cumulative loads; however, limited evidence exists as to the subsequent damages and the role of muscular fatigue. The present study assessed whether muscular fatigue could affect cumulative damage imposed on the L5–S1 joint. Trunk muscle electromyographic (EMG) activities and kinematics/kinetics of 18 healthy male individuals were evaluated during a simulated repetitive lifting task. A traditional EMG-assisted model of the lumbar spine was modified to account for the effect of erector spinae fatigue. L5–S1 compressive loads for each lifting cycle were estimated based on varying (i.e. actual), fatigue-modified, and constant Gain factors. The corresponding damages were integrated to calculate the cumulative damage. Moreover, the damage calculated for one lifting cycle was multiplied by the lifting frequency, as the traditional approach. Compressive loads and the damages obtained through the fatigue-modified model were predicted in close agreement with the actual values. Similarly, the difference between actual damages and those driven by the traditional approach was not statistically significant (p = 0.219). However, damages based on a constant Gain factor were significantly greater than those based on the actual (p = 0.012), fatigue-modified (p = 0.017), and traditional (p = 0.007) approaches. Practitioner summary: In this study, we managed to include the effect of muscular fatigue on cumulative lumbar damage calculations. Including the effect of muscular fatigue leads to an accurate estimation of cumulative damages while eliminating computational complexity. However, using the traditional approach also appears to provide acceptable estimates for ergonomic assessments.

Performance in a Simulated Virtual Reality Anterior Cervical Discectomy and Fusion Task: Disc Residual, Rate of Removal, and Efficiency Analyses.

Anterior cervical discectomy and fusion (ACDF) is among the most common spine procedures. The Sim-Ortho virtual reality simulator platform contains a validated ACDF simulated task for performance assessment. This study aims to develop a methodology to extract three-dimensional data and reconstruct and quantitate specific simulated disc tissues to generate novel metrics to analyze performance metrics of skilled and less skilled participants. We used open-source platforms to develop a methodology to extract three-dimensional information from ACDF simulation data. Metrics generated included, efficiency index, disc volumes removed from defined regions, and rate of tissue removal from superficial, central, and deep disc regions. A pilot study was performed to assess the utility of this methodology to assess expertise during the ACDF simulated procedure. The system outlined, extracts data allowing the development of a methodology which accurately reconstructs and quantitates 3-dimensional disc volumes. In the pilot study, data sets from 27 participants, divided into postresident, resident, and medical student groups, allowed assessment of multiple novel metrics, including efficiency index (surgical time spent in actively removing disc), where the postresident group spent 61.8% of their time compared with 53% and 30.2% for the resident and medical student groups, respectively ( P = .01). During the annulotomy component, the postresident group removed 47.4% more disc than the resident groups and 102% more than the medical student groups ( P = .03). The methodology developed in this study generates novel surgical procedural metrics from 3-dimensional data generated by virtual reality simulators and can be used to assess surgical performance.

Hapstick: A Soft Flexible Joystick for Stiffness Rendering via Fiber Jamming

Continuum robots are well-suited for applications in delicate and constrained environments, such as minimally invasive surgery, due to their inherent compliance and ability to conform to highly curved paths. Yet the kinematic dissimilarity between continuum robots and conventional, off-the-shelf input devices, along with the general lack of haptic feedback available with such devices, can lead to non-intuitive control. In this work, we present Hapstick — a soft, flexible haptic joystick that uses fiber jamming to modulate its stiffness and provide feedback to users during teleoperation tasks. We characterize the performance of Hapstick, showing that the bending stiffness increases linearly with the increase in applied vacuum load. A psychophysical study is also conducted to obtain the just noticeable difference in stiffness that users can perceive using Hapstick. Lastly, we perform a study in which participants use Hapstick to teleoperate a physical tendon-driven continuum robot in a simulated colorectal cancer screening task. Users correctly identify the position and development stages of cancerous tissues in 25 out of 27 trials, illustrating the potential of jamming-based mechanisms as bidirectional interfaces capable of providing effective haptic feedback.

Examining Multitasking Ability on an Interactive Driving Simulator Task to Differentiate Between Fit & Unfit Drivers

Abstract Date Presented 04/21/2023 Driving is a critical instrumental activity of daily living and demands multitasking. Multitasking declines with age and certain medical conditions. An innovative multitasking assessment demonstrated significant differences between medically at-risk drivers and healthy controls. Primary Author and Speaker: Alec Ducharme Additional Authors and Speakers: Meagan T. Denny, Anne E. Dickerson Contributing Authors: Meagan T. Denny, Anne E. Dickerson

Modeling & Simulation SPICE: Assessing the Capability of Credible Simulation Processes

AbstractWhile systems or products are becoming increasingly complex and new technologies emerge to manage this growing complexity, the potential of virtualized or simulation‐based product development is not yet fully exploited. This is mainly due to the fact that simulation results are still not trusted enough. Therefore, this paper outlines a possible approach to derive and ensure required levels of modeling and simulation credibility from the criticality of a simulation task. Furthermore, a framework is presented for assessing the implementation of a credible simulation process. This framework is based on an existing process reference model, ensuring modeling and simulation credibility, and on a process assessment model, which has proven itself in the German automotive industry in the assessment of process capability. Finally, the introduction to this framework is demonstrated using an example from the automotive domain, i.e., the simulation‐based safety approval for the virtual testing of an Automated Lane Keeping System.