Sequential Decision Research Articles

Institutions and cooperation: A meta-analysis of structural features in social dilemmas.

Cooperation underlies the ability of groups to realize collective benefits (e.g., creation of public goods). Yet, cooperation can be difficult to achieve when people face situations with conflicting interests between what is best for individuals versus the collective (i.e., social dilemmas). To address this challenge, groups can implement rules about structural changes in a situation. But what institutional rules can best facilitate cooperation? Theoretically, rules can be made to affect structural features of a social dilemma, such as the possible actions, outcomes, and people involved. We derived 13 preregistered hypotheses from existing work and collected 6 decades of empirical research to test how nine structural features influence cooperation within prisoner's dilemmas and public goods dilemmas. We do this by meta-analyzing mean levels of cooperation across studies (Study 1, k = 2,340, N = 229,528), and also examining how manipulations of these structural features in social dilemmas affect cooperation within studies (Study 2, k = 909). Results indicated that lower conflict of interests was associated with higher cooperation and that (a) the implementation of sanctions (i.e., reward and punishment of behaviors) and (b) allowing for communication most strongly enhanced cooperation. However, we found inconsistent support for the hypotheses that group size and matching design affect cooperation. Other structural features (e.g., symmetry of dilemmas, sequential decision making, payment) were not associated with cooperation. Overall, these findings inform institutions that can (or not) facilitate cooperation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Context-Dependent Criteria for Dirichlet Process in Sequential Decision-Making Problems

In models with insufficient initial information, parameter estimation can be subject to statistical uncertainty, potentially resulting in suboptimal decision-making; however, delaying implementation to gather more information can also incur costs. This paper examines an extension of information-theoretic approaches designed to address this classical dilemma, focusing on balancing the expected profits and the information needed to be obtained about all of the possible outcomes. Initially utilized in binary outcome scenarios, these methods leverage information measures to harmonize competing objectives efficiently. Building upon the foundations laid by existing research, this methodology is expanded to encompass experiments with multiple outcome categories using Dirichlet processes. The core of our approach is centered around weighted entropy measures, particularly in scenarios dictated by Dirichlet distributions, which have not been extensively explored previously. We innovatively adapt the technique initially applied to binary case to Dirichlet distributions/processes. The primary contribution of our work is the formulation of a sequential minimization strategy for the main term of an asymptotic expansion of differential entropy, which scales with sample size, for non-binary outcomes. This paper provides a theoretical grounding, extended empirical applications, and comprehensive proofs, setting a robust framework for further interdisciplinary applications of information-theoretic paradigms in sequential decision-making.

Bayesian Sequential Learning and Decision Making in Bike‐Sharing Systems

Bayesian Sequential Learning and Decision Making in Bike‐Sharing Systems

Rapid computation of survival signature for dynamic fault tree based on sequential binary decision diagram and multidimensional array

Many practical safety-critical systems typically exhibit sequence-dependent failure behaviors, limiting the efficiency of analyzing these systems. Although the survival signature-based method can address this problem to a certain extent, the dependence on Boolean states constrains its application to large systems. In this study, we present a novel method that leverages the sequential binary decision diagram (SBDD) and multidimensional array to rapidly compute survival signatures for dynamic fault trees (DFTs) of these systems. These dynamic nodes in the SBDD are represented through multidimensional arrays, which are then utilized as inputs for the subsequent computations. Ultimately, survival signatures are obtained by iteratively computing the multidimensional arrays. Additionally, two practical engineering cases are examined to highlight the superiority of the proposed methods over other methods. Compared with Boolean state vector-based methods, the proposed method achieves a 689-fold and 209-fold increase in efficiency for calculating survival signatures in their respective cases. Compared with the Monte Carlo (MC) simulation, the simulation efficiency for the reliability results improve by 60-fold and 201-fold in their respective cases.

Designing User Experience Improvement and User Behavior Pattern Recognition Algorithms in Design Operation

Enhancing user experience (UX) is a key component in customer retention and sales promotion in e-commerce platforms. To build an effective UX model it is necessary to predict the user behavior more accurately and develop UX model that is tailored based on those behavior patterns. Existing models lack the ability to integrate advanced Machine Learning (ML) models to address the challenges. This study is an attempt to tackle these limitations that employs advanced AI tools to predict user behavior so that to construct an more effective UX model. The study involved 80 users from China who were aged 26 to 52, with diverse backgrounds in education, occupation, and tech proficiency. The work have employed Google Analytics, Hotjar, and FullStory to collect the user interactions and by using Generalized Sequential Pattern (GSP) algorithm, Decision Trees (DT), and Logistic Regression (LR) the work attempts to accurately predict the user behavior patterns. The results show that the model achieved better accuracy of 0.8795 and an F1 Score of 0.8610 on the test dataset. It also excelled in conversion rate (12.34%) and bounce rate (28.65%) which show effectiveness in retaining users and converting visits into actions.

Sequential Decision Making in Beach Volleyball-A Mixed-Method Approach.

Which opponent player to sequentially serve to in beach volleyball is crucial given the advantage of the attacking team. The sequential choice theory was tested in three studies by analyzing allocation strategies based on the hot hand belief. Study 1 showed strong belief in the hot hand of national coaches. In Study 2, we analyzed Tokyo Olympics data to explore how base rates and sequential selection rates varied in an elite sample. When base rates of players differed by 0.25, low-performing players were frequently selected. In an experiment with elite athletes, Study 3A demonstrated accurate base-rate-difference recognition but low base-rate-change recognition. Study 3B found that the hot hand is believed to be important but is not often detected. We conclude that players and coaches follow predictions of the sequential choice theory and believe in the hot hand, but do not have a shared understanding of how to use it.

Who versus when: Designing decision processes in organizations

This paper analyzes concurrent versus sequential decision-making in a model where two units first communicate and then make decisions, attempting to both adapt to their local conditions and coordinate with their partner. Sequential decision-making improves overall information sharing compared to concurrent decision-making. However, first movers also have an incentive to over-adapt to their state, knowing second movers will conform to their decision. A surplus-maximizing headquarters prefers sequential decision-making to concurrent if and only if (i) the two units' local conditions have sufficiently different volatilities and (ii) their need to coordinate is sufficiently asymmetric or low. Finally, sequential decision-making is shown to be optimal even when allowing for additional governance structures involving the reallocation of decision rights across the units and the headquarters and is shown to render some commonly-analyzed forms of decentralization sub-optimal.

Intuitionistic Fuzzy Sequential Three-Way Decision Model in Incomplete Information Systems

As an effective method for uncertain knowledge discovery and decision-making, the three-way decisions model has attracted extensive attention from scholars. However, in practice, the existing sequential three-way decision model often faces challenges due to factors such as missing data and unbalanced attribute granularity. To address these issues, we propose an intuitionistic fuzzy sequential three-way decision (IFSTWD) model, which introduces several significant contributions: (1) New intuitionistic fuzzy similarity relations. By integrating possibility theory, our model defines similarity and dissimilarity in incomplete information systems, establishing new intuitionistic fuzzy similarity relations and their cut relations. (2) Granulation method innovation. We propose a density neighborhood-based granulation method to partition decision attributes and introduce a novel criterion for evaluating attribute importance. (3) Enhanced decision process. By incorporating sequential three-way decision theory and developing a multi-level granularity structure, our model replaces the traditional equivalent relation in the decision-theoretic rough sets model, thus advancing the model’s applicability and effectiveness. The practical utility of our model is demonstrated through an example analysis of “Chinese + vocational skills” talent competency and validated through simulation experiments on the UCI dataset, showing superior performance compared to existing methods.

Efficient and robust sequential decision making algorithms

AbstractSequential decision‐making involves making informed decisions based on continuous interactions with a complex environment. This process is ubiquitous in various applications, including recommendation systems and clinical treatment design. My research has concentrated on addressing two pivotal challenges in sequential decision‐making: (1) How can we design algorithms that efficiently learn the optimal decision strategy with minimal interactions and limited sample data? (2) How can we ensure robustness in decision‐making algorithms when faced with distributional shifts due to environmental changes and the sim‐to‐real gap? This paper summarizes and expands upon the talk I presented at the AAAI 2024 New Faculty Highlights program, detailing how my research aims to tackle these challenges.

Deep contextual reinforcement learning algorithm for scalable power scheduling

This article introduces a deep contextual reinforcement learning (DCRL) optimization algorithm to tackle the NP-hard power scheduling problem. The algorithm is based on a multi-agent simulation environment that decomposes the power scheduling problem into sequential Markov decision processes (MDPs). The simulation environment inherently corrects incorrect decisions and adjusts supply capacities so that the MDPs adhere to the optimization constraints. A deep reinforcement learning (DRL) model is trained on these MDPs to provide optimal solutions. Demonstrating its applicability and effectiveness, the proposed method is evaluated on various test systems with different unit numbers, constraints, and production cost functions. The experimental results show that the proposed method has better performance relative to alternative methods, such as binary alternative moth-flame optimization (BAMFO), binary particle swarm optimization (BPSO), teaching learning-based optimization (TLBO), new binary particle swarm optimization (NBPSO), binary learning particle swarm optimization (BLPSO), quasi-oppositional teaching learning-based algorithm (QOTLBO), binary-real-coded genetic algorithm (BRCGA), hybrid genetic-imperialist competitive algorithm (HGICA), three-stage priority list (TSPL), and hybridized evolutionary algorithm and sequential quadratic programming (HEPSQP). The novelties of the proposed method lie in its adaptability to longer planning horizons and linear dimensionality complexity, rendering it suitable for large-scale power systems.

Is Reinforcement Learning Good at American Option Valuation?

This paper investigates algorithms for identifying the optimal policy for pricing American Options. The American Option pricing is reformulated as a Sequential Decision-Making problem with two binary actions (Exercise or Continue), transforming it into an optimal stopping time problem. Both the least square Monte Carlo simulation method (LSM) and Reinforcement Learning (RL)-based methods were utilized to find the optimal policy and, hence, the fair value of the American Put Option. Both Classical Geometric Brownian Motion (GBM) and calibrated Stochastic Volatility models served as the underlying uncertain assets. The novelty of this work lies in two aspects: (1) Applying LSM- and RL-based methods to determine option prices, with a specific focus on analyzing the dynamics of “Decisions” made by each method and comparing final decisions chosen by the LSM and RL methods. (2) Assess how the RL method updates “Decisions” at each batch, revealing the evolution of the decisions during the learning process to achieve optimal policy.

EBPVis: Visual Analytics of Economic Behavior Patterns in a Virtual Experimental Environment

AbstractExperimental economics is an important branch of economics to study human behaviours in a controlled laboratory setting or out in the field. Scientific experiments are conducted in experimental economics to collect what decisions people make in specific circumstances and verify economic theories. As a significant couple of variables in the virtual experimental environment, decisions and outcomes change with the subjective factors of participants and objective circumstances, making it a difficult task to capture human behaviour patterns and establish correlations to verify economic theories. In this paper, we present a visual analytics system, EBPVis, which enables economists to visually explore human behaviour patterns and faithfully verify economic theories, e.g. the vicious cycle of poverty and poverty trap. We utilize a Doc2Vec model to transform the economic behaviours of participants into a vectorized space according to their sequential decisions, where frequent sequences can be easily perceived and extracted to represent human behaviour patterns. To explore the correlation between decisions and outcomes, an Outcome View is designed to display the outcome variables for behaviour patterns. We also provide a Comparison View to support an efficient comparison between multiple behaviour patterns by revealing their differences in terms of decision combinations and time‐varying profits. Moreover, an Individual View is designed to illustrate the outcome accumulation and behaviour patterns of subjects. Case studies, expert feedback and user studies based on a real‐world dataset have demonstrated the effectiveness and practicability of EBPVis in the representation of economic behaviour patterns and certification of economic theories.

Robust ordering, production, and replenishment in a supply chain for innovative products: A two-echelon newsvendor problem with partial information in yield and demand

With partial yield and demand information, we investigate a two-echelon newsvendor problem in a supply chain for innovative product, in which the retailer and the supplier make sequential decisions in ordering, production, and replenishment, respectively. We adopt the minimax-regret criterion to assume the risk-averse nature and deduce the minimax-regret inventory decisions. We explore replenishment (emergency) production strategies for the supplier. Then we propose shortage penalty strategies as well as subsidized replenishment strategies for the retailer so that the supplier can maintain the required supply even under higher replenishment production costs. We further reveal the role of information uncertainty in the minimax-regret decisions and the bullwhip effect. Interestingly, the results show that the penalty strategies do not always lead to less regret for the retailer, and the information uncertainty does not necessarily lead to greater regret for the supplier. Moreover, some schemes to reduce the bullwhip effect are proposed, such as improving the accuracy of uncertain yield, increasing shortage penalty or replenishment subsidy, and decreasing replenishment cost. It is verified that the replenishment strategies can increase profits for both the retailer and the supplier under some conditions. In extensions, we reveal the supplier’s preference for emergency production and external procurement, as well as preferences for information sharing and confidentiality. Meanwhile, we find that decentralized decision-making may even outperform centralized decision-making in some situations when the minimax-regret criterion is adopted to deal with the information deficiency. Finally, the robustness of the replenishment strategies in reducing both enterprises’ regrets is checked.

Development of a Canadian Guidance for Reporting Real-world Evidence for Regulatory and Health-Technology Assessment (HTA) Decision Making

Real-world evidence (RWE) can complement and fill knowledge gaps from randomized controlled trials to assist in health-technology assessment (HTA) for regulatory decision-making. However, the generation of RWE is an intricate process with many sequential decision points, and different methods and approaches may impact the quality and reliability of evidence. Standardization and transparency in reporting these decisions is imperative to appraise RWE and incorporate it into HTA decision-making. A partnership between Canadian health system stakeholders, namely Health Canada and Canada's Drug Agency (formerly the Canadian Agency for Drugs and Technologies in Health (CADTH)), was established to develop a guidance for standardization of reporting of RWE for regulatory and HTA decision-making in Canada. In this article, we describe the methods to develop the Guidance for Reporting Real-World Evidence document and checklist for reporting RWE for regulatory and HTA decision-making in Canada. This guidance can be adapted for other jurisdictions and will have future extensions to incorporate emerging issues with RWE and HTA decision-making.

Optimizing sequential decision-making under risk: Strategic allocation with switching penalties

This paper considers the multiarmed bandit (MAB) problem augmented with a critical real-world consideration: the cost implications of switching decisions. Our work distinguishes itself by addressing the largely unexplored domain of risk-averse MAB problems compounded by switching penalties. Such scenarios are not just theoretical constructs but are reflective of numerous practical applications. Our contribution is threefold: firstly, we explore how switching costs and risk aversion influence decision-making in MAB problems. Secondly, we present novel theoretical results, including the development of the Risk-Averse Switching Index (RASI), which addresses the dual challenges of risk aversion and switching costs, demonstrating its near-optimal efficacy. This heuristic solution method is grounded in dynamic coherent risk measures, enabling a time-consistent evaluation of risk and reward. Lastly, through rigorous numerical experiments, we validate our algorithm’s effectiveness and practical applicability, providing decision-makers with valuable insights and tools for navigating the multifaceted landscape of risk-averse environments with inherent switching costs.

EXPRESS: A Machine Learning Approach to Solve the E-commerce Box-Sizing Problem

E-commerce packages are notorious for their inefficient usage of space. More than one-quarter volume of a typical e-commerce package comprises air and filler material. The inefficient usage of space significantly reduces the transportation and distribution capacity increasing the operational costs. Therefore, designing an optimal set of packaging box sizes is crucial for improving efficiency. We present the first learning-based framework to determine the optimal packaging box sizes. In particular, we propose a three-stage optimization framework that combines unsupervised learning, reinforcement learning, and tree search to design box sizes. The package optimization problem is formulated into a sequential decision-making task called the box-sizing game. A neural network agent is then designed to play the game and learn heuristic rules to solve the problem. In addition, a tree-search operator is developed to improve the performance of the learned networks. When benchmarked with company-based optimization formulation and two alternate optimization models, we find that our ML-based approach can effectively solve large-scale problems within a stipulated time. We evaluated our model on real-world datasets supplied by a large e-commerce platform. The framework is currently adopted by a large e-commerce company across its 28 fulfillment centers, which is estimated to save the company about 7.1 million USD annually. In addition, it is estimated that paper consumption will be reduced by 2080 metric tons and greenhouse gas emissions by 1960 metric tons annually. The presented optimization framework serves as a decision support tool for designing packaging boxes at large e-commerce warehouses.

Learning to Move and Plan like the Knight: Sequential Decision Making with a Novel Motor Mapping.

Many skills that humans acquire throughout their lives, such as playing video games or sports, require substantial motor learning and multi-step planning. While both processes are typically studied separately, they are likely to interact during the acquisition of complex motor skills. In this work, we studied this interaction by assessing human performance in a sequential decision-making task that requires the learning of a non-trivial motor mapping. Participants were tasked to move a cursor from start to target locations in a grid world, using a standard keyboard. Notably, the specific keys were arbitrarily mapped to a movement rule resembling the Knight chess piece. In Experiment 1, we showed the learning of this mapping in the absence of planning, led to significant improvements in the task when presented with sequential decisions at a later stage. Computational modeling analysis revealed that such improvements resulted from an increased learning rate about the state transitions of the motor mapping, which also resulted in more flexible planning from trial to trial (less perseveration or habitual responses). In Experiment 2, we showed that incorporating mapping learning into the planning process, allows us to capture (1) differential task improvements for distinct planning horizons and (2) overall lower performance for longer horizons. Additionally, model analysis suggested that participants may limit their search to three steps ahead. We hypothesize that this limitation in planning horizon arises from capacity constraints in working memory, and may be the reason complex skills are often broken down into individual subroutines or components during learning.

Sequential three-way decision with automatic threshold learning for credit risk prediction

Machine learning algorithms treat credit risk prediction as a binary classification problem. However, two-way decisions with a single threshold force to make either a default or non-default decision may be inappropriate. To reduce the risk of decision errors, this study introduces three-way decisions and proposes a sequential three-way decision model with automatic threshold learning to evaluate credit risk. Initially, the model uses the loan amount and interest to determine the decision loss of the three-way decision, assigning distinct decision thresholds to different samples. Subsequently, the model employs decision cost and information gain to formulate an objective for threshold optimisation. Finally, the model continuously optimises the classification process by using the outcomes of certain decisions as supplementary information. In addition, to validate our model, we conduct comparative experiments with various methods on a real credit dataset from a Chinese bank. The results indicate that the model not only enhances classification performance across several metrics but also assists financial institutions in reducing decision error costs.

Marginal improvement procedures for top-[formula omitted] selection

Given a fixed simulation budget, the problem of selecting the best and top-m alternatives among a finite set of alternatives have been studied separately in simulation optimization literature, because the existing sampling procedures are often dedicated to one problem. Under a Bayesian framework, we formulate the top-m selection into a stochastic dynamic program, and characterize the optimal sampling policy via Bellman equations. To determine sequential sampling decisions, we measure the expected marginal improvement from obtaining one additional simulation observation based on predictive distributions, and then develop two cheaply computational approximations to the improvement, thereby yielding two generic sampling procedures that are efficient in selecting top-m alternative(s). The two procedures are proved to be consistent, in a sense that the best and top-m alternatives can be correctly identified as the simulation budget goes to infinity. Numerical experiments on synthetic problems and a coronavirus transmission control application are conducted to demonstrate the efficiency and generality of our procedures.

Enhancing heart disease prediction with reinforcement learning and data augmentation

The study presents a novel method to improve the prediction accuracy of cardiac disease by combining data augmentation techniques with reinforcement learning. The complex nature of cardiac data frequently presents challenges for traditional machine learning models, which results in subpar performance. In response, our fusion methodology improves predictive capabilities by augmenting data and utilizing reinforcement learning's skill at sequential decision-making. Our method predicts cardiac disease with an astounding 94 % accuracy rate, which is an outstanding result. This significant improvement outperforms existing techniques and shows a deeper comprehension of intricate data relationships. The amalgamation of reinforcement learning and data augmentation not only yields superior predictive accuracy but also bears noteworthy consequences for patient care and accurate cardiac diagnosis. Through the efficient combination of these approaches, our method provides a powerful response to the difficulties presented by complicated cardiac data. The potential to transform illness prediction and prevention techniques and ultimately improve patient outcomes is demonstrated by this integration's success.