Two-stage Task Research Articles

Hunger improves reinforcement-driven but not planned action

Human decisions can be reflexive or planned, being governed respectively by model-free and model-based learning systems. These two systems might differ in their responsiveness to our needs. Hunger drives us to specifically seek food rewards, but here we ask whether it might have more general effects on these two decision systems. On one hand, the model-based system is often considered flexible and context-sensitive, and might therefore be modulated by metabolic needs. On the other hand, the model-free system’s primitive reinforcement mechanisms may have closer ties to biological drives. Here, we tested participants on a well-established two-stage sequential decision-making task that dissociates the contribution of model-based and model-free control. Hunger enhanced overall performance by increasing model-free control, without affecting model-based control. These results demonstrate a generalized effect of hunger on decision-making that enhances reliance on primitive reinforcement learning, which in some situations translates into adaptive benefits.

Validating Habitual and Goal-Directed Decision-Making Performance Online in Healthy Older Adults.

Everyday decision-making is supported by a dual-system of control comprised of parallel goal-directed and habitual systems. Over the past decade, the two-stage Markov decision task has become popularized for its ability to dissociate between goal-directed and habitual decision-making. While a handful of studies have implemented decision-making tasks online, only one study has validated the task by comparing in-person and web-based performance on the two-stage task in children and young adults. To date, no study has validated the dissociation of goal-directed and habitual behaviors in older adults online. Here, we implemented and validated a web-based version of the two-stage Markov task using parameter simulation and recovery and compared behavioral results from online and in-person participation on the two-stage task in both young and healthy older adults. We found no differences in estimated free parameters between online and in-person participation on the two-stage task. Further, we replicate previous findings that young adults are more goal-directed than older adults both in-person and online. Overall, this work demonstrates that the implementation and use of the two-stage Markov decision task for remote participation is feasible in the older adult demographic, which would allow for the study of decision-making with larger and more diverse samples.

Learning-Driven Interference-Aware Workload Parallelization for Streaming Applications in Heterogeneous Cluster

In the past few years, with the rapid development of CPU-GPU heterogeneous computing, the issue of task scheduling in the heterogeneous cluster has attracted a great deal of attention. This problem becomes more challenging with the need for efficient co-execution of tasks on the GPUs. However, the uncertainty of heterogeneous cluster and the interference caused by resource contention among co-executing tasks can lead to the unbalanced use of computing resource and further cause the degradation in performance of computing platform. In this article, we propose a two-stage task scheduling approach for streaming applications based on deep reinforcement learning and neural collaborative filtering, which considers fine-grained task division and task interference on the GPU. Specifically, the Learning-Driven Workload Parallelization (LDWP) method selects an appropriate execution node for the mutually independent tasks. By using the deep Q-network, the cluster-level scheduling model is online learned to perform the current optimal scheduling actions according to the runtime status of cluster environments and characteristics of tasks. The Interference-Aware Workload Parallelization (IAWP) method assigns subtasks with dependencies to the appropriate computing units, taking into account the interference of subtasks on the GPU by using neural collaborative filtering. For making the learning of neural network more efficient, we use pre-training in the two-stage scheduler. Besides, we use transfer learning technology to efficiently rebuild task scheduling model referring to the existing model. We evaluate our learning-driven and interference-aware task scheduling approach on a prototype platform with other widely used methods. The experimental results show that the proposed strategy can averagely improve the throughout for distributed computing system by 26.9 percent and improve the GPU resource utilization by around 14.7 percent.

Formulation of Two-Stage Problem of Structural-Parametric Synthesis of Adaptive Electronic Document Management System

The paper scrutinizers the options of optimization and adaption to the individual characteristics of users of Electronic Document Management Systems (EDMS). The problem solution requires further development of the necessary methods, models and criteria. Therefore, the article considers the problem of the structural-parametric synthesis of adaptive EDMS. On the basis of previously conducted research in this area, a new architecture of EDMS is developed, within which a mathematical model of adaptive EDMS is proposed. It includes the main components of the information system, as well as a set of estimates for a number of indicators: total discounted costs, productivity, software quality and, above all, adaptability to the requirements of the user. Using this mathematical model, a two-stage task of structural-parametric synthesis of an adaptive EDMS was set, at the first stage of which the system is synthesized according to the criterion of economic efficiency, and at the second stage the process happens according to its adaptation to each user. The scientific novelty of the proposed approach consists in dividing the optimization task into two stages such as the formalization of the criteria for adaptive EDMS and development of a new architecture and mathematical model of adaptive EDMS. The results can be used to solve problems of design, modernization and adaptation of various information systems

Dynamic pricing in profit-driven task assignment: a domain-of-influence based approach

The development of mobile Internet and sharing economy brings the prosperity of spatial crowdsourcing (SC). Pricing is a crucial step for SC platforms to solve the Profit-driven Task Assignment (PTA) problem to maximize their total profit. However, dynamic pricing is still large unexplored in PTA. In addition, existing works seek solutions without considering the uncertainty of workers’ acceptance for assigned tasks in the task assignment process. To deal with these challenges, we develop a two-stage task assignment framework with dynamic pricing. Specifically, we propose a novel Domain-of-Influence based dynamic pricing algorithm, which can iteratively figure out the price that represents the balance between task demand and worker supply. Then we employ hyperbolic temporal discounting function to estimate the worker’s psychological reward that indicates the acceptance or rejection of assigned task. With considering the driver’s psychological reward, we adopt an optimal algorithm to achieve the optimal task assignment and propose greedy algorithms to improve the computational efficiency. Finally, we evaluate the performance using two road network datasets of Jinan and Luoyang in China. The experimental results show the effectiveness and efficiency of our proposed approaches.

TMaR: a two-stage MapReduce scheduler for heterogeneous environments

In the context of MapReduce task scheduling, many algorithms mainly focus on the scheduling of Reduce tasks with the assumption that scheduling of Map tasks is already done. However, in the cloud deployments of MapReduce, the input data is located on remote storage which indicates the importance of the scheduling of Map tasks as well. In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR. TMaR schedules Map and Reduce tasks on the servers that minimize the task finish time in each stage, respectively. We employ a dynamic partition binder for Reduce tasks in the Reduce stage to lighten the shuffling traffic. Indeed, TMaR minimizes the makespan of a batch of tasks in heterogeneous environments while considering the network traffic. The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount, Sort, and Grep benchmarks. Besides, the power reduction of TMaR is up to 12%.

On the Asynchrony of Computation Offloading in Multi-User MEC Systems

Mobile edge computing (MEC) can extend the computing capability of mobile users (MUs), by offloading computational tasks from MUs to proximal MEC servers. Specifically, offloaded tasks are firstly transmitted to access points (APs) via wireless channels, and then relayed to MEC servers through backhauls. Due to differentiated wireless channels and input data sizes, offloaded tasks from different MUs arrive at the AP and MEC server in an asynchronous manner. Considering such asynchrony, we design sequential execution model for backhaul transmission and task processing in the MEC server and present its merits in terms of maximum task completion time for different backhaul configurations: 1) With ideal backhaul, the optimal task scheduling order in the MEC server follows the first coming and first scheduling rule; 2) With limited backhaul, an effective two-stage task scheduling order on the backhaul and MEC server is determined by the Johnson’s rule. Further, we make suboptimal decisions on wireless resource provisioning and task offloading, with respect to the attained task scheduling order and derived necessary and sufficient conditions under which full local processing and full offloading are optimal. Finally, simulation results demonstrate the superiority of our devised sequential execution model and computation offloading mechanisms.

The influence of internal models on feedback-related brain activity

Decision making relies on the interplay between two distinct learning mechanisms, namely habitual model-free learning and goal-directed model-based learning. Recent literature suggests that this interplay is significantly shaped by the environmental structure as represented by an internal model. We employed a modified two-stage but one-decision Markov decision task to investigate how two internal models differing in the predictability of stage transitions influence the neural correlates of feedback processing. Our results demonstrate that fronto-central theta and the feedback-related negativity (FRN), two correlates of reward prediction errors in the medial frontal cortex, are independent of the internal representations of the environmental structure. In contrast, centro-parietal delta and the P3, two correlates possibly reflecting feedback evaluation in working memory, were highly susceptible to the underlying internal model. Model-based analyses of single-trial activity showed a comparable pattern, indicating that while the computation of unsigned reward prediction errors is represented by theta and the FRN irrespective of the internal models, the P3 adapts to the internal representation of an environment. Our findings further substantiate the assumption that the feedback-locked components under investigation reflect distinct mechanisms of feedback processing and that different internal models selectively influence these mechanisms.

Humans primarily use model-based inference in the two-stage task.

Distinct model-free and model-based learning processes are thought to drive both typical and dysfunctional behaviours. Data from two-stage decision tasks have seemingly shown that human behaviour is driven by both processes operating in parallel. However, in this study, we show that more detailed task instructions lead participants to make primarily model-based choices that have little, if any, simple model-free influence. We also demonstrate that behaviour in the two-stage task may falsely appear to be driven by a combination of simple model-free and model-based learning if purely model-based agents form inaccurate models of the task because of misconceptions. Furthermore, we report evidence that many participants do misconceive the task in important ways. Overall, we argue that humans formulate a wide variety of learning models. Consequently, the simple dichotomy of model-free versus model-based learning is inadequate to explain behaviour in the two-stage task and connections between reward learning, habit formation and compulsivity.

Trait impulsivity and acute stress interact to influence choice and decision speed during multi-stage decision-making

Impulsivity and stress exposure are two factors that are associated with changes in reward-related behavior in ways that are relevant to both healthy and maladaptive decision-making. Nonetheless, little empirical work has examined the possible independent and joint effects of these factors upon reward learning. Here, we sought to examine how trait impulsivity and acute stress exposure affect participants’ choice behavior and decision speed in a two-stage sequential reinforcement-learning task. We found that more impulsive participants were more likely to repeat second-stage choices after previous reward, irrespective of stress condition. Exposure to stress, on the other hand, was associated with an increased tendency to repeat second-stage choices independent of whether these choices previously led to a reward, and this tendency was exacerbated in more impulsive individuals. Such interaction effects between stress and impulsivity were also found on decision speed. Stress and impulsivity levels interacted to drive faster choices overall (again irrespective of reward) at both task stages, while reward received on the previous trial slowed subsequent first-stage choices, particularly among impulsive individuals under stress. Collectively, our results reveal novel, largely interactive effects of trait impulsivity and stress exposure and suggest that stress may reveal individual differences in decision-making tied to impulsivity that are not readily apparent in the absence of stress.

LCBPA: two-stage task allocation algorithm for high-dimension data collecting in mobile crowd sensing network

Mobile crowd sensing (MCS) is a novel emerging paradigm that leverages sensor-equipped smart mobile terminals (e.g., smartphones, tablets, and intelligent wearable devices) to collect information. Compared with traditional data collection methods, such as construct wireless sensor network infrastructures, MCS has advantages of lower data collection costs, easier system maintenance, and better scalability. However, the limited capabilities make a mobile crowd terminal only support limited data types, which may result in a failure of supporting high-dimension data collection tasks. This paper proposed a task allocation algorithm to solve the problem of high-dimensional data collection in mobile crowd sensing network. The low-cost and balance-participating algorithm (LCBPA) aims to reduce the data collection cost and improve the equality of node participation by trading-off between them. The LCBPA performs in two stages: in the first stage, it divides the high-dimensional data into fine-grained and smaller dimensional data, that is, dividing an m-dimension data collection task into k sub-task by K-means, where (k < m). In the second stage, it assigns different nodes with different sensing capability to perform sub-tasks. Simulation results show that the proposed method can improve the task completion ratio, minimizing the cost of data collection.

Reliable Task Offloading for Vehicular Fog Computing Under Information Asymmetry and Information Uncertainty

Vehicular fog computing has emerged as a cost-efficient solution for task processing in vehicular networks. However, how to realize effective server recruitment and reliable task offloading under information asymmetry and uncertainty remains a critical challenge. In this paper, we adopt a two-stage task offloading framework to address this challenge. First, we propose a convex–concave-procedure-based contract optimization algorithm for server recruitment, which aims to maximize the expected utility of the operator with asymmetric information. Then, a low-complexity and stable task offloading mechanism is proposed to minimize the total network delay based on the pricing-based matching. Furthermore, we extend the work to the scenario of information uncertainty and develop a matching-learning-based task offloading mechanism, which takes both occurrence awareness and conflict awareness into consideration. Simulation results demonstrate that the proposed algorithm can effectively motivate resource sharing and guarantee bounded deviation from the optimal performance without the global information.

Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling.

A well-established notion in cognitive neuroscience proposes that multiple brain systems contribute to choice behaviour. These include: (1) a model-free system that uses values cached from the outcome history of alternative actions, and (2) a model-based system that considers action outcomes and the transition structure of the environment. The widespread use of this distinction, across a range of applications, renders it important to index their distinct influences with high reliability. Here we consider the two-stage task, widely considered as a gold standard measure for the contribution of model-based and model-free systems to human choice. We tested the internal/temporal stability of measures from this task, including those estimated via an established computational model, as well as an extended model using drift-diffusion. Drift-diffusion modeling suggested that both choice in the first stage, and RTs in the second stage, are directly affected by a model-based/free trade-off parameter. Both parameter recovery and the stability of model-based estimates were poor but improved substantially when both choice and RT were used (compared to choice only), and when more trials (than conventionally used in research practice) were included in our analysis. The findings have implications for interpretation of past and future studies based on the use of the two-stage task, as well as for characterising the contribution of model-based processes to choice behaviour.

Resource-Aware Task Scheduling and Placement in Multi-FPGA System

With the development of high performance computing, Field Programmable Gate Arrays (FPGAs) are widely used for task acceleration. Partial reconfigurable technology provides the possibility of multi-task concurrent computation on FPGAs. However, it is still difficult to achieve high performance multi-task acceleration in multi-FPGA systems due to the resource contention in limited hardware resources and reconfiguration overhead. In this paper, we propose a two-stage task scheduling approach in multi-FPGA systems to optimize task execution efficiency. At the first scheduling stage, we select an appropriate computing unit for each task considering the subtask similarity and resource requirement similarity to reduce the possibility of reconfiguration and resource contention. At the second scheduling stage, we coordinate the subtask scheduling and placement to make full use of the hardware resources of FPGAs and improve the acceleration performance. Experiments show that our two-stage scheduling approach significantly reduces the tasks makespan and improves the utilization of resources compared with other traditional methods.

A tree ensemble-based two-stage model for advanced-stage colorectal cancer survival prediction

Classification techniques have widely been applied to cancer survival prediction for predicting survival or death of patients. However, little attention has been paid to patients who are predicted to die. In this work, we consider survival prediction as a two-stage task, where the first stage is to predict whether the outcome is survival or not, and the second stage is to predict the remaining lifespan for patients whose predicted outcome is death. To this end, we propose a two-stage machine learning model to enhance cancer survival prediction. At the first stage, a tree-based imbalanced ensemble classification method is proposed for classification of the survivability of advanced-stage cancer patients. At the second stage, a selective ensemble regression method is proposed for survival time prediction, where a priori knowledge is adopted for feature selection and the mean proportion of error interval is proposed for selecting base learners. Extensive computational studies performed on colorectal cancer data from SEER database demonstrate that the proposed two-stage model can achieve a more accurate prediction compared to the one-stage regression model. The results show that the proposed classification approach can effectively handle the imbalanced survivability data, and the proposed regression method outperforms several state-of-the-art regression models.

A note on the analysis of two-stage task results: How changes in task structure affect what model-free and model-based strategies predict about the effects of reward and transition on the stay probability.

Many studies that aim to detect model-free and model-based influences on behavior employ two-stage behavioral tasks of the type pioneered by Daw and colleagues in 2011. Such studies commonly modify existing two-stage decision paradigms in order to better address a given hypothesis, which is an important means of scientific progress. It is, however, critical to fully appreciate the impact of any modified or novel experimental design features on the expected results. Here, we use two concrete examples to demonstrate that relatively small changes in the two-stage task design can substantially change the pattern of actions taken by model-free and model-based agents as a function of the reward outcomes and transitions on previous trials. In the first, we show that, under specific conditions, purely model-free agents will produce the reward by transition interactions typically thought to characterize model-based behavior on a two-stage task. The second example shows that model-based agents’ behavior is driven by a main effect of transition-type in addition to the canonical reward by transition interaction whenever the reward probabilities of the final states do not sum to one. Together, these examples emphasize the task-dependence of model-free and model-based behavior and highlight the benefits of using computer simulations to determine what pattern of results to expect from both model-free and model-based agents performing a given two-stage decision task in order to design choice paradigms and analysis strategies best suited to the current question.

Global Cue Inconsistency Diminishes Learning of Cue Validity.

In daily life, we make decisions that are associated with probabilistic outcomes (e.g., the chance of rain today). People search for and utilize information that validly predicts an outcome (i.e., we look for dark clouds to indicate the possibility of rain). In the current study (N = 107), we present a two-stage learning task that examines how participants learn and utilize predictive information within a probabilistic learning environment. In the first stage, participants select one of three cues that gives predictive information about the outcome of the second stage. Participants then use this information to predict the outcome in stage two, for which they receive feedback. Critically, only one of the three cues in stage one gives valid predictive information about the outcome in stage two. Participants must differentiate the valid from non-valid cues and select this cue on subsequent trials in order to inform their prediction of the outcome in stage two. The validity of this predictive information, however, is reinforced with varying levels of probabilistic feedback (i.e., 75, 85, 95, 100%). A second manipulation involved changing the consistency of the predictive information in stage one and the outcome in stage two. The results show that participants, with higher levels of probabilistic feedback, learned to utilize the valid cue. In inconsistent task conditions, however, participants were significantly less successful in utilizing higher validity cues. We interpret this result as implying that learning in probabilistic categorization is based on developing a representation of the task that allows for goal-directed action.

The Path of Least Resistance: Intertemporal Choice and its Relationship to Choices, Preferences, and Beliefs

The degree to which an individual prefers small immediate versus larger delayed rewards serves as a powerful predictor of their impulsivity towards a number of different kinds of rewards. Here we test the limits of its predictive ability within a variety of cognitive and social domains. Across several large samples of human subjects we find a preference for the path of least resistance in the choices, preferences, and beliefs of individuals who prefer smaller more immediate rewards (steeper discounters). First, steeper discounters used more automatic/less controlled choice strategies, giving more intuitive but incorrect responses on the Cognitive Reflection Test (replicating previous findings); employing a suboptimal probability matching heuristic for a one-shot gamble (rather than maximizing their probability of reward); and relying less on optimal planning in a two-stage reinforcement learning task. Second, steeper discounters preferred to consume information that was less complex and multi-faceted, as suggested by their self-reported Need for Cognitive Closure, their use of short-form social media (i.e., Twitter), and their preferred news sources (in particular, whether or not they preferred National Public Radio over other news sources). Third, steeper discounters had interpersonal and religious beliefs that are associated with reduced epistemic complexity: they were more likely to believe that the behavior of others could be explained by fixed rather than dynamic factors, and they held stronger beliefs in God and in the afterlife. Together these findings provide evidence for a link between individual differences in temporal discounting for monetary rewards and preferences for the path of least resistance (e.g., more automatic modes of processing) across a variety of domains. In so doing, they point toward a more unified account of cognitive style and its influence on judgments and decisions related to academic, occupational, and social success.

Extension of mental preparation positively affects motor imagery as compared to motor execution: A functional near-infrared spectroscopy study

Motor imagery (MI) is widely used to study cognitive action control. Although, the neural simulation theory assumes that MI and motor execution (ME) share many common features, the extent of similarity and whether it spreads into the preparation phase is still under investigation. Here we asked, whether an extension of physiological mental preparation has a comparable effect on MI and ME.Data were recorded using wireless functional near-infrared spectroscopy (fNIRS) in a two-stage task design where subjects were cued with or without preparatory stimuli to either execute or imagine complex sequential thumb-finger tasks.The main finding is that the extended mental preparation has a significant positive effect on oxy-hemoglobin (∆[O2Hb]) in response to MI, which is proportionally larger as that found in response to ME. Furthermore, fNIRS was capable to discriminate within each task whether it was preceded by preparatory stimuli or not. Transition from mental preparation to actual performance (ME or MI) was reflected by a dip of the fNIRS signal presumably related to underlying cortical processes changing between preparation and task performance. Statistically significant main effects of ‘Preparation’ and ‘Task’ showed that ∆[O2Hb] during preparation was preparation-specific, i.e., positively affected by the presence of preparatory stimuli, whereas during task performance ∆[O2Hb] was both preparation- and task-specific, i.e., additionally affected by the task mode.These results are particularly appealing from a practical point of view for making use of MI in neuroscientific applications. Especially neurorehabilitation and neural interfaces may benefit from utilizing positive interactions between mental preparation and MI performance.

Leakage-aware task scheduling for partially dynamically reconfigurable FPGAs

As technology continues to shrink, reducing leakage power of Field-Programmable Gate Arrays (FPGAs) becomes a critical issue for the practical use of FPGAs. In this article, we address the leakage issue of partially dynamically reconfigurable FPGA architectures with sleep transistors embedded into FPGA fabrics. In particular, we focus on eliminating leakage waste due to the delay between reconfiguration and execution time of a task. For partially dynamically reconfigurable FPGAs, the configuration prefetching technique is commonly used to hide runtime reconfiguration overhead. With prefetching, the configuration of a task is loaded into FPGAs as early as possible. Therefore, there is often a delay between reconfiguration and execution time of a task. In this period of time, the SRAM cells allocated to a task cannot be turned off even though they are not utilized. In this article, we propose a two-stage task scheduling methodology to reduce leakage waste due to the delay between reconfiguration and execution time of a task without sacrificing performance. In the first stage, a performance-driven task scheduler that targets at minimizing the schedule length is invoked to generate an initial placement. In the second stage, a postplacement leakage-aware task scheduling is applied to refine the initial placement such that leakage waste is minimized provided that the schedule length is not increased. To solve the postplacement leakage optimization problem, we propose two algorithms. The first one is an optimal algorithm based on Integer Linear Programming (ILP). The second algorithm is a heuristic approach that iteratively refines the placement to reduce leakage waste. Experimental results on real and synthetic designs show that the efficiency and effectiveness of the proposed postplacement leakage reduction techniques.