Discrete Task Research Articles

Learning Diverse Policies with Soft Self-Generated Guidance

Reinforcement learning (RL) with sparse and deceptive rewards is a significant challenge because nonzero rewards are rarely obtained, and hence, the gradient calculated by the agent can be stochastic and without valid information. Recent work demonstrates that using memory buffers of previous experiences can lead to a more efficient learning process. However, existing methods usually require these experiences to be successful and may overly exploit them, which can cause the agent to adopt suboptimal behaviors. This study develops an approach that exploits diverse past trajectories for faster and more efficient online RL, even if these trajectories are suboptimal or not highly rewarded. The proposed algorithm merges a policy improvement step with an additional policy exploration step by using offline demonstration data. The main contribution of this study is that by regarding diverse past trajectories as guidance, instead of imitating them, our method directs its policy to follow and expand past trajectories, while still being able to learn without rewards and gradually approach optimality. Furthermore, a novel diversity measurement is introduced to maintain the diversity of the team and regulate exploration. The proposed algorithm is evaluated on a series of discrete and continuous control tasks with sparse and deceptive rewards. In comparison with the existing RL methods, the experimental results indicate that our proposed algorithm is significantly better than the baseline methods in terms of diverse exploration and avoiding local optima.

Beyond technical mastery: inequality in doctoral research skill development in the biological sciences

PurposeThe purpose of this study is to examine how doctoral students in the biological sciences understand their research skill development and explore potential racial/ethnic and gender inequalities in the scientific learning process.Design/methodology/approachBased on interviews with 87 doctoral students in the biological sciences, this study explores how doctoral students describe development of their research skills. More specifically, a constructivist grounded theory approach is employed to understand how doctoral students make meaning of their research skill development process and how that may vary by gender and race/ethnicity.FindingsThe findings reveal two emergent groups, “technicians” who focus on discrete tasks and data collection, and “interpreters” who combine technical expertise with attention to the larger scientific field. Although both groups are developing important skills, “interpreters” have a broader range of skills that support successful scholarly careers in science. Notably, white men are overrepresented among the “interpreters,” whereas white women and students from minoritized racial/ethnic groups are concentrated among the “technicians.”Originality/valueWhile prior literature provides valuable insights into the inequalities across various aspects of doctoral socialization, scholars have rarely attended to examining inequalities in research skill development. This study provides new insights into the process of scientific learning in graduate school. Findings reveal that research skill development is not a uniform experience, and that doctoral education fosters different kinds of learning that vary by gender and race/ethnicity.

Adjacency Constraint for Efficient Hierarchical Reinforcement Learning.

Goal-conditioned Hierarchical Reinforcement Learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is large. Searching in a large goal space poses difficulty for both high-level subgoal generation and low-level policy learning. In this article, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a k-step adjacent region of the current state using an adjacency constraint. We theoretically prove that in a deterministic Markov Decision Process (MDP), the proposed adjacency constraint preserves the optimal hierarchical policy, while in a stochastic MDP the adjacency constraint induces a bounded state-value suboptimality determined by the MDP's transition structure. We further show that this constraint can be practically implemented by training an adjacency network that can discriminate between adjacent and non-adjacent subgoals. Experimental results on discrete and continuous control tasks including challenging simulated robot locomotion and manipulation tasks show that incorporating the adjacency constraint significantly boosts the performance of state-of-the-art goal-conditioned HRL approaches.

Learning to Search in Task and Motion Planning With Streams

Task and motion planning problems in robotics combine symbolic planning over discrete task variables with motion optimization over continuous state and action variables. Recent works such as PDDLStream [1]garrett2020pddlstream have focused on optimistic planning with an incrementally growing set of objects until a feasible trajectory is found. However, this set is exhaustively expanded in a breadth-first manner, regardless of the logical and geometric structure of the problem at hand, which makes long-horizon reasoning with large numbers of objects prohibitively time-consuming. To address this issue, we propose a geometrically informed symbolic planner that expands the set of objects and facts in a best-first manner, prioritized by a Graph Neural Network that is learned from prior search computations. We evaluate our approach on a diverse set of problems and demonstrate an improved ability to plan in difficult scenarios. We also apply our algorithm on a 7DOF robotic arm in block-stacking manipulation tasks.

A unified theory of discrete and continuous responding.

Understanding the cognitive processes underlying choice requires theories that can disentangle the representation of stimuli from the processes that map these representations onto observed responses. We develop a dynamic theory of how stimuli are mapped onto discrete (choice) and onto continuous response scales. It proposes that the mapping from a stimulus to an internal representation and then to an evidence accumulation process is accomplished using multiple reference points or "anchors." Evidence is accumulated until a threshold amount for a particular response is obtained, with the relative balance of support for each anchor at that time determining the response. We tested this multiple anchored accumulation theory (MAAT) using the results of two experiments requiring discrete or continuous responses to line length and color stimuli. We manipulated the number of options for discrete responses, the number of different stimuli, and the similarity among them, and compared the outcomes to continuous response conditions. We show that MAAT accounts for several key phenomena: more accurate, faster, and more skewed distributions of responses near the ends of a response scale; lower accuracy and slower responses as the number of discrete choice options increases; and longer response times and lower accuracy when alternative responses are more similar to the target response. Our empirical and modeling results suggest that discrete and continuous response tasks can share a common evidence representation, and that the decision process is sensitive to the perceived similarity among the response options. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Differentiating Reasons for Young Adult E-cigarette Use Using Maximum Difference Choice Models.

Understanding the reasons young adults use e-cigarettes (ie, vape)-and whether these motivations vary across groups-is essential for informing tobacco regulatory efforts. An online panel of young adults who vape (n = 230; age = 18-30 years) completed a maximum difference discrete choice task for 15 reasons for vaping. Over 9 choice sets, participants were presented a subset of 5 reasons and selected the most and least important. Hierarchical bayesian analysis estimated the relative importance of each reason. Latent class analysis (LCA) identified groups with similarly ranked reasons for use. Multinomial regression evaluated the association between sample characteristics and class membership. Overall, relaxation had the highest probability of being the most important reason for use (14.8%), followed by harm reduction (13.2%), and flavors (10.3%). LCA identified five distinct classes, based on top reasons for use: 1. cessation (cigarette cessation [20.2%]; n = 80); 2. dependence (relaxation [20.5%] and unable to quit [19.2%]; n = 21); 3. relaxation (relaxation [20.6%]; n = 66); 4. socializing (socializing [22.2%]; n = 27); and 5. variable (boredom [10.5%] and acceptability [10.2%]; n = 36). Participants who were older, smoked cigarettes, or vaped more frequently were more likely to belong to the cessation class while those who were younger or more e-cigarette dependent were more likely to belong to the dependence class. Perceived reasons why young adults vape are highly heterogeneous and dependent on the type of user. Tobacco regulatory efforts targeting distinct types of vapers are needed to minimize the adverse public health impact of vaping without compromising appeal for smoking cessation. E-cigarette use remains high among young adults, with flavors, cost, and harm reduction (vs. combustible cigarettes) among the mostly commonly reported reasons for use. Yet, little is known about how relatively important these reasons are to the individual. Leveraging a maximum difference task, young adults' reasons for use were evaluated on a common interval scale and groups sharing similar reasons identified. Smoking cessation, dependence, relaxation, socialization, and boredom were respectively the most important reasons for use among five classes of vapers. E-cigarette regulatory policies should consider the distinct reasons for use as to not compromise their appeal for smoking cessation.

Temporal relative transformer encoding cooperating with channel attention for EEG emotion analysis

Electroencephalogram (EEG)-based emotion computing has become a hot topic of brain-computer fusion. EEG signals have inherent temporal and spatial characteristics. However, existing studies did not fully consider the two properties. In addition, the position encoding mechanism in the vanilla transformer cannot effectively encode the continuous temporal character of the emotion. A temporal relative (TR) encoding mechanism is proposed to encode the temporal EEG signals for constructing the temporality self-attention in the transformer. To explore the contribution of each EEG channel corresponding to the electrode on the cerebral cortex to emotion analysis, a channel-attention (CA) mechanism is presented. The temporality self-attention mechanism cooperates with the channel-attention mechanism to utilize the temporal and spatial information of EEG signals simultaneously by preprocessing. Exhaustive experiments are conducted on the DEAP dataset, including the binary classification on valence, arousal, dominance, and liking. Furthermore, the discrete emotion category classification task is also conducted by mapping the dimensional annotations of DEAP into discrete emotion categories (5-class). Experimental results demonstrate that our model outperforms the advanced methods for all classification tasks.

Anticipatory Classifier System With Episode-Based Experience Replay

Deep reinforcement learning with Experience Replay (ER), including Deep Q-Network (DQN), has been used to solve many multi-step learning problems. However, in practice, DQN algorithms need better explainability, which limits their applicability in many scenarios. While we can consider DQN as a black-box model, the Learning Classifier Systems (LCSs), including anticipatory versions, also solve multi-step problems, but their operation is subject to interpretation. It seems promising to combine the properties of these two learning approaches. The paper describes an attempt to design and evaluate modification to the Experience Replay extension of the anticipatory classifier system ACS2. The modification is named Episode-based Experience Replay (EER), and its main premise is to replay entire episodes instead of single experience samples. Promising results affirmed by Bayes estimations are obtained on multi-step problems, albeit limited to deterministic and discrete tasks. The experimental results show that the EER extension significantly improves the ACS2’s learning capabilities.

Modeling and Individualizing Continuous Joint Kinematics Using Gaussian Process Enhanced Fourier Series.

Prosthetic discrete controller relies on finite state machines to switch between a set of predefined task-specific controllers. Therefore the prosthesis can only perform a limited number of discrete locomotion tasks and need hours to tune the parameters for each user. In contrast, the continuous controller treats a gait cycle in a unified way. Thus it is expected to better facilitate normative biomechanics by providing a gait predictive model to contribute a non-switching controller that supports a continuum of tasks. Furthermore, a better method is to train a personalized trajectory prediction model suitable for personal characteristics according to personal walking data. This paper proposes a Gaussian process enhanced Fourier series (GPEFS) method to construct a gait prediction model that represents the human locomotion as a continuous function of phase, speed and slope. Firstly the joint trajectories are transformed into the Fourier coefficient space by least square method. Then the relationship between each Fourier coefficient and task input can be learned by multiple Gaussian process regression (GPRs) model respectively. Compared with directly using GPR to fit the joint trajectory under multi task, our method greatly reduces the computational burden, so as to meet the real-time application scenario. In addition, in Fourier coefficient space, the difference in all tasks between the Fourier coefficient of personal data and the one of statistical data follows the same trend. Therefore, a personalized prediction model is built to predict an individual's kinematics over a continuous range of slopes and speeds given only one personalized task at level ground and normal speed. The experimental results show that the gait prediction model and the personalized prediction model are feasible and effective.

Brain Activation Differences of Six Basic Emotions Between 2D Screen and Virtual Reality Modalities.

To our knowledge, it has been widely studied in Screen-2D modality for the six basic emotions proposed by Professor Paul Ekman, but there are only studies on their positive and negative valence in VR-3D modality. In this study, we will investigate whether the six basic emotions have stronger brain activation states in VR-3D modality than in Screen-2D modality. We designed an emotion-inducing experiment with six basic emotions (happiness, surprise, sadness, fear, anger, and disgust) to record the electroencephalogram (EEG) signals during watching VR-3D and Screen-2D videos. The power spectral density (PSD) was calculated to compare the brain activation differences between VR-3D and Screen-2D modalities during the induction of the six basic emotions. The results of statistical analysis of the relative power differences between VR-3D and Screen-2D modalities for each emotion revealed that both happiness and surprise presented greater differences in the α and γ frequency bands, while sad, fear, disgust and anger all presented greater differences in the α and θ frequency bands, which are mainly observed in the frontal and occipital regions. On the other hand, the six emotions all yielded satisfactory classification accuracy (above 85%) by classification from a subset of power feature of the brain activation states in the same emotion between the two modalities. Overall, there are significant differences in the induction of same discrete emotions in VR-3D and Screen-2D modalities, with greater brain activation in VR-3D modalities. These findings provide a better understanding about the neural activity of discrete emotional tasks assessed in VR environments.

What Does It Mean to Measure a Smile? Assigning numerical values to emotions

This article looks at the implications of emotion recognition, zooming in on the specific case of the care robot Pepper introduced at a hospital in Toronto. Here, emotion recognition comes with the promise of equipping robots with a less tangible, more emotive set of skills – from companionship to encouragement. Through close analysis of a variety of materials related to emotion detection software – iMotions – we look into two aspects of the technology. First, we investigate the how of emotion detection: what does it mean to detect emotions in practice? Second, we reflect on the question of whose emotions are measured, and what the use of care robots can say about the norms and values shaping care practices today. We argue that care robots and emotion detection can be understood as part of a fragmentation of care work: a process in which care is increasingly being understood as a series of discrete tasks rather than as holistic practice. Finally, we draw attention to the multitude of actors whose needs are addressed by Pepper, even while it is being imagined as a care provider for patients.

Fresher Experience Plays a More Important Role in Prioritized Experience Replay

Prioritized experience replay (PER) is an important technique in deep reinforcement learning (DRL). It improves the sampling efficiency of data in various DRL algorithms and achieves great performance. PER uses temporal difference error (TD-error) to measure the value of experiences and adjusts the sampling probability of experiences. Although PER can sample valuable experiences according to the TD-error, freshness is also an important character of experiences. It implicitly reflects the potential value of experiences. Fresh experiences are produced by virtue of the current networks and they are more valuable for updating the current network parameters than the past. The sampling of fresh experiences to train the neural networks can increase the learning speed of the agent, but few algorithms can perform this job efficiently. To solve this issue, a novel experience replay method is proposed in this paper. We first define that the experience freshness is negatively correlated with the number of replays. A new hyper-parameter, the freshness discounted factor μ, is introduced in PER to measure the experience freshness. Further, a novel experience replacement strategy in the replay buffer is proposed to increase the experience replacement efficiency. In our method, the sampling probability of fresh experiences is increased by raising its priority properly. So the algorithm is more likely to choose fresh experiences to train the neural networks during the learning process. We evaluated this method in both discrete control tasks and continuous control tasks via OpenAI Gym. The experimental results show that our method achieves better performance in both modes of operation.

Cognitive load in asynchronous discussions of an online undergraduate STEM course

PurposeAs online course enrollments increase, it is important to understand how common course features influence students' behaviors and performance. Asynchronous online courses often include a discussion forum to promote community through interaction between students and instructors. Students interact both socially and cognitively; instructors' engagement often demonstrates social or teaching presence. Students' engagement in the discussions introduces both intrinsic and extraneous cognitive load. The purpose of this study is to validate an instrument for measuring cognitive load in asynchronous online discussions.Design/methodology/approachThis study presents the validation of the NASA-TLX instrument for measuring cognitive load in asynchronous online discussions in an introductory physics course.FindingsThe instrument demonstrated reliability for a model with four subscales for all five discrete tasks. This study is foundational for future work that aims at testing the efficacy of interventions, and reducing extraneous cognitive load in asynchronous online discussions.Research limitations/implicationsNonresponse error due to the unincentivized, voluntary nature of the survey introduces a sample-related limitation.Practical implicationsThis study provides a strong foundation for future research focused on testing the effects of interventions aimed at reducing extraneous cognitive load in asynchronous online discussions.Originality/valueThis is a novel application of the NASA-TLX instrument for measuring cognitive load in asynchronous online discussions.

Balanced task allocation and collision-free scheduling of multi-robot systems in large spacecraft structure manufacturing

The use of multiple cooperating industrial robots provides efficient and flexible solutions to the manufacturing of complex aerospace structures. Such applications require the workloads to be sufficiently shared between neighboring robots, this entails the collision-free scheduling of many discrete tasks, where precedence orders need to be assigned for specific tasks. In this paper, we first present a two-step task allocation method that handles workload balancing, then a scheduling algorithm combining construction heuristic with iterated local search to provide efficient schedules. Our key innovation is a collision model that encodes precedence constraints and a fast heuristic that constructs collision-free schedule under given constraints, the optimization of the schedule is then addressed by an iterated local search. The advantage in terms of minimizing makespan under different problem scales and conditions is validated by computational experiments. Finally, the use of our method is demonstrated by a physical multi-robot system.

Interfinger Synchronization Capability of Paired Fingers in Discrete Fine-Force Control Tasks.

This study examined whether within-a-hand and between-hands finger pairings would exhibit different interfinger synchronization capabilities in discrete fine-force control tasks. Participants were required to perform the designed force control tasks using finger pairings of index and middle fingers on one or two hands. Results demonstrated that the delayed reaction time and the timing difference of paired fingers showed a significant difference among finger pairings. In particular, paired fingers exhibited less delayed reaction time and timing difference in between-hands finger pairings than in within-a-hand finger pairings. Such bimanual advantage of the pairings with two symmetric fingers was evident only in the task types with relatively high amplitudes. However, for a given finger pairing, the asymmetric amplitude configuration, assigning a relatively higher amplitude to either left or right finger of paired fingers, has no significant effect on the interfinger synchronization. Therefore, paired fingers on both hands showed a bimanual advantage in the relatively high force, especially for the pairing of symmetrical fingers, whereas asymmetric amplitude configuration for a finger pairing was able to suppress the bimanual advantage. These findings would enrich the understanding of the interfinger synchronization capability of paired fingers and be referential for interactive engineering applications when leveraging the interfinger synchronization capability in discrete fine-force control tasks.

Mediating Agent Reliability with Human Trust, Situation Awareness, and Performance in Autonomously-Collaborative Human-Agent Teams

When teaming with humans, the reliability of intelligent agents may sporadically change due to failure or environmental constraints. Alternatively, an agent may be more reliable than a human because their performance is less likely to degrade (e.g., due to fatigue). Research often investigates human-agent interactions under little to no time constraints, such as discrete decision-making tasks where the automation is relegated to the role of an assistant. This paper conducts a quantitative investigation towards varying reliability in human-agent teams in a time-pressured continuous pursuit task, and it interconnects individual differences, perceptual factors, and task performance through structural equation modeling. Results indicate that reducing reliability may generate a more effective agent imperceptibly different from a fully reliable agent, while contributing to overall team performance. The mediation analysis shows replication of factors studied in the trust and situation awareness literature while providing new insights: agents with an active stake in the task (i.e., success is dependent on team performance) offset loss of situation awareness, differing from the usual notion of overtrust. We conclude with generalizing implications from an abstract pursuit task, and we highlight challenges when conducting research in time-pressured continuous domains.

Packing-based branch-and-bound for discrete malleable task scheduling

This paper addresses the problem of scheduling chain-like structures of tasks on a single multiprocessor resource. In fact, sub-tasks of unit-time length and predefined size are aggregated to composite tasks that have to be scheduled without preemption, but subject to flexibility concerning resource allocation. This setting most closely resembles the problem of malleable task scheduling, with sub-tasks being the smallest atomic unit of allocation. The specific type of malleability is realized using precedence constraints with minimum and maximum time lags. A bin packing model is established for this scheduling problem and a corresponding, dedicated branch-and-bound algorithm is devised, alongside problem-specific bound tightening, symmetry breaking and dominance concepts. The efficacy of the solution approach is demonstrated based on extensive computational experiments, including randomized instances, adapted benchmark instances from the literature, and a small real-world data set. In comparison to mixed-integer and constraint programming formulations, the new method is able to achieve a considerably higher percentage of optimal solutions at computation times that are up to orders of magnitude smaller.

The effect of enhanced variability after performance stabilization through constant practice.

There is a controversy about the benefits of variable practice on motor learning. This study aimed to analyze the effect of variable practice after the learner stabilized their performance. Thirty-two healthy adults performed a computer-simulated discrete accuracy task in which they had to release a virtual ball to try to hit a target. After a pre-test, the participants were distributed into three training groups: constant practice, variable practice and a group that started practicing in constant conditions, and when their performance stabilized, changed to variable practice. The participants performed 44 sets of 10 repetitions of the task. A post-test and two retention tests were carried out. Variable practice did not show a significant improvement compared to constant practice. Variable practice did not show higher benefits even when performance was stabilized through constant practice, but its effects seem to be modulated by the initial variability of the learners.

Investigation on the effects of presentation modality for spatial signals on human performance using a dual task paradigm

A dual task was designed to involve a tracking mission with various tracking speeds and a spatial compatibility task with various signal-key mappings and presentation modalities. This dual task was used to investigate the effects of workload and resource competition in distinct parts of the dual-task process. The results demonstrated that increasing the tracking speed adversely affected the tracking performance but led to positive arousal to the secondary discrete response task. Visual spatial signals gave the shortest reaction time due to the optimal time-sharing of the visual resources in the focal and ambient channels. Compared with visual spatial signals, spatial signals of auditory and tactile modalities did not lead to an improved performance because of their cross-modality nature. These findings provided practical design guidelines for dual tasks in which the operators need to complete a continuous monitoring task visually and elicit timely and accurate responses to spatial information.

Differentiable Learning of Sequence-Specific Minimizer Schemes with DeepMinimizer.

Minimizers are widely used to sample representative k-mers from biological sequences in many applications, such as read mapping and taxonomy prediction. In most scenarios, having the minimizer scheme select as few k-mer positions as possible (i.e., having a low density) is desirable to reduce computation and memory cost. Despite the growing interest in minimizers, learning an effective scheme with optimal density is still an open question, as it requires solving an apparently challenging discrete optimization problem on the permutation space of k-mer orderings. Most existing schemes are designed to work well in expectation over random sequences, which have limited applicability to many practical tools. On the other hand, several methods have been proposed to construct minimizer schemes for a specific target sequence. These methods, however, only approximate the original objective with likewise discrete surrogate tasks that are not able to significantly improve the density performance. This article introduces the first continuous relaxation of the density minimizing objective, DeepMinimizer, which employs a novel Deep Learning twin architecture to simultaneously ensure both validity and performance of the minimizer scheme. Our surrogate objective is fully differentiable and, therefore, amenable to efficient gradient-based optimization using GPU computing. Finally, we demonstrate that DeepMinimizer discovers minimizer schemes that significantly outperform state-of-the-art constructions on human genomic sequences.