Continuous Task Research Articles

Hypernetworks for Zero-Shot Transfer in Reinforcement Learning

In this paper, hypernetworks are trained to generate behaviors across a range of unseen task conditions, via a novel TD-based training objective and data from a set of near-optimal RL solutions for training tasks. This work relates to meta RL, contextual RL, and transfer learning, with a particular focus on zero-shot performance at test time, enabled by knowledge of the task parameters (also known as context). Our technical approach is based upon viewing each RL algorithm as a mapping from the MDP specifics to the near-optimal value function and policy and seek to approximate it with a hypernetwork that can generate near-optimal value functions and policies, given the parameters of the MDP. We show that, under certain conditions, this mapping can be considered as a supervised learning problem. We empirically evaluate the effectiveness of our method for zero-shot transfer to new reward and transition dynamics on a series of continuous control tasks from DeepMind Control Suite. Our method demonstrates significant improvements over baselines from multitask and meta RL approaches.

Off-Policy Proximal Policy Optimization

Proximal Policy Optimization (PPO) is an important reinforcement learning method, which has achieved great success in sequential decision-making problems. However, PPO faces the issue of sample inefficiency, which is due to the PPO cannot make use of off-policy data. In this paper, we propose an Off-Policy Proximal Policy Optimization method (Off-Policy PPO) that improves the sample efficiency of PPO by utilizing off-policy data. Specifically, we first propose a clipped surrogate objective function that can utilize off-policy data and avoid excessively large policy updates. Next, we theoretically clarify the stability of the optimization process of the proposed surrogate objective by demonstrating the degree of policy update distance is consistent with that in the PPO. We then describe the implementation details of the proposed Off-Policy PPO which iteratively updates policies by optimizing the proposed clipped surrogate objective. Finally, the experimental results on representative continuous control tasks validate that our method outperforms the state-of-the-art methods on most tasks.

Verbal ability in postmenopausal women in relation to age, cognitive and reproductive factors

Word-finding difficulties have been associated with age and, in women, lowered sex hormone levels following menopause. However, there is limited understanding of the ways that specific aspects of word-finding are shaped by women's age, reproductive histories, and background factors such as education. The current study investigated the effects of age, cognitive and reproductive factors on word-finding abilities in 53 healthy postmenopausal women aged 48–79. A questionnaire was used to gather demographic information and reproductive history. A battery of verbal fluency, continuous series, and naming tasks was designed to assess word-finding across different sensory modalities and cognitive demands. Category and letter fluency were quantified as total number of correct words produced on each task. For continuous series, switch rates and switch costs were computed. For the naming tasks, accuracy and latency measures were used. There were three key findings. Firstly, there was a consistent positive association between education and all word-finding measures, i.e., verbal fluency, continuous series, and naming. Secondly, age-related declines were seen on tasks heavily dependent on working memory such as the continuous series task. Thirdly, reproductive factors across the lifespan such as age at menarche and reproductive years showed subtle effects on naming abilities, but not on verbal fluency or continuous series. The results highlight that word-finding abilities in healthy postmenopausal women are shaped by factors associated with their early years (education, age at menarche) and later adult life (age, reproductive years). The study also distinguished between the more global effects of education, and the more task-specific associations with age and reproductive variables, on verbal task performance after menopause.

Age-related decline of online visuomotor adaptation: a combined effect of deteriorations of motor anticipation and execution.

The literature has established that the capability of visuomotor adaptation decreases with aging. However, the underlying mechanisms of this decline are yet to be fully understood. The current study addressed this issue by examining how aging affected visuomotor adaptation in a continuous manual tracking task with delayed visual feedback. To distinguish separate contributions of the declined capability of motor anticipation and deterioration of motor execution to this age-related decline, we recorded and analyzed participants' manual tracking performances and their eye movements during tracking. Twenty-nine older people and twenty-three young adults (control group) participated in this experiment. The results showed that the age-related decline of visuomotor adaptation was strongly linked to degraded performance in predictive pursuit eye movement, indicating that declined capability motor anticipation with aging had critical influences on the age-related decline of visuomotor adaptation. Additionally, deterioration of motor execution, measured by random error after controlling for the lag between target and cursor, was found to have an independent contribution to the decline of visuomotor adaptation. Taking these findings together, we see a picture that the age-related decline of visuomotor adaptation is a joint effect of the declined capability of motor anticipation and the deterioration of motor execution with aging.

Continual Pre-Training of Language Models for Concept Prerequisite Learning with Graph Neural Networks

Prerequisite chains are crucial to acquiring new knowledge efficiently. Many studies have been devoted to automatically identifying the prerequisite relationships between concepts from educational data. Though effective to some extent, these methods have neglected two key factors: most works have failed to utilize domain-related knowledge to enhance pre-trained language models, thus making the textual representation of concepts less effective; they also ignore the fusion of semantic information and structural information formed by existing prerequisites. We propose a two-stage concept prerequisite learning model (TCPL), to integrate the above factors. In the first stage, we designed two continual pre-training tasks for domain-adaptive and task-specific enhancement, to obtain better textual representation. In the second stage, to leverage the complementary effects of the semantic and structural information, we optimized the encoder of the resource–concept graph and the pre-trained language model simultaneously, with hinge loss as an auxiliary training objective. Extensive experiments conducted on three public datasets demonstrated the effectiveness of the proposed approach. Our proposed model improved by 7.9%, 6.7%, 5.6%, and 8.4% on ACC, F1, AP, and AUC on average, compared to the state-of-the-art methods.

Strategy inference during learning via cognitive activity-based credit assignment models

We develop a method for selecting meaningful learning strategies based solely on the behavioral data of a single individual in a learning experiment. We use simple Activity-Credit Assignment algorithms to model the different strategies and couple them with a novel hold-out statistical selection method. Application on rat behavioral data in a continuous T-maze task reveals a particular learning strategy that consists in chunking the paths used by the animal. Neuronal data collected in the dorsomedial striatum confirm this strategy.

Dynamic involvement of premotor and supplementary motor areas in bimanual pinch force control

Many activities of daily living require quick shifts between symmetric and asymmetric bimanual actions. Bimanual motor control has been mostly studied during continuous repetitive tasks, while little research has been carried out in experimental settings requiring dynamic changes in motor output generated by both hands. Here, we performed functional magnetic resonance imaging (MRI) while healthy volunteers performed a visually guided, bimanual pinch force task. This enabled us to map functional activity and connectivity of premotor and motor areas during bimanual pinch force control in different task contexts, requiring mirror-symmetric or inverse-asymmetric changes in discrete pinch force exerted with the right and left hand. The bilateral dorsal premotor cortex showed increased activity and effective coupling to the ipsilateral supplementary motor area (SMA) in the inverse-asymmetric context compared to the mirror-symmetric context of bimanual pinch force control while the SMA showed increased negative coupling to visual areas. Task-related activity of a cluster in the left caudal SMA also scaled positively with the degree of synchronous initiation of bilateral pinch force adjustments, irrespectively of the task context. The results suggest that the dorsal premotor cortex mediates increasing complexity of bimanual coordination by increasing coupling to the SMA while SMA provides feedback about motor actions to the sensory system.

Self-organizing neural network for reproducing human postural mode alternation through deep reinforcement learning

A self-organized phenomenon in postural coordination is essential for understanding the auto-switching mechanism of in-phase and anti-phase postural coordination modes during standing and related supra-postural activities. Previously, a model-based approach was proposed to reproduce such self-organized phenomenon. However, if we set this problem including the process of how we establish the internal predictive model in our central nervous system, the learning process is critical to be considered for establishing a neural network for managing adaptive postural control. Particularly when body characteristics may change due to growth or aging or are initially unknown for infants, a learning capability can improve the hyper-adaptivity of human motor control for maintaining postural stability and saving energy in daily living. This study attempted to generate a self-organizing neural network that can adaptively coordinate the postural mode without assuming a prior body model regarding body dynamics and kinematics. Postural coordination modes are reproduced in head-target tracking tasks through a deep reinforcement learning algorithm. The transitions between the postural coordination types, i.e. in-phase and anti-phase coordination modes, could be reproduced by changing the task condition of the head tracking target, by changing the frequencies of the moving target. These modes are considered emergent phenomena existing in human head tracking tasks. Various evaluation indices, such as correlation, and relative phase of hip and ankle joint, are analyzed to verify the self-organizing neural network performance to produce the postural coordination transition between the in-phase and anti-phase modes. In addition, after learning, the neural network can also adapt to continuous task condition changes and even to unlearned body mass conditions keeping consistent in-phase and anti-phase mode alternation.

Anatomical and functional predictors of disorientation after first-ever brain damage

Background and objectivesDisorientation is a frequent consequence of acute brain injury or diffuse disorders, such as confusional states or dementia. Its anatomical correlates are debated. Impaired memory as its commonly assumed mechanism predicts that disorientation is associated with medial temporal damage. The alternative is that disorientation reflects defective orbitofrontal reality filtering (ORFi) – a specific failure to identify whether thoughts or memories refer to present reality or not. The latter is a function of the posterior orbitofrontal cortex and connected structures. This study examined the mechanisms and anatomical basis of disorientation in an unselected group of patients with first-ever subacute brain injury. MethodsParticipants hospitalized for neurorehabilitation were asked to participate in this observational cohort study if they had first-ever organic hemispheric brain dysfunction as evident in a localizable brain lesion or verbal amnesia (often without localizable brain damage). Orientation to time, place, situation and person was tested with a 20-items questionnaire. To identify the mechanisms of disorientation, we determined its correlations with executive tasks, verbal episodic memory, and ORFi in all patients. ORFi was examined with a continuous recognition task, which measures learning and item recognition in the first run, and ORFi as reflected in the increase of false positive responses in the second run (temporal context confusion). Lesions of patients having localizable brain damage were manually delineated and normalized before entering multivariate lesion-symptom-mapping (LSM) to determine anatomical predictors of orientation. ResultsEighty-four patients (61.1 ± 14.4 years, 29 women) were included. Among measures of memory and executive functioning, a step-wise regression retained temporal context confusion (R = −0.71, p < 0.0001), item recognition (R = 0.67, p < 0.0001) and delayed free recall (R = 0.63, p < 0.0001) as significant predictors of orientation. LSM was possible in 67 participants; it revealed an association of disorientation with damage of the right OFC and the bilateral head of the caudate nucleus. ConclusionDisorientation in non-confused, non-demented patients with first-ever brain damage is associated with impaired orbitofrontal reality filtering and memory dysfunction, but not with executive dysfunction. Its main anatomical determinant is damage to the orbitofrontal cortex and its subcortical relay, the head of the caudate.

Performance-linked visual feedback slows response times during a sustained attention task

In the present study, we tested a visual feedback triggering system based on real-time tracking of response time (RT) in a sustained attention task. In our task, at certain points, brief visual feedback epochs were presented without interrupting the task itself. When these feedback epochs were performance-linked—meaning that they were triggered because participants were responding more quickly than usual—RTs were slowed after the presentation of feedback. However, visual feedback epochs displayed at predetermined times that were independent of participants’ performance did not slow RTs. Results from a second experiment support the idea that this is not simply a return to baseline that would have occurred had the feedback not been presented, but instead suggest that the feedback itself was effective in altering participants’ responses. In a third experiment, we replicated this result across with both written word feedback and visual symbolic feedback, as well as in cases where the participant was explicitly told that the feedback was linked to their performance. All together, these data provide insight into potential mechanisms for detecting and disrupting lapses in sustained attention without interrupting a continuous task.

Distributional reinforcement learning with epistemic and aleatoric uncertainty estimation

Distributional reinforcement learning (RL) differs from conventional RL, which only estimates the expectation of the return. Distributional RL considers the return as a random variable and estimates its distribution. The return distribution can provide more information than its expectation in conventional RL. Thus, distributional RL has been widely studied. However, very few previous works take full advantage of the learned distribution to improve distributional RL. This paper improves distributional RL by introducing epistemic and aleatoric uncertainty estimation. First, an epistemic and aleatoric uncertainty estimation method is introduced using deep ensembles and the learned value distribution. Next, we improve the exploration efficiency of fully parametrized quantile function (FQF) for distributional RL and obtain a FQF-U (uncertainty) algorithm. Then, to overcome the problem that distributional RL cannot operate over continuous control tasks, we propose an epistemic-uncertainty-based distributional soft actor-critic algorithm with an adaptive risk-averse and risk-seeking policy. Finally, experimental results show that our algorithms outperform the baselines in Atari games and Multi-joint dynamics with contact (MuJoCo) environments.

Increased cerebral activity during microsleeps reflects an unconscious drive to re-establish consciousness

BackgroundMicrosleeps are brief instances of sleep, causing complete lapses in responsiveness and partial or total extended closure of both eyes. Microsleeps can have devastating consequences, particularly in the transportation sector. Study objectivesQuestions remain regarding the neural signature and underlying mechanisms of microsleeps. This study aimed to gain a better understanding of the physiological substrates of microsleeps, which might lead to a better understanding of the phenomenon. MethodsData from an earlier study, involving 20 healthy non-sleep-deprived subjects, were analysed. Each session lasted 50 min and required subjects to perform a 2-D continuous visuomotor tracking task. Simultaneous data collection included tracking performance, eye-video, EEG, and fMRI. A human expert visually inspected each participant's tracking performance and eye-video recordings to identify microsleeps. Our interest was in microsleeps of ≥4-s duration, leaving us with a total of 226 events from 10 subjects. The microsleep events were divided into four 2-s segments (pre, start, end, and post) (with a gap in the middle, between start and end segments, for microsleeps >4 s), then each segment was analysed relative to its prior segment by examining changes in source-reconstructed EEG power in the delta, theta, alpha, beta, and gamma bands. ResultsEEG power increased in the theta and alpha bands between the pre and start of microsleeps. There was also increased power in the delta, beta, and gamma bands between the start and end of microsleeps. Conversely, there was a reduction in power between the end and post of microsleeps in the delta and alpha bands. These findings support previous findings in the delta, theta, and alpha bands. However, increased power in the beta and gamma bands has not been previously reported. ConclusionsWe contend that increased high-frequency activity during microsleeps reflects unconscious ‘cognitive’ activity aimed at re-establishing consciousness following falling asleep during an active task.

Real-time task-oriented continuous digging trajectory planning for excavator arms

Current digging trajectory planning methods for excavator arms are limited to a single digging cycle, which does not meet the continuous excavation demands of the task. To address this issue, a real-time task-oriented continuous digging trajectory planning method for autonomous excavators is presented. The method involves optimizing digging trajectory for a single excavation cycle using multi-objective PSO method, building a PINN model using optimization results as training samples for real-time planning, and embedding the PINN model in planning framework for typical tasks. The method was validated using four different cross-section shapes of the trench. Results show that the average time taken to plan a single digging trajectory is less than 4.5ms, which is negligible compared to the time taken by PSO. The overall performance of the trajectory is only about 5% different from those planned by PSO. This method offers a more efficient and effective solution for continuous excavation tasks.

A hybrid safety monitoring framework for industrial FCC disengager coking rate based on FPM, CFD, and ML

This work proposed a hybrid modeling framework for the safety monitoring of coking rates in FCC (Fluidized Catalytic Cracking) disengager. The framework combines CFD (Computational Fluid Dynamics), coking FPM (First Principles Model), machine learning, and industrial data to establish a comprehensive and customizable approach. The FPM of the coking rate, including the UNIFAC condensation ratio model, the coking ratio model and the capture ratio model, are established by combustion experiments, gas-solid two-phase flow simulations and DPM (Discrete Phase Model) simulations. The capture ratio model is built to calculate the capture ratio of heavy oil droplets by each high-risk region, achieved by DPM simulation. By establishing a link between the simulated position and the DCS (Distributed Control System) sensor, the simulation results and the real-time DCS data are matching. An improved LSTM (Long Short-Term Memory) network was then established to predict the real-time temperature and pressure in the high-risk coking regions of the disengager using real-time DCS and LIMS (Laboratory Information Management System) data. The LSTM network can achieve online monitoring of coking rate by coupling FPM. This hybrid coking monitoring framework has been industrially applied and validated with an accuracy of over 90% in a continuous one-month online monitoring task, which is highly interpretable and extensible, providing refining companies with a scalable and customizable approach to monitor coking problems in FCC disengager.

How does online streaming reform e-commerce? An empirical assessment of immersive experience and social interaction in China

The present study examined why live-stream commerce is superior to traditional e-commerce since its retail sales have been booming in China and the communication style with live broadcast technology is entirely different. An integrated model was employed to predict the drivers or restraints of this new business model to substantiate the existence of a lacuna in understanding individual and situational influences. The research described a study that used a combination of information processing theory and flow theory within a Stimulus-Organism-Response (S-O-R) framework to investigate how individual factors and situational factors influence the experience of flow. An integrated model was proposed that stimuli such as individual (i.e., perceived interactivity) and situational influences (i.e., dynamic characteristics and atmosphere clues) can elicit the intrinsic states of organisms (i.e., immersive experience and social interaction), which then result in a response (i.e., online purchase intention). Some shoppers with live-streaming experiences in China were invited to participate in online surveys in 2020. The structural equation modeling results (n = 408) show that atmosphere clues and perceived interactivity of streaming attributes can contribute to both immersive experience and social interaction, predicting consumers’ intention to transact in the new e-commerce. However, the current dynamic/situational infrastructures can only improve the immersive experience but do not assure effective social interaction, leading to the need for continuous tasks in digital development and the directions for future research.

Ensemble and continual federated learning for classification tasks

Federated learning is the state-of-the-art paradigm for training a learning model collaboratively across multiple distributed devices while ensuring data privacy. Under this framework, different algorithms have been developed in recent years and have been successfully applied to real use cases. The vast majority of work in federated learning assumes static datasets and relies on the use of deep neural networks. However, in real-world problems, it is common to have a continual data stream, which may be non-stationary, leading to phenomena such as concept drift. Besides, there are many multi-device applications where other, non-deep strategies are more suitable, due to their simplicity, explainability, or generalizability, among other reasons. In this paper we present Ensemble and Continual Federated Learning, a federated architecture based on ensemble techniques for solving continual classification tasks. We propose the global federated model to be an ensemble, consisting of several independent learners, which are locally trained. Thus, we enable a flexible aggregation of heterogeneous client models, which may differ in size, structure, or even algorithmic family. This ensemble-based approach, together with drift detection and adaptation mechanisms, also allows for continual adaptation in situations where data distribution changes over time. In order to test our proposal and illustrate how it works, we have evaluated it in different tasks related to human activity recognition using smartphones.

Dentate Gyrus Integrity Is Necessary for Behavioral Pattern Separation But Not Statistical Learning.

Pattern separation, the creation of distinct representations of similar inputs, and statistical learning, the rapid extraction of regularities across multiple inputs, have both been linked to hippocampal processing. It has been proposed that there may be functional differentiation within the hippocampus, such that the trisynaptic pathway (entorhinal cortex > dentate gyrus > CA3 > CA1) supports pattern separation, whereas the monosynaptic pathway (entorhinal cortex > CA1) supports statistical learning. To test this hypothesis, we investigated the behavioral expression of these two processes in B. L., an individual with highly selective bilateral lesions in the dentate gyrus that presumably disrupt the trisynaptic pathway. We tested pattern separation with two novel auditory versions of the continuous mnemonic similarity task, requiring the discrimination of similar environmental sounds and trisyllabic words. For statistical learning, participants were exposed to a continuous speech stream made up of repeating trisyllabic words. They were then tested implicitly through a RT-based task and explicitly through a rating task and a forced-choice recognition task. B. L. showed significant deficits in pattern separation on the mnemonic similarity tasks and on the explicit rating measure of statistical learning. In contrast, B. L. showed intact statistical learning on the implicit measure and the familiarity-based forced-choice recognition measure. Together, these results suggest that dentate gyrus integrity is critical for high-precision discrimination of similar inputs, but not the implicit expression of statistical regularities in behavior. Our findings offer unique new support for the view that pattern separation and statistical learning rely on distinct neural mechanisms.

Exploration of the Effect on Continuation Task and Its Teaching Implication in High School

Using primarily a literature-based approach, this paper examines the effects of reading followed by writing on students' language performance (including accuracy, complexity, and fluency) based on the theory of reading followed by writing for learning. The results show that the read-follow-up writing task requires interaction with reading comprehension to promote synergistic effects that effectively enhance learners' language output, especially in terms of accuracy and complexity. On this basis, to explore how reading followed by writing promotes fluency, accuracy and complexity in second language writing, the study shows that 1) progressively increasing the linguistic complexity of input texts contributes to the development of second language accuracy, while simplified texts with constant complexity contribute to the improvement of fluency. 2) The effect of input text linguistic complexity on the complexity of second language writing is inconsistent across indicators, with complexity Constant complexity increases the use of subordinate clauses in linguistic output, while increasing complexity significantly increases the number of parallel sentences used. The study points out that reading followed by writing is an effective way to improve the efficiency of foreign language teaching and learning by closely integrating language output and input and aiming to accelerate students' language use.

Approaching and pointing tracking control for tumbling target under motion constraints

On-orbit servicing for the failed spacecraft involves two necessary stages: on-orbit observing and approaching. For these two stages, a novel approaching and pointing tracking control scheme under motion constraints and input saturation is proposed in this paper. For the security of approaching, a relative position guidance law for specific surface hovering is proposed. Then, a model predictive controller based on quadratic programming algorithm is designed. For the continuous observation task, the optimal maneuver angular velocity under the motion constraint is planned through model predictive control. In order to reduce the dimensionality of the optimization problem to reduce the computational effort, only the attitude kinematics equation is used as the process model. Thereafter, the adaptive anti-saturation attitude controller is designed to track the optimal maneuver angular velocity. The anti-saturation auxiliary system and the adaptive update law are used to cope with the control torque saturation and the integrated uncertainties, respectively. Finally, the stability of the closed-loop system is demonstrated by Lyapunov method. Simulation results demonstrate that the proposed control scheme accomplishes the approaching and pointing tracking task, and finally service spacecraft maintains hovering above the tumbling target and observing its specific surface.

Learning Diverse Policies with Soft Self-Generated Guidance

Reinforcement learning (RL) with sparse and deceptive rewards is a significant challenge because nonzero rewards are rarely obtained, and hence, the gradient calculated by the agent can be stochastic and without valid information. Recent work demonstrates that using memory buffers of previous experiences can lead to a more efficient learning process. However, existing methods usually require these experiences to be successful and may overly exploit them, which can cause the agent to adopt suboptimal behaviors. This study develops an approach that exploits diverse past trajectories for faster and more efficient online RL, even if these trajectories are suboptimal or not highly rewarded. The proposed algorithm merges a policy improvement step with an additional policy exploration step by using offline demonstration data. The main contribution of this study is that by regarding diverse past trajectories as guidance, instead of imitating them, our method directs its policy to follow and expand past trajectories, while still being able to learn without rewards and gradually approach optimality. Furthermore, a novel diversity measurement is introduced to maintain the diversity of the team and regulate exploration. The proposed algorithm is evaluated on a series of discrete and continuous control tasks with sparse and deceptive rewards. In comparison with the existing RL methods, the experimental results indicate that our proposed algorithm is significantly better than the baseline methods in terms of diverse exploration and avoiding local optima.