Stated Goals Research Articles

An algorithm that excavates suboptimal states and improves Q-learning

Abstract Reinforcement learning is inspired by the trial-and-error method in animal learning, where the reward values obtained from the interaction of the agent with the environment are used as feedback signals to train the agent. Reinforcement learning has attracted extensive attention in recent years. It is mainly used to solve sequential decision-making problems and has been applied to various aspects of life, such as autonomous driving, game gaming, and robotics. Exploration and exploitation are the main characteristics that distinguish reinforcement learning methods from other learning methods. Reinforcement learning methods need reward optimization algorithms to better balance exploration and exploitation. Aiming at the problems of unbalanced exploration and a large number of repeated explorations in the Q-learning algorithm in the MDP environment, an algorithm that excavates suboptimal states and improves Q-learning was proposed. It adopts the exploration idea of ‘exploring the potential of the second-best’, and explores the state with suboptimal state value, and calculates the exploration probability value according to the distance between the current state and the goal state. The larger the distance, the higher the exploration demand of the agent. In addition, only the immediate reward and the maximum action value of the next state are needed to calculate the Q value. Through the simulation experiments in two different MDP environments, The frozenLake8x8 environment and the CliffWalking environment, the results verify that the proposed algorithm obtains the highest average cumulative reward and the least total time consumption.

Goal Shifts Structure Memories and Prioritize Event-defining Information in Memory.

Every day, we encounter far more information than we could possibly remember. Thus, our memory systems must organize and prioritize the details from an experience that can adaptively guide the storage and retrieval of specific episodic events. Prior work has shown that shifts in internal goal states can function as event boundaries, chunking experiences into distinct and memorable episodes. In addition, at short delays, memory for contextual information at boundaries has been shown to be enhanced compared with items within each event. However, it remains unclear if these memory enhancements are limited to features that signal a meaningful transition between events. To determine how changes in dynamic goal states influence the organization and content of long-term memory, we designed a 2-day experiment in which participants viewed a series of black-and-white objects surrounded by a color border on a two-by-two grid. The location of the object on the grid determined which of two tasks participants performed on a given trial. To examine if distinct types of goal shifts modulate the effects of event segmentation, we changed the border color, the task, or both after every four items in a sequence. We found that goal shifts influenced temporal memory in a manner consistent with the formation of distinct events. However, for subjective memory representations in particular, these effects differed by the type of event boundary. Furthermore, to examine if goal shifts lead to the prioritization of goal-relevant features in longer lasting memories, we tested source memory for each object's color and grid location both immediately and after a 24-hr delay. On the immediate test, boundaries enhanced the memory for all concurrent source features compared with nonboundary items, but only if those boundaries involved a goal shift. In contrast, after a delay, the source memory was selectively enhanced for the feature relevant to the goal shift. These findings suggest that goals can adaptively structure memories by prioritizing contextual features that define a unique episode in memory.

Why path dependence leads to a fossilized Alberta: regionalism and the climate transition in Canada

ABSTRACT Canada is the world’s fourth largest producer of oil and fifth largest producer of natural gas. Yet, the country faces a ‘twin’ challenge in tackling the climate crisis. On the one hand, Canada’s federal climate goals state that it must achieve net-zero carbon emissions by 2050. On the other hand, regionalism stronghold in fossil-producing provinces lead to difficult trade-offs between an ambitious climate transition and reluctant actors supporting the fossil fuel industry. This situation is characterized as ‘fossilized’ petro-provinces by political scientist Angela Carter. This article explores how Alberta, Canada’s largest oil and natural gas producer, became fossilized. The article develops a path dependence argument organized around the role of regionalism in Alberta’s political development. Regionalism is a crucial persistent feature of Alberta to explain its fossilization because it influenced the formation of an interdependence between the government and the industry, a distinct political culture, and a conservative-led party system.

A computational account of self-control

Self-control is core to human well-being. However, the lack of a well-specified, computationally tractable framework related to self-control makes it difficult to clarify underlying mechanisms, interpret relevant empirical phenomena, or develop interventions helpful in promoting self-control. To help address this gap, we invite consideration of the Comparison with Goal States Model (CGSM) for self-control. The CGSM amplifies activations related to available options whose representations are similar to representations of relevant goals and diminishes activations related to available options whose representations are dissimilar to representations of relevant goals. For example, influenced by healthy eating goals, the CGSM would amplify activations related to an apple and diminish activations related to a cookie, leading to an eventual preference for the apple, even though the cookie might be initially preferred. The CGSM successfully explicates observations related to reaction time in food choice, dynamics reflected in mouse-tracking trajectories, and showcases a mechanism by which hyperbolic discount curves in temporal discounting contexts might emerge. We use the CGSM to propose theoretical constraints on the nature of self-control and describe how multiple strategies have the potential to promote self-control.

Post-transplant renal anemia: a call to action from a national study in routine clinical practice.

Post-transplant anemia is a prevalent yet often overlooked condition that poses significant risks. Current guidelines consider the same treatment recommendations and goals for these patients as for chronic kidney disease patients not on dialysis. Previous reports demonstrated a lack of awareness and suboptimal management, indicating a pressing need for improvement. We therefore wanted to update the information on post-transplant anemia. We aimed to describe the present state of anemia management, goals and adherence to guidelines within a representative sample of the kidney transplant (KTx) population. We designed a retrospective nationwide multicenter study including outpatients from eight KTx hospitals. Nephrologists gathered data from electronic medical records encompassing demographics, comorbidities, KTx characteristics and immunosuppressive therapy, and information pertaining to anemia management (laboratory values, previously prescribed treatments and subsequent adjustments). The European statement on the Kidney Disease: Improving Global Outcomes (KDIGO) guidelines was the reference for definitions, drug prescriptions and targets. Anemia occurring within the initial 6 months post-transplantation was classified as early onset. We included 297 patients with post-transplant anemia aged 62.8 years (standard deviation 13.6), 60% of whom were male. They had received a graft from cardiac death or brain death donors (61.6% and 31.1%, respectively) a median of 2.5 years (0.5-8.7) before. Among them 77% (n=228) were classified as having late post-transplant anemia, characterized by a higher prevalence of microcytic and iron deficiency anemia. A total of 158 patients were on erythropoietic-stimulating agents (ESAs) treatment, yet surprisingly 110 of them lacked iron supplementation. Notably, 44 patients had an indication for iron supplementation and among them, 30 exhibited absolute iron deficiency. Out of the 158 patients receiving ESAs, only 39 surpassed the limit for the ESA resistance index, indicating poor response. This resistance was more frequent among patients with early post-transplant anemia (26.1% vs 9.2%). We have identified iron profile, early post-transplant anemia and estimated glomerular filtration rate as factors associated with the highest risk of resistance. We found that hemoglobin targets are individualized upwards in post-transplant anemia. In this setting, iron therapy continues to be underutilized, especially intravenous, and iron deficiency and prior events (blood transfusion or hospital admission) explain most of the hyporesponsiveness to ESA. This highlights missed opportunities for precise prescription targeting and adherence to established guidelines, suggesting a need for improved management strategies in post-transplant anemia patients.

Obstacle avoidance shape control of deformable linear objects with online parameters adaptation based on differentiable simulation

The manipulation of deformable linear objects (DLOs) such as ropes, cables, and hoses by robots has promising applications in various fields such as product assembly and surgical suturing. However, DLOs are more difficult to manipulate than rigid objects because their shape changes during manipulation. Furthermore, preventing a DLO from colliding with the environment is important to prevent it from becoming entangled and causing shape control to fail. In this paper, we proposed an obstacle avoidance and shape control scheme for DLOs based on differentiable simulation that does not require prior data or a specialized controller. First, we established a dynamic model of the DLO that allows for both forward dynamics transfer and error backpropagation to obtain gradients. Then, we employed model predictive control to optimize the embedded neural network for predicting the actions that would manipulate the DLO. Finally, the control scheme was made applicable to DLOs with different material properties by allowing online adaptation of the model parameters essential to deformation during manipulation. Simulations and real-world experiments demonstrate that the proposed control scheme could manipulate the DLO stably and accurately to avoid obstacles and achieve the goal state. In addition, the online adaptation of parameters helped mitigate the sim-to-real gap.

Co-scale cross-attentional transformer for rearrangement target detection

Rearranging objects (e.g. vase, door) back in their original positions is one of the most fundamental skills for domestic service robots (DSRs). In rearrangement tasks, it is crucial to detect the objects that need to be rearranged according to the goal and current states. In this study, we focus on Rearrangement Target Detection (RTD), where the model generates a change mask for objects that should be rearranged. Although many studies have been conducted in the field of Scene Change Detection (SCD), most SCD methods often fail to segment objects with complex shapes and fail to detect the change in the angle of objects that can be opened or closed. In this study, we propose a Co-Scale Cross-Attentional Transformer for RTD. We introduce the Serial Encoder which consists of a sequence of serial blocks and the Cross-Attentional Encoder which models the relationship between the goal and current states. We built a new dataset consisting of RGB images and change masks regarding the goal and current states. We validated our method on the dataset and the results demonstrated that our method outperformed baseline methods on F 1 -score and mean IoU.

Stress sharing as cognitive glue for collective intelligences: A computational model of stress as a coordinator for morphogenesis

Individual cells have numerous competencies in physiological and metabolic spaces. However, multicellular collectives can reliably navigate anatomical morphospace towards much larger, reliable endpoints. Understanding the robustness and control properties of this process is critical for evolutionary developmental biology, bioengineering, and regenerative medicine. One mechanism that has been proposed for enabling individual cells to coordinate toward specific morphological outcomes is the sharing of stress (where stress is a physiological parameter that reflects the current amount of error in the context of a homeostatic loop). Here, we construct and analyze a multiscale agent-based model of morphogenesis in which we quantitatively examine the impact of stress sharing on the ability to reach target morphology. We found that stress sharing improves the morphogenetic efficiency of multicellular collectives; populations with stress sharing reached anatomical targets faster. Moreover, stress sharing influenced the future fate of distant cells in the multi-cellular collective, enhancing cells’ movement and their radius of influence, consistent with the hypothesis that stress sharing works to increase cohesiveness of collectives. During development, anatomical goal states could not be inferred from observation of stress states, revealing the limitations of knowledge of goals by an extern observer outside the system itself. Taken together, our analyses support an important role for stress sharing in natural and engineered systems that seek robust large-scale behaviors to emerge from the activity of their competent components.

Exploring the goals and outcomes of adults with severe acquired brain injury participating in an extended inpatient brain injury rehabilitation unit in Australia

Purpose To explore the rehabilitation goals and evaluate goal attainment outcomes of people with severe acquired brain injury (ABI), and investigate the relationship between goal engagement and goal attainment. Materials and methods Mixed-methods cohort study with twenty-nine adults with severe ABI in Australia. Demographic data, goal statements and pre-post program Goal Attainment Scale scores as well as Goal Engagement Scale scores were collected. Goals were coded using inductive content analysis and categorised by ICF component and domain. Goal attainment within ICF categories was described and compared using descriptive statistics. Pre-post program change in goal attainment was evaluated using Wilcoxon signed rank tests and correlations between goal engagement and attainment was explored using Spearman’s (rho). Results 94% of 320 goals were categorised as ICF Activity and Participation. There was significant improvement in goal attainment between admission and discharge (z=-0.47, p < 0.01). There was no significant relationship between goal engagement and goal attainment however there was a positive association between engagement in goal setting at admission and discharge. Conclusions: This interdisciplinary, inpatient rehabilitation program underpinned by key-worker facilitated person-centred, role-based goal setting resulted in goal attainment in chosen goals, which were primarily activity and participation-focused.

DIGITALIZATION OF POLITICS AND ELECTRONIC VOTING: FEATURES AND NATIONAL EXPERIENCE

Digitalization of politics is determined not only by the technological capabilities and accessibility of digital benefits for the general population, but also by the interests and goals of political actors, elites and states. The result of digitalization of politics is e-government. The Anglo-American e-government model is based on the idea of universality and serviceability of public service provision. The Asian model of e-government is associated with a multi-level structure of the state apparatus and meeting the information needs of citizens. The continental-European model of e-government is based on the principles of democracy, openness and protection of personal data. The Russian model of e-government seeks to improve the quality of relations between society and the state, as well as to improve the quality of public service provision. A key form of policy digitalization is electronic voting, which is based on blockchain technology. For the first time, e-voting was implemented in the United States and in European countries (Estonia). In 2000, electronic voting was first implemented in Russia, and currently two types of electronic voting are used: remote electronic voting and a ballot processing complex at polling stations. Digitalization of politics opens up new opportunities and horizons for interaction between public and state structures. Digital technologies in politics can make political participation more accessible, increase competition and reduce the level of political absenteeism.

Brain-state mediated modulation of inter-laminar dependencies in visual cortex

Spatial attention is critical for recognizing behaviorally relevant objects in a cluttered environment. How the deployment of spatial attention aids the hierarchical computations of object recognition remains unclear. We investigated this in the laminar cortical network of visual area V4, an area strongly modulated by attention. We found that deployment of attention strengthened unique dependencies in neural activity across cortical layers. On the other hand, shared dependencies were reduced within the excitatory population of a layer. Surprisingly, attention strengthened unique dependencies within a laminar population. Crucially, these modulation patterns were also observed during successful behavioral outcomes that are thought to be mediated by internal brain state fluctuations. Successful behavioral outcomes were also associated with phases of reduced neural excitability, suggesting a mechanism for enhanced information transfer during optimal states. Our results suggest common computation goals of optimal sensory states that are attained by either task demands or internal fluctuations.

Sparse randomized policies for Markov decision processes based on Tsallis divergence regularization

This work investigates a somewhat different point of view on Markov decision processes by reinterpreting them as a randomized shortest paths problem on a bipartite graph, therefore establishing bridges with entropy-regularized reinforcement learning. The graph structure contains the set of states as “left” nodes and the set of actions as “right” nodes. In that context, the action-to-state transition probabilities are provided by the environment whereas the state-to-action probabilities correspond to the (stochastic) policy to be found. The randomized shortest paths formalism (minimizing expected cost to the goal state subject to (Shannon or Tsallis) relative entropy regularization) is then readily applied to this bipartite structure, providing a possibly sparse stochastic policy interpolating between a least-cost and a purely random policy. The algorithm computing the policy is closely related to the dual linear programming formulation of the Markov decision processes to which the relative entropy regularization term, multiplied by a scaling factor balancing exploitation and exploration (the temperature), is added. It is derived from well-known techniques of discrete optimal control, relying on costates (Lagrange parameters) backward computation. In summary, the proposed algorithm allows the design of optimal stochastic – but still sparse – policies, ranging from a purely rational to a random behavior, depending on the temperature parameter.

Child-led goal setting and evaluation tools for children with a disability: A scoping review.

To examine child-led goal setting and evaluation tools and approaches for children with a disability or developmental delay. Six databases were searched for studies that included population (children aged less than 18 years with disability or developmental delay); construct (child-led goal setting tool or approach); and context (developmental therapy or rehabilitation). The utility of tools and approaches across the goal setting and evaluation process was investigated using abductive content analysis. Fifty articles met the inclusion criteria. Three approaches and four tools for child-led goal setting and evaluation were identified. No studies reported the clinimetric properties of tools specifically for child self-respondents. Qualitative analysis revealed six distinct goal phases in which tools and approaches were used, which were synthesized into a new framework for child-led goal setting and evaluation titled DECIDE: Direct children to goal setting; Elicit goal topics and priorities; Construct a goal statement; Indicate baseline goal performance; Develop an action plan to address the goal; and Evaluate goal progress after the intervention. Children actively participated in goal setting and evaluation across six DECIDE goal phases. Further clinimetric information is required to support use of goal setting and evaluation tools with child self-respondents. Future research should emphasize the development of multi-phase goal setting tools and approaches for diverse populations of children.

Proximity to rewards modulates parameters of effortful control exertion.

The now-classic goal-gradient hypothesis posits that organisms increase effort expenditure as a function of their proximity to a goal. Despite nearly a century having passed since its original formulation, goal-gradient-like behavior in human cognitive performance remains poorly understood: Are we more willing to engage in costly cognitive processing when we are near, versus far, from a goal state? Moreover, the computational mechanisms underpinning these potential goal-gradient effects-for example, whether goal proximity affects fidelity of stimulus encoding, response caution, or other identifiable mechanisms governing speed and accuracy-are unclear. Here, in two experiments, we examine the effect of goal proximity, operationalized as progress toward the completion of a rewarded task block, upon task performance in an attentionally demanding oddball task. Supporting the goal-gradient hypothesis, we found that participants responded more quickly, but not less accurately, when rewards were proximal than when they were distal. Critically, this effect was only observed when participants were given information about goal proximity. Using hierarchical drift diffusion modeling, we found that these apparent goal-gradient performance effects were best explained by a collapsing bound model, in which proximity to a goal reduced response caution and increased information processing. Taken together, these results suggest that goal gradients could help explain the oft-observed fluctuations in engagement of cognitively effortful processing, extending the scope of the goal-gradient hypothesis to the domain of cognitive tasks. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The Case to Discard Hospital Mission Statements and Replace It with A Statement of Goals

Background: Hospitals and healthcare institutions spend a considerable amount of time and capital on constructing an appropriate mission statement for themselves. Methods: A review of the purpose of mission statements and the statements made by different healthcare organizations was undertaken. For institutions in the USA, their mission statements were examined in the context of their national ratings. Results: The mission statements of healthcare organizations are often too abstract, using similar phraseology about caring for patients. Concern for the community that the hospital serves is stated by both a hospital listed as one of the best US hospitals and an institution that was considered to be in the lowest tier in the same country. Similarly, the recognition of ‘research’ or ‘innovation and discovery’ was stated by respectively both a top institution in the USA and one in the bottom tier. Conclusions: Healthcare institutions should replace their mission statements with clear and attainable statements of concrete goals, and indicate how successful the institution is at attaining those goals and improving the experience of its patients and staff.

The Priority Goals and Underlying Impairments Contributing to Goal-Related Problems of People with Parkinson's Disease Receiving a Community-Based Rehabilitation Program.

Goal setting is a core rehabilitation practice in Parkinson's disease (PD). Targeting therapy towards specific goals leads to greater improvements in performance and psychosocial outcomes. Goal setting in PD is feasible, and although the nature of goals has been described in previous studies, the underlying impairments related to goals have not been described. Understanding the nature of goals ensures that interventions for people with PD are aligned with their needs and priorities. Understanding the underlying impairments highlights which symptoms have the biggest impact on daily life and is necessary for planning appropriate interventions to target them. To describe the nature of the goals of people with PD; the underlying impairments related to goals; and to compare differences between high and low priority goals. Deductive content analysis was used to map goal statements to the international classification of function (ICF) activity and participation category and to map therapist field notes detailing the primary underlying impairment to the ICF Body Functions category. These results were then compared across goal priority rankings. 88 goals of 22 people with PD were analysed. We found that people with PD set diverse goals across all chapters of the ICF Activity and Participation category, with "self-care" goals making up the highest proportion of goals. The primary underlying impairment related to the goals was predominantly related to impairments in "mental functions" under the Body Functions category. Regardless of goal priority, most goal-related underlying impairments were found to be in the "mental functions" category. The goals of this sample of community-dwelling people with PD highlight their diverse needs and priorities. These findings indicate that nonmotor symptoms, namely, executive dysfunction and amotivation most commonly impact the performance of and participation in activities of greatest importance to people with PD. This trial is registered with ACTRN12621001483842.

Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning

Safety in goal directed Reinforcement Learning (RL) settings has typically been handled through constraints over trajectories and have demonstrated good performance in primarily short horizon tasks. In this paper, we are specifically interested in the problem of solving temporally extended decision making problems such as robots cleaning different areas in a house while avoiding slippery and unsafe areas (e.g., stairs) and retaining enough charge to move to a charging dock; in the presence of complex safety constraints. Our key contribution is a (safety) Constrained Search with Hierarchical Reinforcement Learning (CoSHRL) mechanism that combines an upper level constrained search agent (which computes a reward maximizing policy from a given start to a far away goal state while satisfying cost constraints) with a low-level goal conditioned RL agent (which estimates cost and reward values to move between nearby states). A major advantage of CoSHRL is that it can handle constraints on the cost value distribution (e.g., on Conditional Value at Risk, CVaR) and can adjust to flexible constraint thresholds without retraining. We perform extensive experiments with different types of safety constraints to demonstrate the utility of our approach over leading approaches in constrained and hierarchical RL.

A General Search-Based Framework for Generating Textual Counterfactual Explanations

One of the prominent methods for explaining the decision of a machine-learning classifier is by a counterfactual example. Most current algorithms for generating such examples in the textual domain are based on generative language models. Generative models, however, are trained to minimize a specific loss function in order to fulfill certain requirements for the generated texts. Any change in the requirements may necessitate costly retraining, thus potentially limiting their applicability. In this paper, we present a general search-based framework for generating counterfactual explanations in the textual domain. Our framework is model-agnostic, domain-agnostic, anytime, and does not require retraining in order to adapt to changes in the user requirements. We model the task as a search problem in a space where the initial state is the classified text, and the goal state is a text in a given target class. Our framework includes domain-independent modification operators, but can also exploit domain-specific knowledge through specialized operators. The search algorithm attempts to find a text from the target class with minimal user-specified distance from the original classified object.

Symbolic Reasoning Methods for AI Planning

Planning is the act of deliberative thinking before acting. It is based on a symbolic model of the world and the options to act in it, usually defined in function-free first-order logic. The task is to find a sequence of actions (a plan) that leads from a given current state to a desired goal state. The basic, purely physical description may be augmented with a partially ordered grammar-like structure (a Hierarchical Task Network or HTN), which can describe expert knowledge, or practical, legal, or operational requirements. In this talk, I will survey a variety of methods for automatically deriving plans using symbolic methods for planning -- from both my past and future research. These symbolic methods -- in some sense -- translate planning problems into other, simpler symbolic representations and reason over them to find plans. As a basis for these methods, I will firstly introduce relevant theoretical results on planning. First, I will discuss the expressive power of planning formalisms (ECAI'14, ICAPS'16) and second, the computational complexity of HTN planning and related tasks such as HTN plan verification, plan modification, and plan recognition (ICAPS'15, ICAPS'16). Based on these theoretical results, I will develop why SAT-based HTN planning is possible and how it can be implemented. To this end, I will survey several of my publications at top-tier conferences, including papers at ICAPS'17, AAAI'18, AAAI'19, IJCAI'19, AAAI'20, and ICAPS'21 -- in which I developed an highly SAT-based planner for HTN problems including the ability to find optimal plans as well as the grounding as a preprocessing step. Here I will also give an outlook on future developments and new ideas that I propose for SAT-based planning -- including the exploitation of structures in plan (e.g.\ landmarks or operator-counting constraints). Next, I will present the idea of expressing lifted classical planning as SAT (ICAPS'22). The resulting planner LiSAT was the first lifted SAT-based planner -- and proved highly efficient and outperformed all other lifted planners at the time of publication. Notably, LiSAT was the first planner (lifted or grounded) and still is the only one to solve the challenging OrganicSynthesis benchmark -- and could even prove optimality for all plans. I will also outline future ideas to further improve the efficiency of LiSAT. Lastly, I introduce the notion of planning with symbolic symbolic representations (AAAI'21 and ICAPS'23). Here one uses Binary Decision Diagrams to encode large sets of states efficiently. For expressing the additional structure encoded by HTNs, I show how BDDs can be suitably integrated into finite automata. Based on this representation, an efficient and optimal planning algorithm can be derived. Additionally, I show how this algorithm can be extended to also cover oversubscription planning.

AN EMPIRICAL ANALYSIS OF INDIA'S SMART CITIES INITIATIVE: PROSPECTS AND DIFFICULTIES

The Indian government, along with all state governments, has an ambitious plan called the Smart City Mission (SCM). An urban area that uses many electronic Internet of Things (IoT) sensors to collect data and use it to support services and resources required for effective municipal operations is referred to as a "smart city". The policy might address one issue facing the city, but it might not address all the issues that a city faces. This research will concentrate on the most important and pertinent publications, papers, and articles about the Indian Smart Cities Mission. The theoretical notion of smart cities, the "Indian smart cities mission," and its conceptual framework will also be covered. Ultimately, the research’s conclusions will clarify the goal statement, the stages involved in implementation, and the numerous opportunities and challenges that India's efforts to develop smart cities face. The obstacles and potential in the Indian Smart City Mission will also be highlighted in this paper.