Uncertain Actions Research Articles

UAC: Offline Reinforcement Learning With Uncertain Action Constraint

Offline reinforcement learning (RL) algorithms promise to learn policies directly from offline datasets without environmental interaction. This arrangement enables successful RL applications in the real world, particularly in robots and autonomous driving, where sampling is costly and dangerous. However, the existing offline RL algorithms suffer from insufficient performance attributed to extrapolation error caused by out-of-distribution (OOD) actions. In this work, we propose an offline RL algorithm with an uncertain action constraint (UAC). The design principle of UAC is to minimize the extrapolation error via eliminating unknown and uncertain actions. Concretely, we first theoretically analyze the effects of different types of actions on the extrapolation error. Based on this, we propose an action-constrained strategy that exploits the uncertainty of the environmental dynamics model to eliminate unknown and uncertain actions in the Q-value evaluation process. Furthermore, the convex combination of trajectory information and Gaussian noise is novelly leveraged to enhance the generation probability of the optimal actions. Finally, we carry out the comparison and ablation experiments on the standard D4RL dataset. Experimental results indicate that UAC achieves competitive performance, especially in the field of robotic manipulation.

Space and place for health and care – Nationalist discourses in Swedish daily press during the first year of COVID-19

Sweden’s strategy during COVID-19 with restrictions but no firm closure of the society surprised the rest of the world and was questioned, not least by neighbouring countries. This article analyses public discourses on space and place for health and care in the Swedish daily press during the first year of the pandemic, 2020. Critical discourse analysis was conducted on daily press newspaper articles to approach issues of space, place, health and care during the COVID-10 pandemic. The findings suggest three main discourses. First, a powerful discourse on unity against the threat is articulated, urging citizens in Sweden to be loyal in the national space. Secondly, an affirming national reconstructing discourse is manifested, related to constructions of borders of national space but also in relation to places of family life and social contacts to ‘flatten the curve’ and stay healthy. Thirdly, later in the period the overarching discourse of the nation and its loyal citizens was torn apart and increasing tensions were articulated due to, as it appeared, the uncertain actions from the government. This study adds to the literature on a theoretical and practical level. Raising awareness on nationalist discourses in relation to place, space, health, and care could prove important in combating inequalities in the local society as well as when cooperating on an international level.

Combining deep reinforcement learning with technical analysis and trend monitoring on cryptocurrency markets

Cryptocurrency markets experienced a significant increase in the popularity, which motivated many financial traders to seek high profits in cryptocurrency trading. The predominant tool that traders use to identify profitable opportunities is technical analysis. Some investors and researchers also combined technical analysis with machine learning, in order to forecast upcoming trends in the market. However, even with the use of these methods, developing successful trading strategies is still regarded as an extremely challenging task. Recently, deep reinforcement learning (DRL) algorithms demonstrated satisfying performance in solving complicated problems, including the formulation of profitable trading strategies. While some DRL techniques have been successful in increasing profit and loss (PNL) measures, these techniques are not much risk-aware and present difficulty in maximizing PNL and lowering trading risks simultaneously. This research proposes the combination of DRL approaches with rule-based safety mechanisms to both maximize PNL returns and minimize trading risk. First, a DRL agent is trained to maximize PNL returns, using a novel reward function. Then, during the exploitation phase, a rule-based mechanism is deployed to prevent uncertain actions from being executed. Finally, another novel safety mechanism is proposed, which considers the actions of a more conservatively trained agent, in order to identify high-risk trading periods and avoid trading. Our experiments on 5 popular cryptocurrencies show that the integration of these three methods achieves very promising results.

Fall-Perceived Action Recognition of Persons With Neurological Disorders Using Semantic Supervision

Frequent uncertain falls is one of the common cause of injury among elderly adults and persons suffering from the neurological disorder. It will be costlier to go through <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$24\times 7$ </tex-math></inline-formula> medical monitoring if we monitor a person suffering from the early stage of the neurological disorder. An “uncertain” action classification model can be a less costly and easily scalable. It can help to regularly monitor a person suffering from neurological declines and how frequent it relapse. In this article, we propose a video-based action recognition with fall detection architecture, FallNet, which learns the features of uncertain actions related to day-to-day activities. FallNet first incorporates semantic supervision using the per-class weight of uncertain action through class-wise weighted focal loss. It addresses both the class imbalance problem and the weak interclass separability issue. We design a joint training model to train the overall architecture efficiently in an end-to-end manner. We utilize benchmark data sets, OOPS, HMDB51, and Kinetics-600, for experimentation that has less falling action videos. Therefore, we have collected videos to create a data set, denoted by FallAction, that consists of different 15 falling action classes with an average of 100 videos per class. The proposed network gain an accuracy of 13.2% in OOPs, 2% in HMDB51, and 0.2% in Kinetics-600 data set.

Modeling Defender-Attacker Problems as Robust Linear Programs with Mixed-Integer Uncertainty Sets

We study a class of sequential defender-attacker optimization problems where the defender’s objective is uncertain and depends on the operations of the attacker, which are represented by a mixed-integer uncertainty set. The defender seeks to hedge against the worst possible data realization, resulting in a robust optimization problem with a mixed-integer uncertainty set, which requires the solution of a challenging mixed-integer problem, which can be seen as a saddle-point problem over a nonconvex domain. We study two exact solution algorithms and present two feature applications for which the uncertainty is naturally modeled as a mixed-integer set. Our computational experiments show that the considered algorithms greatly outperform standard algorithms both in terms of computational time and solution quality. Moreover, our results show that modeling uncertainty with mixed-integer sets, instead of approximating the data using convex sets, results in less conservative solutions, which translates to a lower cost for the defender to protect from uncertainty. Summary of Contribution: We consider a class of defender-attacker problems where the defender has to make operational decisions that depend on uncertain actions from an adversarial attacker. Due to the type of information available to the defender, neither probabilistic modeling, nor robust optimization methods with convex uncertainty sets, are well suited to address the defender’s decision-making problem. Consequently, we frame the defender’s problem as a class of robust optimization problems with a mixed-integer uncertainty sets, and devise two exact algorithms that solve this class of problems. A comprehensive computational study shows that for the considered applications, our algorithms improves the performance of existing robust optimization approaches that can be adapted to solve this class of problems. Moreover, we show how mixed-integer uncertainty sets can reduce the level of over-conservatism that is a known issue of robust optimization approaches.

A Multiarmed Bandit Approach to Adaptive Water Quality Management.

ABSTRACTNonpoint source water quality management is challenged with allocating uncertain management actions and monitoring their performance in the absence of state‐dependent decision making. This adaptive management context can be expressed as a multiarmed bandit problem. Multiarmed bandit strategies attempt to balance the exploitation of actions that appear to maximize performance with the exploration of uncertain, but potentially better, actions. We performed a test of multiarmed bandit strategies to inform adaptive water quality management in Massachusetts, USA. Conservation and restoration practitioners were tasked with allocating household wastewater treatments to minimize N inputs to impaired waters. We obtained time series of N monitoring data from 3 wastewater treatment types and organized them chronologically and randomly. The chronological data set represented nonstationary performance based on recent monitoring data, whereas the random data set represented stationary performance. We tested 2 multiarmed bandit strategies in hypothetical experiments to sample from the treatment data through 20 sequential decisions. A deterministic probability‐matching strategy allocated treatments with the highest probability of success regarding their performance at each decision. A randomized probability‐matching strategy randomly allocated treatments according to their probability of success at each decision. The strategies were compared with a nonadaptive strategy that equally allocated treatments at each decision. Results indicated that equal allocation is useful for learning in nonstationary situations but tended to overexplore inferior treatments and thus did not maximize performance when compared with the other strategies. Deterministic probability matching maximized performance in many stationary situations, but the strategy did not adequately explore treatments and converged on inferior treatments in nonstationary situations. Randomized probability matching balanced performance and learning in stationary situations, but the strategy could converge on inferior treatments in nonstationary situations. These findings provide evidence that probability‐matching strategies are useful for adaptive management. Integr Environ Assess Manag 2020;16:841–852. © 2020 The Authors. Integrated Environmental Assessment and Management published by Wiley Periodicals LLC on behalf of Society of Environmental Toxicology & Chemistry (SETAC)

ПРИНЦИПЫ ЭФФЕКТИВНОЙ КОММУНИКАЦИИ И СТИЛИСТИЧЕСКИЕ ОСОБЕННОСТИ АНГЛОЯЗЫЧНОГО ДИСКУРСА НА ПРИМЕРЕ СОВРЕМЕННЫХ ПОЛИТИЧЕСКИХ ДЕБАТОВ

Statement of the problem. Under the conditions of the globalization growing process, English language became one of the leading languages of intercultural interaction and under expanding the influence of the public information exchange it is becoming necessary to consider the modern political communication as the most effective kind of rendering lingocultural and lingosocial reality of western mentality. The special interest presents specific verbal and extra linguistic devices adherent to English political discourse and lingo-stylistic peculiarities of debates, the participants of which are certain representatives of a specific political establishment. Certain lexis and definite construction of pre-election speeches of American politicians have served as an object for the linguistic analysis and were the most effective in helping to influence the electorates’ opinion with the purpose of winning over the opposing party. From the position of discourse, debates reflect a dynamic process of linguistic activity imprinted into its social context. The corresponding constructions of discourse give the borders of communication and organize the way of thinking, cognitive and communicative activity, expressing the authority of the subject within the discourse and his managing and manipulation of the object. The purpose of the article is in pointing out some definite communicative and stylistic devices in the discourse constructions and building methods of lexical-grammatical constructions for achieving the effectiveness of the communicative act in debates conducting technology. The methodology of study includes using the method of rhetorical analysis as well as content analysis of definite speeches by D. Trump and H. Clinton who participated in presidential elections during 2016. Also the method of the purposeful search for examples, used in different kinds of debates which represent a discourse as a complex communicative phenomenon. Research results. The researched linguistic material allows to conclude that there is clear correspondence of the used techniques, speech-cognitive patterns, technologies and strategies with definite political parties presented in the communicative field while conducting debates. The Democrats demonstrated the strategy of using “soft power” and appealing to the rational part of mind along with these tactics: “name calling” in which during the dispute the real names of existing people are drawn to call inner emotional reflection within the minds of electorate, tactic of “solidarization” in which the public representative correlates himself with the civic community and identifies it as a “big family” that has to have a leader with “true democratic values”. With a main motto of “Make America Great Again” republicans were following the line of aggressive strategy opposing the achievements of democrats. In the field of tactics there was appearance of ruthless attempts to get initiative advantage through blaming for ineffective migration policy, uncertain actions in the foreign affairs, on top of that the tactic of “playing on conflicts” to force the opposing figure out of balance was imposed. Speech-cognitive patterns of each party reflected the character of the process of debates as well. Democrats neatly and methodically conducted the discussion, revealing their positions through well-made arguments, while republicans were constructing the image of a “self-made businessman” and built the verbal attitude according to the picture using simple sentences with repetitions, vulgar lexis towards the opponent, emotional extra linguistic signals and exploiting mistakes made in argumentative line of democrats for refutation. Conclusion. The technology of “debates” represents itself as a communicative event between a speaker, a listener (a viewer) in the process of communication within a definite time, space and other context. This communicative act can be oral or written and have verbal and extra linguistic parts which provide effectiveness in achieving persuasion.

Effect of Accident Due to Fall From Height at Construction Sites in Malaysia

Nationally, falls from height (FFH) are a significant threat to the construction fields and are one of the leading causes of a fatal accident to construction workers. Since the construction industry is carried out in hazardous environments, accidents occurring at various severity rates, leading to minor, severe and fatal injuries. Meanwhile, the majority of accidents are caused by a variety of significant causes and uncertain actions or unsafe conditions. The recognition effect of falls from height accidents at construction sites is the focus of this research. Therefore, this paper revealed the major effect of an accident due to a fall from height from past researchers. The reported cases of accidents were investigated by the Malaysian Department for Occupational Safety and Health (DOSH) were reviewed. The finding of this paper indicates a time loss of project execution due to accident investigation was the major effect of the accident due to fall from height at construction sites and cost implication for hiring a new worker, training for a new employee and compensation for injury or settlement of death claims. Discoveries of this paper will enhance the construction industry to improve the performance and regulation of all construction projects in terms of safety.

A novel possibilistic artificial immune-based classifier for course learning outcome enhancement

In this paper, we propose PAIRS3: a possibilistic classification approach based on artificial immune recognition system (AIRS) and the possibility theory. PAIRS3 is applied to address shortcomings in student attainment rates of course learning outcomes by predicting effective remedial actions through learning from assessment rubrics instances. For most of assessment rubric instances, it is difficult to determine the unique most effective remedial action to take. Consequently, each rubric instance will be labeled with uncertain remedial actions which are elicited from quality assurance experts. Elements from possibility theory are used to (1) model the uncertainty about the most effective remedial action labeling each rubric instance and (2) adapt several parts of the standard AIRS algorithm in order to address the uncertainty in class labels. The performance of the proposed method is evaluated against an academic, university level, assessment dataset that has been built progressively over multiple academic semesters. Despite the uncertainty related to the class labels in the dataset, PAIRS3 showed a good performance in terms of accuracy level (close to 75%). Also, when compared to existing state-of-the-art possibilistic classifiers such as PAIRS2, non-specificity possibilistic decision trees (NSPDT), and cluster-based possibilistic decision trees (Clust-PDT), PAIRS3 achieved better accuracy improvement ranging from 10% (in case of Clust-PDT) to 17% (in case of PAIRS2).

Climate Change Adaptation in Natural World Heritage Sites: A Triage Approach

Climate change is a certainty, but the degree and rate of change, as well as impacts of those changes are highly site-specific. Natural World Heritage sites represent a treasure to be managed and sustained for all humankind. Each World Heritage site is so designated on the basis of one or more Outstanding Universal Values. Because climate change impacts are site-specific, adaptation to sustain Universal Values also must be specific. As such, climate change adaptation is a wicked problem, with no clear action strategies available. Further, adaptation resources are limited at every site. Each site management team must decide which adaptations are appropriate investments. A triage approach guides that evaluation. Some impacts will be so large and/or uncertain that the highest probability of adaptation success comes from a series of uncertain actions that reduce investment risk. Others will be small, certain, comfortable and yet have low probable impact on the Universal Value. A triage approach guides the management team toward highest probable return on investment, involving stakeholders from the surrounding landscape, advancing engagement and communication, and increasing transparency and accountability.

Negativity Bias in the European Emissions Market: Evidence from High-Frequency Twitter Sentiment

High-frequency sentiment time series are extracted from Twitter data concerning the European emissions market and are used to explain returns and volatility in European emissions futures. The measures of negative sentiment are shown to Granger-cause EUA futures returns, while positive sentiment does not, indicating the presence of a negativity effect. Using a Threshold GARCH model, it is shown that periods of strong (weak) sentiment correspond with periods of high (low) volatility. Our findings are significant for understanding the role of sentiment in European emissions markets with implications for other markets subject to uncertain political actions and public opinions.

An extended version of opportunity cost algorithm for communication decisions

Decentralized Markov decision processes (DEC-MDPs) provide powerful modeling tools for cooperative multi-agent decision making under uncertainty. In this paper, we tackle particular subclasses of theoretic decision models which operate under time pressure having uncertain actions’ durations. Particularly, we extend a solution method called opportunity cost decentralized Markov decision process (OC-DEC-MDP) to handle more complex precedence constraints where actions of each agent are presented by a partial plan. As a result of local partial plans with precedence constraints between agents, mis-coordination situations may occur. For this purpose, we introduce communication decisions between agents. Since dealing with offline planning for communication increase state space size, we aim at restricting the use of communication. To this end, we propose to exploit problem structure in order to limit communication decisions. Moreover, we study two separate cases about the reliability of the communication. The first case we assume that the communication is always successful (i.e. all messages are always successfully received). The second case, we enhance our policy computation algorithm to deal with possibly missed messages. Experimental results show that even if communication is costly, it improves the degree of coordination between agents and it increases team performances regarding constraints.

174 Total antibody deficiency in B cell sufficient mice enhances atherosclerosis

Background and aim Extensive studies have revealed an important protective function for IgM antibodies from B1a cells in atherosclerosis, whereas the role of B2 cells and IgG antibodies is uncertain, with both pro- and anti-atherogenic actions reported. The aim of this study was to specifically target antibodies but not other B cell functions and determine the effect on mouse atherosclerosis. Methods We created MB1-Cre x XBP1 fl/fl mice that have B cell specific deficiency in the unfolded protein response transcription factor XBP1, previously reported to result in a specific antibody secretion defect. Specific antibody responses were assessed by ELISA and ELISpot after immunisation with NP-chicken gamma globulin. Chimeric Ldlr -/– mice with bone marrow from MB1-Cre x XBP1 fl/fl or control mice were fed a western diet for 8 weeks. Results B cell XBP1 deficiency led to significantly attenuated antibody secretion after NP-CGG immunisation. Analysis of ldlr –/– chimeric mice revealed enhanced atherosclerosis progression in MB1-Cre x XBP1 fl/fl recipients. These mice displayed severely attenuated serum IgM, IgG and IgE levels, but intact spleen, lymph node and peritoneal B cells. There were no differences in total cholesterol levels. Antibody deficiency was associated with enhanced pro-atherogenic Th1 CD4 + T cell immunity. Conclusions Antibodies have an overall protective effect on atherosclerosis and B2 cell antibodies are not responsible for enhanced atherosclerosis in IgM-deficient mice. Other B cell functions may be pathogenic in atherosclerosis.

Output Feedback Nonlinear General Integral Control

This paper proposes an output feedback nonlinear general integral controller for a class of uncertain nonlinear system. By solving Lyapunov equation, we demonstrate a new proposition on Equal ratio gain technique. By using Equal ratio gain technique, Singular perturbation technique and Lyapunov method, theorem to ensure regionally as well as semi-globally exponential stability is established in terms of some bounded information. Moreover, a real time method to evaluate the ratio coefficients of controller and observer are proposed such that their values can be chosen moderately. Theoretical analysis and simulation results show that not only output feedback nonlinear general integral control has the striking robustness but also the organic combination of Equal ratio gain technique and Singular perturbation technique constitutes a powerful tool to solve the output feedback control design problem of dynamics with the nonlinear and uncertain actions.

Planning robot manipulation to clean planar surfaces

This paper presents a new approach to plan high-level manipulation actions for cleaning surfaces in household environments, like removing dirt from a table using a rag. Dragging actions can change the distribution of dirt in an unpredictable manner, and thus the planning becomes challenging. We propose to define the problem using explicitly uncertain actions, and then plan the most effective sequence of actions in terms of time. However, some issues have to be tackled to plan efficiently with stochastic actions. States become hard to predict after executing a few actions, so replanning every few actions with newer perceptions gives the best results, and the trade-off between planning time and plan quality is also important. Finally a learner is integrated to provide adaptation to changes, such as different rag grasps, robots, or cleaning surfaces.We demonstrate experimentally, using two different robot platforms, that planning is more advantageous than simple reactive strategies for accomplishing complex tasks, while still providing a similar performance for easy tasks. We also performed experiments where the rag grasp was changed, and thus the behaviour of the dragging actions, showing that the learning capabilities allow the robot to double its performance with a new rag grasp after a few cleaning iterations.

A novel estimator based learning automata algorithm

Reinforcement learning is one of the subjects of Artificial Intelligence and learning automata have been considered as one of the most powerful tools in this research area. On the evolution of learning automata, the rate of convergence is the most primary goal of designing a learning algorithm. In this paper, we propose a deterministic-estimator based learning automata (LA) of which the estimate of each action is the upper bound of a confidence interval, rather than the Maximum Likelihood Estimate (MLE) that has been widely used in current schemes of Estimator LA. The philosophy here is to assign more confidence on actions that are selected only for a few times, so that the automaton is encouraged to explore the uncertain actions. When all the actions have been fully explored, the automaton behaves just like the Generalized Pursuit Algorithm. A refined analysis is presented to show the &#120598;-optimality of the proposed algorithm. It has been demonstrated by extensive simulations that the presented learning automaton (LA) is faster than any deterministic estimator learning automata that have been reported to date. Moreover, we extend our algorithm to the stochastic estimator schemes. It is also shown that the extended LA has achieved a significant performance improvement, comparing with the current state of the art algorithm of learning automata, especially in complex and confusing environments.

Nonlinear General Integral Control Design via Equal Ratio Gain Technique

This paper proposes two kinds of nonlinear general integral controllers, that is, one is generic and another is practical, for a class of uncertain nonlinear system. By extending equal ratio gain technique to a canonical interval system matrix and using Lyapunov method, theorems to ensure regionally as well as semi-globally asymptotic stability are established in terms of some bounded information. Moreover, for the practical nonlinear integral controller, a real time method to evaluate the equal ratio coefficient is proposed such that its value can be chosen moderately. Theoretical analysis and simulation results demonstrated that not only nonlinear general integral control can effectively deal with the uncertain nonlinear system but also equal ratio gain technique is a powerful and practical tool to solve the control design problem of dynamics with the nonlinear and uncertain actions.

Wilāyah (authority and governance) and its implications for Islamic bioethics: a Sunni Māturīdi perspective

Juridical councils that render rulings on bioethical issues for Muslims living in non-Muslim lands may have limited familiarity with the foundational concept of wilāyah (authority and governance) and its implications for their authority and functioning. This paper delineates a Sunni Māturīdi perspective on the concept of wilāyah, describes how levels of wilāyah correlate to levels of responsibility and enforceability, and describes the implications of wilāyah when applied to Islamic bioethical decision making. Muslim health practitioners and patients living in the absence of political wilāyah may be tempted to apply pragmatic and context-focused approaches to address bioethical dilemmas without a full appreciation of significant implications in the afterlife. Academic wilāyah requires believers to seek authentication of uncertain actions through scholarly opinions. Fulfilling this academic obligation naturally leads to additional mutually beneficial discussions between Islamic scholars, healthcare professionals, and patients. Furthermore, an understanding derived from a Māturīdi perspective provides a framework for Islamic scholars and Muslim health care professionals to generate original contributions to mainstream bioethics and public policy discussions.

Synthesis of nonlinear nonstationary control systems by output under conditions of uncertainty

A problem of synthesizing the control laws for nonlinear nonstationary control systems by output being measured under uncertain external actions and nonlinearities that meet the sector constraints is considered.

PREPROCESSING QUANTIFIED CONSTRAINT SATISFACTION PROBLEMS WITH VALUE REORDERING AND DIRECTIONAL ARC AND PATH CONSISTENCY

The Quantified Constraint Satisfaction Problem (QCSP) is an extension of the CSP that can be used to model combinatorial problems containing contingency or uncertainty. It allows for universally quantified variables that can model uncertain actions and events, such as the unknown weather for a future party, or an opponent's next move in a game. Although interest in QCSPs is increasing in recent years, the development of techniques for handling QCSPs is still at an early stage. For example, although it is well known that local consistencies are of primary importance in CSPs, only arc consistency has been extended to quantified problems. In this paper we contribute towards the development of solution methods for QCSPs in two ways. First, by extending directional arc and path consistency, two popular local consistencies in constraint satisfaction, to the quantified case and proposing an algorithm that achieves these consistencies. Second, by showing how value ordering heuristics can be utilized to speed up computation in QCSPs. We study the impact of preprocessing QCSPs with value reordering and directional quantified arc and path consistency by running experiments on randomly generated problems. Results show that our preprocessing methods can significantly speed up the QCSP solving process, especially on hard instances from the phase transition region.