Stationary Bandit Research Articles

Optimizing decision-making in uncertain environments through analysis of stochastic stationary Multi-Armed Bandit algorithms

Reinforcement learning traditionally plays a pivotal role in artificial intelligence and various practical applications, focusing on the interaction between an agent and its environment. Within this broad field, the multi-armed bandit (MAB) problem represents a specific subset, characterized by a sequential interaction between a learner and an environment where the agents actions do not alter the environment or reward distributions. MABs are prevalent in recommendation systems and advertising and are increasingly applied in sectors like agriculture and adaptive clinical trials. The stochastic stationary bandit problem, a fundamental category of MAB, is the primary focus of this article. Here, we delve into the implementation and analytical comparison of several key bandit algorithmsincluding Explore-then-Commit (ETC), Upper Confidence Bound (UCB), Thompson Sampling (TS), Epsilon-Greedy (-Greedy), SoftMax, and Conservative Lower Confidence Bound (CON-LCB)across various datasets. These datasets vary in the number of options (arms), reward distributions, and specific parameters, offering a broad testing ground. Additionally, this work provides an overview of the diverse applications of bandit problems across different fields, highlighting their versatility and broad impact.

An investigation of progress related to stochastic stationary bandit algorithms

The Multi-armed Bandit algorithm stands as a consequential tool for informed decision-making, distinct from reliance on intuitive selections, given its systematic proclivity to meticulously assess accessible alternatives with the intent of discerning the most auspicious outcome. Amid the repertoire of algorithmic variations, the Stochastic Stationary Bandit algorithm assumes a foundational and enduring role, finding versatile application across diverse domains, including but not limited to digital advertising, price optimization, and recommendation systems. With these considerations in view, the present study embarks upon a comprehensive scrutiny of this subject matter. This paper reviews on the Explore-Then-Commit algorithm, Upper Confidence Bound algorithm, and Thompson Sampling algorithm by explaining, comparing their formulation, features, and expected results. Explore-Then-Commit algorithm has distinct phase to explore all the choices uniformly. Upper Confidence Bound algorithm make decisions by calculate an upper confidence index which is an overestimate for each choice. Thompson Sampling algorithm depends on randomness to make choices. Explore-Then-Commit algorithm faces the problem of when to explore and when to stop. Upper Confidence Bound algorithm and Thompson Sampling algorithm solve this problem by avoid certain phases. Multi-armed Bandit algorithm could deal with the process of displaying items of potential interest to users in a recommendation system, the delivery of resources in resource allocation, or the way to maximize revenue in a business.

Beam Alignment for mmWave Using Non-Stationary Bandits

A wireless system communicating in the millimeter-wave (mmWave) band suffers from severe propagation loss and thus, employs a highly directional beam to mitigate the effect of propagation losses. However, such beams need to be aligned between the transmitter and receiver for any reliable communication to occur. In literature, this problem has been modeled as a multi-armed bandit. Most of the existing solutions for this problem use stationary bandit algorithms, which suit for time-invariant channels, whereas wireless channels are time-variant in nature. Thus, we propose two non-stationary multi-armed bandit based beam alignment BA algorithms, especially designed for time-varying channels. These algorithms perform a distributed BA between the transmitter and the receiver requiring minimal feedback. Using simulations, we have shown that the proposed BA algorithms minimize the outage probabilities and the regret compared to those of the exponential-weights algorithm.

The Stationary Bandit and His Stationary Captives

Mancur Olson’s stationary bandit model of government sees a ruler provide public goods in the form of protection from roving bandits, in exchange for the right to monopolise tax theft from a population. As it stands, this model implicitly treats the exchange as one between ruler, and homogenous citizenry. Yet it is obvious that a citizenry is made up of a heterogeneous group of individuals who have very different capacities to provide labour and other tribute. As such, rulers must be able to distinguish between these individuals. Using this expanded model, I show the way in which states have used means of administrative identity to distinguish between individuals for extractive and other purposes.

Two bandits or more? The case of Viking Age England

The Olsonian distinction between roving and stationary bandits outlines the rationale behind the transition from anarchy to the emergence of the predatory state. This two-bandit model may, however, be expanded to include more bandit types. In the case of Viking Age England, local English kings were unable to monopolize violence and defend their realms against competing Viking raiders. As the Vikings’ time horizon grew, so did the accumulated value of more formal taxation, and bandit types evolved in four steps. The first step is the Olsonian roving bandit, who executed Viking hit-and-run attacks and plunders during the second half of the tenth century. The second step is the gafol bandit; gafol is payment for leaving, paid to, among others, Swein Forkbeard. The third step is the heregeld bandit; heregeld is a tax to support an army for hire; most notably Thorkell the Tall’s. The fourth step is the Olsonian stationary bandit, i.e. the strongest military leader among the Vikings, Cnut the Great, settled down as the new king. Overall, the Olsonian two-bandit model can be expanded to a four-bandit staircase model, in which the new gafol and heregeld bandit types explain the steps from anarchy and short-run raiding to long-run formal taxation in a predatory state.

The Public Authority in the Modern Polity: The Corruption Phenomenon

This paper focuses on the link between the modern authoritarianism and corruption. Even though corruption plays an important role in Communist regimes, post-colonial dictatorships and authoritarian monarchies, coercion - which is a traditional tool used by authoritarian rulers - remains the basis of these regimes. However, a new type of non-democratic regimes, which we call neoauthoritarian, has emerged since the last quarter of the XX century. The new regimes are based on a dynamic interplay between coercion and corruption. That interplay allows authoritarian rulers to bring to the forefront either coercion or corruption, depending on the current political situation in the country and the political, economic and social issues on the political agenda. In this type of regimes, ruling political-economic groups capture the state and the public authority in the country and use all their instruments and resources to achieve their private goals. This paper presents empirical results showing that the Communist regimes, dictatorships and authoritarian monarchies exist in 33 modern non-democratic states, while neoauthoritarian regimes can be found in 19 states. I show that high levels of corruption are typical of all of these regimes, especially in dictatorships and neoauthoritarian ones. I explain a relatively lower level of corruption in the authoritarian monarchies using Olson's theory of stationary bandit. In particular, I speculate that the ruling monarchs fight corruption among bureaucrats since they perceive it as stealing their own property that damages the sources of their administrative rent and their revenues. At the same time, the high-level political corruption persists. Finally, I show that dictatorships are on average more fragile than economically elastic neoauthoritarian regimes, although it might be challenging to differentiate between them. All authoritarian and neoauthoritarian regimes, except a few monarchies, are non-stable regimes, allowing me to hypothesize their coming transformations or collapse.

Контрактные основы современного пенсионирования в России

The paper presents the analysis of pensioning as a social contract. The aim of research is to reveal the features of the Russian pension contract taking into account transformations of pension process in the 2000s. In accordance with the aims this article analyzes the contract bases of pension interactions. As a methodological basis the author applies the theory of social contracts, institutional model of the Principal-Agent and the McGuire-Olson theory of stationary bandit. The conducted research offers to consider pensioning as a special case of a social contract. It also identifies the general trend in changes of the Russian pension contract during the period since 2002 till the present period. The pension policy in 2002-2018 resulted in further strengthening of the state position as a stakeholder of the pension contract, decrease in the role of private structures in pensioning, replacement of pension accumulating with a distribution method, reduction in the value of personified interactions in the pension process and, on the contrary, increase in the role of common pension rules as an implicit basis of the pension contract.

Resource Cursed or Policy Cursed? US Regulation of Conflict Minerals and Violence in the Congo

AbstractThere is widespread belief that civil conflict in poorly governed countries is triggered by surging international demand for their natural resources. We study the consequences of US legislation grounded in this belief, the “conflict minerals” section of the 2010 Dodd-Frank Act. Targeting the eastern Democratic Republic of the Congo, it cuts funding to warlords by discouraging manufacturers from sourcing tin, tungsten, and tantalum from the region. Building from Mancur Olson’s stationary bandit metaphor, we describe some channels through which the legislation could backfire, inciting violence. Using georeferenced data, we find the legislation increased looting of civilians and shifted militia battles toward unregulated gold-mining territories. These findings are a cautionary tale about the possible unintended consequences of imposing boycotts, trade embargoes, and resource certification schemes on war-torn regions.

What does it take for a roving bandit settle down? Theory and an illustrative history of the Visigoths

A roving bandit provides exclusive (rivalrous) collective goods to members of its in-group. A stationary bandit further provides inclusive (non-rivalrous; public) collective goods to the out-group. The inclusive goods are an input to the production of the exclusive goods enjoyed by the in-group. As such, the transition from roving to stationary bandit is likely to involve the redefinition of the in-group, its collective interest, and the type of goods that it provides. To illustrate these points, I employ a case study of the roving Visigothic confederacy as it evolved during the fourth and fifth centuries towards the stationary Visigothic Kingdom. The illustration provides insights into why competition amongst roving bandits does not always (or often) lead to the emergence of a non-predatory state.

The Stationary Bandit Model of Intellectual Property

We propose a new model of intellectual property that presents a different view to the market failure/monopoly rent model advanced by Arrow (1962) in which governments protect inventors from private theft. Instead, using Olson (1993), we represent a public theft model of intellectual property arising when entrepreneurs acting in global markets seek protection from a stationary bandit (their home government) principally against the depredations of other governments (the roving bandits). We argue that this model explains why institutional quality matters to the global location of R&D intensive industries, such as biopharma, and why so much intellectual property is located in tax havens.

When does the emergence of a stationary bandit lead to property insecurity?

The economic theory of the state is often illustrated using examples in which the emergence of a stationary bandit—a ruler who is able to centralize military and fiscal capacity—improves property security. This article argues that the economic theory of the state also provides insight into the conditions when the emergence of a stationary bandit leads to property insecurity. In our reading of the economic theory of the state, the rise of a stationary bandit is only expected to improve property security writ large when coercive power is unquestioned, political institutions constrain rulers, powerful groups do not dominate the political process, and de facto property institutions are inefficient. The framework clarifies why Afghanistan’s first stationary bandit, Abdur Rahman, maintained state ownership over all land, waged violent repopulation campaigns, and drastically increased taxation before, during, and after centralizing state power from 1880 to 1901, while basic land use rights emerged from 1747 to 1880, a period when the state fell short of a monopoly on coercion.

Modern Authoritarianism and Corruption

This paper focuses on the link between the modern authoritarianism and corruption. Even though corruption plays an important role in Communist regimes, post-colonial dictatorships and authoritarian monarchies, coercion - which is a traditional tool used by authoritarian rulers - remains the basis of these regimes. However, a new type of non-democratic regimes, which we call neoauthoritarian, has emerged since the last quarter of the XX century. The new regimes are based on a dynamic interplay between coercion and corruption. That interplay allows authoritarian rulers to bring to the forefront either coercion or corruption, depending on the current political situation in the country and the political, economic and social issues on the political agenda. In this type of regimes, ruling political-economic groups capture the state and the public authority in the country and use all their instruments and resources to achieve their private goals. This paper presents empirical results showing that the Communist regimes, dictatorships and authoritarian monarchies exist in 33 modern non-democratic states, while neoauthoritarian regimes can be found in 19 states. I show that high levels of corruption are typical of all of these regimes, especially in dictatorships and neoauthoritarian ones. I explain a relatively lower level of corruption in the authoritarian monarchies using Olson's theory of stationary bandit. In particular, I speculate that the ruling monarchs fight corruption among bureaucrats since they perceive it as stealing their own property that damages the sources of their administrative rent and their revenues. At the same time, the high-level political corruption persists. Finally, I show that dictatorships are on average more fragile than economically elastic neoauthoritarian regimes, although it might be challenging to differentiate between them. All authoritarian and neoauthoritarian regimes, except a few monarchies, are non-stable regimes, allowing me to hypothesize their coming transformations or collapse.

Visigothic Retinues: Roving Bandits that Succeeded Rome

A roving bandit provides exclusive (rivalrous) collective goods to members of its in- group. A stationary bandit further provides inclusive (nonrivalrous; public) collective goods to the out-group. The inclusive goods are an input to the production of the exclusive goods enjoyed by the in-group. As such, the transition from roving to stationary bandit is likely to involve the redefinition of the in-group, its collective interest, and the type of goods that it provides. To illustrate these points, I employ a case study of the roving Visigothic confederacy as it evolved during the fourth and fifth centuries towards the stationary Visigothic Kingdom. The illustration provides insights into why competition amongst roving bandits does not always (or often) lead to the emergence of a non-predatory state.

State Metallurgy Factories and Direct Taxes in the Urals, 1700–50: Paths to State Building in Early Modern Russia

Historical-sociological descriptions of rise and development of modern bureaucratic attribute great significance to war and taxation. According to so-called military-taxation theory, collects taxes, which grant it resources to survive in geopolitical competition. The ability of a to regularly raise necessary tax revenue to conduct war is viewed as one of most important indicators of successful building. According to this theory, that adopt most productive taxes and institutionalize most modern forms of tax administration are able to mobilize ever greater quantities of labor and materiel and therefore have edge in this perpetual arms race.... In long run, military competition led all surviving states to converge on efficient and productive tax systems, and those tax systems in turn led to militarization and bureaucratization of society. (1) The history of early modern Russian state, too, has been analyzed through prism of military-taxation theory. Chester Dunning and Norman Smith detect a fiscal-military in Russia in 16th century, while Endre Sashalmi prefers period between second half of 17th century to first quarter of (18)th century for emergence of a Russian fiscal-military state. (2) By contrast, Janet Hartley concludes that Russia could hardly be defined as a fiscal-military even in (18)th century. (3) The observations and conclusions of these authors (as well as my own) should be taken as preliminary and open to discussion, because of both current of historical research and existence of important theoretical problems that still require further attention. From 20th century through present day, historians of taxation have reconstructed budget, yearly fluctuation of returns from various types of taxes, and structure of tax apparatus. They have also reconstructed composition of tax apparatus and its daily activities related to collection of taxes and revenue from population. (4) Yet more research is required, working with sources of (16)th through (19)th centuries to resolve new and important theoretical problems. One such question concerns manner in which we approach itself. As Douglass North, John Wallis, and Barry Weingast note, economists and social scientists concerned with understanding how develops and interacts with larger society have modeled as a revenue-maximizing monarch, a stationary bandit, or a single-actor 'representative agent.' By overlooking reality that all states are organizations, this approach misses how internal dynamics of relationships among elites within dominant coalition affect how states interact with larger society. (5) Their approach may not be entirely new, after all, Max Weber already recognized as a human community, one that claims a monopoly of legitimate use of physical force within a given territory. (6) North, Wallis, and Weingast are right, however, to remind us that as a human community is an organization, and that its internal structure and institutions have had a decisive effect on course of building. (7) In Russia, state emerged in late (15) th century as an organization of hereditary landowning class (termed nobility in (18)th century), centralized and united around figure of monarch. Social collaboration within ruling class was in many respects what allowed Russian autocracy to function. Regionally based, independent magnates were absent. As David Goldfrank notes, the aristocracy contributed only glue, not centrifugal counter-force to central establishment. (8) Occupying a office was both right and duty of ruling class. The Russian administrative apparatus of (17)th and (18)th centuries performed function of a corporate structure for ruling class. …

Field experiments on irrigation dilemmas

It is often assumed that irrigation systems require a central authority to solve coordination problems due to the asymmetry in position and influence between those located at the head-end of a system and those located at the tail-end. However, many examples of complex irrigation systems exist that are self-organized without central coordination. Field experiments on asymmetric commons dilemmas are performed with villagers in rural Colombia and Thailand. Our experiments show that there is a dynamic interaction between equality in the use of the common resource, and the level of the contributions to the creation of a common resource. Inequality in the distribution of benefits in one round triggers lower levels of group contributions, reducing efficiency and triggering even more inequality in contributions and distribution of the resource among players.The upstream players act as “stationary bandits”. They take more than an equal share of the common resource, but leave sufficient resources for the downstream players to stimulate them to continue their contributions to the public infrastructure.After 10 rounds, players can vote on one of three allocation rules: equal quota, random and rotating access to appropriation of the resource. The rotating access is most often elected. The resource dynamics in the second part of the experiment depend on the rule elected. With the quota rule, the stationary bandit metaphor is less relevant since taking equal shares of the resource is enforced. With the rotation access rule, the players act strategically on the rotating position. They invest more when having the first access to the resource compared to less favorable access. And when they have first access they extract the main part of the common resource. The rotation rule led to a reduction of the performance of the groups. With the random access rule there is no such strategic investment behavior and participants remain investing equal and similar levels as in the first 10 rounds.The experiments show that a necessary condition of irrigation systems to self-organize is the development of norms to allocate fair shares of the water in order to recruit sufficient labor to construct and maintain the physical infrastructure. The different allocation rules do not increase efficiency, but they did increase equality of the earnings.

Individual rights as a factor of economic convergence

This paper attempts to contribute to the understanding of how democracy and individual rights associated with it affect economic development. The market process approach to innovation (and entrepreneurship) is combined with the stationary bandit approach to property rights to derive a proposition concerning how the lack of exogenous constraints on the government can retard innovation. According to this, a credible commitment to secure property rights through respecting individual or human rights becomes an important factor for the process of catching up for those countries that are relatively close to the most developed ones in terms of technology and thus income. The proposition is tested on cross-country panel data using a measure of civil rights as a proxy for the commitment to secure individual rights. This argument provides one possible explanation for the fact that there is a strong correlation between democracy and income while the correlation between democracy and growth is weak.

An Agent-Based Computational Model of the Predatory State

Olson (1993) has proposed that a self-interested bandit has an incentive to become stationary and provide public goods within his domain, in particular protection of property rights, particularly against other roving bandits. The consequent increased social productivity permits the stationary bandit (dignified with the title of king) to obtain a greater personal return than is obtained by roving bandits preying on a poorer society. At the same time, increased security provides the subjects with a greater incentive to invest, thereby leading to longer term increases in social wealth. Olson`s analysis complements the property-rights view of development most closely associated with Douglass North (North & Thomas (1973), North (1981)). North argues that respect by rulers for the property rights of subjects is responsible for economic development by providing an incentive for productive investment. Olson`s theory gives an explanation of why a self-interested ruler might choose to respect property rights. This paper presents a spatial evolutionary game-theoretic agent-based computational model of the North/Olson argument.

The Political Economy of Stalinism: Evidence from the Soviet Secret Archives by Paul R. s> Gregory (review)

REVIEWS 539 possible mistakes of their leaderships. To do so while underestimating the broadercontext is, however, highlyuncritical.The overwhelmingresponsibility for their fate has to be sought in Stalin's policies and in Britain and America'sfailureeitherto understandor to oppose them. PoliticsDepartment GEORGE SANFORD University ofBristol Gregory, Paul R. ThePoliticalEconomy ofStalinism. Evidence fromtheSoviet Secret Archives. Cambridge University Press, Cambridge, 2004. xi+ 308 pp. Tables. Figures. Notes. Appendices. Bibliographical references. Index. ?2 I.99: $3 I.99 (paperback). THE Soviet Union represented the most ambitious economic experiment of the twentieth century. Its creation dramatically altered the course of world history and its repercussions, political, social and humanitarian as well as economic, were enormous both for the countryitselfand for the wider world. Economically the key change occurredin the early I930S when Stalin and his supporters created, in a rather ad hoc fashion, the economic system that would endure until the collapse of the Soviet Union in I99I. This was an administrative-commandeconomy based on the tripartitepillarsof planning, stateownershipand primitiveaccumulation. Until I99I Western scholars had to rely on published sources for their analysisof the Soviet administrative-commandeconomy. However, since then scholarsfrom both the West and the former Soviet Union have been able to graduallyutilize the unpublishedsourcesof the former Soviet administration. Paul Gregory is one of the leading economic historians of the Soviet Union and has been at the forefront of this new research programme. His book focuses on the first two decades of the administrative-command economy system,when Stalinruled.However, it is Gregory'scontention thatthe system exhibited a greatdeal of inertiaand did not change radicallyuntil its collapse. Thus, the book in effectdealswith the whole period of the Soviet administrative -command economy and it includesa chapteron why the systemultimately failed. After setting out its main themes and explaining how the system came about, the main part of the book (chaptersfour to nine) provides an analysis of how the Soviet economy operated. Gregoryutilizes the work of Olson and Hayek and Arrow'sImpossibilityTheorem to argue that a Stalin-likedictator is inevitable in an administrative-commandeconomy. In practice, following Mancur Olson, he also thinks that dictator Stalin operated like a stationary bandit who, in pursuit of his own self-interest, must maximize growth and development. However, the system that evolved had more than just Stalin's imprinton it. Forexample, one characterwho emergesstronglyin the book is Sergo Ordzhonikidze,head oftheHeavy IndustryMinistryuntilhe committed suicide in February 1937: 'it is difficult to separate the creation of Soviet industryfrom Ordzhonikidze' (p. I56). Also, Gregorystressesthat the system can be characterized as a 'nested dictatorship',in which layer after layer of the system replicated the administrativeand control structuresof the layer 540 SEER, 83, 3, 2005 above itself,with each layer consisting of a superior(in effect, a dictator)and their subordinates. Hence, the system was populated by hundreds, possibly thousands, of 'smaller Stalins'. Endemic to such a system is the principalagent problem, based on the information imbalance between dictator and subordinates.According to Gregory, an important reason for the survivalof the Stalinist system after his death was that his successors could not find a solution to this problem. The result was that the system operated at a lowlevel equilibrium,characterizedby 'planningfromthe achieved level'. Among otherinnovationsin the book, GregoryadaptsAkerloff'sfairwageeffort model to try to explain the investment cycle paradox: that whilst the Soviet system was dedicated to industrialcapital accumulation, periodically investment plans were cut back and, in effect, investment was sacrificedfor consumption. The model predictsthat workerswill reduce their effortif they feel that they are being paid a wage below the fair wage. Thus 'the Soviet leadership chose optimal investment by weighing investment against the loss of workereffort'(p. iIo). He also emphasizes that among the many different forms of rent that opportunisticagents could extract in the systemwere rents derived from illegal, or unauthorized, horizontal transactions.Such transactions were common but they threatened the system by removing the control of transactionsfrom the hands of the planner and by encouraging enterprises to createtheirown unofficialmoney, in theformof surrogatesand commercial credit,which fuelledinflation. This is an importantbook as it presents the firstsystematicaccount of the Soviet economic system using these unpublished sources, drawing together the new researchby Gregory and others, in a form that is easily accessible to the...

Limited Autocracy

All politicians, regardless of the nominal form of government within which they operate, face the trade-off between current period gains and tenure extension. That is, rulers can exploit their power for personal gain, but they risk being removed from their positions of power, either through a popular vote or a coup or revolution. If they temper their exploitation to remain in power, they sacrifice some of their current personal gain. Essentially all politicians are limited autocrats, where the limitations imposed on them differ according to the institutional structure under which they rule. This paper presents a formal model of this trade off in the Mancur Olson stationary bandit framework, where tenure length is explicitly endogenized in the politician’s maximization problem.

Was Mancur a Maoist? An Essay on Kleptocracy and Political Stability

Much of Mancur Olson's work explored the link between government structure and economic development. This paper provides a framework for thinking about this link that exposes both the powerful insights and the deep tensions in Olson's work. In The Rise and Decline of Nations Olson argued that instability was good for democratic accountability because it upset entrenched interests. In contrast, after the fall of the socialist regimes in Europe and the Soviet Union, Olson argued that the stability of a single autocrat or “stationary bandit” was superior to the competitive rent seeking of competing “roving bandits.” I argue that there is a real inconsistency in Olson's thinking on the role of stability and change in political life; I do this by developing the connections between Olson's classic Logic of Collective Action and his subsequent writing. The paper concludes by building on Olson's insights to point the way to a more complete analysis of democracy and transition.