Reward Mechanisms Research Articles

Soft actor-critic DRL algorithm for interval optimal dispatch of integrated energy systems with uncertainty in demand response and renewable energy

The collaborative optimization dispatching of multiple energy flows plays a crucial role in achieving the economic and efficient low-carbon operation of integrated energy systems (IESs). However, the dispatching problem for IESs is characterized by high dimensionality, non-linearity, and complex coupling. Furthermore, the integration of renewable energy sources and flexible loads has transformed the IES into a complex dynamic system with significant uncertainty. Traditional intelligent optimization algorithms exhibit poor adaptability and lengthy solution computation time when tackling such problems. In contrast, deep reinforcement learning (DRL), as an interactive trial-and-error learning method, has shown improved decision-making capabilities. In view of this, a data-driven soft actor-critic (SAC) deep reinforcement learning-based approach is proposed in this paper for interval optimal dispatch of IESs considering multiple uncertainties. First, the basic principle of SAC reinforcement learning is introduced in detail, and the basic framework of reinforcement learning for interval optimal scheduling of IESs is constructed. Then, the environment model of agent interaction is constructed, and the action and state space of DRL, as well as the reward mechanism and neural network structure, are designed. Finally, a typical IES case is experimentally analyzed and compared with three popular DRL algorithms and five state-of-the-art intelligent optimization algorithms. The experimental results demonstrate the advantages and effectiveness of the proposed method in solving the optimal dispatching of IESs.

Towards Understanding Metaverse Engagement via Social Patterns and Reward Mechanism: A Case Study of Nova Empire

With the constant fusion of virtual and reality, a new vision of human beings has emerged—Metaverse. At present, the development of metaverse is still in its infancy, and although the industry has put feverish investment into it, there are still many problems that need to be discussed in academia. The first thing is user engagement. Metaverse relies heavily on massive online users to realize its social value. In other words, user engagement is the foundation of the metaverse ecosystem. Therefore, our research uses <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Nova Empire</i> as a case study, aiming at understanding metaverse engagement via complexity analysis of social patterns and reward mechanisms. Specifically, the behavioral data of 46 954 players in September 2021 are used for analysis. Our observations show that social behavior is not the main factor for user engagement. Then, we perform a correlation analysis of trigger times for different gaming behaviors to verify the above observations. The results prove that the main factors affecting user engagement are game tasks and reward mechanisms in the early stage and social gaming behaviors environment with alliances in the middle and late stages. Finally, we discuss the implications of our findings for the design of future games and propose operation strategies on user engagement to improve the metaverse ecosystem and other online social space.

VerFHS: Verifiable Image Retrieval on Forward Privacy in Blockchain-Enabled IoT

In the scenario of the Internet of Things (IoT) co-located with the cloud, many applications such as face recognition, traffic monitoring and medical diagnosis usually outsource a large amount of generated image data to cloud servers to reduce the burden of local storage. Many secure encrypted image retrieval schemes have been proposed to protect data privacy. However, existing work incurs storage and communication burdens, lacks verifiability of query results and has potential forward security threats. To solve these issues, we propose the VerFHS framework in this paper, which can satisfy Verifiability, Feedback, High-Security. Specifically, we first present an extended secure k-NN algorithm to protect indexes, cleverly design ciphertext inner product for similarity comparison, and use the reward mechanism of blockchain to build a monitoring and feedback mechanism for cloud servers. Then we demonstrate an enhanced VerFHS scheme in the dynamic setting (VerFHSD) that uses a permutation matrix to process image encryption against adaptive attacks during dynamic updates. VerFHSD prevents cloud servers from making search queries over newly added images via previous tokens, thereby achieving forward security. The formal security analysis shows that our schemes protect the privacy of images, indexes & query tokens and forward security. And extensive experiments using the real-world dataset demonstrate that our scheme not only has the highest search accuracy all the time, but also achieves efficient queries at the millisecond level, when compared with other advanced image retrieval schemes.

Cross-regional analysis of RRM design and implementation in mobile games by developers in China, the EU, Japan, and the USA

The broad adoption of random reward mechanisms (RRMs) in digital games requires cross-regional analysis and discussions on RRMs to implement socially acceptable RRM configurations. This study clarifies overall trends, cross-regional common practices, and region-dependent characteristics of RRM design, by empirically and comparatively examining RRM implementations in four regions through an integrated qualitative coding analysis and quantitative principal component analysis (PCA). We constructed an empirically grounded taxonomy of RRMs, consisting of 73 labels in 11 categories and employed these labels in a coding analysis of 40 top-grossing mobile games by Chinese, European, Japanese, and US developers. We found 293 RRMs, to which we applied 3,223 codes and subsequently conducted a PCA with the frequency of assigned codes to identify similarities and differences in RRMs across these regions. The results indicated a higher coherence of RRM designs in Japan than in other regions, arguably owing to a relatively long history of RRMs, higher social acceptance, and robust industry-initiated self-regulation. Our integrated method of qualitative coding analysis and PCA is viable for further comparative studies on game design.

Factors Influencing Crowdworkers’ Continued Participation Behavior in Crowdsourcing Logistics: A Textual Analysis of Comments from Online Platforms

With the lazy economy’s rise in the digital era, the demand for crowdsourcing logistics delivery is increasing. In this process, the continued participation of crowdworkers has been a considerable challenge. In order to further clarify the influencing factors of crowdworkers’ continuous participation and better and targeted incentives for their participative behavior, we use ROST-CM 6.0 software to conduct textual analysis on 3000 comments from crowdworkers on China’s Meituan and Hummingbird crowdsourcing logistics platforms. The results show that the order dispatch system, reward and punishment system, and platforms’ service are the key factors concerned by crowdworkers. The total negative sentiment among crowdworkers regarding crowdsourcing logistics platforms is close to 20%. We also find that crowdsourcing logistics platforms still have room for improvement in the quantity and quality of orders dispatched, the evaluation factors and the appeal system for reward and punishment rules, and the freedom and flexibility of distribution work. Otherwise, this might lead to a trust issue between crowdworkers and the crowdsourcing logistics platform. Based on the research findings, we recommend that the crowdsourcing logistics platforms should enhance service awareness, provide a better work experience for crowdworkers, and optimize platform functions. The government should act as a regulator as well as a service provider. This paper’s innovations include methodologically, from the perspective of the crowdworkers, online comment texts are used to mine the behavioral factors that influence the crowdworker’s continued participation in crowdsourcing logistics; content-wise, it adds fresh insights to existing research on how the order allocation system and platform reward and punishment mechanisms affect the crowdworkers’ continuous participation behavior.

Research on the Architecture of Transactional Smart Contracts Based on Blockchains

At present, smart contracts are designed based on the application field, and their structure and functions are closely related to specific businesses. Even smart contracts within the same field and the same business have different structures and functions due to different developers, resulting in structural confusion, repeated development, and low levels of sharing. In response to this problem, this study conducts a full investigation of smart contracts in various fields, using big data technology to compare and analyze the structures of each contract and extracting the common content of their main bodies to study each feature, as well as to conduct induction and fusion. This study also generally designs a hierarchical structure and formulates structural modules such as transaction rules and the analysis of rights and responsibilities, as well as a reward and punishment mechanism. Data traceability is established, and the overall architectural specification of smart contracts is constructed. Additions, deletions, and improvements are made based on specific application environments to realize the dynamic updates of the architecture of contracts. Experiments show that the architecture of contracts can realize the various functions required in a transaction, solve the problem of the repeated development of current transactional smart contracts, and improve the sharing level.

The Macro Prudential Assessment Framework of China

This paper first introduces the basic concepts of macro-prudential assessment (MPA), and briefly introduces the background of MPA policy implementation, the key indicators of process and MPA assessment, the reward and punishment mechanism, as well as the purpose of China's macro-prudential assessment system. On this basis, this paper selected 15 listed banks as samples to analyze the achievement of the key indicators of MPA. In addition, it analyzes the influence of MPA evaluation system on bank management, introduces the broad credit standard to restrain the asset allocation mode; capital becomes the core constraint; the development mode needs to be adjusted. In addition, the countermeasures of strengthening the capital control, cultivating the awareness of enterprise capital saving, strengthening the strategic management, innovating the asset allocation system, improving the price management system, promoting reasonable pricing, innovating the development model, doing risk prevention and other improvement measures under the MPA mechanism are discussed.

Rewards, Penalties and Real Case References: Improvement of the Last Mile Delivery in USA

The growth of the platform economy has fueled the rapid growth of the cargo transportation industry. However, while it continues to scale up, many problems of the existing freight transportation industry are being exposed - environmental damage, harming consumers, and even depriving producers of profits. The last-mile problem is particularly significant. The United States and China, two of the world's largest countries, have representative platform economy enterprises, namely Amazon and Taobao. These two companies' different freight models represent the two existing mainstream freight models in the world - free home delivery and stagecoach containers. By comparing the two companies' existing last-mile delivery problems and analyzing the two countries' different cultural backgrounds, this paper tries to solve the existing last-mile problems by introducing reward and punishment mechanisms and applying the two models to each other. In this way, enterprises in each country can learn from each other's distribution models and find the optimal solution for the platform economy distribution model.

INNOVATION CULTURE AS A FACTOR OF COMPETITIVENESS IN A PUBLIC COMPANY OF ENERGY SOLUTIONS: A CASE STUDY AT CEMIG SIM

The article discusses the culture of innovation as a factor to stimulate competitiveness in an energy solutions company. The research, of an exploratory, descriptive and qualitative approach, aimed to identify, from the structured interview and the documentary research, which are the elements to stimulate the culture of innovation adopted by Cemig Sim, subsidiary of energy solutions of Companhia Energética de Minas Gerais. The script of questions consisted of 22 questions, being applied to the Marketing, Culture and Customer Success Manager. After data collection, the guidelines to encourage innovation adopted by Cemig Sim were compared with the model proposed by Dutra e Almeida, from 2018, in order to analyze the practices carried out by the company and point out the highlights and possible opportunities for improvement. It was found that of the 33 variables proposed by the aforementioned model, Cemig Sim fully fulfills 26 categories, representing 78.7% of completeness. Initiatives such as error tolerance; learning organizations; cross-functional teamwork; processes and tools for generating new ideas; meet current and potential customers/search for ideas outside the organization; explicit top management support for innovation; creating and maintaining a flexible organizational structure and employee autonomy. However, the company fails to adopt important initiatives for the innovation management process, such as establishing reward and recognition mechanisms for creative behavior; job rotation and; innovation metrics. It is suggested the development of practices that contemplate and encourage such activities in the company.

Studying the association between Gitcoin’s issues and resolving outcomes

The development of open-source software (OSS) projects usually have been driven through collaborations among contributors and strongly relies on volunteering. Thus, allocating software practitioners (e.g., contributors) to a particular task is non-trivial and draws attention away from the development. Therefore, a number of bug bounty platforms have emerged to address this problem through bounty rewards. Especially, Gitcoin, a new bounty platform, introduces a bounty reward mechanism that allows individual issue owners (backers) to define a reward value using cryptocurrencies rather than using crowdfunding mechanisms. Although a number of studies have investigated the phenomenon on bounty platforms, those rely on different bounty reward systems. Our study thus investigates the association between the Gitcoin bounties and their outcomes (i.e., success and non-success). We empirically study over 4,000 issues with Gitcoin bounties using statistical analysis and machine learning techniques. We also conducted a comparative study with the Bountysource platform to gain insights into the usage of both platforms. Our study highlights the importance of factors such as the length of the project, issue description, type of bounty issue, and the bounty value, which are found to be highly correlated with the outcome of bounty issues. These findings can provide useful guidance to practitioners.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Solving combined economic and emission dispatch problems using reinforcement learning-based adaptive differential evolution algorithm

Nowadays, economic and environmental concerns in production have become increasingly significant. To address these issues, the Combined Economic and Emission Dispatch (CEED) problem has been introduced to optimize the power generation process by considering fuel cost and emitted substances. However, due to the nonlinearity and nonconvexity of the objective function, the optimization of CEED remains a challenge. In this paper, we develop a Reinforcement Learning-based Adaptive Differential Evolution (RLADE) algorithm to enhance the optimization performance. The mutation strategy and crossover probability of RLADE are optimized using Reinforcement Learning (RL) to respectively ensure better convergence speed and searchability. Additionally, two modifications of RL, namely the adaptive population size-based state division and fitness-ranking-based reward mechanism, are proposed to improve the accuracy of state division and reward calculation in RL. The experiments conducted in this paper consider two objective formulation methods of CEED problems, namely the quadratic and cubic criterion functions. The mean values and standard deviations of the obtained solutions were utilized to assess the performance of RLADE, as well as other comparative algorithms, namely DE algorithm and two RL-based DE variants. The results clearly demonstrate that RLADE surpasses its counterparts with proportion of 100%, 85.7%, and 100% for the 6-unit and 11-unit quadratic CEED problems, as well as cubic criterion functions, in terms of both search accuracy and convergence ability. Furthermore, the significance of RLADE's superiority is confirmed through the Wilcoxon's signed rank test.

Low-carbon Economic Scheduling of Combined Cooling, Heating and Power Microgrid Considering Demand Response and Shared Energy Storage

In order to further promote the new energy consumption of combined cooling, heating, and power (CCHP) microgrid, reduce carbon emissions, and optimize the total operating cost of the microgrid, a low-carbon optimal scheduling model of CCHP microgrid was proposed. It considered demand response (DR) and shared energy storage (SES). Firstly, the concept of SES and its operation mode are applied to the microgrid. Secondly, in order to take advantage of the flexible nature of multiple loads, an integrated demand response (IDR) model with a ladder subsidy mechanism was established. Then, the optimal scheduling model introduces a reward and punishment ladder carbon trading mechanism (RPL-CTM). Finally, taking energy purchase cost, load reduction subsidy cost, interaction cost with energy storage station, carbon trading cost, and wind abandoning penalty cost as optimization objectives, a multi-energy coupling, and multi-objective collaborative optimization scheduling model was established. Numerical analysis indicates that the model can efficiently lower the microgrid’s operating cost and encourage the use of clean energy and carbon emission reduction.

Crystal: Enhancing Blockchain Mining Transparency With Quorum Certificate

Researchers have discovered a series of theoretical attacks against Bitcoin's Nakamoto consensus; the most damaging ones are selfish mining, double-spending, and consistency delay attacks. These attacks have one common cause: block withholding. This paper proposes Crystal, which leverages quorum certificates to resist block withholding misbehavior. Crystal continuously elects committees from miners and requires each block to have a quorum certificate, i.e., a set of signatures issued by members of its committee. Consequently, an attacker has to publish its blocks to obtain quorum certificates, rendering block withholding impossible. To build Crystal, we design a novel two-round committee election in a Sybil-resistant, unpredictable and non-interactive way, and a reward mechanism to incentivize miners to follow the protocol. Our analysis and evaluations show that Crystal can significantly mitigate selfish mining and double-spending attacks. For example, in Bitcoin, an attacker with 30% of the total computation power will succeed in double-spending attacks with a probability of 15.6% to break the 6-confirmation rule; however, in Crystal, the success probability for the same attacker falls to 0.62%. We provide formal end-to-end safety proofs for Crystal, ensuring no unknown attacks will be introduced. To the best of our knowledge, Crystal is the first protocol that prevents selfish mining and double-spending attacks while providing safety proof.

Unleashing Intrapreneurial Behavior: Exploring Configurations of Influencing Factors among Grassroots Employees.

Effectively promoting employees' intrapreneurial behavior has become the focus of enterprises. This study takes the middle and grassroots employees in enterprises as subjects and explores the configuration effect of multiple influencing factors on employees' intrapreneurial behavior. Based on employee expectation theory and individual-environment matching theory, this study collates six influencing factors: entrepreneurial self-efficacy, entrepreneurial competence, task school level, perceived value, management support, and reward mechanism. A total of 163 samples were obtained, and the qualitative comparative analysis method based on fuzzy set was used to analyze the influence mechanism and result path of employees' intrapreneurial behavior from the perspective of the interaction between individual factors and organizational factors. Six influencing paths of employees' high intrapreneurial behavior were found, which can be divided into ability-driven and value-driven factors, revealing that the six factors can produce equivalent results in different configurations. Furthermore, five influencing paths of employees' non-high intrapreneurial behavior were divided into three types: ability obstacle type, perception obstacle type, and value obstacle type. These have an asymmetric causal relationship with employees' high intrapreneurial behavior. This study provides management support for effectively stimulating employees' intrapreneurial behavior.

Air Channel Planning Based on Improved Deep Q-Learning and Artificial Potential Fields

With the rapid advancement of unmanned aerial vehicle (UAV) technology, the widespread utilization of UAVs poses significant challenges to urban low-altitude safety and airspace management. In the coming future, the quantity of drones is expected to experience a substantial surge. Effectively regulating the flight behavior of UAVs has become an urgent and imperative issue that needs to be addressed. Hence, this paper proposes a standardized approach to UAV flight through the design of an air channel network. The air channel network comprises numerous single air channels, and this study focuses on investigating the characteristics of a single air channel. To achieve optimal outcomes, the concept of the artificial potential field algorithm is integrated into the deep Q-learning algorithm during the establishment of a single air channel. By improving the action space and reward mechanism, the resulting single air channel enables efficient avoidance of various buildings and obstacles. Finally, the algorithm is assessed through comprehensive simulation experiments, demonstrating its effective fulfillment of the aforementioned requirements.

Location and path planning for urban emergency rescue by a hybrid clustering and ant colony algorithm approach

Rescue station setup and rescue path planning are two important tasks in urban emergency rescue. The former task ensures rescue response capability and the latter task provides effective rescue solutions. When emergencies occur in cities, evacuees are distributed along the urban road network. Rescue resources refer to rescue vehicles whose available number and capacity are both limited. With the constraints of rescue resources and the number of rescues, this paper aims to simultaneously optimize the tasks of rescue station setup and rescue path planning. In the addressed scenario, the priority of each evacuee is quantified as a weight value that is used as the main optimization objective. To solve the problem, a comprehensive urban emergency rescue planning approach is proposed. The proposed approach consists of components of road network processing, road network weight calculation, rescue station setup and rescue path planning. For the setup of rescue stations, this paper employs a clustering method to provide a set of high-quality candidate rescue stations for subsequent path planning based on the locations of evacuees and the road network structure. For rescue path planning, an improved ant colony optimization algorithm is developed. The proposed method is called the planning algorithm with clustering and improved ant colony optimization (PA-C-IACO). The proposed PA-C-IACO redefines the degree of heuristic and pheromone concentration increments for transfer between intersections in the ant colony algorithm and incorporates a reward mechanism during the pheromone update process. Experimental results on six different size datasets show that PA-C-IACO outperforms state-of-the-art algorithms and shows good robustness and feasibility.

Fibonacci Group Consensus Algorithm Based on Node Evaluation Mechanisms

In response to challenges posed by the Practical Byzantine Fault Tolerance (PBFT) consensus algorithm, where all nodes are involved in the consensus process, issues such as high communication overhead in the three-phase procedure, random selection of master nodes, and the absence of reward and penalty mechanisms are addressed. This leads to the proposal of a Speculative Practical Byzantine Fault Tolerance (SP-PBFT) consensus algorithm based on a node evaluation mechanism for speculative grouping. Firstly, the consensus protocol was optimized, and a timeout mechanism was proposed to divide the consensus process into an optimistic mode and a pessimistic mode, which reduced the communication overhead under the premise of resisting Byzantine node attacks. Secondly, a node evaluation mechanism was proposed to calculate the node reputation value according to the node’s historical behavior and give the corresponding reward or punishment, which reduced the possibility of malicious nodes participating in the consensus process. Finally, the Fibonacci grouping mechanism was used to reduce the number of nodes participating in the consensus process, fundamentally improving the consensus efficiency, avoiding the problem of centralization of the consensus process caused by the cumulative reputation value of nodes, and improving the enthusiasm of consensus nodes. Simulation experiments using Docker containers to simulate multiple nodes show that the SP-PBFT consensus algorithm proposed in this paper has better performance than the PBFT consensus algorithm and other improved algorithms in terms of consensus delay, throughput, fault tolerance and communication complexity.

Neuronal activity and reward processing in relation to binge eating.

Studies increasingly show the importance of reward processing in binge eating and provide evidence of associated changes in the neurobiological reward system. This review gives an up-to-date overview of the neurobiological substrates of reward processing subconstructs in binge eating. Neural findings are linked to different behavioral theories and the clinical relevance is discussed. Increased neural responses in the orbitofrontal cortex, anterior cingulate cortex as well as striatum during anticipation and receipt of food rewards are found in association to binge eating. Increased model-free learning is also found and associated with altered brain reward reactivity. Data in rest report reduced striatal dopamine release and lower frontostriatal connectivity. Mechanisms of onset of binge eating are less clear, but specific personality traits, related to frontostriatal dysconnectivity, probably increase the risk of binge eating onset. Both structural and task-based imaging studies show differences in the neurobiological reward system in binge eating. These changes are linked to specific reward processing, such as altered reward responsiveness to food cues, reinforcement learning, and habitual behavior. Findings are lined with different behavioral theories of binge eating, and a staging model is described, from onset to full illness development. Understanding the specific underlying aberrant reward mechanism in binge eating, associated with different stages of the illness, enables caregivers to focus their treatment more precisely.

Evolutionary game analysis of rural public-private partnership older adult care project in the context of population aging in China.

Public-private partnership (PPP) older adult care project is an effective means to solve the dilemma of the aging population in China's rural areas, but there are some problems in the operation process, such as a low participation rate and poor service quality, resulting in the needs of rural older adult groups not being met. To alleviate the pressure of the aging population in rural areas, this study establishes an evolutionary game model for the PPP older adult care project, then defines the interests of local government, the private sector, and rural older adult residents, based on which it discusses the strategic choices of the three parties in the evolutionary process, and finally analyzes the influencing factors of the strategic choices of the game parties through simulation. The results suggest that whether the private sector chooses to actively participate in the project will be influenced by the willingness of local government and rural older adult residents to participate in the project. Local government could play the role of supervisor through reward and punishment mechanisms. Whether older rural residents choose to participate in the project depends on the number of benefits they would receive. Based on these findings, local governments should clarify the responsibilities of relevant stakeholders, adopt a regulatory strategy combining subsidies and penalties, improve the participation efficiency of rural older adult residents, promote the effective operation of PPP older adult care projects, and improve the quality of rural older adult care services in the new era.

Teaching and Learning Computational Thinking Through Game-Based Learning: A Systematic Review

Computational thinking (CT) is considered a fundamental skill that everyone in the 21st century should have. Game-based learning (GBL) may be used to teach CT, and it’s necessary to clarify how to design and implement game-based CT teaching. The literature was systematically searched for empirical studies published between 2011 and 2021. Thirty-nine studies were included in the review and findings suggested that GBL has positive effects on CT, but has non-significant effects on some CT elements (e.g., conditions, triggers, and abstraction) because of limited time to learn these elements and students’ preferences for using CT elements. Game elements, particularly clear goals and rules, progressive challenges, immediate feedback, storyline, avatar, social interaction and various reward mechanisms were used to motivate students to engage in activities to develop CT. Furthermore, single or multiple theoretical foundations, such as constructivist learning theory and experiential learning theory, may guide the design and implementation of game-based activities. Problem-solving, project-based approaches were used to encourage students to use CT to solve problems or complete a project. Finally, guidelines for designing and implementing game-based learning activities for promoting CT were discussed.