Temporal Difference Research Articles

The development and testing of a smart sensorized guide wire for catheterization in a "blood" vessel phantom to support aortic valve implementation.

Heart valve disease is commonly treated by minimally invasive procedures with guide wires and catheterization. The main purpose of this study is to find out whether an extension of the guide wire with a sensor can support the surgeon within the blood vessel to reduce X-ray necessity. A smart guide wire is developed by an extension with a flex-bending sensor to evaluate the sensor signal with and without "blood" flow at a constant compression force. Various surgically relevant investigations are performed. For assessment, the mean temporal average of the moving averaged filtered ADC signal and a subsequent FFT are carried out. Results show that there is a smaller sensor signal when the applied force or bending at the sensor is higher. In all investigations, there was a different sensor signal. The flex-bending sensor can detect the effect of pulsatile flow. The smallest temporal averaged signal difference between reference and clamp in the front wire's tip is 1.09%. For example, the mean temporal average of the filtered ADC signal for different clinically relevant scenarios is between 2550 and 2900. The results show that the sensorized guide wire developed for catheterization can support aortic valve implementation. The sensor sensitivity is sufficient to detect even very small variations within the blood vessel and therefore is promising to support catheterization heart valve surgeries in future.

AF-FTTSnet: An end-to-end two-stream convolutional neural network for online quality monitoring of robotic welding

Online welding quality monitoring (WQM) is crucial for intelligent welding, and deep learning approaches considering spatiotemporal features for WQM tasks show great potential. However, one of the important challenges for existing approaches is to balance the spatiotemporal representation learning capability and computational efficiency, which makes it challenging to adapt welding processes with complex and drastic molten pool dynamic behavior. This paper proposes a novel approach for WQM using molten pool visual sensing and deep learning considering spatiotemporal features, the proposed deep learning network called attention fusion based frame-temporality two-stream network (AF-FTTSnet). Firstly, a passive vision sensor is used to acquire continuous dynamic molten pool images. Meanwhile, temporal difference images are computed to provide novel features and temporal representations. Then, a two-stream feature extraction module is designed to concurrently extract rich spatiotemporal features from molten pool images and temporal difference images. Finally, an attention fusion module with the ability to automatically identify and weight the most relevant features is designed to achieve optimal fusion of the two-stream features. The shop welding experimental results indicate that the proposed AF-FTTSnet model can effectively and robustly recognize five typical welding states during helium arc welding, with an accuracy of 99.26%. This model has been demonstrated to exhibit significant performance improvements compared to mainstream temporal sequence models. Available: https://github.com/Just199806/TSCNN/tree/master.

Concrete acoustic emission signal augmentation method based on generative adversarial networks

With the development of concrete structure health monitoring (SHM), the utilization of acoustic emission (AE) as an effective non-destructive testing method has attracted increasing attention among researchers. However, the acquisition of concrete AE signals in real-world scenarios is often limited, which poses challenges to the training as well as the practical application of deep learning (DL) models. To address this problem, a concrete acoustic emission signal augmentation method based on generative adversarial networks (GAN) is proposed in this paper. This method features a unique architecture where both the generator and discriminator are built upon a Bidirectional Long Short-Term Memory (BiLSTM) network. Additionally, the generator integrates an attention mechanism between the BiLSTM layers, enhancing its ability to capture the temporal dynamics of AE signals. To improve the local details of the generated signals, a temporal difference mean square error (TD-MSE) regularization is incorporated into the loss function. Experimental results demonstrate that the method not only substantially increases the amount of data available but also significantly enhances the model’s generalization capabilities and classification accuracy.

Discerning Temporal Difference Learning

Temporal difference learning (TD) is a foundational concept in reinforcement learning (RL), aimed at efficiently assessing a policy's value function. TD(λ), a potent variant, incorporates a memory trace to distribute the prediction error into the historical context. However, this approach often neglects the significance of historical states and the relative importance of propagating the TD error, influenced by challenges such as visitation imbalance or outcome noise. To address this, we propose a novel TD algorithm named discerning TD learning (DTD), which allows flexible emphasis functions—predetermined or adapted during training—to allocate efforts effectively across states. We establish the convergence properties of our method within a specific class of emphasis functions and showcase its promising potential for adaptation to deep RL contexts. Empirical results underscore that employing a judicious emphasis function not only improves value estimation but also expedites learning across diverse scenarios.

A Multimodal, Multi-Task Adapting Framework for Video Action Recognition

Recently, the rise of large-scale vision-language pretrained models like CLIP, coupled with the technology of Parameter-Efficient FineTuning (PEFT), has captured substantial attraction in video action recognition. Nevertheless, prevailing approaches tend to prioritize strong supervised performance at the expense of compromising the models' generalization capabilities during transfer. In this paper, we introduce a novel Multimodal, Multi-task CLIP adapting framework named M2-CLIP to address these challenges, preserving both high supervised performance and robust transferability. Firstly, to enhance the individual modality architectures, we introduce multimodal adapters to both the visual and text branches. Specifically, we design a novel visual TED-Adapter, that performs global Temporal Enhancement and local temporal Difference modeling to improve the temporal representation capabilities of the visual encoder. Moreover, we adopt text encoder adapters to strengthen the learning of semantic label information. Secondly, we design a multi-task decoder with a rich set of supervisory signals, including the original contrastive learning head, a cross-modal classification head, a cross-modal masked language modeling head, and a visual classification head. This multi-task decoder adeptly satisfies the need for strong supervised performance within a multimodal framework. Experimental results validate the efficacy of our approach, demonstrating exceptional performance in supervised learning while maintaining strong generalization in zero-shot scenarios.

A time two-grid algorithm for two-dimensional nonlinear time-fractional partial integro-differential equations

In this paper, a temporal second order two-grid difference scheme is proposed for the two-dimensional nonlinear time-fractional partial integro-differential equations with a weakly singular kernel. The first-order backward difference and L1 formula are used in the temporal direction to estimate the first level of time, the L2−1σ formula and L1-type formula are used in the temporal direction for later time steps, and the central difference formula is used in the spatial directions. To improve the computational efficiency of nonlinear system, an efficient time two-grid algorithm is proposed. This algorithm firstly solves a nonlinear system on the coarse grid, and then the Lagrangian linear interpolation is applied on the coarse grid to estimate the function values on the fine grid. The stability and convergence of the two-grid difference scheme are analyzed by the energy method. The convergence order of the two-grid difference scheme is O(τF2+τC4+hx2+hy2), where τF and τC are the time step sizes of fine grid and coarse grid respectively, while hx and hy are the space step sizes. Numerical experiments show that the accuracy of the theoretical analysis and the efficiency of the two-grid algorithm.

Spatial-temporal constraints for surface soil moisture mapping using Sentinel-1 and Sentinel-2 data over agricultural regions

Surface soil moisture (SSM) information could have important applications in agricultural and regional water management. Remote sensing, particularly synthetic aperture radar (SAR), is an important technology for the estimation of spatial–temporal SSM over larger areas. Using Sentinel-1 and Sentinel-2 data, this research developed a general spatial–temporal constrained machine-learning-based method for surface soil moisture mapping over agricultural regions. Central to this method is the construction of spatial and temporal constraints and their implementation in machine-learning models. We first defined the spatial and temporal constraints for SSM estimation by investigating the spatial division of cultivated crop types and the temporal division of cumulative precipitation. Second, under the presumption that the SSM and associated variables are smoothly changing, we extracted the temporal difference variables from the multi-temporal remote sensing data. Finally, we incorporated two constraints as categorical features and temporal differences into a CatBoost-based model to improve surface soil moisture mapping. We verified the proposed model in a Spain study area with multi-temporal remote sensing observations. The experimental results after incorporating the spatial–temporal constraints demonstrate the efficacy of the proposed model for mapping surface soil moisture over agricultural regions, with significantly improved R2 = 0.7328, RMSE = 0.0451 vol, and MAE = 0.0351 vol This study also concluded that using multiple polarization in the machine-learning-based method could reliably and accurately estimate surface soil moisture.

Robust change detection for remote sensing images based on temporospatial interactive attention module

Change Detection (CD) is a vital monitoring method in Earth observation, especially pertinent for land-use analysis, city management, and disaster damage assessment. However, in the era of constellation interconnection and air-sky collaboration, the changes in the Regions Of Interest (ROI) cause many false detections due to geometric perspective rotation and temporal style difference. In response to these challenges, we introduce CDNeXt, this framework elucidates a robust and efficient method for combining Siamese networks based on the pre-trained backbone with the innovative Temporospatial Interactive Attention Module (TIAM) for remote sensing imagery. The CDNeXt can be categorized into four primary components: Encoder, Interactor, Decoder, and Detector. Notably, the Interactor, powered by TIAM, queries and rebuilds spatial perspective dependencies and temporal style correlations from binary temporal features extracted by the Encoder to enlarge the difference of ROI change. Culminating the process, the Detector integrates the hierarchical features generated by the Decoder, subsequently producing a binary change mask. We have achieved new State-Of-The-Art (SOTA) performance in change detection, with our method surpassing existing techniques on four benchmark datasets: an F1 score of 82.63% on SYSU-CD, 87.14% on LEVIR-CD+, 66.71% on S2Looking, and 71.11% BANDON. To further validate the effectiveness of the TIAM, we compared it to other attention modules in both interactive and non-interactive modes. Our code is available on GitHub: https://github.com/wjj282439449/CDNeXt.

Prioritized experience replay based on dynamics priority

Experience replay has been instrumental in achieving significant advancements in reinforcement learning by increasing the utilization of data. To further improve the sampling efficiency, prioritized experience replay (PER) was proposed. This algorithm prioritizes experiences based on the temporal difference error (TD error), enabling the agent to learn from more valuable experiences stored in the experience pool. While various prioritized algorithms have been proposed, they ignored the dynamic changes of experience value during the training process, merely combining different priority criteria in a fixed or linear manner. In this paper, we present a novel prioritized experience replay algorithm called PERDP, which employs a dynamic priority adjustment framework. PERDP adaptively adjusts the weights of each criterion based on average priority level of the experience pool and evaluates experiences’ value according to current network. We apply this algorithm to the SAC model and conduct experiments in the OpenAI Gym experimental environment. The experiment results demonstrate that the PERDP exhibits superior convergence speed when compared to the PER.

A cross-cultural examination of temporal orientation through everyday language onsocial media.

Past research has shown that culture can form and shape our temporal orientation-the relative emphasis on the past, present, or future. However, there are mixed findings on how temporal orientations vary between North American and East Asian cultures due to the limitations of survey methodology and sampling. In this study, we applied an inductive approach and leveraged big data and natural language processing between two popular social media platforms-Twitter and Weibo-to assess the similarities and differences in temporal orientation in the United States of America and China, respectively. We first established predictive models from annotation data and used them to classify a larger set of English Twitter sentences (NTW = 1,549,136) and a larger set of Chinese Weibo sentences (NWB = 95,181) into four temporal catetories-past, future, atemporal present, and temporal present. Results show that there is no significant difference between Twitter and Weibo on past or future orientations; the large temporal orientation difference between North Americans and Chinese derives from their different prevailing focus on atemporal (e.g., facts, ideas) present (Twitter) or temporal present (e.g., the "here" and "now") (Weibo). Our findings contribute to the debate on cultural differences in temporal orientations with new perspectives following a new methodological approach. The study's implications call for a reevaluation of how temporal orientation is measured in cross-cultural studies, emphasizing the use of large-scale language data and acknowledging the atemporal present category. Understanding temporal orientations can guide effective cross-cultural communication strategies to tailor approaches for different audience based on temporal orientations, enhancing intercultural understanding and engagement.

Parrot Trade and the Potential Risk of Psittacosis as a Zoonotic Disease in Indonesian Bird Markets

Wildlife trade, both legal and illegal, is increasingly recognized as a key factor in the rise of emerging viral infectious diseases, and this is especially apparent in Asia, where large numbers of wildlife are openly offered for sale in bird markets. We here focus on the risk of Psittacosis becoming a zoonotic disease in the wildlife markets of Java and Bali, Indonesia. Psittacosis is particularly prevalent in parrots (hence the name), and the trade in parrots was instrumental in the Great Parrot Fever Pandemic in 1929/1930. Between 2014 and 2023, we conducted 176 surveys of 14 bird markets, during which we recorded 4446 largely wild-caught parrots for sale. On average, each market had nine genera on offer, and the diversity of genera increased with the increasing presence of parrots (up to 16 genera). For most of the bird markets during each survey, parrots from different genera and originating from different parts of the world, were offered for sale alongside each other. Genera offered for sale together did not cluster into natural (geographic) groups. We found no temporal difference in the sale of parrots. We conclude that the omnipresence of wild-caught parrots from various geographic regions in large numbers within the same bird markets increases the risk that psittacosis is present and that this poses a real risk for the zoonotic spread of avian chlamydiosis to humans.

Prospectively accelerated dynamic speech magnetic resonance imaging at 3 T using a self-navigated spiral-based manifold regularized scheme.

This work develops and evaluates a self-navigated variable density spiral (VDS)-based manifold regularization scheme to prospectively improve dynamic speech magnetic resonance imaging (MRI) at 3 T. Short readout duration spirals (1.3-ms long) were used to minimize sensitivity to off-resonance. A custom 16-channel speech coil was used for improved parallel imaging of vocal tract structures. The manifold model leveraged similarities between frames sharing similar vocal tract postures without explicit motion binning. The self-navigating capability of VDS was leveraged to learn the Laplacian structure of the manifold. Reconstruction was posed as a sensitivity-encoding-based nonlocal soft-weighted temporal regularization scheme. Our approach was compared with view-sharing, low-rank, temporal finite difference, extra dimension-based sparsity reconstruction constraints. Undersampling experiments were conducted on five volunteers performing repetitive and arbitrary speaking tasks at different speaking rates. Quantitative evaluation in terms of mean square error over moving edges was performed in a retrospective undersampling experiment on one volunteer. For prospective undersampling, blinded image quality evaluation in the categories of alias artifacts, spatial blurring, and temporal blurring was performed by three experts in voice research. Region of interest analysis at articulator boundaries was performed in both experiments to assess articulatory motion. Improved performance with manifold reconstruction constraints was observed over existing constraints. With prospective undersampling, a spatial resolution of 2.4 × 2.4 mm2/pixel and a temporal resolution of 17.4 ms/frame for single-slice imaging, and 52.2 ms/frame for concurrent three-slice imaging, were achieved. We demonstrated implicit motion binning by analyzing the mechanics of the Laplacian matrix. Manifold regularization demonstrated superior image quality scores in reducing spatial and temporal blurring compared with all other reconstruction constraints. While it exhibited faint (nonsignificant) alias artifacts that were similar to temporal finite difference, it provided statistically significant improvements compared with the other constraints. In conclusion, the self-navigated manifold regularized scheme enabled robust high spatiotemporal resolution dynamic speech MRI at 3 T.

A Distributional Perspective on Multiagent Cooperation With Deep Reinforcement Learning.

Among various value decomposition-based multiagent reinforcement learning (MARL) algorithms, the overall performance of the multiagent system is represented by a scalar global Q value and optimized by minimizing the temporal difference (TD) error with respect to that global Q value. However, the global Q value cannot accurately model the distributed dynamics of the multiagent system, since it is only a simplified representation for different individual Q values of agents. To explicitly consider the correlations between different cooperative agents, in this article, we propose a distributional framework and construct a practical model called distributional multiagent cooperation (DMAC) from a novel distributional perspective. Specifically, in DMAC, we view the individual Q value for the executed action of a random agent as a value distribution, whose expectation can further represent the overall performance. Then, we employ distributional RL to minimize the difference between the estimated distribution and its target for the optimization. The advantage of DMAC is that the distributed dynamics of agents can be explicitly modeled, and this results in better performance. To verify the effectiveness of DMAC, we conduct extensive experiments under nine different scenarios of the StarCraft Multiagent Challenge (SMAC). Experimental results show that the DMAC can significantly outperform the baselines with respect to the average median test win rate.

A model for automatic agent generation based on machine learning in simulation games

With the continuous promotion of Artificial Intelligence (AI) technology in games, researchers are generally willing to pay attention to the effect of AI on game picture quality and man-machine games. However, how to generate a large number of non-player characters (NPC) in a game, especially in a simulation game, is still a problem. Even if some people use machine learning to discover solutions, most answers tend to be limited to copying finite-state machines established by programmers. This paper explores techniques, such as machine learning and maze algorithms, for 2D in-game environments to construct a model that can quickly generate NPCs with 'personality' and adapt to the environment when developing a game. In the process of model construction, the temporal difference algorithm and the depth-first algorithm are used to customize the decision structure for the NPC. At the same time, a simple node-style map generation algorithm is added to restrain NPC. Moreover, an optional neural network model gives the NPC a memory that can be updated. This model successfully realizes the rapid construction of NPC in developing some simple 2D games. In addition, NPCs have a fair amount of intelligence. It can be used during game development or while the game is running and generate a certain number of NPCs in an unfamiliar environment with differences in behavior.

Remote Sensing Observations of a Coastal Water Environment Based on Neural Network and Spatiotemporal Fusion Technology: A Case Study of Hangzhou Bay

The coastal environment is characterized by high, multi-scale dynamics and the corresponding observations from a single remote sensing sensor are still facing challenges in achieving both high temporal and spatial resolution. This study proposed a spatiotemporal fusion model for coastal environments, which could fully enhance the efficiency of remote sensing data use and overcome the shortcomings of traditional spatiotemporal models that are insensitive to small-scale disturbances. The Enhanced Deep Super-Resolution Network (EDSR) was used to reconstruct spatial features in the lower spatial resolution GOCI-II data. The spatial features obtained instead of GOCI-II data were fed into the spatiotemporal fusion model, which enabled the fusion data to achieve an hour-by-hour observation of the water color and morphology information changes at 30 m resolution, including the changes in the spatial and temporal distributions of suspended particulate matter (SPM), the characterization of the vortex street caused by the bridge piers, the inundation process of the tidal flats, and coastline changes. In addition, this study analyzed the various factors affecting fusion accuracy, including spectral difference, errors in both temporal difference and location distance, and the structure of the EDSR model on the fusion accuracy. It is demonstrated that the location distance error and the spectral difference have the most significant impact on the fusion data, which may lead to the introduction of some ambiguous or erroneous spatial features.

Virtual reality perceptual training can improve the temporal discrimination ability of swinging during softball batting.

Perception and action uncoupling in fastball sports anticipatory training is often criticized. Nevertheless, perception-only training offers distinct advantages, such as flexibility concerning time, place, and injury limitations. Therefore, the effectiveness of this training approach warrants evaluation. This study developed a virtual reality (VR) training system based on the idea that the two visual pathways in the brain are associated with visual information attributes, rather than perception or action. A key feature of this study's perception-only training was its presentation of not only the opponent's kinematics but also the ball's flight information (the attributes that guide hitting) to train the visual system necessary for real situations. Seventeen female softball batters were assigned to two groups: a training group (N = 9) and a control group (N = 8). Only the training group underwent the VR anticipatory skill training to discriminate the different ball speeds. Both groups completed a perception test and an action test on the VR during the pre- and post-training periods. The perception test assessed response accuracy (RA) in discriminating ball speed, and the action test assessed the temporal difference in swing onset (delta onset). Comparison of these two outcome variables between groups revealed that perception-only training improved both perceptual and action responses. This study demonstrated the effectiveness of perception-only training and emphasized the need for its thoughtful utilization and design.

Research on decision making of intelligent vehicle based on composite priority experience replay

To address the problems of underutilization of samples and unstable training for intelligent vehicle training in the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, a TD3 algorithm based on the Composite Prioritized Experience Replay (CPR-TD3) mechanism is proposed. It considers experience immediate reward value and Temporal Difference error (TD-error) based and respectively to construct priorities to rank the samples. Subsequently composite average ranking of the samples to recalculate the priorities for sampling, uses the collected samples to train the target network. Then introduces the minimum lane change distance and the variable headway time distance to improve the reward function. Finally, the improved algorithm is proved to be effective by comparing it with the traditional TD3 on the highway scenario, and the CPR-TD3 algorithm improves the training efficiency of intelligent vehicles.

A Temporal Second-Order Difference Scheme for Variable-Order-Time Fractional-Sub-Diffusion Equations of the Fourth Order

In this article, we develop a compact finite difference scheme for a variable-order-time fractional-sub-diffusion equation of a fourth-order derivative term via order reduction. The proposed scheme exhibits fourth-order convergence in space and second-order convergence in time. Additionally, we provide a detailed proof for the existence and uniqueness, as well as the stability of scheme, along with a priori error estimates. Finally, we validate our theoretical results through various numerical computations.

Frontal Norepinephrine Represents a Threat Prediction Error Under Uncertainty

BackgroundTo adapt to threats in the environment, animals must predict them and engage in defensive behavior. While the representation of a prediction error signal for reward has been linked to dopamine, a neuromodulatory prediction error for aversive learning has not been identified. MethodsWe measured and manipulated norepinephrine release during threat learning using optogenetics and a novel fluorescent norepinephrine sensor. ResultsWe found that norepinephrine response to conditioned stimuli reflects aversive memory strength. When delays between auditory stimuli and footshock are introduced, norepinephrine acts as a prediction error signal. However, temporal difference prediction errors do not fully explain norepinephrine dynamics. To explain noradrenergic signaling, we used an updated reinforcement learning model with uncertainty about time and found that it explained norepinephrine dynamics across learning and variations in temporal and auditory task structure. ConclusionsNorepinephrine thus combines cognitive and affective information into a predictive signal and links time with the anticipation of danger.

Reinforcement learning strategies using Monte-Carlo to solve the blackjack problem

Blackjack is a classic casino game in which the player attempts to outsmart the dealer by drawing a combination of cards with face values that add up to just under or equal to 21 but are more incredible than the hand of the dealer he manages to come up with. This study considers a simplified variation of blackjack, which has a dealer and plays no active role after the first two draws. A different game regime will be modeled for everyone to ten multiples of the conventional 52-card deck. Irrespective of the number of standard decks utilized, the game is played as a randomized discrete-time process. For determining the optimum course of action in terms of policy, we teach an agent-a decision maker-to optimize across the decision space of the game, considering the procedure as a finite Markov decision chain. To choose the most effective course of action, we mainly research Monte Carlo-based reinforcement learning approaches and compare them with q-learning, dynamic programming, and temporal difference. The performance of the distinct model-free policy iteration techniques is presented in this study, framing the game as a reinforcement learning problem.