Coarser Time Scales Research Articles

Enhancing the Robustness of Traffic Signal Control with StageLight: A Multiscale Learning Approach

The continuous evolution of artificial intelligence and cyber–physical systems has presented promising opportunities for optimizing traffic signal control in densely populated urban areas, with the aim of alleviating traffic congestion. One area that has garnered significant interest from both researchers and practitioners is the application of deep reinforcement learning (DRL) in traffic signal control. However, DRL-based algorithms often suffer from instability due to the dynamic nature of traffic flows. Discrepancies between the environments used for training and those encountered during deployment often lead to operational failures. Moreover, conventional DRL-based traffic signal control algorithms tend to reveal vulnerabilities when faced with unforeseen events, such as sensor failure. These challenges highlight the need for innovative solutions to enhance the robustness and adaptability of such systems. To address these pertinent issues, this paper introduces StageLight, a novel two-stage multiscale learning approach, which involves learning optimal timings on a coarse time scale in stage 1, while finetuning them on a finer time scale in stage 2. Our experimental results demonstrate StageLight’s remarkable capability to generalize across diverse traffic conditions and its robustness to various sensor-failure scenarios.

Multivariate multiscale entropy (mMSE) as a tool for understanding the resting-state EEG signal dynamics: the spatial distribution and sex/gender-related differences

BackgroundThe study aimed to determine how the resting-state EEG (rsEEG) complexity changes both over time and space (channels). The complexity of rsEEG and its sex/gender differences were examined using the multivariate Multiscale Entropy (mMSE) in 95 healthy adults. Following the probability maps (Giacometti et al. in J Neurosci Methods 229:84–96, 2014), channel sets have been identified that correspond to the functional networks. For each channel set the area under curve (AUC), which represents the total complexity, MaxSlope—the maximum complexity change of the EEG signal at thefine scales (1:4 timescales), and AvgEnt—to the average entropy level at coarse-grained scales (9:12 timescales), respectively, were extracted. To check dynamic changes between the entropy level at the fine and coarse-grained scales, the difference in mMSE between the #9 and #4 timescale (DiffEnt) was also calculated.ResultsWe found the highest AUC for the channel sets corresponding to the somatomotor (SMN), dorsolateral network (DAN) and default mode (DMN) whereas the visual network (VN), limbic (LN), and frontoparietal (FPN) network showed the lowest AUC. The largest MaxSlope were in the SMN, DMN, ventral attention network (VAN), LN and FPN, and the smallest in the VN. The SMN and DAN were characterized by the highest and the LN, FPN, and VN by the lowest AvgEnt. The most stable entropy were for the DAN and VN while the LN showed the greatest drop of entropy at the coarse scales. Women, compared to men, showed higher MaxSlope and DiffEnt but lower AvgEnt in all channel sets.ConclusionsNovel results of the present study are: (1) an identification of the mMSE features that capture entropy at the fine and coarse timescales in the channel sets corresponding to the main resting-state networks; (2) the sex/gender differences in these features.

Evidence for acoustic niche partitioning depends on the temporal scale in two sympatric Bornean hornbill species

AbstractUnderstanding niche partitioning of closely related sympatric species is a fundamental goal in ecology. Acoustic communication space can be considered a resource, and the acoustic niche hypothesis posits that competition between species may lead to partitioning of communication space. Here, we compare the calling behavior of two sympatric Bornean hornbill species—the rhinoceros hornbill (Buceros rhinoceros) and the helmeted hornbill (Rhinoplax vigil)—to test for evidence of acoustic niche partitioning. Both hornbill species emit calls heard over many kilometers and have similar habitat preferences which is predicted to result in interspecific competition. We collected acoustic data on sympatric populations of both hornbill species using 10 autonomous recording units in Danum Valley Conservation Area, Sabah, Malaysia. We found that there was substantial spectral overlap between the calls of the two species, indicating the potential for competition for acoustic space. To test for evidence of acoustic niche partitioning, we investigated spatial and temporal patterns of calling in each species. Both hornbills were strictly diurnal and called throughout the day, and we were equally likely to detect both species at each of our recorders. We did not find evidence of temporal acoustic avoidance at a relatively coarse timescale when we divided our dataset into 1 h bins, but we did find evidence of temporal acoustic avoidance at a finer timescale when we used null models to compare the observed duration of overlapping calls to the expected amount of overlap due to chance.Abstract in Malay is available with online material.

Inhibition of knee joint sensory afferents alters covariation across strides between quadriceps muscles during locomotion.

Sport-related injuries to articular structures often alter the sensory information conveyed by joint structures to the nervous system. However, the role of joint sensory afferents in motor control is still unclear. Here, we evaluate the role of knee joint sensory afferents in the control of quadriceps muscles, hypothesizing that such sensory information modulates control strategies that limit patellofemoreal joint loading. We compared locomotor kinematics and muscle activity before and after inhibition of knee sensory afferents by injection of lidocaine into the knee capsule of rats. We evaluated whether this inhibition reduced the strength of correlation between the activity of vastus medialis (VM) and vastus lateralis (VL) both across strides and within each stride, coordination patterns that limit net mediolateral patellofemoral forces. We also evaluated whether this inhibition altered correlations among the other quadriceps muscle activity, the time-profiles of individual EMG envelopes, or movement kinematics. Neither the EMG envelopes nor limb kinematics was affected by the inhibition of knee sensory afferents. This perturbation also did not affect the correlations between VM and VL, suggesting that the regulation of patellofemoral joint loading is mediated by different mechanisms. However, inhibition of knee sensory afferents caused a significant reduction in the correlation between vastus intermedius (VI) and both VM and VL across, but not within, strides. Knee joint sensory afferents may therefore modulate the coordination between the vasti muscles but only at coarse time scales. Injuries compromising joint afferents might result in altered muscle coordination, potentially leading to persistent internal joint stresses and strains.NEW & NOTEWORTHY Sensory afferents originating from knee joint receptors provide the nervous system with information about the internal state of the joint. In this study, we show that these sensory signals are used to modulate the covariations among the activity of a subset of vasti muscles across strides of locomotion. Sport-related injuries that damage joint receptors may therefore compromise these mechanisms of muscle coordination, potentially leading to persistent internal joint stresses and strains.

Scale-dependent effects of roadways on the movement behavior of a large-bodied pit viper (Crotalus horridus)

Roadways are among the most widespread and disruptive anthropogenic land use features that influence the behavior and movement of wildlife. Negative impacts of roadways have been well documented, but the behavioral impact of roadways on smaller, cryptic species has yet to be thoroughly examined. Using a novel integration of radio telemetry and tri-axial accelerometry, we evaluated the effects of roadways on the movement behavior of 26 adult Timber Rattlesnakes (Crotalus horridus) at coarse and fine time scales in central Georgia between June 2020 and November 2021. To interpret the effect of roads at a coarse time scale, we modeled the effect of mean annual distance to roadways (DTR) on annual measures of movement and space use by C. horridus using both radio telemetry and accelerometry derived metrics (RT and ACT metrics). To explore the fine-scale impact of roadways, we quantified RT and ACT metrics during confirmed road interactions (i.e., instances when individual snakes crossed a road or encountered a road but did not cross) and compared these instances to the RT and ACT metrics calculated across the remainder of the active season within this subset of snakes. Relating the annual RT and ACT metrics to DTR revealed no significant associations at a coarse time scale. However, the evaluation of C. horridus movement behavior during punctuated road encounters revealed that snakes increased RT and ACT metrics during the road interactions compared to metrics calculated across the remainder of the active season. This might indicate that the abundance of contiguous habitat adjacent to roadways at our study site is serving as an adequate buffer to any long-term shifts in movement behavior, but the potential hidden cost of increasing movement when snakes encounter roads could have negative implications for populations that encounter roads more frequently, even in the absence of significant direct road mortality. Overall, integrating radio telemetry and accelerometry and adopting a scale-dependent approach to quantifying movement allowed for a more detailed evaluation of the response of C. horridus to roadways. This approach holds promise for detecting and interpreting previously overlooked short-term alterations in snake movement behavior with potentially significant fitness consequences.

EEG complexity during mind wandering: A multiscale entropy investigation

Our attention often drifts away from the ongoing task to task-unrelated thoughts, a phenomenon commonly referred to as mind wandering. Ample studies dedicated to delineating its electrophysiological correlates have revealed distinct event-related potentials (ERP) and spectral patterns associated with mind wandering. It remains less clear whether the complexity of the electroencephalography (EEG) changes when our minds wander, a metric that captures the predictability of the time series at varying timescales. Accordingly, this study investigated whether mind wandering impacts EEG signal complexity. We further explored whether such effects differ across timescales, and change in a context-dependent manner as indexed by global and local levels of processing. To address this, we recorded participants' EEG while they completed Navon's global and local processing task and occasionally reported whether they were on-task or mind wandering throughout the task. We found that brain signal complexity as indexed by multiscale entropy decreased at medium timescales in centro-parietal regions and increased at coarse timescales in anterior and posterior regions during mind wandering, as compared to the on-task state, for global processing. Moreover, global processing showed increased complexity at fine to medium timescales compared to local processing. Finally, behavioral performance revealed a context-dependent effect in accuracy measures, with mind wandering showing lower accuracy compared to the on-task state only during the local condition. Taken together, these results indicate that changes in brain signal complexity across timescales may be an important feature of mind wandering.

Two-Timescale Dynamic Energy and Reserve Dispatch With Wind Power and Energy Storage

The integration of volatile renewable resources and energy storage entails making dispatch decisions for conventional coal-fired units and fast-response devices in different timescales. This paper studies intraday dynamic energy-reserve dispatch following a two-timescale setting. The coarse timescale determines the hourly reference output and reserve allocation, which offers a sufficient backup for the fine-timescale operation; the fine timescale determines the adjustment of gas-fired units and energy storage system every 15 minutes in response to the actual wind power. A stochastic dynamic programming method is proposed to make decisions at the coarse timescale while guaranteeing the robust feasibility of the fast process via vertex scenarios of uncertainty set and bounding the state-of-charge intervals for energy storage; the fast-response actions at the fine timescale are updated using a truncated rolling-horizon optimization, which incorporates the cost-to-go functions calculated at the coarse timescale to prevent the fast decisions from being myopic. Case studies on the modified IEEE 5-bus system and 118-bus system validate the proposed framework.

Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games With Local Control and Global State Information

Stochastic dynamic teams and games are rich models for decentralized systems and challenging testing grounds for multiagent learning. Previous work that guaranteed team optimality assumed stateless dynamics, or an explicit coordination mechanism, or joint-control sharing. In this article, we present an algorithm with guarantees of convergence to team optimal policies in teams and common interest games. The algorithm is a two-timescale method that uses a variant of Q-learning on the finer timescale to perform policy evaluation while exploring the policy space on the coarser timescale. Agents following this algorithm are “independent learners”: they use only local controls, local cost realizations, and global state information, without access to controls of other agents. The results presented here are the first, to the best of our knowledge, to give formal guarantees of convergence to team optimality using independent learners in stochastic dynamic teams and common interest games.

The modulatory effect of adaptive task-switching training on resting-state neural network dynamics in younger and older adults

With increasing life expectancy and active aging, it becomes crucial to investigate methods which could compensate for generally detected cognitive aging processes. A promising candidate is adaptive cognitive training, during which task difficulty is adjusted to the participants’ performance level to enhance the training and potential transfer effects. Measuring intrinsic brain activity is suitable for detecting possible distributed training-effects since resting-state dynamics are linked to the brain’s functional flexibility and the effectiveness of different cognitive processes. Therefore, we investigated if adaptive task-switching training could modulate resting-state neural dynamics in younger (18–25 years) and older (60–75 years) adults (79 people altogether). We examined spectral power density on resting-state EEG data for measuring oscillatory activity, and multiscale entropy for detecting intrinsic neural complexity. Decreased coarse timescale entropy and lower frequency band power as well as increased fine timescale entropy and higher frequency band power revealed a shift from more global to local information processing with aging before training. However, cognitive training modulated these age-group differences, as coarse timescale entropy and lower frequency band power increased from pre- to post-training in the old-training group. Overall, our results suggest that cognitive training can modulate neural dynamics even when measured outside of the trained task.

Investigating the impact of calibration timescales on streamflow simulation, parameter sensitivity and model performance for Indian catchments

ABSTRACT Hydrological model calibration is a quintessential step in model development, and the time scale of calibration depends on the application. However, the implications of choice of time scale of calibration have not been explored extensively. Here, we evaluate the effect of the time scale of calibration on model sensitivity, best parameter ranges, and predictive uncertainty for three river basins using the Soil and Water Assessment Tool (SWAT) model. Multiple models were set up for three different catchments from southern India. Our results showed that the sensitivity of the parameters, best parameter ranges, and model performance are conditioned on the time scale of calibration. The models calibrated at coarser time scales marginally outperformed the models calibrated at fine time scale in terms of Nash-Sutcliffe efficiency and percentage bias. Transfer of parameters across scales (both from coarse to fine and from fine to coarse) have a general tendency to worsen the model performance in all three catchments, with few exceptions.

Generic Framework for Downscaling Statistical Quantities at Fine Time-Scales and Its Perspectives towards Cost-Effective Enrichment of Water Demand Records

The challenging task of generating a synthetic time series at finer temporal scales than the observed data, embeds the reconstruction of a number of essential statistical quantities at the desirable (i.e., lower) scale of interest. This paper introduces a parsimonious and general framework for the downscaling of statistical quantities based solely on available information at coarser time scales. The methodology is based on three key elements: (a) the analysis of statistics’ behaviour across multiple temporal scales; (b) the use of parametric functions to model this behaviour; and (c) the exploitation of extrapolation capabilities of the functions to downscale the associated statistical quantities at finer scales. Herein, we demonstrate the methodology using residential water demand records and focus on the downscaling of the following key quantities: variance, L-variation, L-skewness and probability of zero value (no demand; intermittency), which are typically used to parameterise a stochastic simulation model. Specifically, we downscale the above statistics down to a 1 min scale, assuming two scenarios of initial data resolution, i.e., 5 and 10 min. The evaluation of the methodology on several cases indicates that the four statistics can be well reconstructed. Going one step further, we place the downscaling methodology in a more integrated modelling framework for a cost-effective enhancement of fine-resolution records with synthetic ones, embracing the current limited availability of fine-resolution water demand measurements.

Implicit Space-Time Domain Decomposition Approach for Solving Multiphase Miscible Flow: Accuracy and Scalability

Summary In this paper, we propose a fully implicit space-time multiscale scheme to improve computational efficiency in solving nonlinear multiphase flow in porous media. Here, error estimators are used for adaptively changing the spatial-temporal mesh. This algorithm applied to the black-oil model is compared to a standard control volume approach using a fine time and spatial mesh. Error estimators are introduced to determine subdomains of the reservoir in which high nonlinearity hinders Newtonian convergence. This is followed by applying local fine timesteps to these marked regions, whereas the remaining regions retain the coarse time scale. Once a temporal discretization is determined for different parts of the reservoir, the spatial mesh is refined for treating saturation fronts. The nonmatching interfaces arising from different temporal and spatial scales are resolved by the enhanced velocity method, which enforces strongly the continuity of fluxes. This whole system is solved monolithically. Results from a three-phase black-oil model are described. The multiscale solution is compared to a uniformly fine spatial mesh and fine timestepping solution to confirm accuracy. Solutions from both Gaussian and channelized permeability fields are presented. In the multiscale solution, we observe temporal refinements being applied to the water and gas saturation fronts by reducing the timestep size to guarantee Newtonian convergence. We also observe that the extent of refinements is balanced between the two saturation fronts based on their respective nonlinearity. For Gaussian permeability fields, the algorithm only treats saturation fronts spatially, whereas for channelized permeability fields, the geological features are also considered. Production profiles for the three phases match well between the two solutions under specific refinement criteria. We also investigate the improvement in computational efficiency, as well as the algorithm scalability in regard to increasing problem sizes. We observe a speedup of approximately 10 for solving the linear system, and such speedup increases as the problem size expands. In reservoir simulation, schemes that attempt to decouple the diffusion and advection process become less robust as the physics becomes more complex. Our approach, which is focused on handling the high nonlinearities, can be treated fully implicitly and provide a reduction in simulation costs. This approach can also be applied to model order reduction in providing accurate snapshot solutions. NOTE: This paper is published as part of the 2021 SPE Reservoir Simulation Conference Special Issue.

Distance to Healthy Metabolic and Cardiovascular Dynamics From Fetal Heart Rate Scale-Dependent Features in Pregnant Sheep Model of Human Labor Predicts the Evolution of Acidemia and Cardiovascular Decompensation.

The overarching goal of the present work is to contribute to the understanding of the relations between fetal heart rate (FHR) temporal dynamics and the well-being of the fetus, notably in terms of predicting the evolution of lactate, pH and cardiovascular decompensation (CVD). It makes uses of an established animal model of human labor, where 14 near-term ovine fetuses subjected to umbilical cord occlusions (UCO) were instrumented to permit regular intermittent measurements of metabolites lactate and base excess, pH, and continuous recording of electrocardiogram (ECG) and systemic arterial blood pressure (to identify CVD) during UCO. ECG-derived FHR was digitized at the sampling rate of 1,000 Hz and resampled to 4 Hz, as used in clinical routine. We focused on four FHR variability features which are tunable to temporal scales of FHR dynamics, robustly computable from FHR sampled at 4 Hz and within short-time sliding windows, hence permitting a time-dependent, or local, analysis of FHR which helps dealing with signal noise. Results show the sensitivity of the proposed features for early detection of CVD, correlation to metabolites and pH, useful for early acidosis detection and the importance of coarse time scales (2.5–8 s) which are not disturbed by the low FHR sampling rate. Further, we introduce the performance of an individualized self-referencing metric of the distance to healthy state, based on a combination of the four features. We demonstrate that this novel metric, applied to clinically available FHR temporal dynamics alone, accurately predicts the time occurrence of CVD which heralds a clinically significant degradation of the fetal health reserve to tolerate the trial of labor.

Space-time modeling of child mortality at the Admin-2 level in a low and middle income countries context.

The Sustainable Development Goals call for a total reduction of preventable child mortality before 2030. Further, the goals state the desirability to have subnational mortality estimates. Estimates at this level are required for health interventions at the subnational level. In a low and middle income countries context, the data on mortality typically consist of household surveys, which are carried out with a stratified, cluster design, and census microsamples. Most household surveys collect full birth history (FBH) data on birth and death dates of a mother's children, but censuses collect summary birth history (SBH) data which consist only of the number of children born and the number that died. In previous work, direct (survey-weighted) estimates with associated variances were derived from FBH data and smoothed in space and time. Unfortunately, the FBH data from household surveys are usually not sufficiently abundant to obtain yearly estimates at the Admin-2 level (at which interventions are often made). In this paper we describe four extensions to previous work: (i) combining SBH data with FBH data, (ii) modeling on a yearly scale, to combine data on a yearly scale with data at coarser time scales, (iii) adjusting direct estimates in Admin-2 areas where we do not observe any deaths due to small sample sizes, (iv) acknowledge differences in data sources by modeling potential bias arising from the various data sources. The methods are illustrated using household survey and census data from Kenya and Malawi, to produce mortality estimates from 1980 to the time of the most recent survey, and predictions to 2020.

A Procedure for Combining Improved Correlated Sampling Methods and a Resampling Strategy to Generate a Multi-Site Conditioned Streamflow Process

In this study, a new method is developed to generate multi-site streamflow series. First, a correlated sampling method based on spectral decomposition is employed to simulate a large number of individual years of multi-site streamflow series. Then, combined with historical design flood data, flood characteristic series considering historical floods are generated. Finally, a subset of the streamflow series corresponding to these flood characteristics is selected from the pool of multi-site simulated streamflow series. The proposed model is applied to multi-site streamflow simulations in the upper Yangtze River basin. Experimental results show that the simulated data not only ensure the fundamental statistical characteristics of the observed data, but also provide good preservation of high order self-dependent, cross-dependent at finer and coarser time scales with any marginal distributions and any correlation structures. In addition, the simulated extreme situations are more reliable for risk and vulnerability analysis of multi-reservoir joint operation systems due to the consideration of the historical design flood data.

Spatiotemporal complexity patterns of resting-state bioelectrical activity explain fluid intelligence: Sex matters.

Neural complexity is thought to be associated with efficient information processing but the exact nature of this relation remains unclear. Here, the relationship of fluid intelligence (gf) with the resting‐state EEG (rsEEG) complexity over different timescales and different electrodes was investigated. A 6‐min rsEEG blocks of eyes open were analyzed. The results of 119 subjects (57 men, mean age = 22.85 ± 2.84 years) were examined using multivariate multiscale sample entropy (mMSE) that quantifies changes in information richness of rsEEG in multiple data channels at fine and coarse timescales. gf factor was extracted from six intelligence tests. Partial least square regression analysis revealed that mainly predictors of the rsEEG complexity at coarse timescales in the frontoparietal network (FPN) and the temporo‐parietal complexities at fine timescales were relevant to higher gf. Sex differently affected the relationship between fluid intelligence and EEG complexity at rest. In men, gf was mainly positively related to the complexity at coarse timescales in the FPN. Furthermore, at fine and coarse timescales positive relations in the parietal region were revealed. In women, positive relations with gf were mostly observed for the overall and the coarse complexity in the FPN, whereas negative associations with gf were found for the complexity at fine timescales in the parietal and centro‐temporal region. These outcomes indicate that two separate time pathways (corresponding to fine and coarse timescales) used to characterize rsEEG complexity (expressed by mMSE features) are beneficial for effective information processing.

Differential Modulation of Brain Signal Variability During Cognitive Control in Athletes with Different Domains of Expertise

Motor expertise has recently been associated with differences in domain-general cognition. Studies using averaged neurophysiological signals (e.g., event-related potentials) have shown varying degree of expertise-related differences in neural activity. As a result, the precise mechanisms underlying these differences remain to be described. Here we used multiscale entropy analysis (MSE) to investigate whether the complexity of underlying neural systems working in a wide-range time scales can better explain the cognitive characteristics of athletes with different domains of expertise. Behavioral and electroencephalograms (EEG) measures of athletes practicing an interceptive sport (badminton; n = 17) or a static sport (long-distance running; n = 17) were assessed during a flanker task with varying degrees of response conflict. The interceptive sport players showed superior behavioral performance overall on the task relative to the static sport players. Although both groups exhibited greater sample entropy across most time scales in high-conflict relative to low-conflict trials over the parietal site, this effect was only evident at coarser time scales over the midfrontal site for the interceptive sport players. Together, our results suggest that individual differences in motor expertise may be associated with difference in information-processing capacity and information integration during cognitive processing, as demonstrated by differential cognitive modulation of brain signal variability.

A Near–Far-Field Model for Bubbles Influenced by External Electrical Fields

In this paper, we present a model that is based on near–far-field charged bubble formation and transportation in an underlying dielectric liquid. The bubbles are controlled by the dielectric liquid, which is influenced by an external electrical field. This allows us to control the shape and volume of the bubbles in the dielectric liquid, such as water. These simulations are important to close the gap between the formation of charged bubbles, which is a fine-scale model and their transport in the underlying liquid, which is a coarse-scale model. In the fine-scale model, the formation of the bubbles and their influence of the electric-stress is approached by a near-field model, which is done by the Young–Laplace equation plus additional force-terms. In the coarse-scale model, the transport of the bubbles is approached by a far-field model, which is done with a convection-diffusion equation. The models are coupled with a bubble in cell scheme, which interpolates between the fine and coarse scales of the different models. Such a scale-dependent approach allows us to apply optimal numerical solvers for the different fine and coarse time and space scales and help to foresee the fluctuations of the charged bubbles in the E-field. We discuss the modeling approaches, numerical solver methods and we present the numerical results for the near–far-field bubble formation and transport model in a dielectric carrier fluid.

Multivariate and multi-scale generator based on non-parametric stochastic algorithms

AbstractA method for generating combined multivariate time series at multiple locations and at different time scales is presented. The procedure is based on three steps: first, the Monte Carlo method generation of data with statistical properties as close as possible to the observed series; second, the rearrangement of the order of simulated data in the series to achieve target correlations; and third, the permutation of series for correlation adjustment between consecutive years. The method is non-parametric and retains, to a satisfactory degree, the properties of the observed time series at the selected simulation time scale and at coarser time scales. The new approach is tested on two case studies, where it is applied to the log-transformed streamflow and precipitation at weekly and monthly time scales. Special attention is given to the extrapolation of non-parametric cumulative frequency distributions in their tail zones. The results show a good agreement of stochastic properties between the simulated and observed data. For example, for one of the case studies, the average relative errors of the observed and simulated weekly precipitation and streamflow statistics (up to skewness coefficient) are in the range of 0.1–9.2% and 0–5.4%, respectively.

Estimating Dynamic Treatment Regimes in Mobile Health Using V-Learning

The vision for precision medicine is to use individual patient characteristics to inform a personalized treatment plan that leads to the best possible healthcare for each patient. Mobile technologies have an important role to play in this vision as they offer a means to monitor a patient’s health status in real-time and subsequently to deliver interventions if, when, and in the dose that they are needed. Dynamic treatment regimes formalize individualized treatment plans as sequences of decision rules, one per stage of clinical intervention, that map current patient information to a recommended treatment. However, most existing methods for estimating optimal dynamic treatment regimes are designed for a small number of fixed decision points occurring on a coarse time-scale. We propose a new reinforcement learning method for estimating an optimal treatment regime that is applicable to data collected using mobile technologies in an outpatient setting. The proposed method accommodates an indefinite time horizon and minute-by-minute decision making that are common in mobile health applications. We show that the proposed estimators are consistent and asymptotically normal under mild conditions. The proposed methods are applied to estimate an optimal dynamic treatment regime for controlling blood glucose levels in patients with type 1 diabetes.