Inference Error Research Articles

Simulation-based, Finite-sample Inference for Privatized Data

Privacy protection methods, such as differentially private mechanisms, introduce noise into resulting statistics which often produces complex and intractable sampling distributions. In this paper, we propose a simulation-based “repro sample” approach to produce statistically valid confidence intervals and hypothesis tests, which builds on the work of Xie and Wang (2022). We show that this methodology is applicable to a wide variety of private inference problems, appropriately accounts for biases introduced by privacy mechanisms (such as by clamping), and improves over other state-of-the-art inference methods such as the parametric bootstrap in terms of the coverage and type I error of the private inference. We also develop significant improvements and extensions for the repro sample methodology for general models (not necessarily related to privacy), including 1) modifying the procedure to ensure guaranteed coverage and type I errors, even accounting for Monte Carlo error, and 2) proposing efficient numerical algorithms to implement the confidence intervals and p-values.

Robustness of Divergence Time Estimation Despite Gene Tree Estimation Error: A Case Study of Fireflies (Coleoptera: Lampyridae).

Genomic data has become ubiquitous in phylogenomic studies, including divergence time estimation, but provide new challenges. These challenges include, amongst others, biological gene tree discordance, methodological gene tree estimation error, and computational limitations on performing full Bayesian inference under complex models. In this study, we use a recently published firefly (Coleoptera: Lampyridae) anchored hybrid enrichment dataset (AHE; 436 loci for 88 Lampyridae species and 10 outgroup species) as a case study to explore gene tree estimation error and the robustness of divergence time estimation. First, we explored the amount of model violation using posterior predictive simulations because model violations are likely to bias phylogenetic inferences and produce gene tree estimation error. We specifically focused on missing data (either uniformly distributed or systematically) and the distribution of highly variable and conserved sites (either uniformly distributed or clustered). Our assessment of model adequacy showed that standard phylogenetic substitution models are not adequate for any of the 436 AHE loci. We tested if the model violations and alignment errors resulted indeed in gene tree estimation error by comparing the observed gene tree discordance to simulated gene tree discordance under the multispecies coalescent model. Thus, we show that the inferred gene tree discordance is not only due to biological mechanism but primarily due to inference errors. Lastly, we explored if divergence time estimation is robust despite the observed gene tree estimation error. We selected four subsets of the full AHE dataset, concatenated each subset and performed a Bayesian relaxed clock divergence estimation in RevBayes. The estimated divergence times overlapped for all nodes that are shared between the topologies. Thus, divergence time estimation is robust using any well selected data subset as long as the topology inference is robust.

PhyKIT: A Multitool for Phylogenomics.

Multiple sequence alignments and phylogenetic trees are rich in biological information and are fundamental to research in biology. PhyKIT is a tool for processing and analyzing the information content of multiple sequence alignments and phylogenetic trees. Here, we describe how to use PhyKIT for diverse analyses, including (i) constructing a phylogenomic supermatrix, (ii) detecting errors in orthology inference, (iii) quantifying biases in phylogenomic data sets, (iv) identifying radiation events or lack of resolution using gene support frequencies, and (v) conducting evolution-based screens to facilitate gene function prediction. Several PhyKIT functions that streamline multiple sequence alignment and phylogenetic processing-such as renaming FASTA entries or tree tips-are also discussed. These protocols demonstrate how simple command-line operations in the unified framework of PhyKIT facilitate diverse phylogenomic data analysis and processing, from supermatrix construction and diagnosis to gaining clues about gene function. © 2024 The Author(s). Current Protocols published by Wiley Periodicals LLC. Basic Protocol 1: Installing PhyKIT and syntax for usage Basic Protocol 2: Constructing a phylogenomic supermatrix Basic Protocol 3: Detecting anomalies in orthology relationships Basic Protocol 4: Quantifying biases in phylogenomic data matrices and related measures Basic Protocol 5: Identifying polytomies Basic Protocol 6: Assessing gene-gene coevolution as a genetic screen.

A Hierarchical Multi-Task Learning Framework for Semantic Annotation in Tabular Data.

To optimize the utilization and analysis of tables, it is essential to recognize and understand their semantics comprehensively. This requirement is especially critical given that many tables lack explicit annotations, necessitating the identification of column types and inter-column relationships. Such identification can significantly augment data quality, streamline data integration, and support data analysis and mining. Current table annotation models often address each subtask independently, which may result in the neglect of constraints and contextual information, causing relational ambiguities and inference errors. To address this issue, we propose a unified multi-task learning framework capable of concurrently handling multiple tasks within a single model, including column named entity recognition, column type identification, and inter-column relationship detection. By integrating these tasks, the framework exploits their interrelations, facilitating the exchange of shallow features and the sharing of representations. Their cooperation enables each task to leverage insights from the others, thereby improving the performance of individual subtasks and enhancing the model's overall generalization capabilities. Notably, our model is designed to employ only the internal information of tabular data, avoiding reliance on external context or knowledge graphs. This design ensures robust performance even with limited input information. Extensive experiments demonstrate the superior performance of our model across various tasks, validating the effectiveness of unified multi-task learning framework in the recognition and comprehension of table semantics.

Species-tree topology impacts the inference of ancient whole-genome duplications across the angiosperm phylogeny.

The history of angiosperms is marked by repeated rounds of ancient whole-genome duplications (WGDs). Here we used state-of-the-art methods to provide an up-to-date view of the distribution of WGDs in the history of angiosperms that considers both uncertainty introduced by different WGD inference methods and different underlying species-tree hypotheses. We used the distribution synonymous divergences (Ks) of paralogs and orthologs from transcriptomic and genomic data to infer and place WGDs across two hypothesized angiosperm phylogenies. We further tested these WGD hypotheses with syntenic inferences and Bayesian models of duplicate gene gain and loss. The predicted number of WGDs in the history of angiosperms (~170) based on the current taxon sampling is largely similar across different inference methods, but varies in the precise placement of WGDs on the phylogeny. Ks-based methods often yield alternative hypothesized WGD placements due to variation in substitution rates among lineages. Phylogenetic models of duplicate gene gain and loss are more robust to topological variation. However, errors in species-tree inference can still produce spurious WGD hypotheses, regardless of method used. Here we showed that different WGD inference methods largely agree on an average of 3.5 WGD in the history of individual angiosperm species. However, the precise placement of WGDs on the phylogeny is subject to the WGD inference method and tree topology. As researchers continue to test hypotheses regarding the impacts ancient WGDs have on angiosperm evolution, it is important to consider the uncertainty of the phylogeny as well as WGD inference methods.

Null models confirm nest site fidelity by male smallmouth bass, Micropterus dolomieu

BackgroundMany animals appear to preferentially renest in proximity to a site they previously occupied. Evidence of nest fidelity is often inferred from a right skewed distribution of distances between the nests of individuals that breed in two consecutive reproduction episodes, where many individuals nest some arbitrarily close distance to their prior nest and others, in the extended right tail of the distribution, nest far from the nest they previously occupied. Because right skewed distributions of inter-nest distances can arise even when individuals choose nest locations randomly, however, such inferences are prone to error. The importance of null models—used to generate patterns of individual inter-nest distances by processes that do not involve site attachment—for inferences about site fidelity has been known for decades but is still often unappreciated or ignored.MethodsThe right skewed distributions of inter-nest distances observed in two earlier studies of male smallmouth bass (Micropterus dolomieu) suggest prima facie that males exhibit nest site fidelity between annual reproduction episodes, but patterns of inter-nest distances have yet to be compared to an adequate null model. Here, we evaluate the nest site fidelity of marked male M. dolomieu in a decade-long dataset, where we apply a randomization procedure based on the rencontre probability problem to generate null models. Eight observed distributions of individual, annual inter-nest distances are compared to a year-specific null model to determine whether random processes are sufficient to explain the observed distributions of inter-nest distances.ResultsThrough contrasts between observed annual inter-nest distances and results derived from null models that imposed realistic constraints on behavior, we show that some males were undoubtedly nest-site faithful. To reinforce the utility of null models and to make these kinds of models more accessible, we also provide a supplemental tutorial. The tutorial illustrates how random site choices, subject to common ecological and behavioral constraints, and even how distance is measured, can produce patterns of inter-nest distances that falsely imply nest site fidelity, or a lack of fidelity. The R code needed to reproduce these null models is included. The inference errors evident in our examples generalize to other forms of site fidelity, such as the apparent patch fidelity of certain sea bird foragers.ConclusionsThe comparisons of observed distributions of inter-nest distances with those generated by null models imply that, as suggested in prior studies, male M. dolomieu indeed exhibit annual nest site fidelity. Procedures like those we apply are necessary first steps in analyses when distributions of distances between the nests of individuals in consecutive reproduction episodes are used to infer nest-site fidelity. Why male M. dolomieu are site faithful is a question yet to be answered.

A rank-based sequential test of independence

Summary We consider the problem of independence testing for two univariate random variables in a sequential setting. By leveraging recent developments on safe, anytime-valid inference, we propose a test with time-uniform Type-I error control and derive explicit bounds on the finite-sample performance of the test. We demonstrate the empirical performance of the procedure in comparison to existing sequential and nonsequential independence tests. Furthermore, since the proposed test is distribution-free under the null hypothesis, we empirically simulate the gap due to Ville’s inequality, the supermartingale analogue of Markov’s inequality, that is commonly applied to control Type-I error in anytime-valid inference, and apply this to construct a truncated sequential test.

Multicultural Counseling Skills Evaluation Form: A Tool for Clinical Supervision

As trainees’ development of multicultural counseling skills has become integral to graduate training in the mental health professions, so has the necessity of competent supervision for developing these skills. A critical component of supervision is evaluation, the process of observing, judging, and providing feedback to supervisees about their development. However, there is a paucity of guidance and resources available to assist supervisors in evaluating the multicultural skill development of supervisees. Such a vacuum opens the door for considerable bias as well as judgmental and inferential errors, resulting in the real possibility of unsound, unreliable, and invalid evaluations of supervisee development. This article presents the Multicultural Counseling Skills Evaluation Form (MCSEF), a 23-item instrument designed to fill the vacuum. The conceptual foundation of the form, development of items and revisions through expert review, potential utility in training and evaluation, and limitations are discussed.

A Hierarchical Data-Driven Predictive Control of Image-Based Visual Servoing Systems With Unknown Dynamics.

In this article, a hierarchical predictive control (PC) algorithm is designed for visual servoing mobile robot systems. At the kinematic level, the image-based visual servoing model of a wheeled mobile robot is established. By defining the corresponding performance index of the PC, an iterative linear quadratic regulator (iLQR) is used to obtain the velocity controller and to provide reference velocity for dynamics. In dynamics, a data-driven PC controller based on the Gaussian process (GP) is proposed to obtain the torque controller with unknown dynamics. The input-to-state practical stability (ISpS) of the system based on the proposed data-driven PC method is proved by introducing reasonable assumptions. The corresponding theorem also analyzes the maximum upper bound of GP inference error. Finally, the effectiveness of the proposed hierarchical controller is verified by simulations and experiments.

MEAformer: An all-MLP transformer with temporal external attention for long-term time series forecasting

Transformer-based models have significantly improved performance in Long-term Time Series Forecasting (LTSF). These models employ various self-attention mechanisms to discover long-term dependencies. However, the computational efficiency is hampered by the inherent permutation invariance of self-attention, and they primarily focus on relationships within the sequence while neglecting potential relationships between different sample sequences. This limits the ability and flexibility of self-attention in LTSF. In addition, the Transformer's decoder outputs sequences in an autoregressive manner, leading to slow inference speed and error accumulation effects, especially for LTSF. Regarding the issues with Transformer-based models for LTSF, we propose a model better suited for LTSF, named MEAformer. MEAformer adopts a fully connected Multi-Layer Perceptron (MLP) architecture consisting of two types of layers: encoder layers and MLP layers. Unlike most encoder layers in Transformer-based models, the MEAformer replaces self-attention with temporal external attention. Temporal external attention explores potential relationships between different sample sequences in the training dataset. Compared to the quadratic complexity of self-attention mechanisms, temporal external attention has efficient linear complexity. Encoder layers can be stacked multiple times to capture time-dependent relationships at different scales. Furthermore, the MEAformer replaces the intricate decoder layers of the original model with more straightforward MLP layers. This modification aims to enhance inference speed and facilitate single-pass sequence generation, effectively mitigating the problem of error accumulation effects. Regarding long-term forecasting, MEAformer achieves state-of-the-art performance on six benchmark datasets, covering five real-world domains: energy, transportation, economy, weather, and disease. Code is available at: https://github.com/huangsiyuan924/MEAformer.

Estimating the price elasticity of gasoline demand in correlated random coefficient models with endogeneity

SummaryWe propose a per‐cluster instrumental variable (PCIV) approach for estimating linear correlated random coefficient models in the presence of contemporaneous endogeneity and two‐way fixed effects. This approach estimates heterogeneous effects and aggregates them to population averages. We demonstrate consistency, showing robustness over standard estimators, and provide analytic standard errors for robust inference. In Monte Carlo simulation, PCIV performs relatively well in finite samples in either dimension. We apply PCIV in estimating the price elasticity of gasoline demand using state fuel taxes as instrumental variables. We find significant elasticity heterogeneity and more elastic gasoline demand on average than with standard estimators.

Dollo Parsimony Overestimates Ancestral Gene Content Reconstructions.

Ancestral reconstruction is a widely used technique that has been applied to understand the evolutionary history of gain and loss of gene families. Ancestral gene content can be reconstructed via different phylogenetic methods, but many current and previous studies employ Dollo parsimony. We hypothesize that Dollo parsimony is not appropriate for ancestral gene content reconstruction inferences based on sequence homology, as Dollo parsimony is derived from the assumption that a complex character cannot be regained. This premise does not accurately model molecular sequence evolution, in which false orthology can result from sequence convergence or lateral gene transfer. The aim of this study is to test Dollo parsimony's suitability for ancestral gene content reconstruction and to compare its inferences with a maximum likelihood-based approach that allows a gene family to be gained more than once within a tree. We first compared the performance of the two approaches on a series of artificial data sets each of 5,000 genes that were simulated according to a spectrum of evolutionary rates without gene gain or loss, so that inferred deviations from the true gene count would arise only from errors in orthology inference and ancestral reconstruction. Next, we reconstructed protein domain evolution on a phylogeny representing known eukaryotic diversity. We observed that Dollo parsimony produced numerous ancestral gene content overestimations, especially at nodes closer to the root of the tree. These observations led us to the conclusion that, confirming our hypothesis, Dollo parsimony is not an appropriate method for ancestral reconstruction studies based on sequence homology.

Median quartet tree search algorithms using optimal subtree prune and regraft

Gene trees can be different from the species tree due to biological processes and inference errors. One way to obtain a species tree is to find one that maximizes some measure of similarity to a set of gene trees. The number of shared quartets between a potential species tree and gene trees provides a statistically justifiable score; if maximized properly, it could result in a statistically consistent estimator of the species tree under several statistical models of discordance. However, finding the median quartet score tree, one that maximizes this score, is NP-Hard, motivating several existing heuristic algorithms. These heuristics do not follow the hill-climbing paradigm used extensively in phylogenetics. In this paper, we make theoretical contributions that enable an efficient hill-climbing approach. Specifically, we show that a subtree of size m can be placed optimally on a tree of size n in quasi-linear time with respect to n and (almost) independently of m. This result enables us to perform subtree prune and regraft (SPR) rearrangements as part of a hill-climbing search. We show that this approach can slightly improve upon the results of widely-used methods such as ASTRAL in terms of the optimization score but not necessarily accuracy.

Incomplete lineage sorting and gene flow within Allium (Amayllidaceae)

The phylogeny and systematics of the genus Allium have been studied with a variety of diverse data types, including an increasing amount of molecular data. However, strong phylogenetic discordance and high levels of uncertainty have prevented the identification of a consistent phylogeny. The difficulty in establishing phylogenetic consensus and evidence for genealogical discordance make Allium a compelling test case to assess the relative contribution of incomplete lineage sorting (ILS), gene flow and gene tree estimation error on phylogenetic reconstruction. In this study, we obtained 75 transcriptomes of 38 Allium species across 10 subgenera. Whole plastid genome, single copy genes and consensus CDS were generated to estimate phylogenetic trees both using coalescence and concatenation methods. Multiple approaches including coalescence simulation, quartet sampling, reticulate network inference, sequence simulation, theta of ILS and reticulation index were carried out across the CDS gene trees to investigate the degrees of ILS, gene flow and gene tree estimation error. Afterward, a regression analysis was used to test the relative contributions of each of these forms of uncertainty to the final phylogeny. Despite extensive topological discordance among gene trees, we found a fully supported species tree that agrees with the most of well-accepted relationships and establishes monophyly of the genus Allium. We presented clear evidence for substantial ILS across the phylogeny of Allium. Further, we identified two ancient hybridization events for the formation of the second evolutionary line and subg. Butomissa as well as several introgression events between recently diverged species. Our regression analysis revealed that gene tree inference error and gene flow were the two most dominant factors explaining for the overall gene tree variation, with the difficulty in disentangling the effects of ILS and gene tree estimation error due to a positive correlation between them. Based on our efforts to mitigate the methodological errors in reconstructing trees, we believed ILS and gene flow are two principal reasons for the oft-reported phylogenetic heterogeneity of Allium. This study presents a strongly-supported and well-resolved phylogenetic backbone for the sampled Allium species, and exemplifies how to untangle heterogeneity in phylogenetic signal and reconstruct the true evolutionary history of the target taxa.

DPIVE: A Regionalized Location Obfuscation Scheme with Personalized Privacy Levels

The popularity of cyber-physical systems is fueling the rapid growth of location-based services. This poses the risk of location privacy disclosure. Effective privacy preservation is foremost for various mobile applications. Recently, geo-indistinguishability and expected inference error are proposed for limiting location leakages. In this article, we argue that personalization means regionalization for geo-indistinguishability, and we propose a regionalized location obfuscation mechanism called DPIVE with personalized utility sensitivities. This substantially corrects the differential and distortion privacy problem of the PIVE framework proposed by Yu et al. on NDSS 2017. We develop DPIVE with two phases. In Phase I, we determine disjoint sets by partitioning all possible positions such that different locations in the same set share the Protection Location Set (PLS). In Phase II, we construct a probability distribution matrix in which the rows corresponding to the same PLS have their own sensitivity of utility (PLS diameter). Moreover, by designing a QK-means algorithm for more search space in 2-D space, we improve DPIVE with a refined location partition and present fine-grained personalization, enabling each location to have its own privacy level endowed with a customized privacy budget. Experiments with two public datasets demonstrate that our mechanisms have the superior performance, typically on skewed locations.

Combining compositional data sets introduces error in covariance network reconstruction

Abstract Microbial communities are diverse biological systems that include taxa from across multiple kingdoms of life. Notably, interactions between bacteria and fungi play a significant role in determining community structure. However, these statistical associations across kingdoms are more difficult to infer than intra-kingdom associations due to the nature of the data involved using standard network inference techniques. We quantify the challenges of cross-kingdom network inference from both theoretical and practical points of view using synthetic and real-world microbiome data. We detail the theoretical issue presented by combining compositional data sets drawn from the same environment, e.g. 16S and ITS sequencing of a single set of samples, and we survey common network inference techniques for their ability to handle this error. We then test these techniques for the accuracy and usefulness of their intra- and inter-kingdom associations by inferring networks from a set of simulated samples for which a ground-truth set of associations is known. We show that while the two methods mitigate the error of cross-kingdom inference, there is little difference between techniques for key practical applications including identification of strong correlations and identification of possible keystone taxa (i.e. hub nodes in the network). Furthermore, we identify a signature of the error caused by transkingdom network inference and demonstrate that it appears in networks constructed using real-world environmental microbiome data.

Reinterpretation of the results of randomized clinical trials.

Randomized clinical trials (RCTs) shape our clinical practice. Several studies report a mediocre replicability rate of the studied RCTs. Many researchers believe that the relatively low replication rate of RCTs is attributed to the high p value significance threshold. To solve this problem, some researchers proposed using a lower threshold, which is inevitably associated with a decrease in the study power. The results of 22 500 RCTs retrieved from the Cochrane Database of Systematic Reviews (CDSR) were reinterpreted using 2 fixed p significance threshold (0.05 and 0.005), and a recently proposed flexible threshold that minimizes the weighted sum of errors in statistical inference. With p < 0.05 criterion, 28.5% of RCTs were significant; p < 0.005, 14.2%; and p < flexible threshold, 9.9% (2/3 of significant RCTs based on p < 0.05 criterion, were found not significant). Lowering the p cut-off, although decreases the false-positive rate, is not generally associated with a lower weighted sum of errors; the false-negative rate increases (the study power decreases); important treatments may be left undiscovered. Accurate calculation of the optimal p value thresholds needs knowledge of the variance in each study arm, a posteriori. Lowering the p value threshold, as it is proposed by some researchers, is not reasonable as it might be associated with an increase in false-negative rate. Using a flexible p significance threshold approach, although results in a minimum error in statistical inference, might not be good enough too because only a rough estimation may be calculated a priori; the data necessary for the precise computation of the most appropriate p significance threshold are only available a posteriori. Frequentist statistical framework has an inherent conflict. Alternative methods, say Bayesian methods, although not perfect, would be more appropriate for the data analysis of RCTs.

MonoAux: Fully Exploiting Auxiliary Information and Uncertainty for Monocular 3D Object Detection.

Monocular 3D object detection plays a pivotal role in autonomous driving, presenting a formidable challenge by requiring the precise localization of 3D objects within a single image, devoid of depth information. Most existing methods in this domain fall short of harnessing the limited information available in monocular 3D detection tasks. They typically provide only a single detection outcome, omitting essential uncertainty analysis and result post-processing during model inference, thus limiting overall model performance. In this paper, we propose a comprehensive framework that maximizes information extraction from monocular images while encompassing diverse depth estimation and incorporating uncertainty analysis. Specifically, we mine additional information intrinsic to the monocular 3D detection task to augment supervision, thereby addressing the information scarcity challenge. Moreover, our framework handles depth estimation by recovering multiple sets of depth values from calculated visual heights. The final depth estimate and 3D confidence are determined through an uncertainty fusion process, effectively reducing inference errors. Furthermore, to address task weight allocation in multi-task training, we present a versatile training strategy tailored to monocular 3D detection. This approach leverages measurement indicators to monitor task progress, adaptively adjusting loss weights for different tasks. Experimental results on the KITTI and Waymo dataset confirm the effectiveness of our approach. The proposed method consistently provides enhanced performance across various difficulty levels compared to the original framework while maintaining real-time efficiency.

Aeolian Sediment Transport Responses to Vegetation Cover Change: Effects of Sampling Error on Model Uncertainty

AbstractAlthough it is widely known that observations of aeolian sediment transport are susceptible to large sampling errors, sample designs are frequently used that do not sufficiently reduce the measurement uncertainties inherent in the study of aeolian processes. Here, we examine the influence of sample size (n) and sampling location on uncertainty in models of aeolian sediment transport responses to vegetation cover change. We compare measurements from a stratified random array of 27 horizontal sediment mass flux samplers to vegetative cover data collected at a 1 ha site over a period of nearly 6 years. To assess the sensitivity of modeled relationships between aeolian transport and vegetative cover to sample design, we analyze statistical regressions for all possible combinations of sample size and sampler locations. We show that at least 17 randomly located samplers are needed to consistently capture the sediment mass flux response to vegetative cover change. We found that multiple statistically significant models can describe the sediment flux‐vegetative cover relationship when using smaller sample sizes, demonstrating the risks of inferring sediment transport response from an underpowered sample design. Across vegetative functional groups, we found that woody cover generally influenced aeolian sediment transport rates more than herbaceous cover, while model uncertainty at large sample sizes (n &gt; 17) showed the limitation of using vegetative cover as an indicator of aeolian sediment transport. Our results suggest an evaluation of sampling practices in aeolian sediment transport studies may be needed to avoid inferential errors that are likely pervasive in this field of study.

Evaluating the Interplay between Trajectory Segmentation and Mode Inference Error

Travel behavior changes are essential to transportation decarbonization. Travel diaries, consisting of sequences of trips between places, are typically used to instrument human travel behavior. However, these diaries are only as accurate as the underlying methods used to construct them. Travel diary algorithms have been a popular research topic since the advent of Global Positioning System tracking surveys. These algorithms have typically been validated using prompted recall of presegmented trips, thus disregarding the continuity of mode inference. Phone operating systems have adopted battery-conserving techniques, but the resulting data collection errors have not been studied extensively. We introduce a framework to evaluate the accuracy of trip length computations and mode inference by analyzing continuous mode-segmented trajectories for groups of trips. We then use the framework to identify the input data quality and the impact of postprocessing. Our primary inputs to this evaluation are MobilityNet, a public dataset containing information from three artificial timelines covering 15 different travel modes, and sample open-source travel diary creation algorithms from the OpenPATH project. Our framework concretely shows that the variance of the distance error drops from [Formula: see text] to [Formula: see text] (Android, iOS) after postprocessing. Similarly, the weighted F-scores for mode inference increase from [Formula: see text] to [Formula: see text] (iOS, Android) between random forest and geographic information system-based models. We hope that this standardized method will be adapted to evaluate other, potentially proprietary, travel diary algorithms. The results can be used to understand and improve the state of the art in the travel diary creation field.