Tree-like Data Research Articles

Bias Analysis and Correction in Weighted-L1 Estimators for the First-Order Bifurcating Autoregressive Model

This study examines the bias in weighted least absolute deviation (WL1) estimation within the context of stationary first-order bifurcating autoregressive (BAR(1)) models, which are frequently employed to analyze binary tree-like data, including applications in cell lineage studies. Initial findings indicate that WL1 estimators can demonstrate substantial and problematic biases, especially when small to moderate sample sizes. The autoregressive parameter and the correlation between model errors influence the volume and direction of the bias. To address this issue, we propose two bootstrap-based bias-corrected estimators for the WL1 estimator. We conduct extensive simulations to assess the performance of these bias-corrected estimators. Our empirical findings demonstrate that these estimators effectively reduce the bias inherent in WL1 estimators, with their performance being particularly pronounced at the extremes of the autoregressive parameter range.

Segmentation of Tree-Like Data Structures and Their Parallel Processing in Authentication Methods Based on Block Coupling Encoding

Purpose of reseach. In the tasks of authenticating groups of messages encoded in the mode of chaining blocks, there is a need for the formation and processing of specific tree-like structures. The contents of such structures, in addition to information about the placement of data in the internal memory of the calculators, describes the relative location of messages in the data stream between subscribers of a peer-to-peer network. This information is necessary to isolate a structured set from the entire message stream to the receiver, for which its source is uniquely determined. Using approaches to segmentation of tree structures allows you to parallelize the processes of adding elements to it and searching for areas corresponding to an authentication error.Methods. The division of the tree structure into areas subject to modification and areas for analysis is based on a metric dynamically formed from message authentication codes – the position of a specific message in a structured set of messages transmitted from the source to the receiver. The value of this metric determines the distance from the root of the tree, which defines the boundary between the two named areasResults. By isolating the modified and analyzed sections of the tree structure, races of processes implementing independent algorithms for working with it are excluded. The possibility of detecting authentication errors before receiving the last message in a structured set of messages is shown. As a result, there is no need to transmit those group messages that were supposed to be sent after the error was detected. Formulas for estimating the average transmission time of multiple messages with sequential and parallel implementation of procedures for the formation and processing of a tree structure containing descriptors of incoming messages to the receiver are given.Conclusion. The paper shows that the parallel implementation of algorithms for adding elements to the tree structure and the algorithm for searching for areas corresponding to an error reduces the average transmission time of a group of messages by 5-12% compared with the sequential implementation of these algorithms. This reduces the load on the communication channel for the target class of systems using block coupling encoding for authentication.

Fidelity of hyperbolic space for Bayesian phylogenetic inference.

Bayesian inference for phylogenetics is a gold standard for computing distributions of phylogenies. However, Bayesian phylogenetics faces the challenging computational problem of moving throughout the high-dimensional space of trees. Fortunately, hyperbolic space offers a low dimensional representation of tree-like data. In this paper, we embed genomic sequences as points in hyperbolic space and perform hyperbolic Markov Chain Monte Carlo for Bayesian inference in this space. The posterior probability of an embedding is computed by decoding a neighbour-joining tree from the embedding locations of the sequences. We empirically demonstrate the fidelity of this method on eight data sets. We systematically investigated the effect of embedding dimension and hyperbolic curvature on the performance in these data sets. The sampled posterior distribution recovers the splits and branch lengths to a high degree over a range of curvatures and dimensions. We systematically investigated the effects of the embedding space's curvature and dimension on the Markov Chain's performance, demonstrating the suitability of hyperbolic space for phylogenetic inference.

Modeling preference heterogeneity using model-based decision trees

This article investigates the usage of a general model-based recursive partitioning algorithm to model preference heterogeneity. We use the algorithm to grow a decision tree based on statistical tests of the stability of individuals’ preference parameters. In particular, we used a Mixed Logit (MIXL) model with alternative-specific attributes at the end leaves of the tree while using individual characteristics as partition variables. This configuration allows us to search for instabilities of the taste parameters across individuals’ characteristics. We conduct a simulation study to investigate the algorithm’s ability to recover different data generating processes with structural breaks in the taste parameters. The results show that the algorithm can correctly recover diverse tree-like data generating processes. Additionally, we applied the algorithm to stated choice data of the preferences for the environmental impact of (hypothetical) energy generation plans in Chile. The results show that the model-based decision tree fits the data better than MIXL in terms of information criteria. Moreover, we show that the derived tree structure depends on the assumptions on the parameters’ distributions. Additionally, we compare the model-based decision tree model with Latent Class (LC) models with and without within-class heterogeneity. Finally, we show that the recursive partitioning algorithm can inform the selection of variables to be included in the LC allocation models.

On the Estimation of Discrete Choice Models to Capture Irrational Customer Behaviors

The random utility maximization model is by far the most adopted framework to estimate consumer choice behavior. However, behavioral economics has provided strong empirical evidence of irrational choice behaviors, such as halo effects, that are incompatible with this framework. Models belonging to the random utility maximization family may therefore not accurately capture such irrational behavior. Hence, more general choice models, overcoming such limitations, have been proposed. However, the flexibility of such models comes at the price of increased risk of overfitting. As such, estimating such models remains a challenge. In this work, we propose an estimation method for the recently proposed generalized stochastic preference choice model, which subsumes the family of random utility maximization models and is capable of capturing halo effects. In particular, we propose a column-generation method to gradually refine the discrete choice model based on partially ranked preference sequences. Extensive computational experiments indicate that our model, explicitly accounting for irrational preferences, can significantly boost the predictive accuracy on both synthetic and real-world data instances. Summary of Contribution: In this work, we propose an estimation method for the recently proposed generalized stochastic preference choice model, which subsumes the family of random utility maximization models and is capable of capturing halo effects. Specifically, we show how to use partially ranked preferences to efficiently model rational and irrational customer types from transaction data. Our estimation procedure is based on column generation, where relevant customer types are efficiently extracted by expanding a treelike data structure containing the customer behaviors. Furthermore, we propose a new dominance rule among customer types whose effect is to prioritize low orders of interactions among products. An extensive set of experiments assesses the predictive accuracy of the proposed approach by comparing it against rank-based methods with only rational preferences and with more general benchmarks from the literature. Our results show that accounting for irrational preferences can boost predictive accuracy by 12.5% on average when tested on a real-world data set from a large chain of grocery and drug stores.

Tuning as convex optimisation: a polynomial tuner for multi-parametric combinatorial samplers

AbstractCombinatorial samplers are algorithmic schemes devised for the approximate- and exact-size generation of large random combinatorial structures, such as context-free words, various tree-like data structures, maps, tilings, RNA molecules. They can be adapted to combinatorial specifications with additional parameters, allowing for a more flexible control over the output profile of parametrised combinatorial patterns. One can control, for instance, the number of leaves, profile of node degrees in trees or the number of certain sub-patterns in generated strings. However, such a flexible control requires an additional and nontrivial tuning procedure. Using techniques of convex optimisation, we present an efficient tuning algorithm for multi-parametric combinatorial specifications. Our algorithm works in polynomial time in the system description length, the number of tuning parameters, the number of combinatorial classes in the specification, and the logarithm of the total target size. We demonstrate the effectiveness of our method on a series of practical examples, including rational, algebraic, and so-called Pólya specifications. We show how our method can be adapted to a broad range of less typical combinatorial constructions, including symmetric polynomials, labelled sets and cycles with cardinality lower bounds, simple increasing trees or substitutions. Finally, we discuss some practical aspects of our prototype tuner implementation and provide its benchmark results.

Palindromic trees for a sliding window and its applications

The palindromic tree (a.k.a. eertree) for a string S of length n is a tree-like data structure that represents the set of all distinct palindromic substrings of S, using O(n) space [Rubinchik and Shur, 2018]. It is known that, when S is over an alphabet of size σ and is given in an online manner, then the palindromic tree of S can be constructed in O(nlog⁡σ) time with O(n) space. In this paper, we consider the sliding window version of the problem: For a sliding window of length at most d, we present two versions of an algorithm which maintains the palindromic tree of size O(d) for every sliding window S[i..j] over S, where 1≤j−i+1≤d. The first version works in O(nlog⁡σ′) time with O(d) space where σ′≤d is the maximum number of distinct characters in the windows, and the second one works in O(n+dσ) time with (d+2)σ+O(d) space. We also show how our algorithms can be applied to efficient computation of minimal unique palindromic substrings (MUPS) and minimal absent palindromic words (MAPW) for a sliding window.

Wepy: A Flexible Software Framework for Simulating Rare Events with Weighted Ensemble Resampling.

Here, we introduce the open-source software framework wepy (https://github.com/ADicksonLab/wepy) which is a toolkit for running and analyzing weighted ensemble (WE) simulations. The wepy toolkit is in pure Python and as such is highly portable and extensible, making it an excellent platform to develop and use new WE resampling algorithms such as WExplore, REVO, and others while leveraging the entire Python ecosystem. In addition, wepy simplifies WE-specific analyses by defining out-of-core tree-like data structures using the cross-platform HDF5 file format. In this paper, we discuss the motivations and challenges for simulating rare events in biomolecular systems. As has previously been shown, high-dimensional WE resampling algorithms such as WExplore and REVO have been successful at these tasks, especially for rare events that are difficult to describe by one or two collective variables. We explain in detail how wepy facilitates implementation of these algorithms, as well as aids in analyzing the unique structure of WE simulation results. To explain how wepy and WE work in general, we describe the mathematical formalism of WE, an overview of the architecture of wepy, and provide code examples of how to construct, run, and analyze simulation results for a protein–ligand system (T4 Lysozyme in an implicit solvent). This paper is written with a variety of readers in mind, including (1) those curious about how to leverage WE rare-event simulations for their domain, (2) current WE users who want to begin using new high-dimensional resamplers such as WExplore and REVO, and (3) expert users who would like to prototype or implement their own algorithms that can be easily adopted by others.

Resonator Networks, 1: An Efficient Solution for Factoring High-Dimensional, Distributed Representations of Data Structures.

The ability to encode and manipulate data structures with distributed neural representations could qualitatively enhance the capabilities of traditional neural networks by supporting rule-based symbolic reasoning, a central property of cognition. Here we show how this may be accomplished within the framework of Vector Symbolic Architectures (VSAs) (Plate, 1991; Gayler, 1998; Kanerva, 1996), whereby data structures are encoded by combining high-dimensional vectors with operations that together form an algebra on the space of distributed representations. In particular, we propose an efficient solution to a hard combinatorial search problem that arises when decoding elements of a VSA data structure: the factorization of products of multiple codevectors. Our proposed algorithm, called a resonator network, is a new type of recurrent neural network that interleaves VSA multiplication operations and pattern completion. We show in two examples-parsing of a tree-like data structure and parsing of a visual scene-how the factorization problem arises and how the resonator network can solve it. More broadly, resonator networks open the possibility of applying VSAs to myriad artificial intelligence problems in real-world domains. The companion article in this issue (Kent, Frady, Sommer, & Olshausen, 2020) presents a rigorous analysis and evaluation of the performance of resonator networks, showing it outperforms alternative approaches.

Refinement kinds: type-safe programming with practical type-level computation

This work introduces the novel concept of kind refinement , which we develop in the context of an explicitly polymorphic ML-like language with type-level computation. Just as type refinements embed rich specifications by means of comprehension principles expressed by predicates over values in the type domain, kind refinements provide rich kind specifications by means of predicates over types in the kind domain. By leveraging our powerful refinement kind discipline, types in our language are not just used to statically classify program expressions and values, but also conveniently manipulated as tree-like data structures, with their kinds refined by logical constraints on such structures. Remarkably, the resulting typing and kinding disciplines allow for powerful forms of type reflection, ad-hoc polymorphism and type-directed meta-programming, which are often found in modern software development, but not typically expressible in a type-safe manner in general purpose languages. We validate our approach both formally and pragmatically by establishing the standard meta-theoretical results of type safety and via a prototype implementation of a kind checker, type checker and interpreter for our language.

Interval-based Recording of Generated Pseudorandom Numbers

For some applications that use pseudorandom numbers it is essential to keep the record of numbers generated so far. Such a representative example is cryptanalytic TMTO approach. In order to save the space, instead of straightforward recording of individual numbers generated, an ordered tree-like data structure which tracks the intervals of generated numbers is proposed. For estimating the memory requirements of this structure as a function of pseudorandom numbers range size, an analytical probabilistic model is established and used. This model determines the maximum number of intervals during recording which corresponds to the tree size. The result obtained from analytical model is fully validated experimentally by means of simulation for a wide spectrum of range sizes.

Parallel construction of binary tree based on sorting

Introduction. Algorithms for the parallel binary tree construction are developed. The algorithms are based on sorting and described in a constructive form. For the Nelement set, the time complexity has T(R) = O(1) and T(R) = O(log2 N) estimates, where R = (N2-N)/2 is the number of processors. The tree is built with the uniqueness property. The algorithms are invariant with respect to the input sequence type. The work objective is to develop and study ways of accelerating the process of organizing and transforming the tree-like data structures on the basis of the stable maximum parallel sorting algorithms for their application to the basic operations of information retrieval on databases.Materials and Methods. A one-to-one relation between the input element set and the binary tree built for it is established using a stable address sorting. The sorting provides maximum concurrency, and, in an operator form, establishes a one-to-one mapping of input and output indices. On this basis, methods for the mutual transformation of the binary data structures are being developed.Research Results. An efficient parallel algorithm for constructing a binary tree based on the address sorting with time complexity of T(N2) = O(log2 N) is obtained. From the well-known analogues, the algorithm differs in structure and logarithmic estimation of time complexity, which makes it possible to achieve the acceleration of O(Nα), α≥1 order analogues. As an advanced version, an algorithm modification, which provides the maximum parallel construction of the binary tree based on a stable address sorting and a priori calculation of the stored subtree root indices is suggested. The algorithm differs in structure and estimation of T(1) = O(1) time complexity. A similar estimate is achieved in a sequential version of the modified algorithm, which allows obtaining the acceleration of known analogs O(Nα), α>1 order.Discussion and Conclusions. The results obtained are focused on the creation of effective methods for the dynamic database processing. The proposed methods and algorithms can form an algorithmic basis for an advanced deterministic search on the relational databases and information systems.

SILK+ Preventing Latency Spikes in Log-Structured Merge Key-Value Stores Running Heterogeneous Workloads

Log-Structured Merge Key-Value stores (LSM KVs) are designed to offer good write performance, by capturing client writes in memory, and only later flushing them to storage. Writes are later compacted into a tree-like data structure on disk to improve read performance and to reduce storage space use. It has been widely documented that compactions severely hamper throughput. Various optimizations have successfully dealt with this problem. These techniques include, among others, rate-limiting flushes and compactions, selecting among compactions for maximum effect, and limiting compactions to the highest level by so-called fragmented LSMs. In this article, we focus on latencies rather than throughput. We first document the fact that LSM KVs exhibit high tail latencies. The techniques that have been proposed for optimizing throughput do not address this issue, and, in fact, in some cases, exacerbate it. The root cause of these high tail latencies is interference between client writes, flushes, and compactions. Another major cause for tail latency is the heterogeneous nature of the workloads in terms of operation mix and item sizes whereby a few more computationally heavy requests slow down the vast majority of smaller requests. We introduce the notion of an Input/Output (I/O) bandwidth scheduler for an LSM-based KV store to reduce tail latency caused by interference of flushing and compactions and by workload heterogeneity. We explore three techniques as part of this I/O scheduler: (1) opportunistically allocating more bandwidth to internal operations during periods of low load, (2) prioritizing flushes and compactions at the lower levels of the tree, and (3) separating client requests by size and by data access path. SILK+ is a new open-source LSM KV that incorporates this notion of an I/O scheduler.

An exact algorithm for the minimum quartet tree cost problem

The minimum quartet tree cost (MQTC) problem is a graph combinatorial optimization problem where, given a set of $$n \ge 4$$ data objects and their pairwise costs (or distances), one wants to construct an optimal tree from the $$3 \cdot {n \atopwithdelims ()4}$$ quartet topologies on n, where optimality means that the sum of the costs of the embedded (or consistent) quartet topologies is minimal. The MQTC problem is the foundation of the quartet method of hierarchical clustering, a novel hierarchical clustering method for non tree-like (non-phylogeny) data in various domains, or for heterogeneous data across domains. The MQTC problem is NP-complete and some heuristics have been already proposed in the literature. The aim of this paper is to present a first exact solution approach for the MQTC problem. Although the algorithm is able to get exact solutions only for relatively small problem instances, due to the high problem complexity, it can be used as a benchmark for validating the performance of any heuristic proposed for the MQTC problem.

Improving generalized inverted index lock wait times

Concurrent operations on tree like data structures is a cornerstone of any database system. Concurrent operations intended for improving read\\write performance and usually implemented via some way of locking. Deadlock-free methods of concurrency control are known as tree locking protocols. These protocols provide basic operations(verbs) and algorithm (ways of operation invocations) for applying it to any tree-like data structure. These algorithms operate on data, managed by storage engine which are very different among RDBMS implementations. In this paper, we discuss tree locking protocol implementation for General inverted index (Gin) applied to multiversion concurrency control (MVCC) storage engine inside PostgreSQL RDBMS. After that we introduce improvements to locking protocol and provide usage statistics about evaluation of our improvement in very high load environment in one of the world’s largest IT company.

Generating derivative superstructures for systems with high configurational freedom

Modeling alloys requires the exploration of all possible configurations of atoms. Additionally, modeling the thermal properties of materials requires knowledge of the possible ways of displacing the atoms. One solution to finding all symmetrically unique configurations and displacements is to generate the complete list of possible configurations and remove those that are symmetrically equivalent. This approach, however, suffers from a combinatorial explosion when the supercell size is large, when there are more than two atom types, or when there are many displaced atoms. This problem persists even when there are only a relatively small number of unique arrangements that survive the elimination process. Here, we extend an existing algorithm to include the extra configurational degrees of freedom from the inclusion of displacement directions. The algorithm uses group theory and a tree-like data structure to eliminate large classes of configurations, avoiding the typical combinatoric explosion. With this approach we can now enumerate previously inaccessible cases, including atomic displacements.

Optimal Polynomial Time Algorithm for Restoring Multicast Cloud Services

The failure-prone nature of data center networks has evoked countless contributions to develop proactive and reactive countermeasures. Yet, most of these techniques were developed with unicast services in mind. When in fact, multiple services hosted in data center networks today rely on multicast communication to disseminate traffic. Hence, the existing survivability schemes fail to cater to the distinctive properties and quality of service requirements that multicast services entail. This letter is devoted to understanding the ramifications of facility node or substrate link failure on multicast services residing in cloud networks. We formally define the multicast virtual network restoration problem and prove its NP-complete nature in arbitrary graphs. Furthermore, we prove that the problem can be solved in polynomial-time in multi-rooted treelike data center network topologies.

A Hierarchical Representation and Computation Scheme of Arbitrary-dimensional Geometrical Primitives Based on CGA

Resolving the conflicts between the high dimensionality of geometry representation and the linear organization and storage of geometrical objects in computer memories plays a key role in spatial data structure constructions. In this paper, a new data structure MVTree is developed based on geometric algebra to support the unified organization and computation of geometrical primitives. The MVTree is a tree-like data structure which has a dimensional hierarchical structure generated by outer product. Multidimensional geometrical primitives is represented as the combination of blades stored in the nodes of MVTrees. The geometric computation between different geometrical objects is operated with GA operators with a judgement-based hierarchical computation. Applications of the MVTree are demonstrated by a topological relation computation and a Delaunay-TIN intersection. The results suggest that the MVTree structure can support unified representation of arbitrary-dimensional geometrical primitives, and can integrate data organization and computation in a unitary structure as well. The application of the new MVTree data structure can not only reduce the complexity of data architectures but also inherit the power of geometric algebra computing to improve the processing ability of computer graphic software.

Cell-type phylogenetics and the origin of endometrial stromal cells.

A challenge of genome annotation is the identification of genes performing specific biological functions. Here, we propose a phylogenetic approach that utilizes RNA-seq data to infer the historical relationships among cell types and to trace the pattern of gene-expression changes on the tree. The hypothesis is that gene-expression changes coincidental with the origin of a cell type will be important for the function of the derived cell type. We apply this approach to the endometrial stromal cells (ESCs), which are critical for the initiation and maintenance of pregnancy. Our approach identified well-known regulators of ESCs, PGR and FOXO1, as well as genes not yet implicated in female fertility, including GATA2 and TFAP2C. Knockdown analysis confirmed that they are essential for ESC differentiation. We conclude that phylogenetic analysis of cell transcriptomes is a powerful tool for discovery of genes performing cell-type-specific functions.

Towards better understanding of frequent itemset relationships through tree-like data structures

A common goal of descriptive data mining techniques is presenting new information in concise, easily interpretable and understandable ways. In this paper we propose a technique for modeling relationships between frequent itemsets through visually descriptive tree-like data structures. We define and discuss algorithms for forming these structures as well as suggest new measures for evaluating their informative value. We also present our visualization tool which implements proposed concepts and solutions. Finally, we apply our research on two different dataset types and discuss the results. The first dataset proves the applicability of our visualization technique for common market basket analysis. The second dataset is an example of a “dense” dataset, a troublesome type for frequent itemset mining since it commonly produces a significantly large number of frequent itemsets. We demonstrate a modified variant of our technique which allows efficient visual representation of such datasets as well.