Top-down Computation Research Articles

Learning Sentence-Level Representations with Predictive Coding

Learning sentence representations is an essential and challenging topic in the deep learning and natural language processing communities. Recent methods pre-train big models on a massive text corpus, focusing mainly on learning the representation of contextualized words. As a result, these models cannot generate informative sentence embeddings since they do not explicitly exploit the structure and discourse relationships existing in contiguous sentences. Drawing inspiration from human language processing, this work explores how to improve sentence-level representations of pre-trained models by borrowing ideas from predictive coding theory. Specifically, we extend BERT-style models with bottom-up and top-down computation to predict future sentences in latent space at each intermediate layer in the networks. We conduct extensive experimentation with various benchmarks for the English and Spanish languages, designed to assess sentence- and discourse-level representations and pragmatics-focused assessments. Our results show that our approach improves sentence representations consistently for both languages. Furthermore, the experiments also indicate that our models capture discourse and pragmatics knowledge. In addition, to validate the proposed method, we carried out an ablation study and a qualitative study with which we verified that the predictive mechanism helps to improve the quality of the representations.

Revisiting the Top-Down Computation of BDD of Spanning Trees of a Graph and Its Tutte Polynomial

Revisiting the Top-Down Computation of BDD of Spanning Trees of a Graph and Its Tutte Polynomial

Scalable distributed data cube computation for large-scale multidimensional data analysis on a Spark cluster

A data cube is a powerful analytical tool that stores all aggregate values over a set of dimensions. It provides users with a simple and efficient means of performing complex data analysis while assisting in decision making. Since the computation time for building a data cube is very large, however, efficient methods for reducing the data cube computation time are needed. Previous works have developed various algorithms for efficiently generating data cubes using MapReduce, which is a large-scale distributed parallel processing framework. However, MapReduce incurs the overhead of disk I/Os and network traffic. To overcome these MapReduce limitations, Spark was recently proposed as a memory-based parallel/distributed processing framework. It has attracted considerable research attention owing to its high performance. In this paper, we propose two algorithms for efficiently building data cubes. The algorithms fully leverage Spark’s mechanisms and properties: Resilient Distributed Top-Down Computation (RDTDC) and Resilient Distributed Bottom-Up Computation (RDBUC). The former is an algorithm for computing the components (i.e., cuboids) of a data cube in a top-down approach; the latter is a bottom-up approach. The RDTDC algorithm has three key functions. (1) It approximates the size of the cuboid using the cardinality without additional Spark action computation to determine the size of each cuboid during top-down computation. Thus, one cuboid can be computed from the upper cuboid of a smaller size. (2) It creates an execution plan that is optimized to input the smaller sized cuboid. (3) Lastly, it uses a method of reusing the result of the already computed cuboid by top-down computation and simultaneously computes the cuboid of several dimensions. In addition, we propose the RDBUC bottom-up algorithm in Spark, which is widely used in computing Iceberg cubes to maintain only cells satisfying a certain condition of minimum support. This algorithm incorporates two primary strategies: (1) reducing the input size to compute aggregate values for a dimension combination (e.g., A, B, and C) by removing the input, which does not satisfy the Iceberg cube condition at its lower dimension combination (e.g., A and B) computed earlier. (2) We use a lazy materialization strategy that computes every combination of dimensions using only transformation operations without any action operation. It then stores them in a single action operation. To prove the efficiency of the proposed algorithms using a lazy materialization strategy by employing only one action operation, we conducted extensive experiments. We compared them to the cube() function, a built-in cube computation library of Spark SQL. The results showed that the proposed RDTDC and RDBUC algorithms outperformed Spark SQL cube().

Hierarchical graphical models for simultaneous tracking and recognition in wide-area scenes.

We present a unified framework to track multiple people, as well localize, and label their activities, in complex long-duration video sequences. To do this, we focus on two aspects: 1) the influence of tracks on the activities performed by the corresponding actors and 2) the structural relationships across activities. We propose a two-level hierarchical graphical model, which learns the relationship between tracks, relationship between tracks, and their corresponding activity segments, as well as the spatiotemporal relationships across activity segments. Such contextual relationships between tracks and activity segments are exploited at both the levels in the hierarchy for increased robustness. An L1-regularized structure learning approach is proposed for this purpose. While it is well known that availability of the labels and locations of activities can help in determining tracks more accurately and vice-versa, most current approaches have dealt with these problems separately. Inspired by research in the area of biological vision, we propose a bidirectional approach that integrates both bottom-up and top-down processing, i.e., bottom-up recognition of activities using computed tracks and top-down computation of tracks using the obtained recognition. We demonstrate our results on the recent and publicly available UCLA and VIRAT data sets consisting of realistic indoor and outdoor surveillance sequences.

Research on Disassembly Sequence Planning in DFD

The problems of automatic disassembly direction computation and disassembly sequencing must be solved in DFD. The computation rules have been set based on freedom intersection. Applying constraints elimination and assembly construction mapping, sequence generation has been realized by heuristic algorithm. To simplify the planning procedure, a modulized construction model has been developed for top-down computation. The principle could be applied for CAD software development in DFD.

Inference and Learning with Hierarchical Shape Models

In this work we introduce a hierarchical representation for object detection. We represent an object in terms of parts composed of contours corresponding to object boundaries and symmetry axes; these are in turn related to edge and ridge features that are extracted from the image. We propose a coarse-to-fine algorithm for efficient detection which exploits the hierarchical nature of the model. This provides a tractable framework to combine bottom-up and top-down computation. We learn our models from training images where only the bounding box of the object is provided. We automate the decomposition of an object category into parts and contours, and discriminatively learn the cost function that drives the matching of the object to the image using Multiple Instance Learning. Using shape-based information, we obtain state-of-the-art localization results on the UIUC and ETHZ datasets.

The Multi-Tree Cubing algorithm for computing iceberg cubes

The computation of data cubes is one of the most expensive operations in on-line analytical processing (OLAP). To improve efficiency, an iceberg cube represents only the cells whose aggregate values are above a given threshold (minimum support). Top-down and bottom-up approaches are used to compute the iceberg cube for a data set, but both have performance limitations. In this paper, a new algorithm, called Multi-Tree Cubing (MTC), is proposed for computing an iceberg cube. The Multi-Tree Cubing algorithm is an integrated top-down and bottom-up approach. Overall control is handled in a top-down manner, so MTC features shared computation. By processing the orderings in the opposite order from the Top-Down Computation algorithm, the MTC algorithm is able to prune attributes. The Bottom Up Computation (BUC) algorithm and its variations also perform pruning by relying on the processing of intermediate partitions. The MTC algorithm, however, prunes without processing such partitions. The MTC algorithm is based on a specialized type of prefix tree data structure, called an Attribute–Partition tree (AP-tree), consisting of attribute and partition nodes. The AP-tree facilitates fast, in-memory sorting and APRIORI-like pruning. We report on five series of experiments, which confirm that MTC is consistently as fast or faster than BUC, while finding the same iceberg cubes.

Recognition by top-down and bottom-up processing in cortex: the control of selective attention.

Visual recognition is achieved by a hierarchy of bidirectionally connected cortical areas. The entry of signals into higher areas involves the serial sampling of information within a movable window of attention. Here we explore how the cortex can move this window and integrate the sampled information. To make this concrete, we modeled the process of visual word recognition by hierarchical cortical areas representing features, letters, and words. At the start of the recognition process, nodes representing all contextually possible words are active. Simple connectivity rules allow a parallel top-down (T-D) computation of the relative probability of each feature at each location given the set of active words. This information is then used to guide the window of attention to information-rich features (e.g., a feature that is present in the visual image but has lowest probability). Bottom-up processing of this feature excludes words that do not contain it and leads to T-D recomputation of feature probabilities. Recognition occurs after several such cycles when all but one word has been excluded. We show that when 950 words are stored in long-term memory, recognition occurs after an average of 4.9 cycles. Because covert attention can be moved every 20-30 ms, word recognition could be as fast as determined experimentally (<200 ms of cortical processing). This model accounts for the findings that recognition time depends logarithmically on set size, recognition time is reduced when context reduces the number of possible targets, the time to classify a nonword decreases when its approximation to English decreases, and in high level cortex, the firing of neurons tuned to an object increases progressively as its recognition occurs. More generally the model provides a physiologically plausible view of how bi-directional signal flow in cortex guides attention to produce efficient recognition.

Symmetry in Syntax: Merge and Demerge

This article proposes a theory of phrase structure which is free from unmotivated stipulations as much as possible and which is also compatible with the most restrictive theory of parametric variation currently available (cf. Fukui (1995)), with a special focus on the nature and role of linear order and functional categories. A hypothesis called the "Symmetry of Derivation" is put forth, according to which the computations in the overt syntax (i.e., the pre-Spell-Out computations) and the computations in the (pre-Morphology) phonological component are "symmetric" in the sense that they form mirror images of each other. More specifically, we propose that language computation maps an array of linguistic elements to an interface representation in such a way that it starts with a lexical item (a head) proceeding in a bottom-up fashion (Merge) and at some point of this step-by-step derivational process (Spell-Out) starts "decomposing" the structures already formed in a top-down fashion ("Demerge") until the derivation reaches a completely unstructured sequence with a fixed linear order. It is shown that this symmetry principle explains the major properties of phrase structure in an elegant way. The principle accounts for the apparently universal "leftness" property of Spec in a straightforward way by attributing its leftness to the fact that a Spec, by definition, is the first maximal projection in a given phrase that the top-down computation Demerge encounters. With respect to the order between a head and its complement, which allows for cross-linguistic variation (head-initial vs. head-last), the symmetry principle predicts that the head-last order reflects the "base" order involving no relevant movement whereas the head-initial order is derived by movement, in clear contrast with Kayne's (1994) approach. The difference between head-initial English and head-last Japanese is, then, attributed to different properties of a "light verb" v (Chomsky (1995b)): v has the property of attracting V in English but not in Japanese. Numerous theoretical and empirical consequences are shown to follow in an interesting way from this hypothesis, coupled with the symmetry principle of derivations.

Efficient top-down computation of queries under the well-founded semantics

The well-founded model provides a natural and robust semantics for logic programs with negative literals in rule bodies. Although various procedural semantics have been proposed for query evaluation under the well-founded semantics, the practical issues of implementation for effective and efficient computation of queries have been rarely discussed. This paper investigates two major implementation issues of query evaluation under the well-founded semantics, namely, (a) to ensure that negative literals be resolved only after their positive counterparts have been completely evaluated, and (b) to detect and handle potential negative loops. We present efficient incremental algorithms for maintaining positive and negative dependencies among subgoals in a top-down evaluation. Both completely evaluated subgoals and potential negative loops are detected by inspecting the dependency information of a single subgoal. Our implementation can be viewed as an effective successor to SLDNF resolution, extending Prolog computation in a natural and smooth way.

Abstract interpretation: A kind of magic

Magic sets and, more recently, magic templates have been used in the field of deductive databases to facilitate efficient bottom-up evaluation of database queries. Roughly speaking a top-down computation of a definite logic program is simulated by first transforming the program and then executing the new program bottom-up. In this paper we give a new and very simple proof that this approach is equivalent to the collecting interpretation of the abstract interpretation framework for logic programs of Mellish. As a side-effect we are also able to show that “bottom-up” abstract interpretation based on the magic templates transformation is equally powerful as Mellish's abstract interpretation framework, but less powerful than other (more precise) abstract interpretation frameworks.