Vertex Programs Research Articles

Fregel: a functional domain-specific language for vertex-centric large-scale graph processing

Abstract The vertex-centric programming model is now widely used for processing large graphs. User-defined vertex programs are executed in parallel over every vertex of a graph, but the imperative and explicit message-passing style of existing systems makes defining a vertex program unintuitive and difficult. This article presents Fregel, a purely functional domain-specific language for processing large graphs and describes its model, design, and implementation. Fregel is a subset of Haskell, so Haskell tools can be used to test and debug Fregel programs. The vertex-centric computation is abstracted using compositional programming that uses second-order functions on graphs provided by Fregel. A Fregel program can be compiled into imperative programs for use in the Giraph and Pregel+ vertex-centric frameworks. Fregel’s functional nature without side effects enables various transformations and optimizations during the compilation process. Thus, the programmer is freed from the burden of program optimization, which is manually done for existing imperative systems. Experimental results for typical examples demonstrated that the compiled code can be executed with reasonable and promising performance.

Social Recommendation With Learning Personal and Social Latent Factors

Due to leveraging social relationships between users as well as their past social behavior, social recommendation becomes a core component in recommendation systems. Most existing social recommendation methods only consider direct social relationships among users (e.g., explicit and observed social relations). Recently, researchers proved that indirect social relationships can be effective to improve the recommendation quality when users only have few social connections, because it can identify the user interesting group even though the users have no observed social connection. In the literature, separate two-stage methods are studied, but they cannot explicitly capture the natural relationship between indirect social relations and latent user/item factors. In this paper, the main contribution is to propose a new joint recommendation model taking advantage of the Indirect Social Relations detection and Matrix Factorization collaborative filtering on social network and rating behavior information, which is called as InSRMF. In our work, the user latent factors can simultaneously and seamlessly capture user's personal preferences and social group characteristics. To optimize the InSRMF model, we develop a parallel graph vertex programming algorithm for efficiently handling large scale social recommendation data. Experiments based on four real-world datasets (Ciao, Epinions, Douban and Yelp) are conducted to demonstrate the performance of the proposed model. The experimental results have shown that InSRMF has ability to mine the proper indirect social relations and improve the recommendation performance compared with the testing methods in the literature, especially on the users with few social neighbors, Near-cold-start Users, Pure-cold-start Users and Long-tail Items.

Characterizing the Twitter network of prominent politicians and SPLC-defined hate groups in the 2016 US presidential election

We characterize the Twitter networks of the major presidential candidates, Donald J. Trump and Hillary R. Clinton, with various American hate groups defined by the US Southern Poverty Law Center (SPLC). We further examined the Twitter networks for Bernie Sanders, Ted Cruz, and Paul Ryan, for 9 weeks around the 2016 election (4 weeks prior to the election and 4 weeks post-election). We carefully account for the observed heterogeneity in the Twitter activity levels across individuals through the null hypothesis of apathetic retweeting that is formalized as a random network model based on the directed, multi-edged, self-looped, configuration model. Our data revealed via a generalized Fisher’s exact test that there were significantly many Twitter accounts linked to SPLC-defined hate groups belonging to seven ideologies (Anti-Government, Anti-Immigrant, Anti-LGBT, Anti-Muslim, Alt-Right, White-Nationalist and Neo-Nazi) and also to @realDonaldTrump relative to the accounts of the other four politicians. The exact hypothesis test uses Apache Spark’s distributed sort and join algorithms to produce independent samples in a fully scalable way from the null model. Additionally, by exploring the empirical Twitter network we found that significantly more individuals had the fewest retweet degrees of separation simultaneously from Trump and each one of these seven hateful ideologies relative to the other four politicians. We conduct this exploration via a geometric model of the observed retweet network, distributed vertex programs in Spark’s GraphX library and a visual summary through neighbor-joined population retweet ideological trees. Remarkably, less than 5% of individuals had three or fewer retweet degrees of separation simultaneously from Trump and one of several hateful ideologies relative to the other four politicians. Taken together, these findings suggest that Trump may have indeed possessed unique appeal to individuals drawn to hateful ideologies; however, such individuals constituted a small fraction of the sampled population.

Improving parallel efficiency for asynchronous graph analytics using Gauss‐Seidel‐based matrix computation

SummaryGraph analytics is extensively used in big‐data applications such as social networks, web analysis, bio‐informatics, etc. Most graph processing frameworks adopt vertex‐centric model due to its ease of use and programming. However, when dealing with asynchronous graph analytics, frameworks based on vertex programming perform inefficiently. The reason is that first, vertex programming must guarantee the sequential consistency, which means frequent use of locks or atomic operations, and second, the algorithms are parallelized in vertex level and latent parallelism of the algorithms cannot be exploited. To improve parallel efficiency of asynchronous graph processing, the Gauss‐Seidel style algorithms in particular, this paper proposes a scheduling model using Gauss‐Seidel‐based matrix computation, which converts the vertex programming into two main matrix operations and then algorithms are parallelized by row and column vectors. Compared to vertex programming, our model parallelizes algorithms in a finer way to exploit more latent parallelism, while retains the ease‐of‐programming advantage of vertex programming. Instead of using locks to guarantee the sequential consistency, our model uses a hybrid synchronization policy to reduce serializability among threads and overheads of context switching. Furthermore, this model strengthens locality of the program. Experiment results show that our model outperforms vertex‐centric asynchronous frameworks in both performance and scalability. Moreover, it even surpasses the matrix‐based synchronous framework GraphMat with some non‐Gauss‐Seidel style algorithms.

HPGraph: High-Performance Graph Analytics with Productivity on the GPU

The growing use of graph in many fields has sparked a broad interest in developing high-level graph analytics programs. Existing GPU implementations have limited performance with compromising on productivity. HPGraph, our high-performance bulk-synchronous graph analytics framework based on the GPU, provides an abstraction focused on mapping vertex programs to generalized sparse matrix operations on GPU as the backend. HPGraph strikes a balance between performance and productivity by coupling high-performance GPU computing primitives and optimization strategies with a high-level programming model for users to implement various graph algorithms with relatively little effort. We evaluate the performance of HPGraph for four graph primitives (BFS, SSSP, PageRank, and TC). Our experiments show that HPGraph matches or even exceeds the performance of high-performance GPU graph libraries such as MapGraph, nvGraph, and Gunrock. HPGraph also runs significantly faster than advanced CPU graph libraries.

Think like a vertex, behave like a function! a functional DSL for vertex-centric big graph processing

The vertex-centric programming model, known as “think like a vertex”, is being used more and more to support various big graph processing methods through iterative supersteps that execute in parallel a user-defined vertex program over each vertex of a graph. However, the imperative and message-passing style of existing systems makes defining a vertex program unintuitive. In this paper, we show that one can benefit more from “Thinking like a vertex” by “Behaving like a function” rather than “Acting like a procedure” with full use of side effects and explicit control of message passing, state, and termination. We propose a functional approach to vertex-centric graph processing in which the computation at every vertex is abstracted as a higher-order function and present Fregel, a new domain-specific language. Fregel has clear functional semantics, supports declarative description of vertex computation, and can be automatically translated into Pregel, an emerging imperative-style distributed graph processing framework, and thereby achieve promising performance. Experimental results for several typical examples show the promise of this functional approach.

John Holland Martin From Picograms to Petagrams and Copepods to Climate: The Class of MS 280, Moss Landing Marine Laboratories

John Holland Martin From Picograms to Petagrams and Copepods to Climate: <i>The Class of MS 280, Moss Landing Marine Laboratories</i>

GraphMat

Given the growing importance of large-scale graph analytics, there is a need to improve the performance of graph analysis frameworks without compromising on productivity. GraphMat is our solution to bridge this gap between a user-friendly graph analytics framework and native, hand-optimized code. GraphMat functions by taking vertex programs and mapping them to high performance sparse matrix operations in the backend. We thus get the productivity benefits of a vertex programming framework without sacrificing performance. GraphMat is a single-node multicore graph framework written in C++ which has enabled us to write a diverse set of graph algorithms with the same effort compared to other vertex programming frameworks. GraphMat performs 1.1-7X faster than high performance frameworks such as GraphLab, CombBLAS and Galois. GraphMat also matches the performance of MapGraph, a GPU-based graph framework, despite running on a CPU platform with significantly lower compute and bandwidth resources. It achieves better multicore scalability (13-15X on 24 cores) than other frameworks and is 1.2X off native, hand-optimized code on a variety of graph algorithms. Since GraphMat performance depends mainly on a few scalable and well-understood sparse matrix operations, GraphMat can naturally benefit from the trend of increasing parallelism in future hardware.

Fast Isosurface Rendering on a GPU by Cell Rasterization

Abstract This paper presents a fast, high‐quality, GPU‐based isosurface rendering pipeline for implicit surfaces defined by a regular volumetric grid. GPUs are designed primarily for use with polygonal primitives, rather than volume primitives, but here we directly treat each volume cell as a single rendering primitive by designing a vertex program and fragment program on a commodity GPU. Compared with previous raycasting methods, ours has a more effective memory footprint (cache locality) and better coherence between multiple parallel SIMD processors. Furthermore, we extend and speed up our approach by introducing a new view‐dependent sorting algorithm to take advantage of the early‐z‐culling feature of the GPU to gain significant performance speed‐up. As another advantage, this sorting algorithm makes multiple transparent isosurfaces rendering available almost for free. Finally, we demonstrate the effectiveness and quality of our techniques in several real‐time rendering scenarios and include analysis and comparisons with previous work.

Fast and simple hardware accelerated voxelizations using simplicial coverings

Voxelization of solids, that is the representation of a solid by a set of voxels that approximates it, is an operation with important applications in fields like solid modeling, physical simulation or volume graphics. Moreover, the new generation of affordable 3D raster displays has renewed the interest on fast voxelization algorithms, as the scan-conversion of a solid is a basic operation on these devices. In this paper a hardware accelerated method for computing a voxelization of a polyhedron is presented. The algorithm is simple, efficient, robust and handles any kind of polyhedron (self-intersecting, with or without holes, manifold or non-manifold). Three different implementations are described in detail. The first is a conventional implementation in the CPU, the second is a hardware accelerated implementation that uses standard OpenGL primitives, and the third exploits the capabilities of modern GPUs by using vertex programs.

Hybrid Visualization for White Matter Tracts using Triangle Strips and Point Sprites

Diffusion tensor imaging is of high value in neurosurgery, providing information about the location of white matter tracts in the human brain. For their reconstruction, streamline techniques commonly referred to as fiber tracking model the underlying fiber structures and have therefore gained interest. To meet the requirements of surgical planning and to overcome the visual limitations of line representations, a new real-time visualization approach of high visual quality is introduced. For this purpose, textured triangle strips and point sprites are combined in a hybrid strategy employing GPU programming. The triangle strips follow the fiber streamlines and are textured to obtain a tube-like appearance. A vertex program is used to orient the triangle strips towards the camera. In order to avoid triangle flipping in case of fiber segments where the viewing and segment direction are parallel, a correct visual representation is achieved in these areas by chains of point sprites. As a result, a high quality visualization similar to tubes is provided allowing for interactive multimodal inspection. Overall, the presented approach is faster than existing techniques of similar visualization quality and at the same time allows for real-time rendering of dense bundles encompassing a high number of fibers, which is of high importance for diagnosis and surgical planning.

Far voxels

We present an efficient approach for end-to-end out-of-core construction and interactive inspection of very large arbitrary surface models. The method tightly integrates visibility culling and out-of-core data management with a level-of-detail framework. At preprocessing time, we generate a coarse volume hierarchy by binary space partitioning the input triangle soup. Leaf nodes partition the original data into chunks of a fixed maximum number of triangles, while inner nodes are discretized into a fixed number of cubical voxels. Each voxel contains a compact direction dependent approximation of the appearance of the associated volumetric subpart of the model when viewed from a distance. The approximation is constructed by a visibility aware algorithm that fits parametric shaders to samples obtained by casting rays against the full resolution dataset. At rendering time, the volumetric structure, maintained off-core, is refined and rendered in front-to-back order, exploiting vertex programs for GPU evaluation of view-dependent voxel representations, hardware occlusion queries for culling occluded subtrees, and asynchronous I/O for detecting and avoiding data access latencies. Since the granularity of the multiresolution structure is coarse, data management, traversal and occlusion culling cost is amortized over many graphics primitives. The efficiency and generality of the approach is demonstrated with the interactive rendering of extremely complex heterogeneous surface models on current commodity graphics platforms.

Geometry engine architecture with early backface culling hardware

Most graphics accelerators waste valuable performance on transforming invisible vertices. To solve this problem, we have performed backface culling (BFC) earlier than transform and lighting (TnL). This paper proposes a survived vertex decision (SVD) algorithm to remove invisible vertices, and also suggests a geometry engine architecture that performs the early BFC with the SVD algorithm. This approach requires less hardware overhead. The SVD algorithm discards a vertex only if all triangles sharing that vertex are invisible in the mesh representing triangle lists or strips. The dedicated hardware performing the early BFC guarantees better performance in our approach, since it runs with the vertex engines in parallel. Particularly for a standalone engine, we introduce a unified architecture named the VP-Engine, which can perform the same tasks of the vertex engine and also handle the early BFC. Our architecture is designed using an instruction set simulator with a C++ library for cycle-accurate simulations. The early BFC removes half of the vertices that are transformed in the conventional approach, and as such the performance of our proposed architecture is twice as fast at maximum. Even with the sequential operations of early BFC and typical TnL, the VP-Engine is faster while the length of a vertex program is larger than 24.

Adaptive extraction of time-varying isosurfaces

We present an algorithm for adaptively extracting and rendering isosurfaces from compressed time-varying volume data sets. Tetrahedral meshes defined by longest edge bisection are used to create a multiresolution representation of the volume in the spatial domain that is adapted over time to approximate the time-varying volume. The reextraction of the isosurface at each time step is accelerated with the vertex programming capabilities of modern graphics hardware. A data layout scheme which follows the access pattern indicated by mesh refinement is used to access the volume in a spatially and temporally coherent manner. This data layout scheme allows our algorithm to be used for out-of-core visualization.

Fat borders: gap filling for efficient view-dependent LOD NURBS rendering

Real-time high-quality rendering of complex NURBS models remains a big challenge. Simply splitting the models into several parts which can be simplified and rendered independently introduces disturbing gaps along the common borders. Recent approaches for view-dependent rendering of huge models either neglect the artifacts introduced by the gaps or try to maintain the connectivity of the models. Unfortunately, in the second case the computational complexity and storage requirements of the algorithms are high. In this work we present a novel solution to the gap problem. Vertex programs are used to generate appropriately shaded fat borders that fill the gaps between neighboring patches.

Efficient Splatting Using Modern Graphics Hardware

Abstract Interactive volume rendering for data set sizes larger than one million samples requires either dedicated hardware, such as three-dimensional texture mapping, or a sparse representation and rendering algorithm. Consumer graphics cards have seen a rapid explosion of performance and capabilities over the past few years. This paper presents a splatting algorithm for direct volume rendering that utilizes the new capabilities of vertex programs and the OpenGL imaging extensions. This paper presents three techniques: immediate mode rendering, vertex shader rendering, and point convolution rendering, to implement splatting on a PC equipped with an NVIDIA GeForce4 display card. Per-splat and per-voxel render time analysis is conducted for these techniques. The results show that vertex-shader rendering offers an order of magnitude speed-up over immediate mode rendering and that interactive volume rendering is becoming feasible on these consumer-level graphics cards for complex volumes with millions of voxels.

A new apparatus for track-analysis in nuclear track emulsion based on a CCD-camera device

A CCD camera-based, image-analyzing system for automatic evaluation of nuclear track emulsion chambers is presented. The stage of a normal microscope moves using three remote controlled stepping motors with a step size of 0.25 microm. A CCD-camera is mounted on top of the microscope in order to register the nuclear emulsion. The camera has a resolution capable of differentiating single emulsion-grains (0.6 microm). The camera picture is transformed from analogue to digital signals and stored by a frame grabber. Some background-picture elements can be eliminated by applying cuts on grey levels. The central computer processes the picture, correlates the single picture points, the coordinates and the grey-levels, such that in the end one has a unique assignment of each picture point to an address on the hard disk for a given plate. After repetition of this procedure for several plates by means of an appropriate software (for instance our vertex program [1]), the coordinates of the points are combined to tracks, and a variety of distributions like pseudorapidity-distributions can be calculated and presented on the terminal.

Labyrinth of doom: A device to minimize the “swimmer” component in sediment trap collections

A simple device used to discriminate between “swimmers” and “nonswimmers” in sediment trap collections has been developed. It consists of a series of funnels which concentrate passively sinking material into a central collection tube, whereas swimmers are distributed randomly between the inner and outer collection tubes. The degree to which swimmer/nonswimmer fractionation is achieved depends on the number and spacing of funnels and the ratio of inner to outer collection tube areas. Although inner collection tube samples contain a few swimmers, there are considerably fewer than are collected in cod ends without this device. Furthermore, outer tube contents are devoid of passively sinking material providing a “pure swimmer” fraction which can be compared to inner collection tube contents. This device has been successfully deployed on many occasions in traps of the Soutar (conical) and Knauer (cylindrical) type as part of the VERTEX program in the NE Pacific.

Organic geochemistry of particulate matter in the ocean: The role of particles in oceanic sedimentary cycles

Organic geochemists often use the occurrence of specific organic compounds in marine sediments as biomarkers to relate sedimentary organic matter to biological processes in the overlying water column. However, organic matter undergoes many diagenetic changes during transport through the water column. Only a small fraction of the organic matter produced in surface waters reaches the sea floor to be incorporated into the sediments. This material is subject to considerable transformation by heterotrophic organisms. The initial structures of individual compounds can be altered and the proportion of various compounds and compound classes can change as a result of varying stabilities. We summarize here the results of our studies of the organic geochemistry of suspended and sinking particles collected as part of the PARFLUX and VERTEX programs and present new results for amino acids, fatty acids, sterols and steroidal ketones from the VERTEX project site north of Hawaii.

Efficient numerical techniques for perturbative lattice gauge theory computations

We discuss a set of methods and numerical tools, which are useful for a computer based approach to perturbative calculations in lattice gauge theory. The topics considered include the automatic generation of gluon vertex programs, a derivation of the Faddeev-Popov determinant on lattices with boundary, the use of a partially finite lattice with twisted boundary conditions as an infrared cutoff without zero modes, and finally the numerical extrapolation of lattice Feynman diagrams to the continuum limit. As an illustration of the methods we describe their implementation in the computation of the on-shell improved lattice action at weak coupling.