Data Organization Strategy Research Articles

An efficient memory data organization strategy for application-characteristic graph processing

An efficient memory data organization strategy for application-characteristic graph processing

A Massive Laser Point Cloud Data Organization Strategy Based on the Mixed Model

In the process of data interaction, neighborhood query, filtering, visualization, and dynamic update of LiDAR point cloud data, how to efficiently organize and process massive point cloud data, and quickly index and locate any point in the point cloud and its neighborhood Search is a key issue to be solved urgently. In this paper, combining the advantages of a virtual grid with no interpolation loss on original data, sT spatial relationship, and low memory occupation of the octree, we design an index method based on the combination of virtual grid and adaptive octree based on dynamic scheduling of internal and external memory. Realize the organization and scheduling of massive LiDAR laser scanning point cloud data.

Relational data paradigms: What do we learn by taking the materiality of databases seriously?

Although databases have been well-defined and thoroughly discussed in the computer science literature, the actual users of databases often have varying definitions and expectations of this essential computational infrastructure. Systems administrators and computer science textbooks may expect databases to be instantiated in a small number of technologies (e.g., relational or graph-based database management systems), but there are numerous examples of databases in non-conventional or unexpected technologies, such as spreadsheets or other assemblages of files linked through code. Consequently, we ask: How do the materialities of non-conventional databases differ from or align with the materialities of conventional relational systems? What properties of the database do the creators of these artifacts invoke in their rhetoric describing these systems—or in the data models underlying these digital objects? To answer these questions, we conducted a close analysis of four non-conventional scientific databases. By examining the materialities of information representation in each case, we show how scholarly communication regimes shape database materialities— and how information organization paradigms shape scholarly communication. These cases show abandonment of certain constraints of relational database construction alongside maintenance of some key relational data organization strategies. We discuss the implications that these relational data paradigms have for data use, preservation, and sharing, and discuss the need to support a plurality of data practices and paradigms.

Evaluating partitioning and bucketing strategies for Hive-based Big Data Warehousing systems

Hive has long been one of the industry-leading systems for Data Warehousing in Big Data contexts, mainly organizing data into databases, tables, partitions and buckets, stored on top of an unstructured distributed file system like HDFS. Some studies were conducted for understanding the ways of optimizing the performance of several storage systems for Big Data Warehousing. However, few of them explore the impact of data organization strategies on query performance, when using Hive as the storage technology for implementing Big Data Warehousing systems. Therefore, this paper evaluates the impact of data partitioning and bucketing in Hive-based systems, testing different data organization strategies and verifying the efficiency of those strategies in query performance. The obtained results demonstrate the advantages of implementing Big Data Warehouses based on denormalized models and the potential benefit of using adequate partitioning strategies. Defining the partitions aligned with the attributes that are frequently used in the conditions/filters of the queries can significantly increase the efficiency of the system in terms of response time. In the more intensive workload benchmarked in this paper, overall decreases of about 40% in processing time were verified. The same is not verified with the use of bucketing strategies, which shows potential benefits in very specific scenarios, suggesting a more restricted use of this functionality, namely in the context of bucketing two tables by the join attribute of these tables.

Identifying potential biomarkers in LC‐MS data

AbstractIn this paper an application of the uninformative variable elimination–partial least squares (UVE‐PLS) method extended by the Monte Carlo approach for selection of possible biomarkers from the liquid chromatography coupled with mass spectrometry (LC‐MS) data is reported. The main challenge consists not in the chemometrics analysis of LC‐MS data, but in the data organization. However, as demonstrated in our study, the selected variables are similar regardless of the data organization strategy. The best results are obtained for the standard normal variate (SNV) transformed data. Copyright © 2007 John Wiley & Sons, Ltd.