Large Numerical Data Sets Research Articles

Principal component analysis based combustion model in the context of a lifted methane/air flame: Sensitivity to the manifold parameters and subgrid closure

The present work advances the PC-transport approach in the context of Large Eddy Simulation (LES) of turbulent combustion. Accurate modeling of combustion systems requires large kinetic mechanisms. However, realistic high-fidelity simulations of turbulent reacting flows still represent a big challenge on the current computational tools. Therefore, a parameterization of the thermo-chemical state-space using a reduced number of variables is needed. To this end, the potential offered by Principal Component Analysis (PCA) in identifying low-dimensional manifolds is very appealing. The present paper extends the PC-transport approach, coupled with Gaussian Process Regression (GPR), to a lifted methane/air flame in LES. Previous investigations by the authors showed the great potential of the PC-GPR model in the context of Sandia flames. This study investigated some key features of the model: the sensitivity to the training data set and the scaling methods . To this end, two different canonical reactors were used: unsteady counter-flow laminar flames (CFLF) and unsteady perfectly stirred reactor (PSR). Moreover, the authors proposes an approach to address the issue of data density inherent to large numerical data sets, by means of a kernel density weighting of the data set before applying PCA. Finally, a subgrid scale (SGS) closure model was coupled to the PC-transport approach to treat complex turbulence/chemistry interactions.

ReVisE: Remote visualization environment for large numerical simulation datasets.

The paper presents a new open-source visualization system, named ReVisE, aimed to provide interactive visualization of large datasets, which are results of complex numerical simulations. These datasets are hosted on a remote server or a supercomputer. The design of the system is briefly described. Dataset representation, proposed for interactive visualization and implemented in the system, is discussed. The effectiveness of our approach is confirmed by results of performance measurements on test and real-life large datasets. A comparison with other visualization systems is presented. Future plans of system development are outlined.

Online Reviews Analysis for Customer Segmentation through Dimensionality Reduction and Deep Learning Techniques

Digital social media has played a key role in tourism and hospitality industry. The use of machine and deep learning has been effective in market segmentation and customers' preference prediction through social big data analysis. This paper develops a new method to analyze large set of open data in social networking sites for travellers segmentation and predict tourists' choice preferences using dimensionality reduction and deep learning techniques. Deep belief network was used for predicting the travellers’ choice preferences from their past ratings and online reviews. Self-organizing map was also used for clustering the travellers’ online ratings and reviews. The feature extraction is performed using latent Dirichlet allocation as an unsupervised learning technique. To improve the effectiveness of learning, a dimensionality reduction technique, higher-order singular value decomposition, is performed on the clusters for the prediction of missing values and traveller–traveller similarity calculation. The proposed method was evaluated on travellers’ online reviews and ratings which were crawled from TripAdvisor. The results showed the robustness of the proposed method in analysing the large text-based reviews and numerical datasets in tourism context.

Computer vision, human senses, and language of art

What is the most important reason for using Computer Vision methods in humanities research? In this article, I argue that the use of numerical representation and data analysis methods offers a new language for describing cultural artifacts, experiences and dynamics. The human languages such as English or Russian that developed rather recently in human evolution are not good at capturing analog properties of human sensorial and cultural experiences. These limitations become particularly worrying if we want to compare thousands, millions or billions of artifacts—i.e. to study contemporary media and cultures at their new twenty-first century scale. When we instead use numerical measurements of image properties standard in Computer Vision, we can better capture details of a single artifact as well as visual differences between a number of artifacts–even if they are very small. The examples of visual dimensions that numbers can capture better then languages include color, shape, texture, contours, composition, and visual characteristics of represented faces, bodies and objects. The methods of finding structures and relationships in large numerical datasets developed in statistics and machine learning allow us to extend this analysis to very big datasets of cultural objects. Equally importantly, numerical image features used in Computer Vision also give us a new language to represent gradual and continuous temporal changes—something which natural languages are also bad at. This applies to both single artworks such as a film or a dance piece (describing movement and rhythm) and also to changes in visual characteristics in millions of artifacts over decades or centuries.

Parallel spectral element method for guided wave based structural health monitoring

Parallel implementation of the spectral element method is developed in which flat shell spectral elements are utilized for spatial domain representation. The implementation is realised by using Matlab Parallel Computing Toolbox and optimized for Graphics Processing Unit (GPU) computation. In this way, considerable computation speed-up can be achieved in comparison to computation on conventional processors. The implementation includes an interpolation of wavefield on a uniform grid. The method was tested on experimental data set available on the Open Guided Waves platform. The qualitative comparison was performed on full wavefield data, whereas quantitative comparison was made directly on signals of propagating Lamb waves registered by piezoelectric transducers. In both cases, good agreement between numerical and experimental results was achieved. The proposed method is particularly useful for structural health monitoring algorithms in which signal parameters are required such as wave velocity dependence on the angle of propagation. Moreover, it enables model–assisted damage size estimation in which the damage influence curve is estimated based on large numerical data sets and sparse experimental data. Due to relatively short computation times, large data sets can be generated and used for machine learning or other soft computing methods opening up new possibilities in health monitoring of metallic and composite structures.

Biospark: scalable analysis of large numerical datasets from biological simulations and experiments using Hadoop and Spark.

Source code is licensed under the Apache 2.0 open source license and is available at the project website: https://www.assembla.com/spaces/roberts-lab-public/wiki/Biospark CONTACT: eroberts@jhu.eduSupplementary information: Supplementary data are available at Bioinformatics online.

Fast Dimension-based Partitioning and Merging clustering algorithm

Clustering multi-dense large scale high dimensional numeric datasets is a challenging task duo to high time complexity of most clustering algorithms. Nowadays, data collection tools produce a large amount of data. So, fast algorithms are vital requirement for clustering such data. In this paper, a fast clustering algorithm, called Dimension-based Partitioning and Merging (DPM), is proposed. In DPM, first, data is partitioned into small dense volumes during the successive processing of dataset dimensions. Then, noise is filtered out using dimensional densities of the generated partitions. Finally, merging process is invoked to construct clusters based on partition boundary data samples. DPM algorithm automatically detects the number of data clusters based on three insensitive tuning parameters which decrease the burden of its usage. Performance evaluation of the proposed algorithm using different datasets shows its fastness and accuracy compared to other clustering competitors.

Fast data-oriented microaggregation algorithm for large numerical datasets

Microaggregation is a successful mechanism to solve the tension between respondent privacy and data quality in the context of Statistical Disclosure Control. Microaggregation, for numerical datasets, is defined as a clustering problem with the constraint of having at least k records in each group, such that the sum of the within-group squared error (SSE) is minimized. Unfortunately, the data publisher has to execute an algorithm iteratively for different values of k to investigate a good trade-off between privacy and utility. Multiple execution of an algorithm on large numerical datasets is resource wasting, since most of the computations are repetitive. In this paper, we propose a Fast Data-oriented Microaggregation algorithm (FDM) that efficiently anonymizes large multivariate numerical datasets for multiple successive values of k. Experimental results on real world datasets demonstrate the superiority of the method in terms of both the data quality and time complexity. Moreover, the method usually achieves a better trade-off between disclosure risk and information loss of the protected dataset in comparison with previous techniques.

Numerical Investigation of Heat Transfer in Rectangular Microchannels Under H2 Boundary Condition During Developing and Fully Developed Laminar Flow

Study of fluid flow characteristics at microscale is gaining importance with shrinking device sizes. Better understanding of fluid flow and heat transfer in microchannels will have important implications in electronic chip cooling, heat exchangers, MEMS, and microfluidic devices. Due to short lengths employed in microchannels, entrance header effects can be significant and need to be investigated. In this work, three dimensional model of microchannels, with aspect ratios (α = a/b) ranging from 0.1 to 10, are numerically simulated using CFD software tool fluent. Heat transfer effects in the entrance region of microchannel are presented by plotting average Nusselt number as a function of nondimensional axial length x*. The numerical simulations with both circumferential and axial uniform heat flux (H2) boundary conditions are validated for existing data set for four wall heat flux case. Large numerical data sets are generated in this work for rectangular cross-sectional microchannels with heating on three walls, two opposing walls, one wall, and two adjacent walls under H2 boundary condition. This information can provide better understanding and insight into the transport processes in the microchannels. Although the results are seen as relevant in microscale applications, they are applicable to any sized channels. Based on the numerical results obtained for the whole range, generalized correlations for Nusselt numbers as a function of channel aspect ratio are presented for all the cases. The predicted correlations for Nusselt numbers can be very useful resource for the design and optimization of microchannel heat sinks and other microfluidic devices.

Grand Challenges in Bioinformatics and Computational Biology

Grand Challenges in Bioinformatics and Computational Biology

An Efficient Data Format for Mass Spectrometry-Based Proteomics

The diverse range of mass spectrometry (MS) instrumentation along with corresponding proprietary and nonproprietary data formats has generated a proteomics community driven call for a standardized format to facilitate management, processing, storing, visualization, and exchange of both experimental and processed data. To date, significant efforts have been extended towards standardizing XML-based formats for mass spectrometry data representation, despite the recognized inefficiencies associated with storing large numeric datasets in XML. The proteomics community has periodically entertained alternate strategies for data exchange, e.g., using a common application programming interface or a database-derived format. However, these efforts have yet to gain significant attention, mostly because they have not demonstrated significant performance benefits over existing standards, but also due to issues such as extensibility to multidimensional separation systems, robustness of operation, and incomplete or mismatched vocabulary. Here, we describe a format based on standard database principles that offers multiple benefits over existing formats in terms of storage size, ease of processing, data retrieval times, and extensibility to accommodate multidimensional separation systems.

MzAPI: a new strategy for efficiently sharing mass spectrometry data

To The Editor: The call for data access standards in mass spectrometry-based proteomics has led to proposals focused on the extraction of native data to XML-based formats. 1,2 While self-describing and human-readable formats represent laudable goals, particularly for archival purposes, they are not well suited to large numeric datasets. Consequently, while metadata in mzML2 remain human-readable, the vast majority of the file is devoted to a hexadecimal representation of the mass spectra. Moreover, the transition from mzXML to a true XML format (mzML2) eliminates embedded indexing schemes; consequently, extracted files are compromised in both content and access efficiency.1,3 Based on similarities in data structure and access patterns, we suggest that fields such as astronomy are better models for proteomics data analysis (Figure 1). These fields also rely on common formats, but typically utilize binary standards such as HDF54 or netCDF5. By contrast, the commercial nature of mass spectrometry has led to the evolution of proprietary binary file formats. In light of these observations, we propose that a common and redistributable application programming interface (API) represents a more viable approach to data access in mass spectrometry. In effect, we propose to shift the burden of standards compliance to the manufacturers’ existing data access libraries. Figure 1 Array Scanners, Telescopes, and Mass Spectrometers: XML, HDF, or API? While our proposal for abstraction through a common API represents a clear departure from current attempts to define a universal file format, it is by no means unique within the broader scientific community. For example, standardized APIs have enabled the development of portable applications in such diverse areas as computer graphics (OpenGL7) and parallel processing (Message Passing Interface, MPI8). More importantly, we believe that a common API will benefit all stakeholders. For example, free redistribution of compiled, vendor-supplied dynamically linked libraries (DLLs) will protect the proprietary layout of native files and provide users with direct and flexible access to data system- and instrument-specific functionality which are typically ignored by lowest common denominator export routines. In addition, we note that mzAPI naturally supports the FDA’s 21 CFR part 11 regulatory requirements for electronic records9 Finally, a community-driven API standard will facilitate manufacturer support of UNIX, in addition to Windows, by highlighting the subset of procedures, from each data system (Xcalibur™, Analyst™, etc.), which need to be ported. Motivated originally by our desire to provide a more intimate environment for flexible and in-depth exploration of mass spectrometry data, particularly from low-throughput experiments, we developed a preliminary common API (mzAPI) – consisting of just five procedures (http://blais.dfci.harvard.edu/mzAPI). To maximize accessibility we exposed mzAPI in the form of a Python library within a flexible, mass-informatics desktop framework called multiplierz (http://blais.dfci.harvard.edu/multiplierz). We are encouraged by results from this test harness, in particular how well mzAPI and our desktop environment support a variety of data analytic operations. Equally impressive is how quickly non-programmers can customize scripts for their specific tasks. Despite success to date in our own lab, we recognize that mzAPI will benefit from further refinement and stress testing. Accordingly, we are actively seeking input from the research community with respect to both concept and implementation of a comprehensive API-based standard for mass spectrometry data access and analysis.

Evaluation of SOVAT: An OLAP-GIS decision support system for community health assessment data analysis

BackgroundData analysis in community health assessment (CHA) involves the collection, integration, and analysis of large numerical and spatial data sets in order to identify health priorities. Geographic Information Systems (GIS) enable for management and analysis using spatial data, but have limitations in performing analysis of numerical data because of its traditional database architecture.On-Line Analytical Processing (OLAP) is a multidimensional datawarehouse designed to facilitate querying of large numerical data. Coupling the spatial capabilities of GIS with the numerical analysis of OLAP, might enhance CHA data analysis. OLAP-GIS systems have been developed by university researchers and corporations, yet their potential for CHA data analysis is not well understood. To evaluate the potential of an OLAP-GIS decision support system for CHA problem solving, we compared OLAP-GIS to the standard information technology (IT) currently used by many public health professionals.MethodsSOVAT, an OLAP-GIS decision support system developed at the University of Pittsburgh, was compared against current IT for data analysis for CHA. For this study, current IT was considered the combined use of SPSS and GIS ("SPSS-GIS"). Graduate students, researchers, and faculty in the health sciences at the University of Pittsburgh were recruited. Each round consisted of: an instructional video of the system being evaluated, two practice tasks, five assessment tasks, and one post-study questionnaire. Objective and subjective measurement included: task completion time, success in answering the tasks, and system satisfaction.ResultsThirteen individuals participated. Inferential statistics were analyzed using linear mixed model analysis. SOVAT was statistically significant (α = .01) from SPSS-GIS for satisfaction and time (p < .002). Descriptive results indicated that participants had greater success in answering the tasks when using SOVAT as compared to SPSS-GIS.ConclusionUsing SOVAT, tasks were completed more efficiently, with a higher rate of success, and with greater satisfaction, than the combined use of SPSS and GIS. The results from this study indicate a potential for OLAP-GIS decision support systems as a valuable tool for CHA data analysis.

Report on the First International Workshop on Database Preservation (PresDB'07)

The need to preserve scientific, scholarly and cultural data has long been recognized. These data sets are valuable and many of them are either impossible to reproduce (e.g. climate and demographic data) or can only be recovered at enormous costs (e.g. data from high energy physics experiments). While substantial investment has been made in archiving and preserving conventional forms of these objects, such as documents, images and numerical data in some file format, the need to preserve entire databases has only recently emerged. Databases differ from fixed digital objects studied in the past, in that they change over time, they have internal structure, and they include schemas and integrity constraints, which are basic for the current and future interpretation of the data. Increasingly, database technology is being used in the storage of large numerical scientific data sets.

The Field Representation Language

The complexity of quantitative biomedical models, and the rate at which they are published, is increasing to a point where managing the information has become all but impossible without automation. International efforts are underway to standardise representation languages for a number of mathematical entities that represent a wide variety of physiological systems. This paper presents the Field Representation Language (FRL), a portable representation of values that change over space and/or time. FRL is an extensible mark-up language (XML) derivative with support for large numeric data sets in Hierarchical Data Format version 5 (HDF5). Components of FRL can be reused through unified resource identifiers (URI) that point to external resources such as custom basis functions, boundary geometries and numerical data. To demonstrate the use of FRL as an interchange we present three models that study hyperthermia cancer treatment: a fractal model of liver tumour microvasculature; a probabilistic model simulating the deposition of magnetic microspheres throughout it; and a finite element model of hyperthermic treatment. The microsphere distribution field was used to compute the heat generation rate field around the tumour. We used FRL to convey results from the microsphere simulation to the treatment model. FRL facilitated the conversion of the coordinate systems and approximated the integral over regions of the microsphere deposition field.

Algebraic manipulation of scientific datasets

We investigate algebraic processing strategies for large numeric datasets equipped with a possibly irregular grid structure. Such datasets arise, for example, in computational simulations, observation networks, medical imaging, and 2-D and 3-D rendering. Existing approaches for manipulating these datasets are incomplete: The performance of SQL queries for manipulating large numeric datasets is not competitive with specialized tools. Database extensions for processing multidimensional discrete data can only model regular, rectilinear grids. Visualization software libraries are designed to process gridded datasets efficiently, but no algebra has been developed to simplify their use and afford optimization. Further, these libraries are data dependent - physical changes to data representation or organization break user programs. In this paper, we present an algebra of grid-fields for manipulating both regular and irregular gridded datasets, algebraic optimization techniques, and an implementation backed by experimental results. We compare our techniques to those of spatial databases and visualization software libraries, using real examples from an Environmental Observation and Forecasting System. We find that our approach can express optimized plans inaccessible to other techniques, resulting in improved performance with reduced programming effort.

K-MEANS-TYPE ALGORITHMS ON DISTRIBUTED MEMORY COMPUTER

Partitioning a set of objects into homogeneous clusters is a fundamental operation in data mining. The k-means-type algorithm is best suited for implementing this operation because of its efficiency in clustering large numerical and categorical data sets. An efficient parallel k-means-type algorithm for clustering data sets on a distributed share-nothing parallel system is considered. It has a simple communication scheme which performs only one round of information exchange in every iteration. We show that the speedup of our algorithm is asymptotically linear when the number of objects is sufficiently large. We implement the parallel k-means-type algorithm on an IBM SP2 parallel machine. The performance studies show that the algorithm has nice parallelism in experiments.

JGR‐Solid Earth moves toward electronic communication

The AGU Council and the Publications Committee have given a high priority to fostering improved AGU communications by electronic transmission. As a result, the Journal of Geophysical Research‐Solid Earth has taken several initial steps to implement electronic communication. These include electronic data supplements to journal articles. For large numeric data sets, the author will be responsible for depositing the data with an appropriate, approved data center. In the case of small data sets that will fit onto a single 3.5 disk and are inappropriate for a data center, AGU will serve as a storage/distribution facility as it currently does for microfiche. In addition, templates of editorial review forms for referees and associate editors are now available on Kosmos through Internet. Use of these electronic forms will expedite the manuscript review process. The forms can be used with conventional word processing programs and the reviews can be transmitted via e‐mail.

A numerical data base for VAX and personal computers for the storage, reconstruction and display of global thermospheric and ionospheric models

Theoretical three-dimensional and time-dependent atmospheric models (also called General Circulation Models) describe, in a forward simulation fashion, the response of the atmosphere to complex external driving forces. Their advent has created a need to disseminate the large volume of data presented by such models to potential users in a convenient and abbreviated form. Taking as an example the theoretical thermospheric model developed at University College London, England, it is inconvenient to transmit the data by print-out, magnetic tape, disk media, electronic mail or networks. The number of individual parameters computed in a data set corresponding to a “steady-state” 24 h simulation, contains 3.8 × 10 6 real numbers, describing 15 pressure levels, 91 steps in latitude, 20 steps in longitude, 20 steps in Universal Time, and at least seven geophysical (atmospheric) parameters. These large numerical data sets exist for each of some 50 complete thermospheric simulations, covering a range of solar, geomagnetic and seasonal conditions. A process of high energy Fourier filtering has been used to reduce the data volume by a factor of up to 150, a data set size which is convenient for storage and electronic transmission purposes. A four-dimensional array, i.e. latitude vs longitude vs height vs time, can be quickly and conveniently reconstructed with high fidelity, for each individual atmospheric parameter using a VAX or PC computer. This provides a convenient and rapid means of disseminating verified data from the global simulations to end users.