Large Data Repositories Research Articles

Large scale ontology tailoring and simulation in the Semantic Grid Environment

Ontologies contain semantic information often annotated with metadata. They can get quite complex and very large, causing difficulties in using these ontologies on a daily basis. A large cluster computer is required to tailor these base ontologies into smaller and/or extracted sub-ontologies, but this is expensive and not a viable solution for most users. The Semantic Grid has the ability to process large semantic data repositories as fast as or faster than some of the largest cluster computers at a fraction of the cost. This paper explores the viability of using the Semantic Grid to tailor large ontologies, efficiently and cheaply.

TMA-Combiner, a simple software tool to permit analysis of replicate cores on tissue microarrays

We have previously published a suite of software tools that facilitates the reformulation of tissue microarray (TMA) data so that it may be analyzed using techniques originally devised for analysis of cDNA microarray data. However, current microarray data often feature multiple scores for a given tissue sample and antibody combination. Furthermore, an efficient and systematic method for combining scores that takes into account the differing staining properties of tissue epitopes has not been described. We thus present the TMA-Combiner, a new Microsoft Excel-based macro that permits analysis of data for which tissues may have two or more scores per antibody, and permits combination of data from multiple different tissue microarrays. It accomplishes this by rendering one score per tissue per antibody from two or more scores, using one of multiple user-selectable combination rules developed to account for the differing staining properties of tissue epitopes. This greatly facilitates analysis of tissue microarrays, particularly for users with large repositories of data, and may facilitate discovery of biological trends and help refine diagnostic accuracy of tissue markers in clinical samples.

Association rule hiding

Large repositories of data contain sensitive information that must be protected against unauthorized access. The protection of the confidentiality of this information has been a long-term goal for the database security research community and for the government statistical agencies. Recent advances in data mining and machine learning algorithms have increased the disclosure risks that one may encounter when releasing data to outside parties. A key problem, and still not sufficiently investigated, is the need to balance the confidentiality of the disclosed data with the legitimate needs of the data users. Every disclosure limitation method affects, in some way, and modifies true data values and relationships. We investigate confidentiality issues of a broad category of rules, the association rules. In particular, we present three strategies and five algorithms for hiding a group of association rules, which is characterized as sensitive. One rule is characterized as sensitive if its disclosure risk is above a certain privacy threshold. Sometimes, sensitive rules should not be disclosed to the public since, among other things, they may be used for inferring sensitive data, or they may provide business competitors with an advantage. We also perform an evaluation study of the hiding algorithms in order to analyze their time complexity and the impact that they have in the original database.

The utility of multivariate design in PLS modeling

AbstractWe discuss the use of multivariate design to ensure representativity and balance of the training set data for PLS multivariate modeling. Three application areas are used to illustrate the discussion, namely multivariate calibration in process analytical chemistry, quantitative structure activity relationships (QSAR) in medicinal and pharmaceutical chemistry, and data mining. In both QSAR and data mining, the multivariate design is also useful for the balanced sampling of data from a large, complex, and unbalanced data repository. Copyright © 2004 John Wiley & Sons, Ltd.

Deep scientific computing requires deep data

Increasingly, scientific advances require the fusion of large amounts of complex data with extraordinary amounts of computational power. The problems of deep science demand deep computing and deep storage resources. In addition to teraflop-range computing engines with their own local storage, facilities must provide large data repositories of the order of 10–100 petabytes, and networking to allow the movement of multi-terabyte files in a timely and secure manner. This paper examines such problems and identifies associated challenges. The paper discusses some of the storage systems and data management methods that are needed for computing facilities to address the challenges and describes some ongoing improvements.

Improving Rule Set Based Software Quality Prediction: A Genetic Algorithm-based Approach.

The object-oriented (OO) paradigm has now reached maturity. OO software products are becoming more complex which makes their evolution effort and time consuming. In this respect, it has become important to develop tools that allow assessing the stability of OO software (i.e., the ease with which a software item can evolve while preserving its design). In general, predicting the quality of OO software is a complex task. Although many predictive models are proposed in the literature, we remain far from having reliable tools that can be applied to real industrial systems. The main obstacle for building reliable predictive tools for real industrial systems is the lackof representative samples. Unlike other domains where such samples can be drawn from available large repositories of data, in OO software the lack of such repositories makes it hard to generalize, to validate and to reuse existing models. Since universal models do not exist, selecting an appropriate quality model is a difficult, non-trivial decision for a company. In this paper, we propose two general approaches to solve this problem. They consist of combining/adapting a set of existing models. The process is driven by the context of the target company. These approaches are applied to OO software stability prediction.

Handheld computers in radiology.

The next phase of the digital revolution in medicine is taking place through the dissemination of powerful handheld computers. Handheld computers, or personal digital assistants (PDAs), are no longer considered either a curiosity or a toy. The current handheld computer has many features (Internet access, simple e-mail client software, spreadsheet and database programs, word processing, and digital media) that make it an ideal tool for healthcare providers. Improvements in handwriting recognition, display characteristics, and wireless networking capabilities provide a platform for real-time review of both large static and dynamic repositories of patient data. Although earlier PDA models lacked the ability to display medical images appropriately, current PDAs boast display characteristics that approach low-resolution computer monitors. Although the handheld computer is not yet a reliable option for soft-copy reading, it offers many features that can improve work flow and efficiency for the radiologist. These features include improved personal information management, decision support via access to educational materials, and remote access to radiology-related information systems.

IMoViS: A System for Mobile Visualization of Intrusion Detection Data

Intrusion detection applications often produce large amount of data. The visualization of this information is a key task in order to allow the user to effectively detect attacks and intrusions. Information visualization is an important sub-discipline within the field of scientific visualization and focuses on visual mechanisms designed to communicate clearly to the user the structure of information and to facilitate the access to large data repositories. Information visualization helps people in dealing with all this information by taking advantage of the visual perception capabilities of the human being. By presenting the information in a more graphically oriented fashion, it is possible for the human brain to take advantage of its perceptual system in the initial information acquiring, rather than immediately relying entirely on the cognitive system. Some of the most important papers in the field are collected in a book. Information visualization algorithms require merging of data visualization methods, computer graphics solutions, and graphical interface design.

On learning to predict Web traffic

The ease of collecting data about customers through the Internet has facilitated the process of developing large repositories of data. These data can and do contain patterns that are useful for the decision maker. Knowledge discovery and data mining methods have been widely used to extract these patterns. It is acknowledged that about 80% of the resources in a majority of data mining applications are spent on cleaning and preprocessing the data. However, there have been relatively few studies on preprocessing data used as input in these data mining systems. In this study, we present a feature selection method based on the Hausdorff distance measure, and evaluate its effectiveness in preprocessing input data for inducing decision trees. Message traffic data from a Web site are used to illustrate performance of the proposed method.

A Web‐Based Query Interface for Health‐Care Data Warehouses

This paper presents a new type of computer interface to large population health‐care data repositories via the Internet. It does not require programming knowledge but is flexible enough to handle the ad‐hoc queries common in research settings. The interface allows for the automated creation of SAS programs by the end users using a GUI‐based questionnaire. The interface is the front end of a three‐tier client/server system for on‐line information retrieval for large health‐care data warehouses.

Object-based representations of spatial images

Object based representations of image data enable new content-related functionalities while facilitating management of large image databases. Developing such representations for multi-date and multi-spectral images is one of the objectives of the second phase of the Alexandria Digital Library (ADL) project at UCSB. Image segmentation and image registration are two of the main issues that are to be addressed in creating localized image representations. We present in this paper some of the recent and current work by the ADL's image processing group on robust image segmentation, registration, and the use of image texture for content representation. Built upon these technologies are techniques for managing large repositories of data. A texture thesaurus assists in creating a semantic classification of image regions. An object-based representation is proposed to facilitate data storage, retrieval, analysis, and navigation.

Internet 2 – WWW: Where, When and Why?

Introduction General During the past decade, federal R&D agencies, the academic community and private companies have worked together to develop many of the Internet's technologies. This partnership has created a multi-billion dollar industry. Internet 2 (I2) is comprised of over 181 US research universities. These universities, working with federal research agencies and leaders of the information technology industry, teamed up to create the next stage of Internet development. The Internet 2 project is a collaboration for the development of a Next Generation Internet (NGI) for research and education that will include enhanced network services focusing on multimedia applications. The work is developmental and pre-competitive in nature (Bakos 1998). It was agreed upon that charter membership in the project remains open for a limited time to additional institutions that are in a position to commit the resources necessary for full participation. Today more than 250 institutions have committed themselves to participate in Internet 2. Objectives The objectives of the Internet 2 project were set as follows: * To maintain a common bearer service to support new and existing applications * To move from best effort packet delivery to a differentiated communications service * To provide the capabilities of tailoring network service characteristics to meet specific applications requirements * To achieve an advanced communications infrastructure for the Research and Education communities The need for a wide range of possibilities for data transmission is obvious and includes mainly current and future applications from the domains of Medicine, Education and Commerce. The list below emphasizes some of the current and future needs that encouraged the professionals to establish and develop the project (Brown 1999). * The idea that transmission rates, error rates, and other characteristics can, to some extent, be guaranteed in advance is known as Quality of Service or QoS. High Quality of Service and an efficient one-to-many broadband data transport including support for multimedia and shared information processing will be required for the broad use of distance learning (a.k.a. eLearning). * Leaders of the international research community are in need of high capacity infrastructure and selectable quality of service so that they can make effective use of national laboratories, computational facilities and large data repositories. * Medical researchers require support for remote consultation and diagnoses over highly reliable and predictable communications lines. * Physicists, and especially those who deal with vast astronomical or geophysical datasets, have similar needs. * As commercial transaction data reaches research focus, financial and economic analysts will need real-time access to masses of data. A major impediment to the realization of these applications is the lack of advanced communications services in the current commodity Internet--there is no test case, as yet, by which to prove their effectiveness, much less to implement them on a wide scale. The Novelty A number of technical and practical considerations underlie the overall architecture of the Internet 2 infrastructure. One of these is the need to minimize the overall costs of participating campuses. This is achieved by providing access to both the commodity Internet and advanced services through the same high-capacity local connection circuit. Another catered need is the accommodation of other campus programs and projects. This is done by means of a flexible regional interconnection architecture. For example, a metropolitan area network service might offer high capacity Internet services to students and faculty members. The new, key element in this architecture is the Gigapop (gigabit capacity point of presence) that presents a high capacity, state-of-the-art interconnection point where I2 participants may exchange advanced services traffic with other I2 participants. …

Data Preparation for Mining World Wide Web Browsing Patterns

The World Wide Web (WWW) continues to grow at an astounding rate in both the sheer volume of traffic and the size and complexity of Web sites. The complexity of tasks such as Web site design, Web server design, and of simply navigating through a Web site have increased along with this growth. An important input to these design tasks is the analysis of how a Web site is being used. Usage analysis includes straightforward statistics, such as page access frequency, as well as more sophisticated forms of analysis, such as finding the common traversal paths through a Web site. Web Usage Mining is the application of data mining techniques to usage logs of large Web data repositories in order to produce results that can be used in the design tasks mentioned above. However, there are several preprocessing tasks that must be performed prior to applying data mining algorithms to the data collected from server logs. This paper presents several data preparation techniques in order to identify unique users and user sessions. Also, a method to divide user sessions into semantically meaningful transactions is defined and successfully tested against two other methods. Transactions identified by the proposed methods are used to discover association rules from real world data using the WEBMINER system [15].

A methodology for risk-based inspection of pressurized systems

This paper describes a methodology for risk-based inspection and focuses on the planning of inspection activities. Brunei Shell Petroleum (BSP) operates a large number of ageing production and evacuation facilities in the South China Sea, and is facing the challenge to identify the optimal inspection effort in a period of time when more production is required and operating costs have to be minimized. Since 1989 major investments (annual costs about 15 million US$) were made to perform base line inspections which resulted in corrective work and a large repository of equipment integrity data. These data require analysis and updating, as defined in this methodology. The challenge is for BSP to implement risk-based inspection of static equipment, which so far has been very successful. The methodology is based on the determination of risk by evaluating the consequences and the likelihood of equipment failure. Likelihood of failure is assessed, by means of extrapolation, at the future planned maintenance campaign to identify the necessary corrective work. The extrapolation is based on historical inspection and maintenance data as well as on anticipated future operating conditions. Such an analysis requires in-depth knowledge of the equipment regarding functions, operating conditions, materials, imposed potential failure modes and causes, and their controlling parameters (integrity parameters). The novelty of the method is that equipment integrity is expressed in terms of integrity parameters, which can be measured and trended. Consequently, based on defined limits for the integrity parameters (i.e. anomaly classifications), actions for managing risk to an acceptable low level can be identified. The transparent linkage between monitoring activities (e.g. inspection) and integrity parameters offers an effective framework to manage and to demonstrate integrity against clear quantifiable acceptance limits. By pursuing the above methodology for all equipment, a list of inspection activities, prioritized on equipment condition and the risk of failure, can be derived. Within BSP this has resulted in a demonstrated increased efficiency of the inspection effort, a focused strategy for repairs executed on a campaign basis, and an improved quantification of BSP's facilities integrity. The approach has been proven in the company to be practical and it can be implemented relatively quickly.

Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem

The problem of merging multiple databases of information about common entities is frequently encountered in KDD and decision support applications in large commercial and government organizations. The problem we study is often called the Merge/Purge problem and is difficult to solve both in scale and accuracy. Large repositories of data typically have numerous duplicate information entries about the same entities that are difficult to cull together without an intelligent ’’equational theory‘‘ that identifies equivalent items by a complex, domain-dependent matching process. We have developed a system for accomplishing this Data Cleansing task and demonstrate its use for cleansing lists of names of potential customers in a direct marketing-type application. Our results for statistically generated data are shown to be accurate and effective when processing the data multiple times using different keys for sorting on each successive pass. Combing results of individual passes using transitive closure over the independent results, produces far more accurate results at lower cost. The system provides a rule programming module that is easy to program and quite good at finding duplicates especially in an environment with massive amounts of data. This paper details improvements in our system, and reports on the successful implementation for a real-world database that conclusively validates our results previously achieved for statistically generated data.

Systems for KDD: From concepts to practice

The considerable interest in knowledge discovery in databases (KDD) has led to several techniques and tools for the automated extraction of useful information from large data repositories. In order to use these developments in practical settings, there is need to consider the computing systems that would support the complete KDD process. In this regard, we identify and discuss important computing systems issues, and we compare and contrast some available research and commercial efforts. We suggest several enhancements to the underlying database systems that may significantly benefit the KDD process. We indicate why it is important that tools to handle different aspects of the KDD effort need to be integrated. Finally, we briefly describe our experience in implementing a prototype KDD system for a large corporate environment.

Multidimensional ranking for data in digital spatial libraries

Digital spatial libraries currently under development are generating large repositories of data which will continue to grow. As these repositories grow, the situation will inevitably arise in which a digital library user may be confronted with several hundred spatial data sets in response to a particular query. The question then arises as to how the results from this search can be most easily assimilated by the user. Text based materials have benefited from substantial research and experience on ranking of search results. Ranking of spatial data sets has not received the same attention since there has been little motivation for such activity until recently. In this paper we propose a multidimensional ranking scheme based on the three dimensions of space, time, and theme. The multidimensional rank is presented graphically to inform users about how well data sets from a digital spatial library meet their spatial, temporal, and thematic targets.

Information visualization

Information visualization , an increasingly important subdiscipline within the field of Human Computer Interaction (HCI) [13], focuses on visual mechanisms designed to communicate clearly to the user the structure of information and improve on the cost of access to large data repositories. In printed form, information visualization has included the display of numerical data (e.g., bar charts, plot charts, pie charts), combinatorial relations (e.g., drawings of graphs), and geographic data (e.g., encoded maps) [1, 9, 16]. In addition to these “static” displays, computer-based systems, such as the Information Visualizer [2] and Dynamic Queries [15] have coupled powerful visualization techniques (e.g., constraints, 3D, animation) with near real-time interactivity, i.e., the ability of the system to respond quickly to the user's direct manipulation commands. Another important aspect of computer-based information systems concerns the dual communication with user and machine, which motivates the concept of visual formalism that has been introduced by Harel: “The intricate nature of a variety of computer-related systems and situations can, and in our opinion should, be represented via visual formalisms; visual because they are to be generated, comprehended, and communicated by humans; and formal, because they are to be manipulated, maintained, and analyzed by computers” [12]. While historically HCI and database research were kept separate, the interests of both research communities have been converging, mainly in what concerns the topic of information visualization. In the database community, the focus on information visualization started with research in visual query languages, where the visualization of schema and/or database instances is common (for a survey, see [4]). Recently, a new generation of database systems is emerging, which tightly combine querying capabilities with visualization techniques and are information visualization systems in their own right [3, 8, 11]. Database applications that access large data repositories, such as data mining and data warehousing, and the enormous quantity of information sources on the WWW available to users with diverse capabilities also provide HCI researchers with new opportunities for information visualization (see, for instance, the ACM report on Strategic Directions in HCI [13], the reports of the “FADIVA” Working Group [10], and the work presented in [14]). The objective of this special issue is two-fold: to compile some of the most recent research on information visualization from both communities, and to make it available to a large readership whose main interests lie in the management of data. The eight papers in this issue cover fundamental topics in information visualization, including tailorable multi-visualizations (i.e., the ability of the system to provide the user with alternative visualizations, depending on their suitability to different data, tasks, and users' preferences); near real-time interactivity when dealing with very large data sizes; effective display and color usage, and multidimensionality. A short overview of each paper follows.