Disk Storage Space Research Articles

Concentration Field Estimation on Effluent Mixing and Transport With a Parameter‐Based Field Reconstruction Convolutional Neural Network and Random Forests

AbstractEstimating the concentration field of effluent mixing and transport is a challenging task because of the complex mechanisms and three‐dimensional (3D) variability, but it is of great importance for water quality assessment and water resources management. Sophisticated numerical models based on the 3D computational fluid dynamics (CFD) technique can provide accurate predictions, but they are computationally expensive and require high‐level 3D CFD expertise, impeding their widespread usage. This work primarily develops a Parameter‐based Field Reconstruction convolutional neural network (PFR‐CNN) that can predict the complete concentration field of a target variable using the main influencing parameters, and explores the possibility of employing the proposed network to model the process of effluent mixing and transport. A validated numerical solver, TwoLiquidMixingFoam, was utilized to provide extensive data for multiple vertical buoyant effluents. A PFR‐CNN was developed to estimate the concentration fields based on the densimetric Froude number (F) and normalized port spacing (S/D). The network was developed using the training‐validating dataset from the numerical dataset, and further assessed using additional testing data. A random forest model was also developed, and the results showed that both methods can provide good performance, but the PFR‐CNN method was more robust and required less disk storage space. The results of our study demonstrated that the proposed PFR‐CNN approach is a promising tool for estimating effluent mixing and transport, and can be potentially leveraged to estimate other variables in the field of water resources with the governing parameters.

A Novel Big Data Storage Reduction Model for Drill Down Search

Multi-level searching is called Drill down search. Right now, no drill down search feature is available in the existing search engines like Google, Yahoo, Bing and Baidu. Drill down search is very much useful for the end user to find the exact search results among the huge paginated search results. Higher level of drill down search with category based search feature leads to get the most accurate search results but it increases the number and size of the file system. The purpose of this manuscript is to implement a big data storage reduction binary file system model for category based drill down search engine that offers fast multi-level filtering capability. The basic methodology of the proposed model stores the search engine data in the binary file system model. To verify the effectiveness of the proposed file system model, 5 million unique keyword data are stored into a binary file, thereby analysing the proposed file system with efficiency. Some experimental results are also provided based on real data that show our storage model speed and superiority. Experiments demonstrated that our file system expansion ratio is constant and it reduces the disk storage space up to 30% with conventional database/file system and it also increases the search performance for any levels of search. To discuss deeply, the paper starts with the short introduction of drill down search followed by the discussion of important technologies used to implement big data storage reduction system in detail.

NGS-Logistics: data infrastructure for efficient analysis of NGS sequence variants across multiple centers

Background Next-Generation Sequencing (NGS) is a key tool in genomics, in particular in research and diagnostics of human Mendelian, oligogenic, and complex disorders [1]. Multiple projects now aim at mapping the human genetic variation on a large scale, such as the 1,000 Genomes Project, the UK 100k Genome Project. Meanwhile with the dramatic decrease of the price and turnaround time, large amounts of human sequencing data have been generated over the past decade [2]. As of January 2014, about 2,555 sequencers were spread over 920 centers across the world [3]. As a result, about 100,000 human exome have been sequenced so far [4]. Crucially, the speed at which NGS data is produced greatly surpasses Moore’s law [5] and challenges our ability to conveniently store, exchange, and analyze this data. Data pre-processing is needed to extract reliable information from sequencing data and it can be divided into two major steps: primary analysis (image analysis and base calling) and secondary analysis. When looking for variation in the human genome, secondary analysis consists of aligning/mapping the reads against the reference genome and scanning the alignment for variation. Both raw data and mapped reads are large files occupying significant disk storage space. The collection of files resulting from the analysis of a single whole genome study can take up to 50Gb of disk space. This raises significant issues in terms of computing and data storage and transfer, with off-site data transfer currently being a key bottleneck. Moreover, the analysis of NGS data also raises the major challenge of how to reconcile federated analysis of personal genomic data and confidentiality of data to protect privacy. In many situations, the analysis of data from a single study alone will be much less powerful than if it can be correlated with other studies. In particular, when investigating a mutation of interest, it is extremely useful to obtain data about other patients or controls sharing similar mutations. However, personal genome data (whole genome, exome, transcriptome data, etc.) is sensitive personal data. Confidentiality of this data must be guaranteed at all times and only duly authorized researchers should access such personal data.

Monitoring ground surface deformation over the North China Plain using coherent ALOS PALSAR differential interferograms

North China Plain (Huabei Plain) is one of the most densely populated regions on earth. Due to excessive underground water extraction, the North China Plain has been experiencing severe ground deformation over the last three decades. Therefore, for the purpose of hazard mitigation, it is necessary to monitor the ground displacement occurred in this region. As an extension of the differential radar interferometry (DInSAR) technique, advanced DInSAR techniques involving multiple images have demonstrated the potential to effectively map ground displacement. Such techniques are able to measure the temporal evolution of ground deformation with millimetre-level accuracy by using a stack of differential interferograms. In this study, the ALOS PALSAR data acquired over the North China Plain, which cover an area of approximately 16,000 km $$^2$$ , were processed based on the concept of advanced DInSAR techniques. Because of the large size of the PALSAR images, a targeted processing strategy was designed. This strategy is able to reduce required disk storage space and I/O operations, leading to the improvement of the computational efficiency. The resulting mean deformation velocity map demonstrates that a large portion of the area covered by the data was affected by various degrees of ground deformation between January 2007 and April 2010. The ground deformation is mostly distributed in rural areas, while the downtown areas are generally stable.

A Case for Continuous Data Protection at Block Level in Disk Array Storages

This paper presents a study of data storages for continuous data protection (CDP). After analyzing the existing data protection technologies, we propose a new disk array architecture that provides Timely Recovery to Any Point-in-time, referred to as TRAP. TRAP stores not only the data stripe upon a write to the array but also the time-stamped Exclusive ors (xors) of successive writes to each data block. By leveraging the xor operations that are performed upon each block write in today's RAID4/5 controllers, TRAP does not incur noticeable performance overhead. More importantly, TRAP is able to recover data very quickly to any point-in-time upon data damage by tracing back the sequence and history of xors resulting from writes. What is interesting is that the TRAP architecture is very space efficient. We have implemented a prototype of the new TRAP architecture using software at the block level and carried out extensive performance measurements using TPC-C benchmarks running on Oracle and Postgres databases, TPC-W running on a MySQL database, and file system benchmarks running on Linux and Windows systems. Our experiments demonstrated that TRAP not only is able to recover data to any point-in-time very quickly upon a failure but also uses less storage space than traditional daily incremental backup/snapshot. Compared to the state-of-the-art CDP technologies, TRAP saves disk storage space by one to two orders of magnitude with a simple and a fast encoding algorithm. In addition, TRAP can provide two-way data recovery with the availability of only one reference image in contrast to the one-way recovery of snapshot and incremental backup technologies.

Extracting the cores of implicit communities

In this paper, we improve the trawling and point out some communities missed by trawling. We use the DBG (Dense Bipartite Graph) to identify a structure of a potential community instead of CBG (Complete Bipartite Graph). Based on DBG, we proposed a new method based on edge removal to extract cores from a web graph. Moreover, we improve the crawler to save only potential pages as fans of a core and save a lot of disk storage space. To evaluate the set of cores whether or not belong to a community, the statistics of term frequency is used. In the paper, the dataset of experiment were crawled under domain “.cn”. The result show that the our algorithm works properly and some new cores can be found by our method.

TRAP-Array

RAID architectures have been used for more than two decades to recover data upon disk failures. Disk failure is just one of the many causes of damaged data. Data can be damaged by virus attacks, user errors, defective software/firmware, hardware faults, and site failures. The risk of these types of data damage is far greater than disk failure with today's mature disk technology and networked information services. It has therefore become increasingly important for today's disk array to be able to recover data to any point in time when such a failure occurs. This paper presents a new disk array architecture that provides Timely Recovery to Any Point-in-time, referred to as TRAP-Array. TRAP-Array stores not only the data stripe upon a write to the array, but also the time-stamped Exclusive-ORs of successive writes to each data block. By leveraging the Exclusive-OR operations that are performed upon each block write in today's RAID4/5 controllers, TRAP does not incur noticeable performance overhead. More importantly, TRAP is able to recover data very quickly to any point-in-time upon data damage by tracing back the sequence and history of Exclusive-ORs resulting from writes. What is interesting is that TRAP architecture is amazingly space-efficient. We have implemented a prototype TRAP architecture using software at block device level and carried out extensive performance measurements using TPC-C benchmark running on Oracle and Postgress databases, TPC-W running on MySQL database, and file system benchmarks running on Linux and Windows systems. Our experiments demonstrated that TRAP is not only able to recover data to any point-in-time very quickly upon a failure but it also uses less storage space than traditional daily differential backup/snapshot. Compared to the state-of-the-art continuous data protection technologies, TRAP saves disk storage space by one to two orders of magnitude with a simple and a fast encoding algorithm. From an architecture point of view, TRAP-Array opens up another dimension for storage arrays. It is orthogonal and complementary to RAID in the sense that RAID protects data in the dimension along an array of physical disks while TRAP protects data in the dimension along the time sequence.

D-CRANE: a database system for utilization of cranes

Crane selection is a time consuming process that involves extensive data manipulation. Several systems have been developed to assist in selecting cranes and in planning their lifts. These systems lack the support of a comprehensive database to provide information about crane configurations, their lift capacity settings, and rigging equipment. Although crane manufacturers provide data for their cranes, these data are not always consistent and do not follow a standard format. This creates frequent problems for crane users, especially when interpolating the load charts. This requires the users to make decisions based on job conditions and categories of cranes, which can lead to costly mistakes. This paper describes the development of a comprehensive database (called D-CRANE) designed to support efficient selection of cranes. D-CRANE has been developed in collaboration with an industrial partner. It includes operational information about crane geometry, lift configurations, lift capacity settings, accessories, and attachments. D-CRANE has a number of interesting features: (i) powerful graphics capabilities, featuring a multimedia environment and a practical user-friendly interface; (ii) capacity to accommodate different types of commercially available cranes; (iii) powerful storage, sorting, and query routines; (iv) flexibility in using metric and empirical units; (v) capability of operating in a network environment, and (vi) minimum disk storage space. D-CRANE is a relational database designed using entity relation diagram and is implemented using MS-Access database management system. A case example is presented to demonstrate the use and capabilities of D-CRANE.Key words: database management system, crane selection, planning critical and heavy lift.

Video staging: a proxy-server-based approach to end-to-end video delivery over wide-area networks

Real-time distribution of stored video over wide-area networks (WANs) is a crucial component of many emerging distributed multimedia applications. The heterogeneity in the underlying network environments is an important factor that must be taken into consideration when designing an end-to-end video delivery system. We present a novel approach to the problem of end-to-end video delivery over WANs using proxy servers situated between local-area networks (LANs) and a backbone WAN. A major objective of our approach is to reduce the backbone WAN bandwidth requirement. Toward this end, we develop an effective video delivery technique called video staging via intelligent utilization of the disk bandwidth and storage space available at proxy servers. Using this video staging technique, only part of a video stream is retrieved directly from the central video server across the backbone WAN whereas the rest of the video stream is delivered to users locally from proxy servers attached to the LANs. In this manner, the WAN bandwidth requirement can be significantly reduced, particularly when a large number of users from the same LAN access the video data. We design several video staging methods and evaluate their effectiveness in trading the disk bandwidth of a proxy server for the backbone WAN bandwidth. We also develop two heuristic algorithms to solve the problem of designing a multiple video staging scheme for a proxy server with a given video access profile of a LAN. Our results demonstrate that the proposed proxy-server-based approach provides an effective and scalable solution to the problem of the end-to-end video delivery over WANs.

Scheduling and data layout policies for a near-line multimedia storage architecture

Recent advances in computer technologies have made it feasible to provide multimedia services, such as news distribution and entertainment, via high-bandwidth networks. The storage and retrieval of large multimedia objects (e.g., video) becomes a major design issue of the multimedia information system. While most other works on multimedia storage servers assume an on-line disk storage system, we consider a two-tier storage architecture with a robotic tape library as the vast near-line storage and an on-line disk system as the front-line storage. Magnetic tapes are cheaper, more robust, and have a larger capacity; hence, they are more cost effective for large scale storage systems (e.g., videoon-demand (VOD) systems may store tens of thousands of videos). We study in detail the design issues of the tape sub-system and propose some novel tape-scheduling algorithms which give faster response and require less disk buffer space. We also study the disk-striping policy and the data layout on the tape cartridge in order to fully utilize the throughput of the robotic tape system and to minimize the on-line disk storage space.

Scattering from an axisymmetric structure insonified at any incidence using a coupled finite-element/boundary element method

In the field of the scattering of acoustic waves from structures immersed in an infinite fluid medium, a coupling between the EQI boundary element code and the ATILA finite-element code is available. The boundary element method used is the Helmholtz integral equation. At some frequencies, called irregular frequencies, this integral formulation presents an infinite number of solutions. This problem is solved with the help of the Jones method: The integral equations system is overdetermined by null-field equations. Many structures are axisymmetric. So, it is attractive to develop a modeling that is able to keep the advantages of the axis of symmetry and to take into account an incident wave of any direction. This numerical tool is performed using a harmonic cylindrical decomposition. In this case, the mesh is restricted to the line of the structure included in the meridian plane. Compared with three-dimensional modeling, this new possibility provides the following advantages: either a reduction of the computation time, the memory size, and the disk storage space and an improvement of the numerical behavior; or an increase of the maximum frequency. Comparison between numerical results and experimental data is presented.

FOUR WORTHWHILE SOFTWARE PRODUCTS

There are several new or revised software products that I want to talk about in this column. I've found them to be worth the time and expense and think you might, too. The first is Ami Pro Version 2.0 which is a graphical word processor that permits you to do sophisticated desktop publishing. The second is Quattro Pro 4.0 the latest upgrade to the now venerable Quattro spreadsheet series. The third product is Stacker, a data compression utility that can more than double your hard disk storage space. Finally, there is Windows 3 Secrets which is a book with two disks of software. (As an aside, what kind of “ware” is a book—paperware?)

The DairyCHAMP program: a computerised recording system for dairy herds.

The DairyCHAMP program is an animal health and management software program that helps daily animal management, herd performance monitoring and problem analysis. Data entry to the program uses a data dictionary and includes an error-checking system that ensures the consistency and appropriateness of data entered. DairyCHAMP performs health management functions, provides a convenient user interface, ensures uniform data across farms by using a standard data dictionary, can be fully integrated with decision-making software programs like DairyORACLE, and is flexible enough to be useful for many types of dairy facilities. Data are entered via a menu-based system. Animal events are organised around reproduction and lactation cycles and health records. Farm records include inventories for drug, feed and semen. Farm parameters can be established which customize the program for an individual farm. The database system is an integration of three schemas: the individual user's view, the community view and the storage system. The individual user's view must be easy to use, while the storage system must be compact enough to fit within the disc storage space on a microcomputer. This conflict requires a translation from one schema to another. The DairyCHAMP program accomplishes this through a coding system which assigns a code number to each event. The program can add synonyms to this event dictionary by assigning the same code number to the synonym the user chooses. The DairyCHAMP program provides access to the large amounts of data required to aid in daily animal management, allow performance monitoring and analyse problems. Its highly integrated system is efficient and easy to use and maintain.

Solution of a large structural analysis problem on a minicomputer

A large structural analysis problem, involving the solution of 11448 equations with an average front width of about 350, was solved on a minicomputer. This mathematical model resulted from the application of the finite element method, using flat shell elements, to determine the stress levels induced in the carriage of a stacker-reclaimer due to design loads. Computer programs were developed for analysis; the frontal solution procedure with a restart facility and other modifications were used for the solution of the finite element equations. A scheme was developed and employed to solve such large problems on a minicomputer in spite of the limited computing speed and disc storage space available. Seven loads cases were analyzed and areas of high tensile and compressive stress identified.

Retrieval performance versus disc space utilization on WORM optical discs

Steady progress in the development of optical disc technology over the past decade has brought it to the point where it is beginning to compete directly with magnetic disc technology. WORM optical discs in particular, which permanently register information on the disc surface, have significant advantages over magnetic technology for applications that are mainly archival in nature but require the ability to do frequent on-line insertions. In this paper, we propose a class of access methods that use rewritable storage for the temporary buffering of insertions to data sets stored on WORM optical discs and we examine the relationship between the retrieval performance from WORM optical discs and the utilization of disc storage space when one of these organizations is employed. We describe the performance trade off as one of fast sequential retrieval of the contents of a block versus wasted space owing to data replication. A model of a specific instance of such an organization (a buffered hash file scheme) is described that allows for the specification of retrieval performance objectives. Alternative strategies for managing data replication that allow trade offs between higher consumption rates and better average retrieval performance are also described. We then provide an expected value analysis of the amount of disc space that must be consumed on a WORM disc to meet specified performance limits. The analysis is general enough to allow easy extension to other types of buffered files systems for WORM optical discs.

Getting More from Diskette Storage

In today's world of microcomputing, there is one given: however much disk storage space you have today, you'll need much, much more tomorrow. Just four years ago Arthur Naiman wrote about microcomputer hard disk storage in Word Processing Buyer's Guide (BYTE/McGraw‐Hill, Peter‐borough, NH: 1983. p.235). He related that “An optional 8.4MB hard disk drive is available for S4500. In fact, you can connect up to three of them, although what you'd need 26 million characters worth of storage for is beyond me.” Today, 20MB of storage is available on an expansion card for under $500 and 20MB is considered to be the minimum acceptable hard disk capacity by most experts.

MULREG: A multiple regression analysis system for microcomputers

The package described here is a complete system for data entry, edition, transformation, and analysis using either standard or stepwise (backward elimination procedure) multiple regression with a maximum of more than 50 variables and the number of subjects limited only by the amount of disk storage space. The system is menu driven to allow for maximum ease of use and reliability of operation. The critical matrix inversion routine performs all computations in double precision to reduce the computation errors to a minimum. The system includes four programs designed respectively to: (1) create, edit, and transform data; (2) run analysis and output to the video display; (3) run analysis and output to the line printer; and (4) produce a scattergram. Computational procedures follow those described by Pedhazur (1982) and Younger (1979). Input. Data are entered using a routine called CREATEF3/BAS. Each variable is stored as a separate sequential data flle. This routine also allows for complete edition of the data (erasing, adding, changing any data piece), transformation of a data file using a user-defined formula that can include another data flle (variable), and some other functions related to the management of disk data file (display directory of data file, print data file content, and purge data files). Analysis. Three options are provided to the user in the context of data analysis. Using the first option, the user can run the analysis and route the output to the video display. The stepwise analysis and some descriptive statistics are not available, but all other features of the system are provided. The main use of this option is to perform an analysis on a limited number of variables, when speed is of some concern. With the second option (run analysis and output to the line printer), it is possible to select either standard or stepwise regression analysis. The stepwise procedure used is the backward.elimination method. If the stepwise option is selected, the standardized residuals and the predicted scores (Y') are saved to disk on each step for later plotting on the line printer. The third analysis option allows for the production of scattergrams. With this option, it is possible to obtain a scattergram of the relationship between any pair of variables created by the CREATEF3/BAS routine or

A small university allocates resources

The Feldberg Computer Center which houses a Decsystem-10 timesharing computer opened in 1972 with the purpose of serving the research community at Brandeis University. Since its opening the user community has grown to include students in academic courses, a few users from the administrative community, and some non-profit agencies.One of the problems facing a center with a heterogeneous user base is the equitable allocation of resources. We must decide when users can access the machine, how much disk storage space each may have, etc. Tlne problem has been complicated by the shifts in the types of users from each of these communities. Professors have become more sophisticated about the value of using computers in conjunction with courses. Students are requesting more courses in Computer Science. A growing percentage of our researchers are social scientists. And, of course, professors as well as students have discovered that computer editors and text processing programs make producing papers, theses, and books easier. Further complications are the federal regulations relating to the way overhead is calculated for grants.We have developed a policy which we believe is humane and equitable. We have found that most people are satisfied. This paper will describe that resource allocation policy.

Interactive display of molecular models using a microcomputer system

An interactive graphics display system (IGDS) which presents perspective views of wire frame models is described. The display unit is a microprocessor driven TERAK 8510a system with a grid of 320 · 240 points. A brief discussion of the the hardware is given. While the system is quite general — it can draw a perspective view of any type of wire frame model — the emphasis here is on molecular models. The graphics software has been written in UCSD PASCAL. The program employs traditional computer graphics algorithms available in the literature. The data structure used by the program was designed to occupy minimal disk storage space. Data files can be constructed locally or can be obtained through a communications link with a central computer facility. Despite the inherent slowness (due to matrix operations being performed by the software calculations rather than hardware), the low cost of the system suggests general applicability. The program is easy to use, interactive, and produces good results for systems of up to 200 points.

Pathology accessioning and retrieval system with encoding by computer (PARSEC). A microcomputer-based system for anatomic pathology featuring automated SNOP coding and multiple administrative functions.

A pathology accessioning and retrieval system with encoding by computer (PARSEC) has been developed, employing a relatively inexpensive microcomputer. PARSEC performs a variety of administrative functions for anatomic pathology, including accessioning of surgical specimens, storage of patient demographic information, editing, retrieval, and archiving of patient data, as well as CAP (college of American Pathologists) workload units, billing, and inventory functions for histopathology. In addition, appropriate gross and microscopic descriptions and pathologic diagnoses can be entered into the system by a text editor. Automatic assignment of SNOP (Systematized Nomenclature of Pathology) codes, is accomplished via an online SNOP lexicon, allowing the ultimate generation of completed surgical pathology reports. The data base management system employed makes optimum use of disk storage space, while permitting rapid data retrieval. Data file maintenance is automatically accomplished by the system, requiring no user intervention.