Web-based Data Services Research Articles

GIS preprocessing for rapid initialization of HEC-HMS hydrological basin models using web-based data services

Hydrological GIS preprocessing tools are used to spatially relate different datasets for producing topologically connected watersheds and deriving attributes for model parameterization. Obtaining such information requires disparate data-sources from various websites with lengthy processing requirements. This research describes the development of HMS-PrePro, an open-source GIS preprocessing tool that was developed for the U.S. Army Corps of Engineers' (USACE) Hydrologic Engineering Center's Hydraulic Modeling System (HEC-HMS) software. HMS-PrePro provides data as a service by connecting to Esri's Living Atlas web service and extracting authoritative hydrological datasets for any watershed of interest in the contiguous United States. The datasets are saved to a local geodatabase, and spatial operations are executed to automatically delineate the watershed, derive topological connections, and estimate several common hydrologic parameters. An output basin text file is produced in HEC-HMS format for rapid model initialization.

HydroDS: Data services in support of physically based, distributed hydrological models

Physically based distributed hydrologic models require geospatial and time-series data that take considerable time and effort in processing them into model inputs. Tools that automate and speed up input processing facilitate the application of these models. In this study, we developed a set of web-based data services called HydroDS to provide hydrologic data processing ‘software as a service.’ HydroDS provides functions for processing watershed, terrain, canopy, climate, and soil data. The services are accessed through a Python client library that facilitates developing simple but effective data processing workflows with Python. Evaluations of HydroDS by setting up the Utah Energy Balance and TOPNET models for multiple headwater watersheds in the Colorado River basin show that HydroDS reduces the input preparation time compared to manual processing. It also removes the requirements for software installation and maintenance by the user, and the Python workflows enhance reproducibility of hydrologic data processing and tracking of provenance.

CMS Space Monitoring

During the first LHC run, CMS stored about one hundred petabytes of data. Storage accounting and monitoring help to meet the challenges of storage management, such as efficient space utilization, fair share between users and groups and resource planning. We present a newly developed CMS space monitoring system based on the storage metadata dumps produced at the sites. The information extracted from the storage dumps is aggregated and uploaded to a central database. A web based data service is provided to retrieve the information for a given time interval and a range of sites, so it can be further aggregated and presented in the desired format. The system has been designed based on the analysis of CMS monitoring requirements and experiences of the other LHC experiments. In this paper, we demonstrate how the existing software components of the CMS data placement system, PhEDEx, have been re-used, dramatically reducing the development effort.

The PhEDEx next-gen website

PhEDEx is the data-transfer management solution written by CMS. It consists of agents running at each site, a website for presentation of information, and a web-based data-service for scripted access to information.The website allows users to monitor the progress of data-transfers, the status of site agents and links between sites, and the overall status and behaviour of everything about PhEDEx. It also allows users to make and approve requests for data-transfers and for deletion of data. It is the main point-of-entry for all users wishing to interact with PhEDEx.For several years, the website has consisted of a single perl program with about 10K SLOC. This program has limited capabilities for exploring the data, with only coarse filtering capabilities and no context-sensitive awareness. Graphical information is presented as static images, generated on the server, with no interactivity. It is also not well connected to the rest of the PhEDEx codebase, since much of it was written before the data-service was developed. All this makes it hard to maintain and extend.We are re-implementing the website to address these issues. The UI is being rewritten in Javascript, replacing most of the server-side code. We are using the YUI toolkit to provide advanced features and context-sensitive interaction, and will adopt a Javascript charting library for generating graphical representations client-side. This relieves the server of much of its load, and automatically improves server-side security. The Javascript components can be re-used in many ways, allowing custom pages to be developed for specific uses. In particular, standalone test-cases using small numbers of components make it easier to debug the Javascript than it is to debug a large server program.Information about PhEDEx is accessed through the PhEDEx data-service, since direct SQL is not available from the clients’ browser. This provides consistent semantics with other, externally written monitoring tools, which already use the data-service. It also reduces redundancy in the code, yielding a simpler, consolidated codebase.In this talk we describe our experience of re-factoring this monolithic server-side program into a lighter client-side framework. We describe some of the techniques that worked well for us, and some of the mistakes we made along the way. We present the current state of the project, and its future direction.

Low Cost VoIP Architecture Using Open Source Software Component in Tertiary Institutions

Governments and their agencies are often challenged by high cost and flexible telephonic, Web based data services. Emerging technologies, such as those of Voice over Internet Protocol (VoIP) that allow convergent systems where voice and Web technologies can utilize the same network to provide both services, can be used to improve such services. This paper describe VoIP system for the enterprise network (e.g. company, university) that have been developed based on Asterisk which is a kind of open source software to implement IP-PBX system. Through the development and evaluation, we have confirmed that VoIP system based on Asterisk is very powerful as a whole and most PBX functions to be required for the enterprise network can be realized. Interesting findings include that the University of Kufa has a potential to implement the project. By connecting multiple Asterisk servers located in different sites based on IAX2, large scale enterprise network can be developed. Since the software recommended for installation is open source, the project could be used as a source of valuable information by students who specialize in real-time multi-media systems.

COASTAL OCEAN OBSERVING NETWORK – OPEN SOURCE ARCHITECTURE FOR DATA MANAGEMENT AND WEB-BASED DATA SERVICES

Abstract. The observations from the oceans are the backbone for any kind of operational services, viz. potential fishing zone advisory services, ocean state forecast, storm surges, cyclones, monsoon variability, tsunami, etc. Though it is important to monitor open Ocean, it is equally important to acquire sufficient data in the coastal ocean through coastal ocean observing systems for re-analysis, analysis and forecast of coastal ocean by assimilating different ocean variables, especially sub-surface information; validation of remote sensing data, ocean and atmosphere model/analysis and to understand the processes related to air-sea interaction and ocean physics. Accurate information and forecast of the state of the coastal ocean at different time scales is vital for the wellbeing of the coastal population as well as for the socio-economic development of the country through shipping, offshore oil and energy etc. Considering the importance of ocean observations in terms of understanding our ocean environment and utilize them for operational oceanography, a large number of platforms were deployed in the Indian Ocean including coastal observatories, to acquire data on ocean variables in and around Indian Seas. The coastal observation network includes HF Radars, wave rider buoys, sea level gauges, etc. The surface meteorological and oceanographic data generated by these observing networks are being translated into ocean information services through analysis and modelling. Centralized data management system is a critical component in providing timely delivery of Ocean information and advisory services. In this paper, we describe about the development of open-source architecture for real-time data reception from the coastal observation network, processing, quality control, database generation and web-based data services that includes on-line data visualization and data downloads by various means.

Protecting personal data from untrusted web-based data services

Our society places a heavy emphasis on security and the privacy of personal data. Personal data at the hands of a malicious entity could spell disastrous consequences for the data's owner. There is a whole slew of attacks on the Internet aimed at procuring and misusing personal information for profit.1, 2 And so, recently, there has been a huge thrust towards developing technologies that protect personal data from being misused.3, 4 Governments across the world are passing laws to protect personal information.5 Outsourcing personal information to web-based data services (WDS) is a common recent trend. These services have become very popular, but they require us to trust the service provider and may be vulnerable to both outsider and insider attacks. Ravi Chandra Jammalamadaka of eBay and Sharad Mehrotra and Nalini Venkatasubramanian of the University of California, Irvine, present the design and initial implementation of DataProtector, security middleware that acts as a transparent middleman between browser and web server and enforces security requirements at the level of HTTP requests.

SWBDS: Web Data Sources Management Infrastructure for Semantic Web Applications

The Semantic Web envisions a World Wide Web in which datasources are encapsulated and described with rich semantics, and demander can issue complex queries. A critical problem in the situation is how to efficiently describe, organize and search these encapsulated datasources. This paper describes the Semantic Web-Based Data service(SWBDS for short) approach, which addresses these challenges. SWBDS introduces an ontology-based approach to, mapping web data sources to data service (DS for short), publishing DS with the shared domain ontology, and answering queries through DS provided interface. We define the domain ontology to illustrate the DS interface. The domain ontology is described in OWL that can be understood and processed by machines. Therefore, SWBDS can provide reasoning functions and facilitate datasources management with little human effort. This new system model makes full use of legacy applications and is flexible for future extensions.