Data Processing Environment Research Articles

Implementation of an Intelligent Indoor Environmental Monitoring and management system in cloud

Indoor environmental monitoring and management system has played an essential role in public health sustainability. By monitoring the indoor air quality in public areas such as schools, offices, home or other buildings, the authorities will be given a better picture of the indoor air quality to take the right steps to ensure the better air quality for people inside the buildings. This system also can give information about indoor air quality information for the society. Therefore, to achieve this goal, we need to develop a monitoring system by using Information Technology based on Big Data and Cloud Computing environment to give warning. In this paper, we propose Intelligent Indoor Environment Monitoring System (iDEMS) combined with ZigBee wireless sensor network technology to store and process environmental data in HBase. The mechanism of the proposed system is classified into three stages: data collection, data processing, and information monitoring. To understand an Intelligent Indoor Environmental Monitoring, first, we collected the gas from Intelligent Indoor Environmental Monitoring through the environmental sensors with ZigBee wireless sensor network technology. We build a platform for iDEMS to collect the data related to the indoor gases. Second, the environmental data collected in the first stage will be stored and processed in HBase which support massive data storage and free to increase storage capacity for the analysis and processing of Big Data. In this stage, we also compared several data-input methods to import data in HBase much more efficient. Third, the intelligent-control socket is integrated into iDEMS and give the warning if the air quality exceeds the absolute legal limit based on the air quality index rule from the authority. Finally, iDEMS presents the resulting information by a web-based Monitoring Platform so that users can use the Internet to monitor the environment and enable them to utilize these informed decisions on managing and improving the environments.

A Parallel and Optimization Approach for Land-Surface Temperature Retrieval on a Windows-Based PC Cluster

Land-surface temperature (LST) is a very important parameter in the geosciences. Conventional LST retrieval is based on large-scale remote-sensing (RS) images where split-window algorithms are usually employed via a traditional stand-alone method. When using the environment to visualize images (ENVI) software to carry out LST retrieval of large time-series datasets of infrared RS images, the processing time taken for traditional stand-alone servers becomes untenable. To address this shortcoming, cluster-based parallel computing is an ideal solution. However, traditional parallel computing is mostly based on the Linux environment, while the LST algorithm developed within the ENVI interactive data language (IDL) can only be run in the Windows environment in our project. To address this problem, we combine the characteristics of LST algorithms with parallel computing, and propose the design and implementation of a parallel LST retrieval algorithm using the message-passing interface (MPI) parallel-programming model on a Windows-based PC cluster platform. Furthermore, we present our solutions to the problems associated with performance bottlenecks and fault tolerance during the deployment stage. Our results show that, by improving the parallel environment of the storage system and network, one can effectively solve the stability issues of the parallel environment for large-scale RS data processing.

Critical Care Health Informatics Collaborative (CCHIC): Data, tools and methods for reproducible research: A multi-centre UK intensive care database

ObjectiveTo build and curate a linkable multi-centre database of high resolution longitudinal electronic health records (EHR) from adult Intensive Care Units (ICU).To develop a set of open-source tools to make these data ‘research ready’ while protecting patient’s privacy with a particular focus on anonymisation. Materials and methodsWe developed a scalable EHR processing pipeline for extracting, linking, normalising and curating and anonymising EHR data. Patient and public involvement was sought from the outset, and approval to hold these data was granted by the NHS Health Research Authority’s Confidentiality Advisory Group (CAG). The data are held in a certified Data Safe Haven. We followed sustainable software development principles throughout, and defined and populated a common data model that links to other clinical areas. ResultsLongitudinal EHR data were loaded into the CCHIC database from eleven adult ICUs at 5 UK teaching hospitals. From January 2014 to January 2017, this amounted to 21,930 and admissions (18,074 unique patients). Typical admissions have 70 data-items pertaining to admission and discharge, and a median of 1030 (IQR 481–2335) time-varying measures. Training datasets were made available through virtual machine images emulating the data processing environment. An open source R package, cleanEHR, was developed and released that transforms the data into a square table readily analysable by most statistical packages. A simple language agnostic configuration file will allow the user to select and clean variables, and impute missing data. An audit trail makes clear the provenance of the data at all times. DiscussionMaking health care data available for research is problematic. CCHIC is a unique multi-centre longitudinal and linkable resource that prioritises patient privacy through the highest standards of data security, but also provides tools to clean, organise, and anonymise the data. We believe the development of such tools are essential if we are to meet the twin requirements of respecting patient privacy and working for patient benefit. ConclusionThe CCHIC database is now in use by health care researchers from academia and industry. The ‘research ready' suite of data preparation tools have facilitated access, and linkage to national databases of secondary care is underway.

Multi-Level Attribute-Based Encryption Access Control Scheme for Big Data

There are many security threats such as data’s confidentiality and privacy protection in the new application scenario of big data processing, and for the problems such as coarse granularity and low sharing capability existing in the current research on big data access control, a new model to support fine-grained access control and flexible attribute change is proposed. Based on CP-ABE method, a multi-level attribute-based encryption scheme is designed to solve fine-grained access control problem. And to solve the problem of attribute revocation, the technique of re-encryption and version number tag is integrated into the scheme. The analysis shows that the proposed scheme can meet the security requirement of access control in big data processing environment, and has an advantage in computational overhead compared with the previous schemes.

The IoT for smart sustainable cities of the future: An analytical framework for sensor-based big data applications for environmental sustainability

The Internet of Things (IoT) is one of the key components of the ICT infrastructure of smart sustainable cities as an emerging urban development approach due to its great potential to advance environmental sustainability. As one of the prevalent ICT visions or computing paradigms, the IoT is associated with big data analytics, which is clearly on a penetrative path across many urban domains for optimizing energy efficiency and mitigating environmental effects. This pertains mainly to the effective utilization of natural resources, the intelligent management of infrastructures and facilities, and the enhanced delivery of services in support of the environment. As such, the IoT and related big data applications can play a key role in catalyzing and improving the process of environmentally sustainable development. However, topical studies tend to deal largely with the IoT and related big data applications in connection with economic growth and the quality of life in the realm of smart cities, and largely ignore their role in improving environmental sustainability in the context of smart sustainable cities of the future. In addition, several advanced technologies are being used in smart cities without making any contribution to environmental sustainability, and the strategies through which sustainable cities can be achieved fall short in considering advanced technologies. Therefore, the aim of this paper is to review and synthesize the relevant literature with the objective of identifying and discussing the state-of-the-art sensor-based big data applications enabled by the IoT for environmental sustainability and related data processing platforms and computing models in the context of smart sustainable cities of the future. Also, this paper identifies the key challenges pertaining to the IoT and big data analytics, as well as discusses some of the associated open issues. Furthermore, it explores the opportunity of augmenting the informational landscape of smart sustainable cities with big data applications to achieve the required level of environmental sustainability. In doing so, it proposes a framework which brings together a large number of previous studies on smart cities and sustainable cities, including research directed at a more conceptual, analytical, and overarching level, as well as research on specific technologies and their novel applications. The goal of this study suits a mix of two research approaches: topical literature review and thematic analysis. In terms of originality, no study has been conducted on the IoT and related big data applications in the context of smart sustainable cities, and this paper provides a basis for urban researchers to draw on this analytical framework in future research. The proposed framework, which can be replicated, tested, and evaluated in empirical research, will add additional depth to studies in the field of smart sustainable cities. This paper serves to inform urban planners, scholars, ICT experts, and other city stakeholders about the environmental benefits that can be gained from implementing smart sustainable city initiatives and projects on the basis of the IoT and related big data applications.

SDMtoolbox 2.0: the next generation Python-based GIS toolkit for landscape genetic, biogeographic and species distribution model analyses

SDMtoolbox 2.0 is a software package for spatial studies of ecology, evolution, and genetics. The release of SDMtoolbox 2.0 allows researchers to use the most current ArcGIS software and MaxEnt software, and reduces the amount of time that would be spent developing common solutions. The central aim of this software is to automate complicated and repetitive spatial analyses in an intuitive graphical user interface. One core tenant facilitates careful parameterization of species distribution models (SDMs) to maximize each model’s discriminatory ability and minimize overfitting. This includes carefully processing of occurrence data, environmental data, and model parameterization. This program directly interfaces with MaxEnt, one of the most powerful and widely used species distribution modeling software programs, although SDMtoolbox 2.0 is not limited to species distribution modeling or restricted to modeling in MaxEnt. Many of the SDM pre- and post-processing tools have ‘universal’ analogs for use with any modeling software. The current version contains a total of 79 scripts that harness the power of ArcGIS for macroecology, landscape genetics, and evolutionary studies. For example, these tools allow for biodiversity quantification (such as species richness or corrected weighted endemism), generation of least-cost paths and corridors among shared haplotypes, assessment of the significance of spatial randomizations, and enforcement of dispersal limitations of SDMs projected into future climates—to only name a few functions contained in SDMtoolbox 2.0. Lastly, dozens of generalized tools exists for batch processing and conversion of GIS data types or formats, which are broadly useful to any ArcMap user.

Crowdsourcing Analysis of Twitter Data on Climate Change: Paid Workers vs. Volunteers

Web based crowdsourcing has become an important method of environmental data processing. Two alternatives are widely used today by researchers in various fields: paid data processing mediated by for-profit businesses such as Amazon’s Mechanical Turk, and volunteer data processing conducted by amateur citizen-scientists. While the first option delivers results much faster, it is not quite clear how it compares with volunteer processing in terms of quality. This study compares volunteer and paid processing of social media data originating from climate change discussions on Twitter. The same sample of Twitter messages discussing climate change was offered for processing to the volunteer workers through the Climate Tweet project, and to the paid workers through the Amazon MTurk platform. We found that paid crowdsourcing required the employment of a high redundancy data processing design to obtain quality that was comparable with volunteered processing. Among the methods applied to improve data processing accuracy, limiting the geographical locations of the paid workers appeared the most productive. Conversely, we did not find significant geographical differences in the accuracy of data processed by volunteer workers. We suggest that the main driver of the found pattern is the differences in familiarity of the paid workers with the research topic.

A Survey Report on Wireless Sensor Network

Wireless Sensor Network (WSN) is the combination of various sensing nodes. WSN is used to produce meaningful information from raw local data collected by each sensor. The characteristics of sensor network create many new and existing application areas for remote sensing. Also now a day there is much application in many field and WSN faces security challenges so I have discussed it with architecture of WSN. With advancement in WSN, small and low cost sensor nodes become available, which have capabilities of wireless communication, sensing various types of environmental conditions and data processing. Now a day, we are living in advance technology environment, which has a great advantage on every person life. Wireless sensor network is one of them which is the collection of nodes organized into a cooperative network and in every node consist of processing capability, and may contains multiple types of memory. In this paper there is review of introduction of Wireless Sensor Network and its application in various fields with security challenges. Also I have discussed various factors that affect the sensor network design. Wireless Sensor Network has tremendous advantages in the worlds which makes every person life much easier and also have various applications.

Application of a Relative Humidity Sensor for Monitoring Water Vapor Concentration inside Enclosures

The presence of water can have adverse effects on the performance of mechanical and electrical hardware. Standing or condensed water can cause shorting or grounding issues in the short term and can accelerate corrosion in the long term. High humidity can also accelerate corrosion as well as cause swelling of electrical components, seals, or composite materials. Water exposure data can be used to actively intervene prior to degradation(e.g. to fix a water leak) or to calculate the remaining life of a system that has been exposed to water or high humidity. The U.S. Army has recognized the importance of environmental monitoring and failure prediction in weapon systems to ensure readiness, to enhance safety, and toimprove weapon performance. Since 2000, Picatinny Arsenal has sponsored the development of specialized environmental monitoring sensors with on-board diagnostics in various new weapon/ammunition systems. In this article we present an effort at the Pacific Northwest National Laboratory to develop an asset health monitor to assess water concentration inside the storage container of an artillery shell. Desiccant is used in the container to maintain the water vapor concentration below 5000 parts per million; a higher water concentration would indicate container leakage or the need to replace the desiccant. Various types of water sensors, including standing water indicators, corrosion indicators, and humidity sensors, were identified and categorized according to type of water exposure, accuracy, and response time. A commercially available relative humidity sensor was identified as a cost effective approach to measuring water vapor concentration inside the container. Relative humidity can be converted to waterconcentration using the water vapor saturation pressure at ambient temperature. Using diurnal temperature simulations, it was determined that the accuracy of the relative humidity sensor would be unsatisfactory in high temperature/low humidity situations where a small change in relative humidity results is a large change in water vapor concentration. To overcome this obstacle, a strategic sampling technique was developed to increase the accuracy of the water concentration measurement that is based on relative humidity sensor data. The relative humidity sensorwas integrated into the Remote Readiness Asset Prognostic/Diagnostic System (RRAPDS), which is an asset health monitor developed by PNNL to measure environmental conditions in munition containers. RRAPDS will be secured to the container with a probe that extends into the container to monitor internal temperature and humidity. RRAPDS includes a microcontroller to process environmental data and execute diagnostic routines. Thispaper will describe the approach that was developed to estimate water concentration using relative humidity sensor data and describe the sampling technique used to maximize the accuracy of the water vapor concentration assessment.

Managing and Analyzing the Constructive and Functional Parameters on Fiberglass Custom Sensor Design for an Aerodynamic Balance

An external three component strain gauge balance, designed, analyzed and manufactured in the Military Technical Academy is presented. The balance is a multi-piece type and employs the use of six custom-made load cells (sensors) to transfer the aerodynamic forces and moments from the wing to the computer software environment for data processing and analysis. The relations between the loads acting on the wing model and the measured forces are also underlined. The placement of the strain gauge on the composite laminate is an important issue for increased sensor precision; therefore, it is given a thorough analysis. The positioning of the force vector and its influence on the strain distribution over the lift sensor is also numerically and experimentally analyzed.

Unified Platform for AI and Big Data Analytics

This paper describes an integrated platform for machine learning and big data analysis. The integrated platform is configured in a way that builds a large distributed data processing environment in the computing environment that makes up the NVIDIA AI platform. In addition, this paper describes the background of this idea selection and the use of the software to build the unified platform. The technical details are shown in terms of how to create the proposed platform. In the anlaysis section, the methodology is provided and also the steps are explained as to how to use this integration platform. Finally, the expected effects are elaborated in the conclusion section.

Health Monitoring and Management for Manufacturing Workers in Adverse Working Conditions.

In adverse working conditions, environmental parameters such as metallic dust, noise, and environmental temperature, directly affect the health condition of manufacturing workers. It is therefore important to implement health monitoring and management based on important physiological parameters (e.g., heart rate, blood pressure, and body temperature). In recent years, new technologies, such as body area networks, cloud computing, and smart clothing, have allowed the improvement of the quality of services. In this article, we first give five-layer architecture for health monitoring and management of manufacturing workers. Then, we analyze the system implementation process, including environmental data processing, physical condition monitoring and system services and management, and present the corresponding algorithms. Finally, we carry out an evaluation and analysis from the perspective of insurance and compensation for manufacturing workers in adverse working conditions. The proposed scheme will contribute to the improvement of workplace conditions, realize health monitoring and management, and protect the interests of manufacturing workers.

Distributed data processing and analysis environment for neutron scattering experiments at CSNS

China Spallation Neutron Source (CSNS) is the first high-performance pulsed neutron source in China, which will meet the increasing fundamental research and technique applications demands domestically and overseas. A new distributed data processing and analysis environment has been developed, which has generic functionalities for neutron scattering experiments. The environment consists of three parts, an object-oriented data processing framework adopting a data centered architecture, a communication and data caching system based on the C/S paradigm, and data analysis and visualization software providing the 2D/3D experimental data display. This environment will be widely applied in CSNS for live data processing.

Evaluation Performance of Task Scheduling Algorithms in Heterogeneous Environments

A heterogeneous computing environment is a large-scale distributed data processing environment, it is depends to some extent parameters on the application and that classified in three main categories such as the hardware, the communication layer, and the software. A computer system is consists of hardware and software from two or more different manufacturers. Scheduling is one of the important factors in the heterogeneous environment and the aim of task scheduling in the processing environment is to move computation towards data. In order to achieve improve performance, increase the throughput and minimizing the makespan; scheduler must avoid unnecessary data transmission. Hence, different scheduling algorithms for heterogeneous computing environment are necessary to provide good performance. How to speedup scheduling the service resources to achieve the lowest cost becomes more and more important. This paper tries to illustrate and analyze the overview of eighteen different scheduling algorithms for heterogeneous computing environment and their scheduling issues and problems.

Processing and Unification of Environmental Noise Data from Road Traffic with Spatial Dimension Collected through Mobile Phones

Noise measurement using mobile phones is now developed very well. While there are some good applications for the measurement of noise from road traffic, thus on processing of measured data is only paid a very little attention. The data, however, are burdened by specific errors and for further work with them it is necessary to adjust and determine their uncertainty. One of the biggest problems is inaccuracy in position versus the noise source and the shortest length of measurement that can be regarded as representative. Imprecision in terms of location can be determined by calculating the variance of possible distance from the noise source, which for measurement of traffic noise requires a map-matching data points both transverse to the street (sidewalk) network and in the longwise direction. During typical urban measurements, this error can even reach 7 - 10 dB. Three basic types of algorithms for the calculation of uncertainty and positional correction based on the type of input and output data (raster, vector, vector-oriented) were tested. Uncertainty in the variability of the measurement data is necessary to determine from the number of passing vehicles per time unit. The presented solutions are implemented in the Mobile Noise system.

Workflow Enabled data Processing in a Concurrent Engineering Environment

Knowledge, information and data hold the key for growth and continuous sustenance in any organization. However, flow of this information and data across inter disciplinary departments to authorized personnel ensures the authencity and accuracy of this data while in transit. Workflow can be devised as a tool used purely in electronic data processing environment where data is to be transmitted across heterogeneous departments handled by personnel belonging to diverse engineering backgrounds. This proves itself to have a catalytic effect in product oriented, process oriented and data processing modes of working in an industry through its pre-set rules, procedures and flexible modes of operation. Further, the same workflow methodology can be used in web based applications which only requires installation of the particular tool in server machine, thus providing the freedom from independent client installations. Enhancing the operational ability of workflow can provide information on the approximate time taken for data processing in various inter-disciplinary departments involving process oriented, product oriented and data processing modes of working.

Survey on Hierarchical Cluster Routing Protocols of WSN

Approaches in wireless sensor network (WSN) technology has gained the availability of small and low-cost sensor nodes with sensing capability of various types of physical and environmental conditions, data processing, and wireless communication. In wireless sensor networks have conduct to many new protocols particularly designed for sensor networks where energy awareness is an essential issue? Most of the observation, however, has been given to the routing protocols since they just differ depending on the application and network architecture. This paper surveys recent routing protocols for sensor networks and presents a classification [2] for the various approaches pursued. There are three main types data-centric, hierarchical and location-based.the only one type explored in this paper is hierarchical. Each hierarchical routing protocol is described.

Quality assessment and improvement of the EUMETSAT Meteosat Surface Albedo Climate Data Record

Abstract. Surface albedo has been identified as an important parameter for understanding and quantifying the Earth's radiation budget. EUMETSAT generated the Meteosat Surface Albedo (MSA) Climate Data Record (CDR) currently comprising up to 24 years (1982–2006) of continuous surface albedo coverage for large areas of the Earth. This CDR has been created within the Sustained, Coordinated Processing of Environmental Satellite Data for Climate Monitoring (SCOPE-CM) framework. The long-term consistency of the MSA CDR is high and meets the Global Climate Observing System (GCOS) stability requirements for desert reference sites. The limitation in quality due to non-removed clouds by the embedded cloud screening procedure is the most relevant weakness in the retrieval process. A twofold strategy is applied to efficiently improve the cloud detection and removal. The first step consists of the application of a robust and reliable cloud mask, taking advantage of the information contained in the measurements of the infrared and visible bands. Due to the limited information available from old radiometers, some clouds can still remain undetected. A second step relies on a post-processing analysis of the albedo seasonal variation together with the usage of a background albedo map in order to detect and screen out such outliers. The usage of a reliable cloud mask has a double effect. It enhances the number of high-quality retrievals for tropical forest areas sensed under low view angles and removes the most frequently unrealistic retrievals on similar surfaces sensed under high view angles. As expected, the usage of a cloud mask has a negligible impact on desert areas where clear conditions dominate. The exploitation of the albedo seasonal variation for cloud removal has good potentialities but it needs to be carefully addressed. Nevertheless it is shown that the inclusion of cloud masking and removal strategy is a key point for the generation of the next MSA CDR release.

Creation and Use of Internal Matrix Database Functions in VBA MS Excel Environment for Bulk Data Processing

Creation and Use of Internal Matrix Database Functions in VBA MS Excel Environment for Bulk Data Processing

CER Hub: An informatics platform for conducting comparative effectiveness research using multi-institutional, heterogeneous, electronic clinical data

ObjectivesComparative effectiveness research (CER) requires the capture and analysis of data from disparate sources, often from a variety of institutions with diverse electronic health record (EHR) implementations. In this paper we describe the CER Hub, a web-based informatics platform for developing and conducting research studies that combine comprehensive electronic clinical data from multiple health care organizations. MethodsThe CER Hub platform implements a data processing pipeline that employs informatics standards for data representation and web-based tools for developing study-specific data processing applications, providing standardized access to the patient-centric electronic health record (EHR) across organizations. ResultsThe CER Hub is being used to conduct two CER studies utilizing data from six geographically distributed and demographically diverse health systems. These foundational studies address the effectiveness of medications for controlling asthma and the effectiveness of smoking cessation services delivered in primary care. DiscussionThe CER Hub includes four key capabilities: the ability to process and analyze both free-text and coded clinical data in the EHR; a data processing environment supported by distributed data and study governance processes; a clinical data-interchange format for facilitating standardized extraction of clinical data from EHRs; and a library of shareable clinical data processing applications. ConclusionCER requires coordinated and scalable methods for extracting, aggregating, and analyzing complex, multi-institutional clinical data. By offering a range of informatics tools integrated into a framework for conducting studies using EHR data, the CER Hub provides a solution to the challenges of multi-institutional research using electronic medical record data.