Big Climate Data Research Articles

A shifting climate: New paradigms and challenges for (early career) scientists in extreme weather research

AbstractResearch on weather and climate extremes has become integral to climate science due to their increasing societal relevance and impacts in the context of anthropogenic climate change. In this perspective we examine recent changes and evolving paradigms in the study of extreme events, emphasizing the increasingly interdisciplinary nature of research and their societal implications. We discuss the importance of understanding the physical basis of extreme events and its linkages to climate impacts, highlighting the need for collaboration across multiple disciplines. Furthermore, we explore the challenge of big climate data analysis and the application of novel statistical methods, such as machine learning, in enhancing our understanding of extreme events. Additionally, we address the engagement with different stakeholder groups and the evolving landscape of climate services and private‐sector involvement. We conclude with reflections on the risks and opportunities for early career researchers in navigating these interdisciplinary and societal demands, stressing the importance of meaningful scientific engagement, and removing barriers to inclusivity and collaboration in climate research.

An Expanded Spatial Durbin Model with Ordinary Kriging of Unobserved Big Climate Data

Spatial models are essential in the prediction of climate phenomena because they can model the complex relationships between different locations. In this study, we discuss an expanded spatial Durbin model with ordinary kriging on unobserved locations (ESDMOK) to predict rainfall patterns in Java Island. The classical spatial Durbin model needed to be expanded to obtain a parameter estimation for each location. We combined this with ordinary kriging because the data were not available in some locations. The data were taken from the National Aeronautics and Space Administration Prediction of Worldwide Energy Resources (NASA POWER) website. Since climate data are big data, we implement a big data analytics approach, namely the data analytics life cycle method. As the exogenous variables, we used air temperature, humidity, solar irradiation, wind speed, and surface pressure. The authors developed an R-Shiny web applications to implement our proposed technique. Using our proposed technique, we obtained more accurate and reliable climate data prediction, indicated by the mean absolute percentage error (MAPE), which was equal to 1.956%. The greatest effect on rainfall was given by the surface pressure variable, and the smallest was wind speed.

Skillful seasonal prediction of Afro-Asian summer monsoon precipitation with a merged machine learning and large ensemble approach

Afro-Asian summer monsoon precipitation (AfroASMP) is the life blood of billions of people living in many developing countries covering West Africa and Asia. Its complex variabilities are always accompanied by natural disasters like floods, landslides and droughts. Reliable AfroASMP prediction several months in advance is valuable for not only decision-makers but also regional socioeconomic sustainability. To address the current predicament of the AfroASMP seasonal prediction, this study provides an effective machine-learning model (Y-model). Y-model uses the monsoon related big climate data for searching the potential predictors, encompassing atmospheric internal factors and external forcings. Only the predictors associated with significant anomalies in summer horizonal winds at 850 hPa over the monsoon domain are retained. These selected predictors are then reorganized into a large ensemble based upon different thresholds of four fundamental principles. These principles include the focused sample sizes, the relationships between predictors and predictand, the independence among predictors, and the extremities of predictors in the forecast year. Real-time predictions can be generated based on the ensemble mean of skillful members during an independent hindcast period. Y-model skillfully predicts four monsoon precipitation indices of AfroASMP during 2011–2022 at lead 4–12 months, correlation skills range from 0.58 to 0.90 and root mean square error skills are reduced by 11–53% compared to CFS v2 model at lead 1 month. This study offers an effective method for preprocessing predictors in seasonal climate prediction.

Integrated flood risk assessment in Hunza-Nagar, Pakistan: unifying big climate data analytics and multi-criteria decision-making with GIS

Floods are a widespread natural disaster with substantial economic implications and far-reaching consequences. In Northern Pakistan, the Hunza-Nagar valley faces vulnerability to floods, posing significant challenges to its sustainable development. This study aimed to evaluate flood risk in the region by employing a GIS-based Multi-Criteria Decision Analysis (MCDA) approach and big climate data records. By using a comprehensive flood risk assessment model, a flood hazard map was developed by considering nine influential factors: rainfall, regional temperature variation, distance to the river, elevation, slope, Normalized difference vegetation index (NDVI), Topographic wetness index (TWI), land use/land cover (LULC), curvature, and soil type. The analytical hierarchy process (AHP) analysis assigned weights to each factor and integrated with geospatial data using a GIS to generate flood risk maps, classifying hazard levels into five categories. The study assigned higher importance to rainfall, distance to the river, elevation, and slope compared to NDVI, TWI, LULC, curvature, and soil type. The weighted overlay flood risk map obtained from the reclassified maps of nine influencing factors identified 6% of the total area as very high, 36% as high, 41% as moderate, 16% as low, and 1% as very low flood risk. The accuracy of the flood risk model was demonstrated through the Receiver Operating Characteristics-Area Under the Curve (ROC-AUC) analysis, yielding a commendable prediction accuracy of 0.773. This MCDA approach offers an efficient and direct means of flood risk modeling, utilizing fundamental GIS data. The model serves as a valuable tool for decision-makers, enhancing flood risk awareness and providing vital insights for disaster management authorities in the Hunza-Nagar Valley. As future developments unfold, this study remains an indispensable resource for disaster preparedness and management in the Hunza-Nagar Valley region.

Probabilistic simulation of big climate data for robust quantification of changes in compound hazard events

Understanding changes in extreme compound hazard events is important for climate mitigation and policy. By definition, such events are rare so robust quantification of their future changes is challenging. An approach is presented, for probabilistic modelling and simulation of climate model data, which is invariant to the event definition since it models the underlying weather variables. The approach is based on the idea of a ‘moving window’ in conjunction with Generalised Additive Models (GAMs) and Bayesian inference. As such, it is robust to the data size and completely parallelizable, while it fully quantifies uncertainty allowing also for comprehensive model checking. Lastly, Gaussian anamorphosis is used to capture dependency across weather variables. The approach results in probabilistic simulations to enable extrapolation beyond the original data range and thus robust quantification of future changes of rare events. We illustrate by application to daily temperature, humidity and precipitation from a regional climate model.

Novel Approach for Spatiotemporal Weather Data Analysis

Massive volumes of multidimensional array-based spatiotemporal data are generated by climate observations and model simulations. The growth in climate data leads to new opportunities for climate studies at multiple spatial and temporal scales. Managing, analyzing and processing of big climate data is considered to be challenging because of huge data volume. In this work multidimensional climate data such as precipitation and temperature are processed and analyzed in the Spark MapReduce Platform, since Spark platform is computationally more efficient than Hadoop-MapReduce Platform of same configuration. In temporal scale monthly and seasonal analysis of climate data has been carried out to understand the regional climate. The prediction of Rainfall on monthly and seasonal time scales is very much important for planning and devising agricultural strategies and disaster management, etc. As the prediction of climate state is very challenging, in this study an attempt is being made for the prediction of the rainfall using the time series analysis in the same framework. As a test case the time series approach such as Auto Regression Integrated Moving Average (ARIMA) has been used for the prediction of rainfall. The proposed approach is evaluated and found to be accurate in the analysis and prediction of climate data and this will surely guide for the researcher for better understanding of the climate and its application to multiple sectors.

Probabilistic Simulationof Big Climate Data for Robust Quantification of Changes in Compound Hazard Events

Probabilistic Simulationof Big Climate Data for Robust Quantification of Changes in Compound Hazard Events

Consensus Forecast of Rainfall Using Hybrid Climate Learning Model

Rainfall event forecasting is prominently done using climate models (CMs) to produce multiple forecasts for the same rainfall event. The best forecast is complicated to find and hence has not yet been explored in the CMs. Recent advances in deep learning methods have provided an exceptional ability to investigate intricate weather patterns from big climate data. In this article, a hybrid climate learning model (HCLM) is proposed that utilizes both the CM and the deep learning models for improving the rainfall forecast. More specifically, a probabilistic multilayer perceptron (PMLP) network evaluates multiple forecasts from the CM-generated forecasts and selects the best one. The selected forecast is next passed onto a hybrid deep long short-term memory (HD-LSTM) network, which looks back and learns the relationship of the selected forecast with corresponding rainfall and temperature observations to produce the next-day rainfall forecast. The experimental results from various climate zones in Australia show that the HCLM outperforms existing state-of-the-art climate and deep learning models.

Big climate data assessment of viticultural conditions for wine quality determination in France

Grapevine is one of the economically most important crops especially in Europe. Although its investment value has been widely recognized, the complex interactions between climate and viticulture remain immaturely understood and modeled, which largely limits a reliable investing strategy by using the observed climate conditions to estimate the wine quality. Therefore, with the aim of comprehensively analyzing the climate-viticulture relationship, compared to most previous studies which employed a few climate factors derived from sparsely located meteorological stations, in the present study, we include 22 climate factors, including temperature, water balance, atmosphere, and radiation data provided by a global land assimilation system covering a period of 40 years (1970 to 2010) as well as two large-scale atmospheric teleconnection indices to establish a holistic climate-wine quality model. Moreover, instead of the conventionally used simple regression methods, to deal with the comprehensive but volume climate dataset, we employ the Least Absolute Shrinkage and Selection Operator (LASSO) regression method, which excels in ingesting a massive amount of variables having complex collinearities. In the pre-analysis of correlations between utilized climate factors, it is found that sunlight has the strongest connections with other factors as it correlates with the most number of climate factors. On the contrary, temperature, the conventionally most commonly employed factor, correlates with much fewer factors. Finally, via validation with wine vintage scores derived from two authoritative rating systems, it is ensured that our proposed approach can accurately establish the climate-wine quality models for four well-known wine-growing regions in France, including Alsace, Bordeaux, Burgundy, and Champagne. Due to the more complex climate pattern of Bordeaux compared to other regions, two bank-wise models instead of a bank-merged model is vital for Bordeaux to achieve a similar modeling accuracy. Eventually, a satisfactory vintage deviance explaining accuracy with one standard deviation score residual within ± 6 points can be achieved in all regions. Therefore, based on the established climate-wine quality model together with the observed climate conditions, the wine quality of each region can be reliably predicted, which provides a reliable reference for wine investment.

SOVAS: a scalable online visual analytic system for big climate data analysis

ABSTRACTBig climate data offers great opportunities for scientific discovery but demands efficient and effective analytics to investigate unknown and complex patterns. Most existing online processing and analytics systems for climate studies only support fixed user interface with predefined functions. These systems are often not scalable to handle massive climate data that could easily accumulate terabytes daily. To address the major limitations of existing online systems for climate studies, this paper presents a scalable online visual analytic system, known as SOVAS, to balance both usability and flexibility. SOVAS, enabled by a set of key techniques, supports large-scale climate data analytics and knowledge discovery in a scalable and sharable environment. This research not only contributes to the community an efficient tool for analyzing big climate data but also contributes to the literature by providing valuable technical references for tackling spatiotemporal big data challenges.

Big Data and Climate Change

Climate science as a data-intensive subject has overwhelmingly affected by the era of big data and relevant technological revolutions. The big successes of big data analytics in diverse areas over the past decade have also prompted the expectation of big data and its efficacy on the big problem—climate change. As an emerging topic, climate change has been at the forefront of the big climate data analytics implementations and exhaustive research have been carried out covering a variety of topics. This paper aims to present an outlook of big data in climate change studies over the recent years by investigating and summarising the current status of big data applications in climate change related studies. It is also expected to serve as a one-stop reference directory for researchers and stakeholders with an overview of this trending subject at a glance, which can be useful in guiding future research and improvements in the exploitation of big climate data.

In-Mapper combiner based MapReduce algorithm for processing of big climate data

Big data refers to a collection of massive volume of data that cannot be processed by conventional data processing tools and technologies. In recent years, the data production sources are enlarged noticeably, such as high-end streaming devices, wireless sensor networks, satellite, wearable Internet of Things (IoT) devices. These data generation sources generate a massive volume of data in a continuous manner. The large volume of climate data is collected from the IoT weather sensor devices and NCEP. In this paper, the big data processing framework is proposed to integrate climate and health data and to find the correlation between the climate parameters and incidence of dengue. This framework is demonstrated with the help of MapReduce programming model, Hive, HBase and ArcGIS in a Hadoop Distributed File System (HDFS) environment. The following weather parameters such as minimum temperature, maximum temperature, wind, precipitation, solar and relative humidity are collected for the study are Tamil Nadu with the help of IoT weather sensor devices and NCEP. Proposed framework focuses only on climate data for 32 districts of Tamil Nadu where each district contains 1,57,680 rows and so there are 50,45,760 rows in total. Batch view precomputation for the monthly mean of various climate parameters would require 50,45,760 rows. Hence, this would create more latency in query processing. In order to overcome this issue, batch views can precompute for a smaller number of records and involve more computation to be done at query time. The In-Mapper based MapReduce framework is used to compute the monthly mean of climate parameter for each latitude and longitude. The experimental results prove the effectiveness of the response time for the In-Mapper based combiner algorithm is less when compared with the existing MapReduce algorithm.

ClimateSpark: An in-memory distributed computing framework for big climate data analytics

The unprecedented growth of climate data creates new opportunities for climate studies, and yet big climate data pose a grand challenge to climatologists to efficiently manage and analyze big data. The complexity of climate data content and analytical algorithms increases the difficulty of implementing algorithms on high performance computing systems. This paper proposes an in-memory, distributed computing framework, ClimateSpark, to facilitate complex big data analytics and time-consuming computational tasks. Chunking data structure improves parallel I/O efficiency, while a spatiotemporal index is built for the chunks to avoid unnecessary data reading and preprocessing. An integrated, multi-dimensional, array-based data model (ClimateRDD) and ETL operations are developed to address big climate data variety by integrating the processing components of the climate data lifecycle. ClimateSpark utilizes Spark SQL and Apache Zeppelin to develop a web portal to facilitate the interaction among climatologists, climate data, analytic operations and computing resources (e.g., using SQL query and Scala/Python notebook). Experimental results show that ClimateSpark conducts different spatiotemporal data queries/analytics with high efficiency and data locality. ClimateSpark is easily adaptable to other big multiple-dimensional, array-based datasets in various geoscience domains.

Embarrassingly Parallel GPU Based Matrix Inversion Algorithm for Big Climate Data Assimilation

Attempts to harness the big climate data that come from high-resolution model output and advanced sensors to provide more accurate and rapidly-updated weather prediction, call for innovations in the existing data assimilation systems. Matrix inversion is a key operation in a majority of data assimilation techniques. Hence, this article presents out-of-core CUDA implementation of an iterative method of matrix inversion. The results show significant speed up for even square matrices of size 1024 X 1024 and more, without sacrificing the accuracy of the results. In a similar test environment, the comparison of this approach with a direct method such as the Gauss-Jordan approach, modified to process large matrices that cannot be processed directly within a single kernel call shows that the former is twice as efficient as the latter. This acceleration is attributed to the division-free design and the embarrassingly parallel nature of every sub-task of the algorithm. The parallel algorithm has been designed to be highly scalable when implemented with multiple GPUs for handling large matrices.

4D Visual Delivery of Big Climate Data: A Fast Web Database Application System

This paper develops a web database application to make space-time 4D visual delivery (4DVD) of big climate data. The delivery system shows climate data in a 4D space-time box and allows users to visualize the data. Users can zoom in or out to help identify desired information for particular locations. Data can then be downloaded for the spatial maps and historical climate time series of a given location after the maps and time series are identified to be useful. These functions enable a user to quickly reach the core interested features without downloading the entire dataset in advance, which saves both time and storage space. The 4DVD system has many graphical display options such as displaying data on a round globe or on a 2D map with detailed background topographic images. It can animate maps and show time series. The combination of these features makes the system a convenient and attractive multimedia tool for classrooms, museums, and households, in addition to climate research scientists, industrial applicants, and policy makers. To demonstrate the 4DVD’s usage, several application examples are included in this paper, such as the La Nina’s influence on land temperature.

A Gaussian process based big data processing framework in cluster computing environment

Machine learning algorithms play a vital role in the prediction of an outbreak of diseases based on climate change. Dengue outbreak is caused by improper maintenance of water storages, lack of urbanization, deforestation, lack of vaccination and awareness. Moreover, a number of dengue cases are varying based on climate season. There is a need to develop the prediction model for modeling the dengue outbreak based climate change. To model the dengue outbreak, Gaussian process regression (GPR) model is applied in this paper that uses the seasonal average of various climate parameters such as maximum temperature, minimum temperature, precipitation, wind, relative humidity and solar. The number of dengue cases and climate data for each block of Tamil Nadu, India are collected from Integrated Disease Surveillance Project and Global Weather Data for SWAT Inc respectively. Local Moran’s I spatial autocorrelation is used in this paper for geographical visualization of hotspot regions. The outbreak of dengue and its hot spot regions are geographically visualized with the help of ArcGIS 10.1 software. The day wise big climate data is collected and stored in the Hadoop cluster computing environment. MapReduce framework is used to reduce the day wise climate data into seasonal climate averages such as winter, summer, and monsoon. The seasonal climate data and number of dengue incidence (health data) are integrated based on the geo-location (latitude and longitude). GPR is used to develop the prediction model for dengue based on the integrated data (climate and health data). The proposed Gaussian process based prediction model is compared with various machine learning approaches such as multiple regression, support vector machine and random forests. Experimental results demonstrate the effectiveness of our Gaussian process based prediction framework.

Disease Surveillance System for Big Climate Data Processing and Dengue Transmission

Ambient intelligence is an emerging platform that provides advances in sensors and sensor networks, pervasive computing, and artificial intelligence to capture the real time climate data. This result continuously generates several exabytes of unstructured sensor data and so it is often called big climate data. Nowadays, researchers are trying to use big climate data to monitor and predict the climate change and possible diseases. Traditional data processing techniques and tools are not capable of handling such huge amount of climate data. Hence, there is a need to develop advanced big data architecture for processing the real time climate data. The purpose of this paper is to propose a big data based surveillance system that analyzes spatial climate big data and performs continuous monitoring of correlation between climate change and Dengue. Proposed disease surveillance system has been implemented with the help of Apache Hadoop MapReduce and its supporting tools.

PolarGlobe: A web-wide virtual globe system for visualizing multidimensional, time-varying, big climate data

ABSTRACTThe increasing research interest in global climate change and the rise of the public awareness have generated a significant demand for new tools to support effective visualization of big climate data in a cyber environment such that anyone from any location with an Internet connection and a web browser can easily view and comprehend the data. In response to the demand, this paper introduces a new web-based platform for visualizing multidimensional, time-varying climate data on a virtual globe. The web-based platform is built upon a virtual globe system Cesium, which is open-source, highly extendable and capable of being easily integrated into a web environment. The emerging WebGL technique is adapted to support interactive rendering of 3D graphics with hardware graphics acceleration. To address the challenges of transmitting and visualizing voluminous, complex climate data over the Internet to support real-time visualization, we develop a stream encoding and transmission strategy based on video-compression techniques. This strategy allows dynamic provision of scientific data in different precisions to balance the needs for scientific analysis and visualization cost. Approaches to represent, encode and decode processed data are also introduced in detail to show the operational workflow. Finally, we conduct several experiments to demonstrate the performance of the proposed strategy under different network conditions. A prototype, PolarGlobe, has been developed to visualize climate data in the Arctic regions from multiple angles.

A high performance query analytical framework for supporting data-intensive climate studies

Climate observations and model simulations produce vast amounts of data. The unprecedented data volume and the complexity of geospatial statistics and analysis requires efficient analysis of big climate data to investigate global problems such as climate change, natural disasters, diseases, and other environmental issues. This paper introduces a high performance query analytical framework to tackle these challenges by leveraging Hive and cloud computing technologies. With this framework, we propose grid transformation, a new perspective for complex climate analysis that applies a series of atomic transformations to terabytes of climate data using SQL-style query (HiveQL). Specifically, we introduce four types of grid transformations (temporal, spatial, local, and arithmetic) to support a broad range of climate analyses, from the basic spatiotemporal aggregation to more sophisticated anomaly detection. Each query is processed as MapReduce tasks in a highly scalable Hadoop cluster as the parallel processing engine. Big climate data are directly stored and managed in a Hadoop Distributed File System without any data format conversion. A prototype is developed to evaluate the feasibility and performance of the framework. Experimental results show that complex and data-intensive climate analysis can be conducted using intuitive SQL queries with good flexibility and performance. This research provides a building block and practical insights in establishing a cyberinfrastructure that provides a high performance and collaborative environment for data-intensive geospatial applications in climate science.

A spatiotemporal indexing approach for efficient processing of big array-based climate data with MapReduce

ABSTRACTClimate observations and model simulations are producing vast amounts of array-based spatiotemporal data. Efficient processing of these data is essential for assessing global challenges such as climate change, natural disasters, and diseases. This is challenging not only because of the large data volume, but also because of the intrinsic high-dimensional nature of geoscience data. To tackle this challenge, we propose a spatiotemporal indexing approach to efficiently manage and process big climate data with MapReduce in a highly scalable environment. Using this approach, big climate data are directly stored in a Hadoop Distributed File System in its original, native file format. A spatiotemporal index is built to bridge the logical array-based data model and the physical data layout, which enables fast data retrieval when performing spatiotemporal queries. Based on the index, a data-partitioning algorithm is applied to enable MapReduce to achieve high data locality, as well as balancing the workload. The proposed indexing approach is evaluated using the National Aeronautics and Space Administration (NASA) Modern-Era Retrospective Analysis for Research and Applications (MERRA) climate reanalysis dataset. The experimental results show that the index can significantly accelerate querying and processing (~10× speedup compared to the baseline test using the same computing cluster), while keeping the index-to-data ratio small (0.0328%). The applicability of the indexing approach is demonstrated by a climate anomaly detection deployed on a NASA Hadoop cluster. This approach is also able to support efficient processing of general array-based spatiotemporal data in various geoscience domains without special configuration on a Hadoop cluster.