Automatic Cloud Research Articles

Diurnal and nocturnal cloud segmentation of all-sky imager (ASI) images using enhancement fully convolutional networks

Abstract. Cloud segmentation plays a very important role in astronomical observatory site selection. At present, few researchers segment cloud in nocturnal all-sky imager (ASI) images. This paper proposes a new automatic cloud segmentation algorithm that utilizes the advantages of deep-learning fully convolutional networks (FCNs) to segment cloud pixels from diurnal and nocturnal ASI images; it is called the enhancement fully convolutional network (EFCN). Firstly, all the ASI images in the data set from the Key Laboratory of Optical Astronomy at the National Astronomical Observatories of Chinese Academy of Sciences (CAS) are converted from the red–green–blue (RGB) color space to hue saturation intensity (HSI) color space. Secondly, the I channel of the HSI color space is enhanced by histogram equalization. Thirdly, all the ASI images are converted from the HSI color space to RGB color space. Then after 100 000 iterative trainings based on the ASI images in the training set, the optimum associated parameters of the EFCN-8s model are obtained. Finally, we use the trained EFCN-8s to segment the cloud pixels of the ASI image in the test set. In the experiments our proposed EFCN-8s was compared with four other algorithms (OTSU, FCN-8s, EFCN-32s, and EFCN-16s) using four evaluation metrics. Experiments show that the EFCN-8s is much more accurate in cloud segmentation for diurnal and nocturnal ASI images than the other four algorithms.

Spectral Albedo Estimation of Snow Covers in Pakistan Using Landsat Data

Accurate snow spectral albedo measurements from satellite data can essentially help to determine the changes over the snow surface area. In the current study, spectral snow albedo is measured over the northern region of Pakistan using spatio-temporal imagery datasets taken by Landsat satellite series equipped with TM and ETM+ sensors. An Automatic Cloud Cover Assessment (ACCA) algorithm is used to mask cloudy pixels from further processing. Furthermore, an image independent model, Second Simulation of the Satellite Signal in the Solar Spectrum (6S) is used in the current research work to atmospherically correct the satellite data. The snow cover albedo of northern Pakistan is estimated using pixel values prior to correction, ACCA output values, and 6S model output values. The results of the study show that highest albedo values are estimated using Landsat band 4 data with albedo model. The results of the study also show that the albedo values measured (band 4) in the year 1992 (0.989) are reduced in the year 2000 (0.931). It is expected that the results of the study could be utilized to predict climatic variations for spring runoff estimation.

Automatic cloud service testing and bottleneck detection system with scaling recommendation

SummaryPerformance problems in a cloud service are difficult to diagnose because they may be caused by various system components. This study proposes an automatic cloud service testing and bottleneck detection system that is applicable to different types of services. With the proposed test module, the user can customize test scenarios to automatically test and collect the corresponding metrics of the target service. Afterward, the proposed bottleneck detection algorithm analyzes the collected metrics and determines whether a bottleneck is presented in the target system. The bottleneck detection module also provides a scaling recommendation for the service provider to facilitate the service system reconfiguration. The experimental results reveal that the proposed system could detect a potential bottleneck in a service system accurately. In accordance with the scaling recommendation, the performance of the target cloud service can be improved efficiently after reconfiguration. Therefore, usage of the proposed system can ensure a high quality of service, and the service level objective could be fulfilled.

Determination of cloud transmittance for all sky imager based solar nowcasting

The demand for accurate solar irradiance nowcast increases together with the rapidly growing share of solar energy within our electricity grids. Intra-hour variabilities, mainly caused by clouds, have a significant impact on solar power plant dispatch and thus on electricity grids. All sky imager (ASI) based nowcasting systems, with a high temporal and spatial resolution, can provide irradiance nowcasts that can help to optimize CSP plant operation, solar power plant dispatch and grid operation. The radiative effect of clouds is highly variable and depends on micro- and macrophysical cloud properties. Frequently, nowcasting systems have to measure/estimate the radiative effect during complex multi-layer conditions with strong variations of the optical properties between individual clouds.We present a novel approach determining cloud transmittance from measurements or from correlations of transmittance with cloud height information. The cloud transmittance is measured by a pyrheliometer when shaded, as the ratio of shaded direct normal irradiance (DNI) and clear sky DNI. However, for most clouds, direct transmittance measurements are not available, as these clouds are not shading the used pyrheliometers. These clouds receive an estimated transmittance value based on (1) their height, (2) results of a probability analysis with historical cloud height and transmittance measurements as well as (3) recent transmittance measurements and their corresponding cloud height. Cloud heights are measured by a stereoscopic approach utilizing two ASIs. We discuss site dependencies of the presented transmittance estimation method and the potential integration of automatic cloud classification approaches.We validated the cloud transmittance estimation over two years (2016 and 2017) and compare the probabilistic cloud transmittance estimation approach with four simple approaches. The overall mean-absolute deviation (MAD) and root-mean-square deviation (RMSD) are 0.11 and 0.16 respectively for transmittance. The deviations are significantly lower for optically thick or thin clouds and larger for clouds with moderate transmittance between 0.18 and 0.585.Furthermore we validated the overall DNI forecast quality of the entire nowcasting system, using this transmittance estimation method, over the same data set with three spatially distributed pyrheliometers. Overall deviations of 13% and 21% are reached for the relative MAD and RMSD with a lead time of 10 min. The effects of the chosen data set on the validation results are demonstrated by means of the skill score.

Advancements in the Aerosol Robotic Network (AERONET) Version 3 database – automated near-real-time quality control algorithm with improved cloud screening for Sun photometer aerosol optical depth (AOD) measurements

Abstract. The Aerosol Robotic Network (AERONET) has provided highly accurate, ground-truth measurements of the aerosol optical depth (AOD) using Cimel Electronique Sun–sky radiometers for more than 25 years. In Version 2 (V2) of the AERONET database, the near-real-time AOD was semiautomatically quality controlled utilizing mainly cloud-screening methodology, while additional AOD data contaminated by clouds or affected by instrument anomalies were removed manually before attaining quality-assured status (Level 2.0). The large growth in the number of AERONET sites over the past 25 years resulted in significant burden to the manual quality control of millions of measurements in a consistent manner. The AERONET Version 3 (V3) algorithm provides fully automatic cloud screening and instrument anomaly quality controls. All of these new algorithm updates apply to near-real-time data as well as post-field-deployment processed data, and AERONET reprocessed the database in 2018. A full algorithm redevelopment provided the opportunity to improve data inputs and corrections such as unique filter-specific temperature characterizations for all visible and near-infrared wavelengths, updated gaseous and water vapor absorption coefficients, and ancillary data sets. The Level 2.0 AOD quality-assured data set is now available within a month after post-field calibration, reducing the lag time from up to several months. Near-real-time estimated uncertainty is determined using data qualified as V3 Level 2.0 AOD and considering the difference between the AOD computed with the pre-field calibration and AOD computed with pre-field and post-field calibration. This assessment provides a near-real-time uncertainty estimate for which average differences of AOD suggest a +0.02 bias and one sigma uncertainty of 0.02, spectrally, but the bias and uncertainty can be significantly larger for specific instrument deployments. Long-term monthly averages analyzed for the entire V3 and V2 databases produced average differences (V3–V2) of +0.002 with a ±0.02 SD (standard deviation), yet monthly averages calculated using time-matched observations in both databases were analyzed to compute an average difference of −0.002 with a ±0.004 SD. The high statistical agreement in multiyear monthly averaged AOD validates the advanced automatic data quality control algorithms and suggests that migrating research to the V3 database will corroborate most V2 research conclusions and likely lead to more accurate results in some cases.

基于曲率突变分析的点云特征线自动提取

基于曲率突变分析的点云特征线自动提取

Improving NDVI by removing cirrus clouds with optical remote sensing data from Landsat-8 – A case study in Quito, Ecuador

The Andean region has a high cloud density throughout the year. The use of optical remote sensing data in the computation of environmental indices of this region has been hampered by the presence of clouds. To maximize accuracy in the computation of several environmental indices including the normalized difference vegetation index (NDVI), we compared the performance of two algorithms in removing clouds in Landsat-8 Operational Land Imager (OLI) data of a high-elevation area. The study area was Quito, Ecuador, which is a city located close to the equator and in a high-elevation area crossed by the Andes Mountains. The first algorithm was the automatic cloud removal method (ACRM), which employs a linear regression between the different spectral bands and the cirrus band. The second algorithm was independent component analysis (ICA), which considers the noise (clouds) as part of independent components applied over the study area. These methods were evaluated based on several images from different years with different cloud conditions. The results indicate that neither algorithm is effective over this region for the removal of clouds or for NDVI computation. However, after improving ACRM, the NDVI computed using ACRM showed a better correlation than ICA with the MODIS NDVI product.

Automatic cloud resource management for interactive remote geovisualization

AbstractRemote geovisualization has gained momentum to support large‐scale geospatial data analysis and complex decision‐making over the last few years. Cloud computing, due to its capabilities to deliver on‐demand computing resources, has been embraced to develop and deploy interactive and scalable remote geovisualization applications. However, current cloud computing frameworks do not offer a versatile resource management scheme that is readily applicable for online remote visualization services, which usually require maintaining a satisfactory service level over time under dynamic workloads. To address this gap, we propose an automatic cloud resource management approach based on a bi‐level scheduling and horizontal scaling scheme to exploit cloud resources efficiently. At the lower level, a dynamic task‐scheduling scheme using collaborative filtering techniques is proposed to allocate virtual cloud resources to execute sub‐tasks. The scheduling scheme considers spatio‐temporal patterns presented in visualization views. At the upper level, reinforcement learning is adopted to perform resource auto‐scaling based on a reward function that integrates three different facets, namely: time cost, resource cost, and service stability. The original reinforcement learning algorithm is improved in two main aspects: (1) considering the delay of resource provisioning that is common in cloud environments; and (2) using online Gaussian estimation to estimate Q values. Task scheduling and auto‐scaling interact with each other and are integrated to deliver a comprehensive and responsive resource management solution. Experimental results demonstrate that our approach outperforms several existing cloud resource management methods. The proposed approach is also applicable for other interactive visualization applications, which have similar workload characteristics and performance requirements as interactive remote geovisualization.

Cloud classification of ground-based infrared images combining manifold and texture features

Abstract. Automatic cloud type recognition of ground-based infrared images is still a challenging task. A novel cloud classification method is proposed to group images into five cloud types based on manifold and texture features. Compared with statistical features in Euclidean space, manifold features extracted on symmetric positive definite (SPD) matrix space can describe the non-Euclidean geometric characteristics of the infrared image more effectively. The proposed method comprises three stages: pre-processing, feature extraction and classification. Cloud classification is performed by a support vector machine (SVM). The datasets are comprised of the zenithal and whole-sky images taken by the Whole-Sky Infrared Cloud-Measuring System (WSIRCMS). Benefiting from the joint features, compared to the recent two models of cloud type recognition, the experimental results illustrate that the proposed method acquires a higher recognition rate with an increase of 2 %–10 % on the ground-based infrared datasets.

FFT features and hierarchical classification algorithms for cloud images

Cloud-type recognition is useful in preventing losses caused by adverse weather conditions. This paper presents a methodology and algorithms for automatic recognition of cloud types from color cloud images taken from the ground. We propose a number of eight algorithms for automatic cloud classification of seven cloud types defined by meteorological organization. Recognition is based on a number of features extracted from images which are related to color, texture, and shape. We introduce three new features based on the Fourier transform, namely, the modified k-FFTPX, the half k-FFTPX, and the h×k-FFT. The classification technique is based on artificial neural network (ANN) with tree algorithm to extract features. The proposed classification tree algorithm uses the technique called hierarchical classification which is composed of three levels of tree. Each level of the tree is capable of classifying up to four classes. We show that this method provides the highest accuracy at 98.08% through a series of four experiments. For accuracy assessment, each experiment splits dataset into train and test images using Leave-One-Out Cross-Validation (LOOCV). The result confirms that the hierarchical classification performs better than a single classification. In addition, the tree can be adapted to classify lesser number of cloud types. Our experiment reveals that the accuracy for classifying two classes, cloud and no-cloud, is high as 100%.

A Cloud Detection Algorithm for MODIS Images Combining Kmeans Clustering and Otsu Method

This paper proposes a method to choose the initial clustering center for the kmeans algorithm and recognize the clustering results without artificial interpretation: Otsu method are used on the MODIS data to determine cloud pixels and not cloud pixels. Take the nearest pixels from each class’s mean value as the initial clustering center of the kmeans algotrithm and determine the category of the clustering results according to the category of the initial clustering center. Theoretically, the method reduce the error caused by the random choice of the initial clustering center in traditional kmeans algorithm and realized automatic cloud detection without artificial interpretation. The experimental results verify the effectiveness of the proposed method.

THIN CLOUD DETECTION METHOD BY LINEAR COMBINATION MODEL OF CLOUD IMAGE

Abstract. The existing cloud detection methods in photogrammetry often extract the image features from remote sensing images directly, and then use them to classify images into cloud or other things. But when the cloud is thin and small, these methods will be inaccurate. In this paper, a linear combination model of cloud images is proposed, by using this model, the underlying surface information of remote sensing images can be removed. So the cloud detection result can become more accurate. Firstly, the automatic cloud detection program in this paper uses the linear combination model to split the cloud information and surface information in the transparent cloud images, then uses different image features to recognize the cloud parts. In consideration of the computational efficiency, AdaBoost Classifier was introduced to combine the different features to establish a cloud classifier. AdaBoost Classifier can select the most effective features from many normal features, so the calculation time is largely reduced. Finally, we selected a cloud detection method based on tree structure and a multiple feature detection method using SVM classifier to compare with the proposed method, the experimental data shows that the proposed cloud detection program in this paper has high accuracy and fast calculation speed.

A Cloud height observing system using high resolution whole sky image

Several methods were known to observe the cloud height. Since the meteorological administration mainly has been using human eye method, an automatic cloud height observer is proposed to reduce human errors.The proposed cloud observer system is for the cloud automatic observation system which has been operating at weathers stations in Korea since 1993. The high hand camera is adapted to get high resolution images without a capture board that has the low resolution. And the man-made passive filter for extracting sky reign is replaced by the active filter using image processing. The famous BRISK matching is selected instead of the block mating to get more precise disparity form a pair of stereo image.The Cloud cover announced by the Korean Meteorological Agency is the eye observation result nowadays, even though an intelligent cloud observation system was supplied. The reason is the impermissible error in the observed data of the automatic observation system. The current system is applied a man-made passive filter has to be changed periodically due to the boundary transition, but the proposed system is maintenance-free, since the active filter using edge detection algorithm automatically adjusts itself. The measuring of disparity which is important to analyze spatial information, has improved up to the resolution of observing height is 100m by getting image data directly form the high resolution camera, the old resolution from the capture board. Also, the disparity analyzing method is changed to the BRISK matching algorithm instead of the block mating algorism to get more deliberate disparity analysis since the dimension of the block is too big relatively compare to the cloud in the stereo all sky image. The test results are acceptable range which is human observation value ± 10% since the resolution of observing height is 100m at lower cloud in the sky. The proposed system is conducting the comparison research to compare with the result of reading satellite images and ceilometer.

Stochastic Model for Generation of High-Resolution Irradiance Data and Estimation of Power Output of Photovoltaic Plants

This paper presents a method for the generation of synthetic time series of solar radiation with a one-minute time resolution using an automatic cloud classification procedure. Solar radiation data from ground-based measurements are used to determine the correlation between the irradiance and the prevailing cloud classes extracted from satellite images. A specific set of Markov chains for each cloud class is adjusted empirically and used for a stochastic simulation of clear sky index time series. The different models are then combined to convert the irradiance into power output time series of photovoltaic power plants of different sizes. Tests to determine the methodology's performance showed positive results in reproducing the statistical characteristics of observed time series data. Estimates of a selected set of metrics were obtained for several sites in Brazil and allowed the characterization of the solar resource for photovoltaic plants of various sizes and mounting methods considering the generation potential and short-term variability.

First results of cirrus clouds properties by means of a pollyxt raman lidar at two measurement sites

Geometrical and optical characteristics of cirrus clouds using Raman lidar PollyXT measurements at different locations are presented. The PollyXT has been participated in two long-term experimental campaigns, one close to New Delhi in India and one at Elandsfontein in South Africa, providing continuous measurements and covering a wide range of cloud types. First results of cirrus cloud properties at different latitudes, as well as their temporal distributions are presented in this study. An automatic cirrus clouds detection algorithm is applied based on the wavelet covariance transform. The measurements at New Delhi performed from March 2008 to February 2009, while at Elandsfontein measurements were performed from December 2009 to January 2011.

Automatic cloud service monitoring and management with prediction-based service provisioning

As the number of users is increasing day by day over the cloud environment, service providers might be unable to offer service as per the agreement. Hence, cloud services being offered should be monitored to detect service level agreement violation. Most existing cloud service monitoring frameworks are developed to support the service provider only, which raises the question of correctness and fairness. On the other hand, if monitoring is applied at user side, then it creates overhead to the clients. To manage such issue, ontology-based automatic cloud services monitoring and management with prediction-based service provisioning approach is proposed in this paper. In this approach, monitoring is performed at cloud broker, which is an intermediate entity between the user and service provider. When service level agreement violation is detected, it sends alert to both clients and service providers, and broker automatically reschedules the tasks to reduce the further violations.

Automatic cloud detection for GF-4 series images

Automatic cloud detection for GF-4 series images

Automatic cloud service monitoring and management with prediction-based service provisioning

Automatic cloud service monitoring and management with prediction-based service provisioning

A Cloud Detection Method Based on Relationship Between Objects of Cloud and Cloud-Shadow for Chinese Moderate to High Resolution Satellite Imagery

Cloud detection of satellite imagery is very important for quantitative remote sensing research and remote sensing applications. However, many satellite sensors do not have enough bands for a quick, accurate, and simple detection of clouds. Particularly, the newly launched moderate to high spatial resolution satellite sensors of China, such as the charge-coupled device on-board the Chinese Huan Jing 1 (HJ-1/CCD) and the wide field of view (WFV) sensor on-board the Gao Fen 1 (GF-1), only have four available bands including blue, green, red, and near infrared bands, which are far from the requirements of most could detection methods. In order to solve this problem, an improved and automated cloud detection method for Chinese satellite sensors called object-oriented cloud and cloud-shadow matching method (OCM) is presented in this paper. It first modified the automatic cloud cover assessment (ACCA) method, which was developed for Landsat-7 data, to get an initial cloud map. The modified ACCA method is mainly based on threshold and different threshold settings produce different cloud maps. Subsequently, a strict threshold is used to produce a cloud map with high confidence and large amount of cloud omission and a loose threshold is used to produce a cloud map with low confidence and large amount of commission. Second, a corresponding cloud-shadow map is also produced using the threshold of near-infrared band. Third, the cloud maps and cloud-shadow map are transferred to cloud objects and cloud-shadow objects. Cloud and cloud-shadow are usually in pairs; consequently, the final cloud and cloud-shadow maps are made based on the relationship between cloud and cloud-shadow objects. The OCM method was tested using almost 200 HJ-1/CCD and GF-1/WFV images across China and the overall accuracy of cloud detection is close to 90%.

Automatic Cloud‐Type Classification Based On the Combined Use of a Sky Camera and a Ceilometer

AbstractA methodology, aimed to be fully operational, for automatic cloud classification based on the synergetic use of a sky camera and a ceilometer is presented. The random forest machine learning algorithm was used to train the classifier with 19 input features: 12 extracted from the sky camera images and 7 from the ceilometer. The method was developed and tested based on a set of 717 images collected at the radiometric stations of the Univ. of Jaén (Spain). Up to nine different types of clouds (plus clear sky) were considered (clear sky, cumulus, stratocumulus, nimbostratus, altocumulus, altostratus, stratus, cirrocumulus, cirrostratus, and cirrus) plus an additional category multicloud, aiming to account for the frequent cases in which the sky is covered by several cloud types. A total of eight experiments was conducted by (1) excluding/including the ceilometer information, (2) including/excluding the multicloud category, and (3) using six or nine different cloud types, aside from the clear‐sky and multicloud category. The method provided accuracies ranging from 45% to 78%, being highly dependent on the use of the ceilometer information. This information showed to be particularly relevant for accurately classifying “cumuliform” clouds and to account for the multicloud category. In this regard, the camera information alone was found to be not suitable to deal with this category. Finally, while the use of the ceilometer provided an overall superior performance, some limitations were found, mainly related to the classification of clouds with similar cloud base height and geometric thickness.