Evaluating fidelity of lossy compression on spatiotemporal data from an IoT enabled smart farm

Abstract

As the volume of data collected by various IoT sensors used in smart farm applications increases, the storing and processing of big data for agricultural applications become a huge challenge. The insight of this paper is that lossy compression can unleash the power of compression to IoT because, as compared with its counterpart (a lossless one), it can significantly reduce the data volume when the spatiotemporal characteristics of IoT sensor data are properly exploited. However, lossy compression faces the challenge of compressing too much data thus losing data fidelity, which might affect the quality of the data and potential analytics outcomes. To understand the impact of lossy compression on IoT data management and analytics, we evaluated four classification algorithms with reconstructed agricultural sensor data based on various energy concentration. Specifically, we applied three transformation-based lossy compression mechanisms to five real-world weather datasets collected at different sampling granularities from IoT weather stations. Our experimental results indicate that there is a strong positive correlation between the concentrated energy of the transformed coefficients and the compression ratio as well as the data quality. While we observed a general trend where much higher compression ratios can be achieved at the cost of a decrease in quality, we also observed that the impact on the classification accuracy varies among the data sets and algorithms we evaluated. Lastly, we show that the sampling granularity also influences the data fidelity in terms of the prediction performance and compression ratio.