Abstract
Abstract. This paper discusses a current issue for several experimental science disciplines, which is the Big Data Problem (BDP). This research study focused on light intensity and ranging (LiDAR) datasets, which are collected for modelling spatial features found on the surface of the earth. Currently, LiDAR datasets are known to be extremely redundant for many applications. Using a formula that allows for calculating the variance of the target-induced error (so-called T-error) caused by the discretisation and quantisation of a 3D surface as a criterion for the quantitative assessment of the fidelity of a model, the use of a Q-tree-based split of the surface is proposed for cells of various sizes depending on the fidelity requirements. A LiDAR dataset representing a 1 km x 1 km terrain surface tile using approximately 12 x 106 points was used during the experiments. The initial LiDAR dataset was used to produce a digital terrain model (DTM) at a 0.5 m x 0.5 m resolution, which was used as a reference model. Subsequently, the initial LiDAR dataset was decimated at various rates, and the resulting DTMs were compared with the reference model. The Q-tree based data structure was utilised to illustrate that the Q-tree approach allows for the production of DTMs at a ‘controlled’ fidelity with a considerable reduction in data volume.
Highlights
The Big Data Problem (BDP) is an unwelcome by-product of the acquisition of spatial data at continuously increasing spatial, spectral radiometric and temporal resolution levels (e.g., Jianping et al 2009)
One way to reduce the impact of the BDP on business and science fields is to use the precise amount of data that is necessary for a given task, perhaps by observing a certain ‘safety margin’
This is only possible if a criterion exists to effectively translate the assumed fidelity or accuracy level into a procedure to select a subset of data required for the task
Summary
The Big Data Problem (BDP) is an unwelcome by-product of the acquisition of spatial data at continuously increasing spatial, spectral radiometric and temporal resolution levels (e.g., Jianping et al 2009). One way to reduce the impact of the BDP on business and science fields is to use the precise amount of data that is necessary for a given task, perhaps by observing a certain ‘safety margin’. This is only possible if a criterion exists to effectively translate the assumed fidelity or accuracy level into a procedure to select a subset of data required for the task. This allows for an increased fidelity of reality modelling; the volume of data produced by the state-of-the-art equipment significantly contributes to the increasing challenge related to data storage and processing time as well as to the management and dissemination of data, which is known as the BDP
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
