Usability of Broadband Performance Measurements for Statistical Surveys

Abstract

This paper addresses the usability of a broadband performance measurement application for end users, as introduced in Europe since 2003 by iPing Research B.V. The small Application, executes various automated performance measurements at regular intervals in the background of a Laptop/PC and data was acquired on bandwidth, price, technology, location, as well as voluntarily supplied gender and age of the App-installer. Panels were acquired in cooperation with Consumer Associations and by general magazine publicity (self-selection), and via a market research bureau (a-select panel).The dataset for the Netherlands for 2003-2010 is compared with data acquired via household surveys by Statistics Netherlands (CBS) and regulatory filings of ISPs to Dutch regulator OPTA. We evaluated the representativeness for the self-select and a-select panels from iPing Research by comparing them with the results for a-select CBS household survey panels on geography, technology, gender and age as well as broadband capacities from the OPTA filings. Using Chi-square testing for geography and technology, we had to reject the hypothesis that iPing and CBS data measure the same population for iPing’s self-select panels. Instead iPing’s a-select panel was within 95% confidence limits of CBS. It was found that in the self-select panels users of higher bitrate offerings are overrepresented, as well as males and persons aged 30 – 60. However, there was no significant difference in gender for average measured bitrates.A comparison of these outcomes and performance measurements results from Speedtest.net and Google M-Labs tools for the Netherlands, reveal implications for proper panel selection that extends to hardware based performance measurements now operational in the UK, USA and EU, as they also use volunteers and risk the found self-selection bias.The recommendation for good statistical data quality on broadband is joint use of a performance measurement tool in combination with a survey among an a-select drawn panel. This allows for the actual broadband access in use to be verified through measurement as part of the survey answer and then a proper weighing of the panel data. Software tools included in an online survey have attractions in cost and less intrusiveness in comparison to hardware.Funding for this research has been provided by the Statistics Netherlands program Impact ICT and the Netherlands Ministry of Economic affairs, Agriculture and Innovation.