Mobile Web Surveys Research Articles

Protocol for a web survey experiment studying the feasibility of asking respondents to capture and submit photos of the books they have at home and the resulting data quality.

This document presents the protocol of a study conducted as a part of the WEB DATA OPP project, which is funded by the H2020 program. The study aimed to investigate different aspects of the collection of images through web surveys. To do this, we implemented a mobile web survey in an opt-in online panel in Spain. The survey had various questions, some of which were about the books that the participants have at their main residence. The questions related to books were asked in three different ways: regular survey questions showing visual examples of how different numbers of books fit in a 74 centimetre wide shelf depending on their thickness, regular survey questions without the visual examples, and questions where participants were asked to send photos of the books at their home. This report explains how the study was designed and conducted. It covers important aspects such as the experimental design, the questionnaire used, the characteristics of the participants, ethical considerations, and plans for disseminating the results.

Display of battery items in web and mail surveys: Grids versus item-by-item and radio versus wide buttons

How to display questions that are part of a battery in self-administered surveys is an important decision. Battery items may be displayed in a grid in a mail survey or computer web survey, but are often displayed as individual items on mobile devices. Although past research has compared grids to item-by-item displays in computer and mobile web surveys, almost no work has compared these displays in mail surveys. Additionally, many web survey templates use wide rectangular buttons to select response options in individual items using a mobile-optimized design, different from the standard round answer space format typically used in mail surveys. In this study, we experimentally test grid versus item-by-item displays and round radio buttons versus wide rectangular buttons for battery items in a probability-based general population mixed-mode mail + web survey of adults in Nebraska. Consistent with past research, we find that item-by-item displays reduce straightlining rates compared to grid designs. We also find that respondents are less likely to select the last two response categories in the item-by-item displays than in the grid displays. Smartphone and computer web respondents have higher item nonresponse rates than mail respondents, and web respondents have lower straightlining rates than mail respondents, accounting for respondent characteristics. Reassuringly, there is no difference in data quality outcomes across radio button versus wide button formats. These findings replicate past research that item-by-item displays reduce straightlining but may shift answer categories. These findings suggest that questionnaire designers can combine round radio button answer spaces on mail surveys with wide buttons on web surveys on battery items with little difference in data quality.

Rapid assessments of the impact of COVID-19 on the availability of quality seed to farmers: Advocating immediate practical, remedial and preventative action

Rapid assessments of the impacts of the COVID-19 crisis on the seed sector were conducted by a coalition of partners in Ethiopia, Myanmar, Nigeria, and Uganda in May and June 2020. The method was rapid, iterative, inclusive and valuable in revealing threats to the availability and timely access of farmers to quality seed and to food, nutrition and income security, and in advocating for remedial and preventative action. Via mobile application and web survey, and focus group discussions on virtual conferencing platforms a panel of 36 or more local experts operating particularly in formal seed systems in each country identified potential disruptions to activities in the seed sector and recommended immediate practical action to ensure continuity in performance. Recommendations, and the stakeholders best positioned to propel their action, were proposed to and approved by senior leadership in the sector. The entire process from survey to publication of a seed alert in each iteration was completed within two weeks. Due to the highly seasonal nature of agriculture, and recognition that activities are time-bound, quick turnaround on assessments was essential. Dashboards indicated where impact was felt the hardest, also showing how dynamic the situation was. Countries were at different stages in their agricultural seasons, which made the data highly contextual, but also interesting for getting a glimpse into the future. Lessons were offered from one country to another. Reduced mobility was the root cause of many disruptions in supplying seed to farmers. Disruptions caused seed and related industry to operate at reduced capacity. The cost of transactions and doing business during these times may have increased the scarcity and price of inputs beyond what farmers can recover. Sales of quality seed in formal markets were perceived to decline due to delays in distribution, weakened promotion efforts and fewer farmers present. Farmers are less likely to benefit from investments in crop improvement for more seasons to come due to delays in the development and release of new varieties. Social distancing prevents stakeholders from meeting to exchange goods, services and information, but the sector is gradually getting up to speed with information technology. For all concerns, practical options were offered and often implemented. The pandemic has exacerbated structural weaknesses in the organization of the seed sector, for which reforms are not only justified, but overdue.

Using emojis in mobile web surveys for Millennials? A study in Spain and Mexico

To involve Millennials in survey participation, and obtain high-quality answers from them, survey designers may require new tools that better catch Millennials' interest and attention. One key new tool that could improve the communication and make the survey participation more attractive to young respondents are the emojis. We used data from a survey conducted among Millennials by the online fieldwork company Netquest in Spain and Mexico (n = 1614) to determine how emojis can be used in mobile web surveys, in particular in open-ended questions, and how their use can affect data quality, completion time, and survey evaluation. Overall, results show a high willingness of Millennials to use emojis in surveys (both stated and actual use) and a positive impact of encouraging Millennials to use emojis in open-ended questions on the amount of information conveyed, the completion time and the survey enjoyment.

Using Buttons as Response Options in Mobile Web Surveys

Survey designers are starting to optimize online questionnaires for smartphones by enlarging the control elements used to accept answers (radio buttons and checkboxes) or by replacing these controls with alternatives such as stacked groups of wide buttons, but these approaches have not yet been carefully evaluated. In the lab experiment reported here, 61 older adults (age 59+) were randomly assigned to complete a survey on an iPhone with response options displayed either with conventional controls (radio buttons and checkboxes) or one of three contemporary formats: larger controls, larger controls enclosed in wide buttons, and wide buttons without any controls. The three optimization approaches were equally effective at improving respondents’ tapping accuracy but were not equally rated in post-survey evaluations. Respondents tended to prefer the larger HTML-style controls over wide buttons. The results suggest that one simple way for researchers to effectively optimize their questionnaires for smartphones is to increase the size of conventional input elements without changing their basic shape or design.

Using Smartphone Technology for Research on Refugees: Evidence from Germany

Researchers attempting to survey refugees over time face methodological issues because of the transient nature of the target population. In this article, we examine whether applying smartphone technology could alleviate these issues. We interviewed 529 refugees and afterward invited them to four follow-up mobile web surveys and to install a research app for passive mobile data collection. Our main findings are as follows: First, participation in mobile web surveys declines rapidly and is rather selective with significant coverage and nonresponse biases. Second, we do not find any factor predicting types of smartphone ownership, and only low reading proficiency is significantly correlated with app nonparticipation. However, obtaining sufficiently large samples is challenging—only 5 percent of the eligible refugees installed our app. Third, offering a 30 Euro incentive leads to a statistically insignificant increase in participation in passive mobile data collection.

SurveyMotion: what can we learn from sensor data about respondents’ completion and response behavior in mobile web surveys?

ABSTRACTParticipation in web surveys via smartphones increased continuously in recent years. The reasons for this increase are a growing proportion of smartphone owners and an increase in mobile Internet access. However, research has shown that smartphone respondents are frequently distracted and/or multitasking, which might affect completion and response behavior in a negative way. We propose ‘SurveyMotion (SMotion)’, a JavaScript-based tool for mobile devices that can gather information about respondents’ motions during web survey completion by using sensor data. Specifically, we collect data about the total acceleration (TA) of smartphones. We conducted a lab experiment and varied the form of survey completion (e.g. standing or walking). Furthermore, we employed questions with different response formats (e.g. radio buttons and sliders) and measured response times. The results reveal that SMotion detects higher TAs of smartphones for respondents with comparatively higher motion levels. In addition, respondents’ motion level affects response times and the quality of responses given. The SMotion tool promotes the exploration of how respondents complete mobile web surveys and could be employed to understand how future mobile web surveys are completed.

Answering Mobile Surveys With Images: An Exploration Using a Computer Vision API

Most mobile devices nowadays have a camera. Besides, posting and sharing images have been found as one of the most frequent and engaging Internet activities. However, to our knowledge, no research has explored the feasibility of asking respondents of online surveys to upload images to answer survey questions. The main goal of this article is to investigate the viability of asking respondents of an online opt-in panel to upload during a mobile web survey: First, a photo taken in the moment, and second, an image already saved on their smartphone. In addition, we want to test to what extent the Google Vision application programming interface (API), which can label images into categories, produces similar tags than a human coder. Overall, results from a survey conducted among millennials in Spain and Mexico ( N = 1,614) show that more than half of the respondents uploaded an image. Of those, 77.3% and 83.4%, respectively, complied with what the question asked. Moreover, respectively, 52.4% and 65.0% of the images were similarly codified by the Google Vision API and the human coder. In addition, the API codified 1,818 images in less than 5 min, whereas the human coder spent nearly 35 hours to complete the same task.

Grid and Item-by-Item Formats in PC and Mobile Web Surveys

While grids or matrix questions are a widely used format in PC web surveys, there is no agreement on the format in mobile web surveys. We conducted a two-wave experiment in an opt in panel in Russia, varying the question format (grid format and item-by-item format) and device respondents used for survey completion (smartphone and PC). The 1,678 respondents completed the survey in the assigned conditions in the first wave and 1,079 in the second wave. Overall, we found somewhat higher measurement error in the grid format in both mobile and PC web conditions. We found almost no significant effect of the question format on test–retest correlations between the latent scores in two waves and no differences in breakoff rates between the question formats. The multigroup comparison showed some measurement equivalence between the question formats. However, the difference varied depending on the length of a scale with a longer scale producing some differences in the measurement equivalence between the conditions. The levels of straightlining were higher in the grid than in the item-by-item format. In addition, concurrent validity was lower in the grid format in both PC and mobile web conditions. Finally, subjective indicators of respondent burden showed that the grid format increased reported technical difficulties and decreased subjective evaluation of the survey.

Mobile Web Survey in the International Setting (Presentation Slides)

There is no doubt that more people are taking web surveys using their mobile devices. In non-English speaking countries, in particular, more respondents complete online surveys using mobile devices than using desktop or laptop computers. This phenomenon has a huge implication on survey research as different devices are associated with different survey measurement and nonresponse errors, which in turn could affect the survey estimates themselves. Also, data quality, as measured by survey satisficing theory, could also be affected by the survey-taking device. An increasing amount of research has been devoted to examining and comparing survey data collected from mobile and non-mobile devices. This study will expand the existing literature by examining mobile web survey data from six countries, up to three samples within each country, and across three waves of data collection. Specifically, the study was conducted in the U.S., U.K. Australia, China, Brazil, and India. Within each country, data were collected through up to three online panel providers, including SurveyMonkey Audience, Cint, and TapResearch. In addition, for each country/sample combination, three waves of data collection were performed, with each wave one month apart. The samples were independent across the three waves. Respondents could choose the device (as opposed to been assigned to) for taking the survey. The same questionnaire (except minor changes to the demographic questions) was used in the study. In the presentation, we will present whether the mobile web usage different by country and sample. Also, we will show the demographic and data quality differences, if any, between mobile and non-mobile respondents by country and sample. This is the first study of this kind that examines the mobile web in such large scale and cross-cultural perspective.

Device and Internet Use Among Spanish-Dominant Hispanics: Implications for Web Survey Design and Testing

This paper focuses on the lessons learned for design and administration of mobile Web surveys for the Spanish-dominant Hispanic population in the United States. The scholarship on completion of Web surveys has recently started to examine Web usage on mobile devices and Internet browsing patterns to better understand the behavior of mobile users. Yet, we still know little about how Spanish-dominant Hispanics in the United States use their electronic devices to answer mobile Web surveys. In this piece, we use findings from usability and cognitive interviews conducted in 2015. The interviews were part of U.S. Census Bureau tests that will inform design decisions for the 2020 Census. For this project, we recruited and interviewed Hispanic or Latino participants who reported not speaking English well, owned smartphones or tablets, reported accessing the Internet with those devices, and reported feeling somewhat or very comfortable using the Internet. We found that access to an electronic device such as those used for completing Web surveys (i.e., computers, smartphones, tablets) did not necessarily indicate full familiarity with the devices, with Internet activities (e.g., browsing, use of Internet applications) or with responding to mobile Web surveys. Moreover, we found evidence that most participants used a very limited number of Internet applications (e.g., only Facebook or Google). Future research should focus on these issues and their implications for the design of Web surveys. Advancing research on these issues is important for mobile-dependent populations who are still not familiar with answering Web surveys in general.

Web Versus Mobile Web

Due to a rising mobile device penetration, Web surveys are increasingly accessed and completed on smartphones or tablets instead of desktop computers or laptops. Mobile Web surveys are also gaining popularity as an alternative self-administered data collection mode among survey researchers. We conducted a methodological experiment among iPhone owners and compared the participation and response behavior of three groups of respondents: iPhone owners who started and completed our survey on a desktop or laptop PC, iPhone owners who self-selected to complete the survey on an iPhone, and iPhone owners who started on a PC but were requested to switch to iPhone. We found that respondents who completed the survey on a PC were more likely to be male, to have a lower educational level, and to have more experience with Web surveys than mobile Web respondents, regardless of whether they used the iPhone voluntarily or were asked to switch from a PC to an iPhone. Overall, iPhone respondents had more missing data and took longer to complete the survey than respondents who answered the questions on a PC, but they also showed less straightlining behavior. There are only minimal device differences on survey answers obtained from PCs and iPhones.

Smartphones vs PCs: Does the Device Affect the Web Survey Experience and the Measurement Error for Sensitive Topics? - A Replication of the Mavletova & Couper’s 2013 Experiment

More and more respondents use mobile devices to complete web surveys. These devices have different characteristics, if compared to PCs (e.g. smaller screen sizes and higher portability). These characteristics can affect the survey responses, mostly when a questionnaire includes sensitive questions. This topic was already studied by Mavletova and Couper (2013), through a two-wave experiment comparing PCs and mobile devices results for the same respondents in a Russian opt-in panel. We replicated this cross-over design, focusing on an opt-in panel for Spain, involving 1,800 panellists and comparing PCs and smartphones. Our results support most of Mavletova and Couper’s (2013) findings (e.g. generally the used device does not significantly affect the reporting of sensitive information), confirming their robustness over the two studied countries. For other results (e.g. trust in data confidentiality), we found differences that can be justified by the diverse context/culture or by the quick changes that are still characterizing the mobile web survey participation.

Grouping of Items in Mobile Web Questionnaires

There is some evidence that a scrolling design may reduce breakoffs in mobile web surveys compared to a paging design, but there is little empirical evidence to guide the choice of the optimal number of items per page. We investigate the effect of the number of items presented on a page on data quality in two types of questionnaires: with or without user-controlled skips. Three versions of a 30-item instrument were compared, with 5, 15, or all 30 questions presented on a page, in two different surveys, one with skips and one without. We found that displaying 30 items on a page reduced the breakoff rate by almost one-third compared to presenting five items per page in the questionnaire without skips, but the difference was not statistically significant. In both surveys with and without skips, the completion times were significantly lower in the 30-item per page condition; however, item nonresponse rates were also higher. We give some practical recommendations to guide choices while designing questionnaires for mobile web surveys.

The Current State of Affairs in Mobile Market Research

The aim of this working paper is to give an overview over the current state of affairs on the topic of Mobile Market Research (MMR). At first, recent trends in stationary and mobile internet usage and the ownership of different web-enabled devices in Germany are depicted. It is carved out that the rise in mobile device ownership and mobile internet usage might have similar impact on the global market research industry as the introduction of the stationary internet at the end of the last millennium. Moreover, a first systematization of the extensive field of MMR is given. Thereafter, the advantages and disadvantages of different web survey modes will be discussed. At first, focus is put on web surveys to which the largest potential in the market research mass market has been attributed. Here, the advantages and disadvantages of stationary web surveys are highlighted in comparison to other, already established survey modes. Thereafter, the various advantages and disadvantages of mobile web surveys are similarly discussed and compared to stationary web surveys. At this, it is pointed out that mobile web surveys differ from their stationary counterparts. Hence, research results regarding the latter cannot be transferred to the prior and vice versa. Moreover, possible areas of application of MMR are introduced to the reader. Finally, the new field of MMR is identified as an extensive research gap in the large body on survey research literature. This gap will then be narrowed down to outcome rates of surveys and individual participation as primary future research targets. At the end of the chapter a literature review on outcome rates in MMR will be conducted. At this, special focus is put on the shortcomings of past studies which a future best practice shall overcome.

Mobile Web Survey Design: Scrolling versus Paging, SMS versus E-mail Invitations

There is some evidence that questionnaire design (scrolling or paging) and invitation mode (SMS or e-mail) have an impact on response rates in web surveys completed on personal computers (PCs). This paper examines whether these findings can be generalized to mobile web surveys. First, we explore the effect of scrolling versus paging design on the breakoff rate, item nonresponse, and completion time in mobile web surveys. Second, we investigate which type of invitation and reminder mode (SMS or e-mail) is more effective in terms of producing higher participation rates and maximizing the percentage of respondents who complete the survey via a mobile device rather than a PC. The paper summarizes the results of an experiment conducted among members of a volunteer online access panel in Russia, who were asked to complete the survey using a mobile device. We find that the scrolling design leads to significantly faster completion times, lower (though not significantly lower) breakoff rates, fewer technical problems, and higher subjective ratings of the questionnaire. We also find that SMS invitations are more effective than e-mail invitations in mobile web surveys.

Sensitive Topics in PC Web and Mobile Web Surveys: Is There a Difference?

A large number of findings in survey research suggest that misreporting in sensitive questions is situational and can vary in relation to context. The methodological literature demonstrates that social desirability biases are less prevalent in self-administered surveys, particularly in Web surveys, when there is no interviewer and less risk of presenting oneself in an unfavorable light. Since there is a growing number of users of mobile Web browsers, we focused our study on the effects of different devices (PC or cell phone) in Web surveys on the respondents’ willingness to report sensitive information. To reduce selection bias, we carried out a two-wave cross-over experiment using a volunteer online access-panel in Russia. Participants were asked to complete the questionnaire in both survey modes: PC and mobile Web survey. We hypothesized that features of mobile Web usage may affect response accuracy and lead to more socially desirable responses compared to the PC Web survey mode. We found significant differences in the reporting of alcohol consumption by mode, consistent with our hypothesis. But other sensitive questions did not show similar effects. We also found that the presence of familiar bystanders had an impact on the responses, while the presence of strangers did not have any significant effect in either survey mode. Contrary to expectations, we did not find evidence of a positive impact of completing the questionnaire at home and trust in data confidentiality on the level of reporting. These results could help survey practitioners to design and improve data quality in Web surveys completed on different devices.

Comparison of Smartphone and Online Computer Survey Administration

The dramatic rise of smartphones has profound implications for survey research. Namely, can smartphones become a viable and comparable device for self-administered surveys? The current study is based on approximately 1,500 online U.S. panelists who were smartphone users and who were randomly assigned to the mobile app or online computer mode of a survey. Within the survey, we embedded several experiments that had been previously tested in other modes (mail, PC web, mobile web). First, we test whether responses in the mobile app survey are sensitive to particular experimental manipulations as they are in other modes. Second, we test whether responses collected in the mobile app survey are similar to those collected in the online computer survey. Our mobile survey experiments show that mobile survey responses are sensitive to the presentation of frequency scales and the size of open-ended text boxes, as are responses in other survey modes. Examining responses across modes, we find very limited evidence for mode effects between mobile app and PC web survey administrations. This may open the possibility for multimode (mobile and online computer) surveys, assuming that certain survey design recommendations for mobile surveys are used consistently in both modes.

Data Quality in PC and Mobile Web Surveys

The considerable growth in the number of smart mobile devices with a fast Internet connection provides new challenges for survey researchers. In this article, I compare the data quality between two survey modes: self-administered web surveys conducted via personal computer and those conducted via mobile phones. Data quality is compared based on five indicators: (a) completion rates, (b) response order effects, (c) social desirability, (d) non-substantive responses, and (e) length of open answers. I hypothesized that mobile web surveys would result in lower completion rates, stronger response order effects, and less elaborate answers to open-ended questions. No difference was expected in the level of reporting in sensitive items and in the rate of non-substantive responses. To test the assumptions, an experiment with two survey modes was conducted using a volunteer online access panel in Russia. As expected, mobile web was associated with a lower completion rate, shorter length of open answers, and similar level of socially undesirable and non-substantive responses. However, no stronger primacy effects in mobile web survey mode were found.

Comparing Survey Results Obtained via Mobile Devices and Computers

With the growing popularity of smartphones and tablet PCs (tablets) equipped with mobile browsers, the possibilities to administer surveys via mobile devices have expanded. To investigate the possible mode effect on answer behavior, results are compared between a mobile device–assisted web survey and a computer-assisted web survey. First, a premeasurement in the CentERpanel is conducted to analyze the user group of mobile devices. Second, the users are randomly allocated one of the three conditions: (1) conventional computer-assisted web survey, (2) hybrid version: a computer-assisted web survey with a layout similar to mobile web survey, and (3) mobile web survey. Special attention is given to the design of the mobile web questionnaire, taking small screen size, and typical functionalities for touchscreens into account. The findings suggest that survey completion on mobile devices need not lead to different results than on computers, but one should be prepared for a lower response rate and longer survey completion time. Further, the study offers considerations for researchers on survey satisfaction, location during survey completion, and preferred device to access Internet. With adaptations, surveys can be conducted on the newest mobile devices, although new challenges are emerging and further research is called for.