Use Of Crowdsourced Data Research Articles

Response Satisficing Across Online Data Sources: Effects of Satisficing on Data Quality and Policy-Relevant Results

The use of crowdsourced data has become extremely popular in marketing and public policy research. However, there are concerns about the validity of studies that source data from crowdsourcing platforms such as Amazon Mechanical Turk (MTurk). Using five different online sample sources, including multiple MTurk samples and professionally managed panels, the authors address issues related to online data quality and its effects on results for a policy-based 2 × 2 between-subjects experiment. They show that survey response satisficing, as well as multitasking, is related to attention check performance measures beyond demographic differences, and there are substantial differences across the five different online data sources. The authors specifically identify segments of high and low response satisficers using a multi-item measure and show that there are critical differences in the policy-relevant results of the experiment for these segments of online respondents. Findings suggest implications for concerns about failures to replicate results in the policy and consumer well-being, business, and social science literatures. The authors offer some suggestions for attempting to reduce problematic effects of response satisficing and data quality that are shown to differ substantially across the sample sources examined.

Crowdsourced data in public administration research: A review and look to the future

AbstractCrowdsourcing platforms such as MTurk and Prolific have emerged as data sources for researchers in the social sciences. This article delves into the past, present, and future use of crowdsourced data in public administration scholarship. Through a review of published articles in top public administration journals (years 2013–2022), we uncover a general growth in the use of crowdsourced data over time. Additionally, we document how researchers have leveraged crowdsourced data to study a diverse range of themes and topics, with particular emphasis on survey experimental approaches and the examination of citizen attitudes and responses. Moreover, drawing on insights from a survey among quantitative public administration researchers, we discuss why the use of crowdsourced data is unlikely to diminish in the foreseeable future—despite ongoing debates regarding data quality and validity. We provide a set of guiding questions for researchers to consider when using crowdsourced data in public administration studies.

FROM MULTIPLE POLYGONS TO SINGLE GEOMETRY: OPTIMIZATION OF POLYGON INTEGRATION FOR CROWDSOURCED DATA

Abstract. Paid crowdsourcing is a popular approach for creating training data in machine learning, but output quality can suffer from various drawbacks, such as noisy data. One solution is to obtain multiple acquisitions of the same dataset and perform integration steps, which can be challenging for geometries such as polygons. In this paper, we propose a raster-based polygon integration approach for the use of crowdsourced data, providing a solution for integrating multiple geometric shapes into single geometries. We analyze the effects of the choice of the integration threshold parameter for different sample sizes on the quality measures intersection over union (IoU) and Hausdorff distance, and provide a recommendation for its optimal selection based on empirical analysis. Additionally, further possibilities to improve integration results are explored, i.e., methods of filtering data before integration by outlier detection.

Examining data imbalance in crowdsourced reports for improving flash flood situational awareness

The use of crowdsourced data has been finding practical use for enhancing situational awareness during disasters. While recent studies have shown promising results regarding the potential of crowdsourced data (such as user-generated flood reports) for flash flood mapping and situational awareness, little attention has been paid to data imbalance issues that could introduce biases in data and assessment. To address this gap, in this study, we examine biases present in crowdsourced reports to identify data imbalance with a goal of improving disaster situational awareness. Three biases are examined: sample bias, spatial bias, and demographic bias. To examine these biases, we analyzed reported flooding from 3-1-1 reports (which is a citizen hotline allowing the community to report problems such as flooding) and Waze reports (which is a GPS navigation app that allows drivers to report flooded roads) with respect to FEMA damage data collected in the aftermaths of Tropical Storm Imelda in Harris County, Texas, in 2019 and Hurricane Ida in New York City in 2021. First, sample bias is assessed by expanding the flood-related categories in 3-1-1 reports. Integrating other flooding related topics into the Global Moran's I and Local Indicator of Spatial Association (LISA) revealed more communities that were impacted by floods. To examine spatial bias, we perform the LISA and BI-LISA tests on the data sets—FEMA damage, 3-1-1 reports, and Waze reports—at the census tract level and census block group level. By looking at two geographical aggregations, we found that the larger spatial aggregations, census tracts, show less data imbalance in the results. Through a regression analysis, we found that 3-1-1 reports and Waze reports have data imbalance limitations in areas where minority populations and single parent households reside. The findings of this study advance understanding of data imbalance and biases in crowdsourced datasets that are growingly used for disaster situational awareness. Through addressing data imbalance issues, researchers and practitioners can proactively mitigate biases in crowdsourced data and prevent biased and inequitable decisions and actions.

Crowdsourced Data Highlight Precipitation Phase Partitioning Variability in Rain‐Snow Transition Zone

AbstractTo increase the number of direct observations of rain and snow, we started a citizen science project that crowdsources precipitation phase reports from volunteers using a smartphone app. We focused on the Lake Tahoe region of California and Nevada, USA which forms part of the rain‐snow transition zone, an area where both solid and liquid precipitation occur in winter months. In two study years, we received 2,495 reports, of which 2,248 (90.1%) passed our quality control checks. Snow was the most frequent phase (64.0%), followed by rain (21.0%) and mixed precipitation (15.0%). We compared these values to estimates from 14 common precipitation phase partitioning methods that use near‐surface meteorology as well as to two remote sensing products from the Global Precipitation Measurement mission (GPM). We found the meteorology‐based methods tended to underestimate snowfall on average (60.9%) with a sizable standard deviation of 18%. The Integrated Multi‐satellitE Retrievals for GPM level 3 probabilityLiquidPrecipitation product also underestimated snowfall (57.5%) relative to the crowdsourced data, while the Dual‐frequency Precipitation Radar level 2A phaseNearSurface product had little spatiotemporal overlap with the observations. We also found slight differences in the rain‐snow line elevations measured by a freezing‐level radar versus those estimated from the crowdsourced data, with the former being 165 m lower than the latter on average. These findings underscore the importance of collecting ground‐truth observations of precipitation phase in the rain‐snow transition zone. We hope future studies will consider the use of crowdsourced data for improved insights into and better representation of hydrometeorological processes.

COVID-19 Misinformation Detection: Machine-Learned Solutions to the Infodemic.

The volume of COVID-19-related misinformation has long exceeded the resources available to fact checkers to effectively mitigate its ill effects. Automated and web-based approaches can provide effective deterrents to online misinformation. Machine learning-based methods have achieved robust performance on text classification tasks, including potentially low-quality-news credibility assessment. Despite the progress of initial, rapid interventions, the enormity of COVID-19-related misinformation continues to overwhelm fact checkers. Therefore, improvement in automated and machine-learned methods for an infodemic response is urgently needed. The aim of this study was to achieve improvement in automated and machine-learned methods for an infodemic response. We evaluated three strategies for training a machine-learning model to determine the highest model performance: (1) COVID-19-related fact-checked data only, (2) general fact-checked data only, and (3) combined COVID-19 and general fact-checked data. We created two COVID-19-related misinformation data sets from fact-checked "false" content combined with programmatically retrieved "true" content. The first set contained ~7000 entries from July to August 2020, and the second contained ~31,000 entries from January 2020 to June 2022. We crowdsourced 31,441 votes to human label the first data set. The models achieved an accuracy of 96.55% and 94.56% on the first and second external validation data set, respectively. Our best-performing model was developed using COVID-19-specific content. We were able to successfully develop combined models that outperformed human votes of misinformation. Specifically, when we blended our model predictions with human votes, the highest accuracy we achieved on the first external validation data set was 99.1%. When we considered outputs where the machine-learning model agreed with human votes, we achieved accuracies up to 98.59% on the first validation data set. This outperformed human votes alone with an accuracy of only 73%. External validation accuracies of 96.55% and 94.56% are evidence that machine learning can produce superior results for the difficult task of classifying the veracity of COVID-19 content. Pretrained language models performed best when fine-tuned on a topic-specific data set, while other models achieved their best accuracy when fine-tuned on a combination of topic-specific and general-topic data sets. Crucially, our study found that blended models, trained/fine-tuned on general-topic content with crowdsourced data, improved our models' accuracies up to 99.7%. The successful use of crowdsourced data can increase the accuracy of models in situations when expert-labeled data are scarce. The 98.59% accuracy on a "high-confidence" subsection comprised of machine-learned and human labels suggests that crowdsourced votes can optimize machine-learned labels to improve accuracy above human-only levels. These results support the utility of supervised machine learning to deter and combat future health-related disinformation.

How to Include Crowd-Sourced Photogrammetry in a Geohazard Observatory—Case Study of the Giant’s Causeway Coastal Cliffs

The Causeway Coast World Heritage Site (Northern Ireland) is subject to rockfalls occurring on the coastal cliffs, thus raising major safety concerns given the number of tourists visiting the site. However, such high tourist frequentation makes this site favorable to implement citizen science monitoring programs. Besides allowing for the collection of a larger volume of data, better distributed spatially and temporally, citizen science also increases citizens’ awareness—in this case, about risks. Among citizen science approaches, Structure-from-Motion photogrammetry based on crowd-sourced photographs has the advantage of not requiring any particular expertise on the part of the operator who takes photos. Using a mock citizen survey for testing purposes, this study evaluated different methods relying on crowd-sourced photogrammetry to integrate surveys performed by citizens into a landslide monitoring program in Port Ganny (part of the touristic site of the Giant’s Causeway). Among the processing scenarios that were tested, the Time-SIFT method allows the use of crowd-sourced data in a very satisfactory way in terms of reconstruction quality, with a standard deviation of 8.6 cm.

Early-stage spatial disease surveillance of novel SARS-CoV-2 variants of concern in Germany with crowdsourced data

The emergence and rapid spread of novel variants of concern (VOC) of the coronavirus 2 constitute a major challenge for spatial disease surveillance. We explore the possibility to use close to real-time crowdsourced data on reported VOC cases (mainly the Alpha variant) at the local area level in Germany. The aim is to use these data for early-stage estimates of the statistical association between VOC reporting and the overall COVID-19 epidemiological development. For the first weeks in 2021 after international importation of VOC to Germany, our findings point to significant increases of up to 35–40% in the 7-day incidence rate and the hospitalization rate in regions with confirmed VOC cases compared to those without such cases. This is in line with simultaneously produced international evidence. We evaluate the sensitivity of our estimates to sampling errors associated with the collection of crowdsourced data. Overall, we find no statistical evidence for an over- or underestimation of effects once we account for differences in data representativeness at the regional level. This points to the potential use of crowdsourced data for spatial disease surveillance, local outbreak monitoring and public health decisions if no other data on new virus developments are available.

HD Map Update for Autonomous Driving With Crowdsourced Data

Current self-driving cars can perform precise localization and generate collision-free trajectories using high definition (HD) maps which provide accurate road information. Therefore, keeping HD maps up to date is important for safe autonomous driving. In general, automotive HD maps are built by the use of expensive mapping systems. In addition, a lot of manual modifications are required in many cases. The conventional HD mapping cannot be frequently carried out due to the high cost. In this letter, we used a large amount of road data collected by crowdsourcing devices. Crowdsourcing devices consist of low-cost sensors. The devices are mounted on repeatedly traveling vehicles such as buses. Although collected data shows high uncertainty and low accuracy, a large amount of data can be obtained in a short time with low expense. We present a solution that keeps HD maps up to date by using crowdsourced data. The developed solution concentrates on landmark information among crowdsourced data and HD maps. By using uncertainty information, we chose reliable observations for map updating. Observation learner algorithms were carefully designed under the consideration of differences between discrete and continuous landmarks. The triggering condition for the map update can be adjusted by the proposed update mode selection strategy. The proposed map updating scheme has been experimentally verified by the use of crowdsourced data collected from real road environments.

An Examination of the Strava Usage Rate—A Parameter to Estimate Average Annual Daily Bicycle Volumes on Rural Roadways

In Spain, a new challenge is emerging due to the increase of many recreational bicyclists on two-lane rural roads. These facilities have been mainly designed for motorized vehicles, so the coexistence of cyclists and drivers produces an impact, in terms of road safety and operation. In order to analyze the occurrence of crashes and enhance safety for bicycling, it is crucial to know the cycling volume. Standard procedures recommend using data from permanent stations and temporary short counts, but bicycle volumes are rarely monitored in rural roads. However, bicyclists tend to track their leisure and exercise activities with fitness apps that use GPS. In this context, this research aims at analyzing the daily and seasonal variability of the Strava Usage Rate (SUR), defined as the proportion of bicyclists using the Strava app along a certain segment on rural highways, to estimate the Annual Average Daily Bicycle (AADB) volume on rural roads. The findings of this study offer possible solutions to policy makers in terms of planning and design of the cycling network. Moreover, the use of crowdsourced data from the Strava app will potentially save costs to public agencies, since public data could replace costly counting campaigns.

A geographically weighted regression model to examine the spatial variation of the socioeconomic and land-use factors associated with Strava bike activity in Austin, Texas

Despite evidence showing the spatial nonstationarity of the determinants of bike activity, very few studies have addressed the phenomena, probably due to the limited sample size of the traditional count data. To address this gap, this study demonstrated the applicability of Strava bike activity data by developing a geographically weighted Poisson regression (GWPR) model that can reveal how the influence of socioeconomic and land-use factors vary across a region. The city of Austin was selected as a case study, and Strava bike volume was gleaned from 1494 intersections. The representativeness of the Strava data was first examined by comparing those data with the video-based actual bicycle volume data from 43 intersections in the study area. Despite the high deviation in several locations, Strava volume exhibited moderate linear relationships with actual volume. The GWPR model developed in this study outperformed the traditional global model and revealed significant spatial variability of nine variables related to age, income, education, transit stops, hub locations, offices, schools, trails, and sidewalk facilities. Notable spatial variations on bike activity were observed across the study area in terms of magnitude, direction, and significance of the impact for all model variables. The analysis and discussion offer guidance to practitioners and policy makers in tailoring policies and programs that consider the spatial context. The study also provides insights for understanding the potential use of crowdsourced data in examining bike activity, especially when resources are limited.

Mapping Crop Types in Southeast India with Smartphone Crowdsourcing and Deep Learning

High resolution satellite imagery and modern machine learning methods hold the potential to fill existing data gaps in where crops are grown around the world at a sub-field level. However, high resolution crop type maps have remained challenging to create in developing regions due to a lack of ground truth labels for model development. In this work, we explore the use of crowdsourced data, Sentinel-2 and DigitalGlobe imagery, and convolutional neural networks (CNNs) for crop type mapping in India. Plantix, a free app that uses image recognition to help farmers diagnose crop diseases, logged 9 million geolocated photos from 2017–2019 in India, 2 million of which are in the states of Andhra Pradesh and Telangana in India. Crop type labels based on farmer-submitted images were added by domain experts and deep CNNs. The resulting dataset of crop type at coordinates is high in volume, but also high in noise due to location inaccuracies, submissions from out-of-field, and labeling errors. We employed a number of steps to clean the dataset, which included training a CNN on very high resolution DigitalGlobe imagery to filter for points that are within a crop field. With this cleaned dataset, we extracted Sentinel time series at each point and trained another CNN to predict the crop type at each pixel. When evaluated on the highest quality subset of crowdsourced data, the CNN distinguishes rice, cotton, and “other” crops with 74% accuracy in a 3-way classification and outperforms a random forest trained on harmonic regression features. Furthermore, model performance remains stable when low quality points are introduced into the training set. Our results illustrate the potential of non-traditional, high-volume/high-noise datasets for crop type mapping, some improvements that neural networks can achieve over random forests, and the robustness of such methods against moderate levels of training set noise. Lastly, we caution that obstacles like the lack of good Sentinel-2 cloud mask, imperfect mobile device location accuracy, and preservation of privacy while improving data access will need to be addressed before crowdsourcing can widely and reliably be used to map crops in smallholder systems.

Using geographically weighted models to explore how crowdsourced landscape perceptions relate to landscape physical characteristics

This study explores how formal measures of landscape wildness (i.e. absence of human artefacts, perceived naturalness of land cover, remoteness from mechanised access, and ruggedness of the terrain) correlate with crowdsourced measures of landscape aesthetic quality as captured in Scenic-Or-Not data for Great Britain. It evaluates multiple linear regression (MLR) and two spatially varying coefficients models: geographically weighted regression (GWR) and multiscale geographically weighted regression (MGWR). The MLR provided a baseline model in an analysis of national data, exhibiting the presence of spatially autocorrelated residuals and suggesting that geographically weighted models may be appropriate. A standard GWR was found to exacerbate local collinearity between covariates, both overfitting and underfitting the model with highly varied and localised results. This was due to its single one-size-fits-all bandwidth and the assumption that all relationships between the target and predictor variables operate over the same spatial scale. MGWR relaxes this assumption by determining parameter-specific bandwidths, mitigating the local collinearity issues found in a standard GWR and resulting in more spatially stable and consistent coefficient estimates. The findings also indicated that the relationship between some covariates (such as remoteness) and perceived landscape quality varied little spatially, while clear gradients were found for other covariates. For example, naturalness was stronger in the north and west, ruggedness was stronger in the south and east, and the absence of human artefacts was weaker in Scotland and the north than in England and the south. Overall, the study showed that MGWR is more sensitive than GWR to the spatial heterogeneity in the statistical relationships between landscape factors and public perceptions. These findings provide nuanced understandings of how these relationships vary spatially, underscoring the value of such approaches in landscape scale analyses to support policy and planning. The discussion section of this paper considers the MGWR as the default geographically weighted model, assessing the potential for the use of crowdsourced data in landscape studies. In so doing, it illustrates how such approaches could be used to explore both subjective and objective landscape evaluations.

Patterns of Oral Microbiota Diversity in Adults and Children: A Crowdsourced Population Study

Oral microbiome dysbiosis has been associated with various local and systemic human diseases such as dental caries, periodontal disease, obesity, and cardiovascular disease. Bacterial composition may be affected by age, oral health, diet, and geography, although information about the natural variation found in the general public is still lacking. In this study, citizen-scientists used a crowdsourcing model to obtain oral bacterial composition data from guests at the Denver Museum of Nature & Science to determine if previously suspected oral microbiome associations with an individual’s demographics, lifestyle, and/or genetics are robust and generalizable enough to be detected within a general population. Consistent with past research, we found bacterial composition to be more diverse in youth microbiomes when compared to adults. Adult oral microbiomes were predominantly impacted by oral health habits, while youth microbiomes were impacted by biological sex and weight status. The oral pathogen Treponema was detected more commonly in adults without recent dentist visits and in obese youth. Additionally, oral microbiomes from participants of the same family were more similar to each other than to oral microbiomes from non-related individuals. These results suggest that previously reported oral microbiome associations are observable in a human population containing the natural variation commonly found in the general public. Furthermore, these results support the use of crowdsourced data as a valid methodology to obtain community-based microbiome data.

Collecting and utilising crowdsourced data for numerical weather prediction: Propositions from the meeting held in Copenhagen, 4–5 December 2018

AbstractIn December 2018, the Danish Meteorological Institute organised an international meeting on the subject of crowdsourced data in numerical weather prediction (NWP) and weather forecasting. The meeting, spanning 2 days, gathered experts on crowdsourced data from both meteorological institutes and universities from Europe and the United States. Scientific presentations highlighted a vast array of possibilities and progress being made globally. Subjects include data from vehicles, smartphones, and private weather stations. Two groups were created to discuss open questions regarding the collection and use of crowdsourced data from different observing platforms. Common challenges were identified and potential solutions were discussed. While most of the work presented was preliminary, the results shared suggested that crowdsourced observations have the potential to enhance NWP. A common platform for sharing expertise, data, and results would help crowdsourced data realise this potential.

Validating crowdsourced bicycling mobility data for supporting city planning

Crowdsourced data hold great potential in creating metrics which can support city planning. Devices such as smartphones, with their sensor capabilities including GPS can be used to capture a wealth of mobility data at the scale of individual trips. However, the use of crowdsourced data for city planning is still hindered by doubts about their accuracy, objectivity and representativeness. This study proposes a validation process with five criteria – geographic coverage, origin-destination match, demographic match, distance–duration distributions, and route match – to assess different representativeness aspects of mobility data. The validation process is demonstrated using a crowdsourced data on bicycling in Sydney, Australia, compared with Census Journey to Work data. Results indicate a good overall fit of the sample against the population, but variations across the five criteria. Implications of these variations on the suitable uses for the crowdsourced data are discussed, and current limitations of the proposed approach are identified.

Using Mechanical Turk Data in IS Research: Risks, Rewards, and Recommendations

With the increasing use of crowdsourced data in behavioral research fields, it is important to examine their appropriateness and desirability for IS research. Extending recent work in the IS literature, this tutorial discusses the risks and rewards of using data gathered on Amazon’s Mechanical Turk. We examine the characteristics of MTurk workers and the resulting method biases that may be exacerbated in MTurk data. Based on this analysis, we present a 2x2 matrix to illustrate the categories of IS research questions that are and are not amenable to MTurk data. We suggest that MTurk data is more appropriate for generalizing studies that examine diverse cognition than for contextualizing studies or those involving shared cognition. Finally, we offer a set of practical recommendations for researchers who wish to collect data on MTurk.

TWEETS AND FACEBOOK POSTS, THE NOVELTY TECHNIQUES IN THE CREATION OF ORIGIN-DESTINATION MODELS

Social media and big data have emerged to be a useful source of information that can be used for planning purposes, particularly transportation planning and trip-distribution studies. Cities in developing countries such as South Africa often struggle with out-dated, unreliable and cumbersome techniques such as traffic counts and household surveys to conduct origin and destination studies. The emergence of ubiquitous crowd sourced data, big data, social media and geolocation based services has shown huge potential in providing useful information for origin and destination studies. Perhaps such information can be utilised to determine the origin and destination of commuters using the Gautrain, a high-speed railway in Gauteng province South Africa. To date little is known about the origins and destinations of Gautrain commuters. Accordingly, this study assesses the viability of using geolocation-based services namely Facebook and Twitter in mapping out the network movements of Gautrain commuters. Explorative Spatial Data Analysis (ESDA), Echo-social and ArcGis software were used to extract social media data, i.e. tweets and Facebook posts as well as to visualize the concentration of Gautrain commuters. The results demonstrate that big data and geolocation based services have the significant potential to predict movement network patterns of commuters and this information can thus, be used to inform and improve transportation planning. Nevertheless use of crowd sourced data and big data has privacy concerns that still need to be addressed.

TWEETS AND FACEBOOK POSTS, THE NOVELTY TECHNIQUES IN THE CREATION OF ORIGIN-DESTINATION MODELS

Abstract. Social media and big data have emerged to be a useful source of information that can be used for planning purposes, particularly transportation planning and trip-distribution studies. Cities in developing countries such as South Africa often struggle with out-dated, unreliable and cumbersome techniques such as traffic counts and household surveys to conduct origin and destination studies. The emergence of ubiquitous crowd sourced data, big data, social media and geolocation based services has shown huge potential in providing useful information for origin and destination studies. Perhaps such information can be utilised to determine the origin and destination of commuters using the Gautrain, a high-speed railway in Gauteng province South Africa. To date little is known about the origins and destinations of Gautrain commuters. Accordingly, this study assesses the viability of using geolocation-based services namely Facebook and Twitter in mapping out the network movements of Gautrain commuters. Explorative Spatial Data Analysis (ESDA), Echo-social and ArcGis software were used to extract social media data, i.e. tweets and Facebook posts as well as to visualize the concentration of Gautrain commuters. The results demonstrate that big data and geolocation based services have the significant potential to predict movement network patterns of commuters and this information can thus, be used to inform and improve transportation planning. Nevertheless use of crowd sourced data and big data has privacy concerns that still need to be addressed.

Crowdsourcing indicators for cultural ecosystem services: A geographically weighted approach for mountain landscapes

Integrating cultural dimensions into the ecosystem service framework is essential for appraising non-material benefits stemming from different human–environment interactions. This study investigates how the actual provision of cultural services is distributed across the landscape according to spatially varying relationships. The final aim was to analyse how landscape settings are associated to people's preferences and perceptions related to cultural ecosystem services in mountain landscapes. We demonstrated a spatially explicit method based on geo-tagged images from popular social media to assess revealed preferences. A spatially weighted regression showed that specific variables correspond to prominent drivers of cultural ecosystem services at the local scale. The results of this explanatory approach can be used to integrate the cultural service dimension into land planning by taking into account specific benefiting areas and by setting priorities on the ecosystems and landscape characteristics which affect the service supply. We finally concluded that the use of crowdsourced data allows identifying spatial patterns of cultural ecosystem service preferences and their association with landscape settings.