Crowdsourced Data Collection Research Articles

SpaceMaze: incentivizing correct mobile crowdsourced sensing behaviour with a sensified minigame

ABSTRACT Modern mobile phones are equipped with many sensors, which can increasingly be used to sense various environmental phenomena. In particular, mobile sensing has enabled crowdsourced data collection at an unprecedented scale. However, as laypersons are involved in this, concerns regarding the data quality arise. This work explores the gamification of smartphone-based measurement processes in practice by embedding a sensing task into a mobile minigame. The underlying idea is – rather than to educate the user on how to correctly perform a measurement task – to opportunistically execute the measurement in the background once the smartphone is in a suitable context. To this end, this paper presents the design and evaluation of SpaceMaze, a smartphone game with the goal of minimising user error by introducing appropriate game mechanics to influence the phone context, using the example of mobile noise level monitoring. A large user study that compares SpaceMaze to two non-gamified apps for noise level monitoring (N = 360 in total) shows that SpaceMaze can successfully reduce user errors when compared to simple non-gamified ambient noise level monitoring applications and that the minigame is generally perceived as being enjoyable. Solutions for remaining problems, such as noise generated by the players, are discussed.

A temporally-calibrated method for crowdsourcing based mapping of intra-urban PM2.5 concentrations

As a primary air pollutant, fine particulate matter (PM2.5) is increasingly attracting attention. Crowdsourcing observations based methods are thought to be the best solutions for identifying the spatio-temporal distribution of PM2.5 in intra-urban areas. However, inconsistent timing in the collection of crowdsourced data has typically been ignored in previous studies. To address this issue, a temporally calibrated method (TCM) was introduced in this study. By interpolating TCM-estimated observations using the inverse distance weighted (IDW) method, variations of PM2.5 concentrations across the urban areas of Changsha City were captured. The results demonstrate that TCM can efficiently resolve the inconsistent timing defects of raw crowdsourcing observations (R2 was 0.73 and the RMSE was 7.65 μg/m3). Furthermore, PM2.5 distributions developed using TCM-based interpolations are of a finer spatial scale than those developed from raw observations at crowdsourcing locations. With a lack of funds to build sufficient stationary monitoring sites, developing crowdsourcing observation-based technology is the most promising solution for revealing intra-urban PM2.5 variations at a higher spatio-temporal- resolution.

Adaptive Indoor Area Localization for Perpetual Crowdsourced Data Collection.

The accuracy of fingerprinting-based indoor localization correlates with the quality and up-to-dateness of collected training data. Perpetual crowdsourced data collection reduces manual labeling effort and provides a fresh data base. However, the decentralized collection comes with the cost of heterogeneous data that causes performance degradation. In settings with imperfect data, area localization can provide higher positioning guarantees than exact position estimation. Existing area localization solutions employ a static segmentation into areas that is independent of the available training data. This approach is not applicable for crowdsoucred data collection, which features an unbalanced spatial training data distribution that evolves over time. A segmentation is required that utilizes the existing training data distribution and adapts once new data is accumulated. We propose an algorithm for data-aware floor plan segmentation and a selection metric that balances expressiveness (information gain) and performance (correctly classified examples) of area classifiers. We utilize supervised machine learning, in particular, deep learning, to train the area classifiers. We demonstrate how to regularly provide an area localization model that adapts its prediction space to the accumulating training data. The resulting models are shown to provide higher reliability compared to models that pinpoint the exact position.

Using Synthetic Data to Enhance the Accuracy of Fingerprint-Based Localization: A Deep Learning Approach

Human-centered data collection is typically costly and implicates issues of privacy. Various solutions have been proposed in the literature to reduce this cost, such as crowdsourced data collection, or the use of semi-supervised algorithms. However, semi-supervised algorithms require a source of unlabeled data, and crowd-sourcing methods require numbers of active participants. An alternative passive data collection modality is fingerprint-based localization. Such methods use received signal strength (RSS) or channel state information (CSI) in wireless sensor networks to localize users in indoor/outdoor environments. In this paper, we introduce a novel approach to reduce training data collection costs in fingerprint-based localization by using synthetic data. Generative adversarial networks (GANs) are used to learn the distribution of a limited sample of collected data and, following this, to produce synthetic data that can be used to augment the real collected data in order to increase overall positioning accuracy. Experimental results on a benchmark dataset show that by applying the proposed method and using a combination of 10% collected data and 90% synthetic data, we can obtain essentially similar positioning accuracy to that which would be obtained by using the full set of collected data. This means that by employing GAN-generated synthetic data, we can use 90% less real data, thereby reduce data-collection costs while achieving acceptable accuracy.

SPARR: Spintronics-based private aggregatable randomized response for crowdsourced data collection and analysis

The rapid development of information and communication technologies has revolutionized people’s lives. A large amount of user behavioral data is generated by client software and is valuable for organizations (e.g., Apple, Google, and Samsung) to adjust their commercial strategies or to promote the quality of service. However, data collection raises privacy concerns. Either malicious or accidental leaking of sensitive information may cause serious consequences. Randomized response is a candidate solution, and it can provide “plausible deniability” for each individual. The aggregators can only collect sanitized data, but they can still conduct analyses and make predictions. Existing randomized response solutions can either apply pseudo-randomized functions with a predicable period of random numbers or have severely restricted functionality rather than providing practical privacy protection and efficient usability. We propose the spintronics-based private aggregatable randomized response (SPARR) for crowdsourced data collection while satisfying differential privacy. SPARR utilizes a spintronics-based true random number generator and multilayer randomized response to guarantee true data randomness and high data utility. Through multilayer randomized response and the corresponding subtle decoding algorithm, SPARR can obtain exceptionally high accuracy and efficiency in the frequency estimation of client strings even for small collections of applications, compared to state-of-the-art methods.

The Impact of Perceived Privacy, Accuracy and Security on the Adoption of Mobile Self-Checkout Systems

Traditional brick-and-mortar retailers face increasing competitive pressure from online alternatives. One promising technology to help retailers cope with the ongoing retail crisis is mobile self-checkout systems (MSCOS). However, there have been issues with MSCOS implementation. This study aims at gaining a better understanding of consumer needs related to MSCOS. Using diffusion of innovation as its theoretical lens, we explore the impact of relative advantage, trialability, compatibility, and ease of use on MSCOS adoption. Since MSCOS involves financial transactions, we also consider the impact of perceived privacy, accuracy, and security. A crowdsourcing data collection resulted in 302 responses from US respondents. Findings support that relative advantage, trialability, and perceived security positively affect usage intention. Moreover, compatibility and ease of use positively affect relative advantage, and perceived privacy and accuracy positively affect perceived security. This study provides significant insights and implications for both innovation adoption research and retail practice. JEL Codes: O300, O330, M150

Knowledge Production, Big Data and Data-Driven Customary International Law

This chapter investigates the role of “Big Data” analysis and data crowdsourcing in shifting power relations with respect to the identification of customary international law. Evidence of states’ practice and legal positions is required in order to determine that a new norm of customary international law has crystallized. And yet, international courts have often settled for anecdotal evidence and impressionistic analysis. However, recent academic works have crowdsourced data collection, compiled big datasets and applied computerized analysis methods to make comprehensive and systematic evaluation of the development of customary norms. I argue that this new mode of knowledge production may democratize both the data collected (giving greater weight to smaller states from the global periphery) and the potential contributors to the production process (including lawyers from different countries and language capabilities). Nevertheless, such production requires scientific sophistication and resources, which once more give actors from rich, developed countries a greater role in developing the law.

Platform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks

Crowdsourcing platforms provide a convenient and scalable way to collect human-generated labels on-demand. This data can be used to train Artificial Intelligence (AI) systems or to evaluate the effectiveness of algorithms. The datasets generated by means of crowdsourcing are, however, dependent on many factors that affect their quality. These include, among others, the population sample bias introduced by aspects like task reward, requester reputation, and other filters introduced by the task design.In this paper, we analyse platform-related factors and study how they affect dataset characteristics by running a longitudinal study where we compare the reliability of results collected with repeated experiments over time and across crowdsourcing platforms. Results show that, under certain conditions: 1) experiments replicated across different platforms result in significantly different data quality levels while 2) the quality of data from repeated experiments over time is stable within the same platform. We identify some key task design variables that cause such variations and propose an experimentally validated set of actions to counteract these effects thus achieving reliable and repeatable crowdsourced data collection experiments.

Citizen science in hydrological monitoring and ecosystem services management: State of the art and future prospects.

Hydrological monitoring is essential to guide evidence-based decision making necessary for sustainable water resource management and governance. Limited hydrometric datasets and the pressure on long-term hydrological monitoring networks make it paramount to explore alternative methods for data collection. This is particularly the case for low-income countries, where data scarcity is more pronounced, and where conventional monitoring methods are expensive and logistically challenging. Citizen science in hydrological research has recently gained popularity and crowdsourced monitoring is a promising cost-effective approach for data collection. Citizen science also has the potential to enhance knowledge co-creation and science-based evidence that underpins the governance and management of water resources. This paper provides a comprehensive review on citizen science and crowdsourced data collection within the context of hydrology, based on a synthesis of 71 articles from 2001 to 2018. Application of citizen science in hydrology is increasing in number and breadth, generating a plethora of scientific data. Citizen science approaches differ in scale, scope and degree of citizen involvement. Most of the programs are found in North America and Europe. Participation mostly comprises a contributory citizen science model, which engages citizens in data collection. In order to leverage the full potential of citizen science in knowledge co-generation, future citizen science projects in hydrology could benefit from more co-created types of projects that establish strong ties between research and public engagement, thereby enhancing the long-term sustainability of monitoring networks.

Open-source software for mouse-tracking in Qualtrics to measure category competition

Mouse-tracking is a sophisticated tool for measuring rapid, dynamic cognitive processes in real time, particularly in experiments investigating competition between perceptual or cognitive categories. We provide user-friendly, open-source software (https://osf.io/st2ef/) for designing and analyzing such experiments online using the Qualtrics survey platform. The software consists of a Qualtrics template with embedded JavaScript and CSS along with R code to clean, parse, and analyze the data. No special programming skills are required to use this software. As we discuss, this software could be readily modified for use with other online survey platforms that allow the addition of custom JavaScript. We empirically validate the provided software by benchmarking its performance on previously tested stimuli (android robot faces) in a category-competition experiment with realistic crowdsourced data collection.

Integration of social spatial data to assess conservation opportunities and priorities

Effective wildlife conservation requires consideration of ecological and social factors, including social acceptability of conservation actions. Using the threatened koala (Phascolarctos cinereus) as a case study, we demonstrate a novel, socio-ecological approach for identifying conservation opportunity that spatially connects landscapes with community preferences to prioritize koala recovery strategies at a regional scale. We conceptualize conservation opportunity as the spatial integration of three sustainability criteria—ecological potential, social acceptability, and economic feasibility. The social acceptability criterion was assessed using a crowdsourced spatial survey that identified spatial preferences for koalas and land uses that impact koala conservation. As a novel approach, we addressed important research questions regarding the design, collection, and analysis of crowdsourced mapping data for identifying socially acceptable conservation opportunities. Public preferences for koalas were mapped closer to home, in higher suitable koala habitats than expected, were more pronounced in conservation and natural areas on public lands, and were mapped less frequently in modified agricultural landscapes. When the multiple criteria (ecological, social, and economic) were included in the conservation assessment, we found the social acceptability criterion exerted the greatest influence on spatial conservation priorities. The systematic assessment of social criteria for conservation using spatial surveys provides information that can be integrated with ecological information to prioritize conservation opportunities. Potential enhancements include expanding survey recruitment efforts and using alternative social data collection methods to achieve greater geographic and socio-demographic representation, and augmenting the economic feasibility assessment with private property values and transaction data from voluntary conservation agreements with private landowners.

Collecting and utilising crowdsourced data for numerical weather prediction: Propositions from the meeting held in Copenhagen, 4–5 December 2018

AbstractIn December 2018, the Danish Meteorological Institute organised an international meeting on the subject of crowdsourced data in numerical weather prediction (NWP) and weather forecasting. The meeting, spanning 2 days, gathered experts on crowdsourced data from both meteorological institutes and universities from Europe and the United States. Scientific presentations highlighted a vast array of possibilities and progress being made globally. Subjects include data from vehicles, smartphones, and private weather stations. Two groups were created to discuss open questions regarding the collection and use of crowdsourced data from different observing platforms. Common challenges were identified and potential solutions were discussed. While most of the work presented was preliminary, the results shared suggested that crowdsourced observations have the potential to enhance NWP. A common platform for sharing expertise, data, and results would help crowdsourced data realise this potential.

Using crowd sourcing to locate and characterize conflicts for vulnerable modes

Most agencies and decision-makers rely on crash and crash severity (property damage only, injury or fatality) data to assess transportation safety; however, in the context of public health where perceptions of safety may influence the willingness to adopt active transportation modes (e.g. bicycling and walking), pedestrian-motor vehicle and other similar conflicts types may define a better performance measure for safety assessment. In the field of transportation safety, an absolute conflict occurs when two parties’ paths cross and one of the parties must undertake an evasive maneuver (e.g. change direction or stop) to avoid a crash. Other less severe conflicts where paths cross but no evasive maneuver is required may also impact public perceptions of safety especially for vulnerable modes. Most of the existing literature focuses on vehicle conflicts. While in the past several years, more research has investigated bicycle and pedestrian conflicts, most of this has focused on the intersection environment. A comprehensive analysis of conflicts appears critical. The major objective of this study is two fold: 1) Development of an innovative and cost effective conflict data collection technique to better understand the conflicts (and their severity) involving vulnerable road users (e.g. bicycle/pedestrian, bicycle/motor vehicle, and pedestrian/motor vehicle) and their severity. 2) Test the effectiveness and practicality of the approach taken and its associated crowd sourced data collection. In an endeavor to undertake these objectives, the researchers developed an android-based crowd-sourced data collection app. The crowd-source data collected using the app is compared with traditional fatality data for hot spot analysis. At the end, the app users provide feedback about the overall competency of the app interface and the performance of its features to the app developers. If widely adopted, the app will enable communities to create their own data collection efforts to identify dangerous sites within their neighborhoods. Agencies will have a valuable data source at low-cost to help inform their decision making related to bicycle and pedestrian education, encouragement, enforcement, programs, policies, and infrastructure design and planning.

Migration as a livelihood strategy in Ethiopia: fallacy or reality?

PurposeThe purpose of this paper is to investigate the impact of migration on the livelihoods of Ethiopians. It is widely acclaimed that migration has positive effects on livelihoods. This paper investigates whether this claim is a fallacy or a reality. Can migration be conceptualized as a strategy for livelihood enhancement? Although Ethiopia has a large number of migrants both internally and externally, this paper focuses on the impact of external migration on the livelihoods of Ethiopian migrants and their families.Design/methodology/approachUsing primary data, the paper attempts to establish whether migration enhances livelihoods. Qualitative data are used. Primary data were collected and analyzed using SurveyMonkey. SurveyMonkey is an internet-based software that has a facility for interview questions and it analyses data automatically on submission of responses. The survey achieved a response rate of 52 percent (218 out of 420). A follow-up survey, done between March 20 and April 16, 2018 to validate the online responses, involved 12 respondents.FindingsResults show that migration is important in the sustenance of livelihoods. Both pecuniary and non-pecuniary benefits have been realised. In addition, migration also benefits development at home.Practical implicationsThe Ethiopian Government should develop policy options that promote the inflow of remittances for livelihood enhancement.Originality/valueThe paper uses SurveyMonkey to gather data from a number of respondents (crowdsourcing data collection). The SurveyMonkey made possible a crowd data gathering process.

Webcams as an untapped opportunity to conduct citizen science: Six years of the American Kestrel Partnership's KestrelCam

Hundreds of zoo-based or wildlife webcams have become available during the past twenty years, mostly with the goal of educating the public. However, there has been virtually no peer-reviewed research that evaluates the education, conservation, or scientific impact of webcams. Here, we provide one of the few examples of a webcam used for citizen science, and the only test of efficacy for crowd-sourced data collection using webcams. The Peregrine Fund streamed six seasons of American Kestrel (Falco sparverius) nests using the same nest box from 2012 through 2017 and viewers input observations into an online portal. We analyze trends in participant and kestrel behavior and test for sources of bias in this citizen scientist-generated dataset by independently reviewing a subset of recordings to determine accuracy of viewer-logged data. Citizen scientists logged a maximum of approximately 5.25% of all footage, but with an accuracy of 88%. Although number of participants declined yearly, on average, participants became more engaged. Sources of bias were related to people's daily activity periods (i.e., less participation at night) and activity within the nest box (i.e., less participation when there were no birds in the box). This citizen scientist-generated dataset generally corroborated the literature regarding American Kestrel biology. Researchers may be cautiously optimistic that datasets generated by citizen scientists can provide valuable information on a given system or study species. Given the ubiquity of webcams and their potential competition for conservation dollars, more research evaluating any aspect of their impact or application is sorely needed.

Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations

While peer-agreement and gold checks are well-established methods for ensuring quality in crowdsourced data collection, we explore a relatively new direction for quality control: estimating work quality directly from workers’ behavioral traces collected during annotation. We propose three behavior-based models to predict label correctness and worker accuracy, then further apply model predictions to label aggregation and optimization of label collection. As part of this work, we collect and share a new Mechanical Turk dataset of behavioral signals judging the relevance of search results. Results show that behavioral data can be effectively used to predict work quality, which could be especially useful with single labeling or in a cold start scenario in which individuals’ prior work history is unavailable. We further show improvement in label aggregation and reducing labeling cost while ensuring data quality.

Modeling aggregated expertise of user contributions to assess the credibility of OpenStreetMap features

AbstractThe emergence of volunteered geographic information (VGI) during the past decade has fueled a wide range of research and applications. The assessment of VGI quality and fitness‐of‐use is still a challenge because of the non‐standardized and crowdsourced data collection process, as well as the unknown skill and motivation of the contributors. However, the frequent approach of assessing VGI quality against external data sources using ISO quality standard measures is problematic because of a frequent lack of available external (reference) data, and because for certain types of features, VGI might be more up‐to‐date than the reference data. Therefore, a VGI‐intrinsic measure of quality is highly desirable. This study proposes such an intrinsic measure of quality by developing the concept of aggregated expertise based on the characteristics of a feature's contributors. The article further operationalizes this concept and examines its feasibility through a case study using OpenStreetMap (OSM). The comparison of model OSM feature quality with information from a field survey demonstrates the successful implementation of this novel approach.

Crowd-sourced data collection to support automatic classification of building footprint data

Abstract. Human settlements are mainly formed by buildings with their different characteristics and usage. Despite the importance of buildings for the economy and society, complete regional or even national figures of the entire building stock and its spatial distribution are still hardly available. Available digital topographic data sets created by National Mapping Agencies or mapped voluntarily through a crowd via Volunteered Geographic Information (VGI) platforms (e.g. OpenStreetMap) contain building footprint information but often lack additional information on building type, usage, age or number of floors. For this reason, predictive modeling is becoming increasingly important in this context. The capabilities of machine learning allow for the prediction of building types and other building characteristics and thus, the efficient classification and description of the entire building stock of cities and regions. However, such data-driven approaches always require a sufficient amount of ground truth (reference) information for training and validation. The collection of reference data is usually cost-intensive and time-consuming. Experiences from other disciplines have shown that crowdsourcing offers the possibility to support the process of obtaining ground truth data. Therefore, this paper presents the results of an experimental study aiming at assessing the accuracy of non-expert annotations on street view images collected from an internet crowd. The findings provide the basis for a future integration of a crowdsourcing component into the process of land use mapping, particularly the automatic building classification.

3D Printing of Digitally Traced Neurons for Neuroanatomy Education

PurposeThree dimensional (3D) printing technology is being increasingly utilized in anatomy education, and its efficacy as a teaching tool is a growing area of education research. Most curriculum guides for 3D printing activities have students download pre‐designed digital anatomical 3D object files from websites (i.e. www.thingiverse.comor www.neuromorpho.org) rather than generate their own 3D models from original data. Recently a technique was published for tracing and 3D‐printing neurons, but it required expensive software, as well as extensive computer programing experience (McDougal and Shepard, 2015). The goal of our study was to develop a simple, intuitive, and inexpensive method to allow students to generate an original reconstructed neuron suitable for 3D‐printing, and to evaluate this method among targeted learners.MethodsNeurons from mouse brain and retina were labeled with fluorescent dye, fixed and imaged sequentially in 1 μm sections with a Zeiss LSM laser scanning microscope. The image stack was loaded into ImageJ, traced, and morphologically analyzed using the free ImageJ plugin Simple Neurite Tracer (SNT). Traced neurons were then exported as an object file into Blender, a free 3D graphics software program, edited, and printed with PLA, ABA, and gypsum powder material, in various sizes and color patterns. Based on this work, a “3D Printing Neurons Made Easy” instruction guide and 5‐part tutorial video was crafted; the raw neuron image stack was published in The Cell Image Library, a freely accessible online public repository that students can access from any computer and trace to create their own digital 3D model. The curriculum was piloted in a neuroanatomy laboratory session for high school students in a summer neuroscience outreach program. Students were given a pre‐ and post‐test on topics related to microscopy, neuroanatomy and image analysis, as well as an exit survey.ResultsOur “3D Printing Neurons Made Easy” curriculum guide enabled students to participate in an optional, self‐guided activity to learn principles in confocal microscopy, neuroanatomy, and image analysis. By using open source software (Image J and Blender) and a public repository, this self‐guided activity engages students in neuroanatomy research at a relatively low cost. 70% students attempted (31/44) and of the 31 students that attempted the project, 87% of the students successfully printed a 3D neuron. 97% of these students had a positive experience with the activity and 80% agreed that they would recommend the activity to their high school biology class. The curriculum is now being piloted among undergraduate and graduate anatomy students, and analysis is ongoing.ConclusionsWe have developed an easy method for students to reconstruct z‐stack images of neurons for 3D printing. These 3D models provide dramatic examples of the complex structure of neurons, and may help communicate complex 3‐dimensional concepts to students. The activity also allows students with no prior research experience to learn principles in microscopy and 3D image analysis in a way that is fun, engaging, and low cost. Our “3D Printing Neurons Made Easy” instructions guide and tutorial videos may be useful for crowd‐sourcing data collection, anatomy education, training students in anatomical research methods, or attracting students to careers in research.

Using citizen science to bridge taxonomic discovery with education and outreach.

Premise of the StudyBiological collections are uniquely poised to inform the stewardship of life on Earth in a time of cataclysmic biodiversity loss. Efforts to fully leverage collections are impeded by a lack of trained taxonomists and a lack of interest and engagement by the public. We provide a model of a crowd‐sourced data collection project that produces quality taxonomic data sets and empowers citizen scientists through real contributions to science. Entitled MicroPlants, the project is a collaboration between taxonomists, citizen science experts, and teachers and students from universities and K–12.MethodsWe developed an online tool that allows citizen scientists to measure photographs of specimens of a hyper‐diverse group of liverworts from a biodiversity hotspot.ResultsUsing the MicroPlants online tool, citizen scientists are generating high‐quality data, with preliminary analysis indicating non‐expert data can be comparable to expert data.DiscussionMore than 11,000 users from both the website and kiosk versions have contributed to the data set, which is demonstrably aiding taxonomists working toward establishing conservation priorities within this group. MicroPlants provides opportunities for public participation in authentic science research. The project's educational component helps move youth toward engaging in scientific thinking and has been adopted by several universities into curriculum for both biology and non‐biology majors.