Crowdsourcing Research Articles

Searching for or reviewing evidence improves crowdworkers’ misinformation judgments and reduces partisan bias

Can crowd workers be trusted to judge whether news-like articles circulating on the Internet are misleading, or does partisanship and inexperience get in the way? And can the task be structured in a way that reduces partisanship? We assembled pools of both liberal and conservative crowd raters and tested three ways of asking them to make judgments about 374 articles. In a no research condition, they were just asked to view the article and then render a judgment. In an individual research condition, they were also asked to search for corroborating evidence and provide a link to the best evidence they found. In a collective research condition, they were not asked to search, but instead to review links collected from workers in the individual research condition. Both research conditions reduced partisan disagreement in judgments. The individual research condition was most effective at producing alignment with journalists’ assessments. In this condition, the judgments of a panel of sixteen or more crowd workers were better than that of a panel of three expert journalists, as measured by alignment with a held out journalist’s ratings.

A Post-Processing Framework for Crowd Worker Responses Using Large Language Models

To develop quality crowdsourcing systems, aggregating responses from workers is a critical issue. However, it has been difficult to construct an automatic mechanism that flexibly aggregates worker responses in natural language. Accordingly, responses need to be collected in a standardized format, such as binary-choice or multiple categorizations, to avoid large aggregation costs. Recently, with the advent of large language models (LLMs), natural language responses can be automatically and flexibly aggregated. We propose a framework that uses LLMs to flexibly aggregate natural language responses from workers and, as a promising example, consider this framework for crime detection from surveillance cameras using crowdsourced cognitive abilities. In an experiment using subjective evaluation, our proposed framework is shown to be effective for automatically aggregating natural language responses from crowd workers.

An Online Inference-Aided Incentive Framework for Information Elicitation Without Verification

We study the design of incentive mechanisms for the problem of information elicitation without verification (IEWV). In IEWV, a data requester seeks to design proper incentives to optimize the tradeoff between the quality of information (collected from distributed crowd workers) and the total cost of incentives (provided to crowd workers) without verifiable ground truth. While prior work often relies on sufficient knowledge of worker information, we study a scenario where the data requester cannot access workers’ heterogeneous information quality and costs ex-ante. We propose a continuum-armed bandit-based incentive mechanism that dynamically learns the optimal reward level from workers’ reported information. A key challenge is that the data requester cannot evaluate the workers’ information quality without verification, which motivates the design of an inference algorithm. The inference problem is non-convex, yet we reformulate it as a bi-convex problem and derive an approximate solution with a performance guarantee, which ensures the effectiveness of our online reward design. We further enhance the inference algorithm using part of the workers’ historical reports. We also propose a novel rule for the data requester to aggregate workers’ solutions more effectively. We show that our mechanism achieves a sub-linear regret <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\tilde {O}(T^{1/2})$ </tex-math></inline-formula> and outperforms several celebrated benchmarks.

User Selection and Engagement for Climate Services Coproduction

Abstract Climate services are high on the international agenda for their potential to help combat the effects of climate change. However, climate science is rarely directly incorporated into the decision-making processes of societal actors, due to what has been identified as the usability gap. This gap is partially due to a failure to timely and meaningfully engage users in the production of climate services, as well as misperceptions as to which users can best benefit from climate service uptake. In this article, we propose user selection and engagement guidelines that integrate important values from participatory science such as those of legitimacy, representativity, and agency. The guidelines consist of 5 + 1 steps: defining why, where, whom, which attributes, and which intensity and how to select and engage with stakeholders. While these steps may be initially implemented by an ideally interdisciplinary team of scientists and service designers, the final step consists of an iterative process by which each decision is agreed on together with the identified users and stakeholders under a coproduction approach. We believe this systematic user selection and engagement practice is key to support the design of climate services aligned to the actual needs of a wide and inclusive range of empowered societal agents. Significance Statement A review of the climate science and services literature and related research projects reveals that, despite the insistence to include users in all stages of the research process, users are often involved only sporadically and inconsistently and when there is little room to change the climate service suitable for decision-making. Here, we argue that a reason for this is the lack of user selection and engagement guidelines. Failure to implement a research design strategy for these decisions can lead to a lack of usability and applicability of the produced climate-related services, as well as hampering their long-term uptake. These guidelines can thus support the development of usable, coproduced, actionable climate science.

Determinants of Female Genital Mutilation Among Under Five-Year Children in Motta Town, Northwest Ethiopia, 2022: Unmatched Case Control Study.

Female genital mutilation is the removal of a woman's external genitalia in whole or in part for a non-obvious medical reason. Female genital mutilation causes short- and long-term complications like bleeding, pain, infection and exposes girls to sexually transmitted diseases. The determinants of female genital mutilation among children under the age of five have received less attention. As a result, the purpose of this study was to determine the factors that influence female genital mutilation in children under the age of five. A community-based unmatched case control study design was used. The study participants were chosen using computer-generated simple random sampling technique. With a ratio of 1:4 between cases and controls, 323 participants were recruited. Data were collected using an interviewer-administered questionnaire. The association between each independent variable and the dependent variable was determined using binary logistic regression. In a multivariable analysis, variables were considered statistically significant if they had a P-value of less than 0.05 at a 95% confidence interval. In this study, mothers' circumcision status (AOR = 4.6; 95% CI: 2.29-9.25), mothers who had an unfavorable attitude (AOR = 4.15; 95% CI: 1.96-8.82), households in the poorest wealth quintile (AOR = 3.65; 95% CI: 1.2-11.54), mothers who had inadequate knowledge (AOR = 3.31; 95% CI: 1.51-7.25) and antenatal care visit of mothers (AOR = 2.46; 95% CI: 1.03-5.83) were found to be determinant factors of female genital mutilation. Mothers' circumcision status, mother's attitude, wealth quintile, knowledge of mothers, and number of antenatal care visits were factors associated with female genital mutilation. Regular awareness-building on the impacts of the practice and special attention to the mother's attitude are important to eliminate female genital mutilation.

Art‐based perceptual ecology

Much discussion and effort have been directed toward developing a more diverse membership at the Ecological Society of America (ESA) and other professional scientific societies. To better meet diversity, equity, and inclusion (DEI) goals, these societies must expand their views of diversity, yielding broader participation in the sciences. I am the founder of and research director at Citizen Artist, a participatory science research platform employing art-based perceptual ecology (ABPE) research methodologies to study ecological systems change. At Citizen Artist, we honor multiple ways of knowing – exploring and investigating differently; engaging in unbounded curiosity, imagination, and creativity; and embracing holistic, integrative approaches to scientific practice and thinking. We offer alternative methodologies, tools, and field practices – non-conventional approaches that attract underrepresented populations to participate in science. Extending the parameters of how ecologists conduct field-based science may bolster participation and lead to unexpected and revolutionary solutions for our most pressing issues, such as the impacts of climate change, loss of habitat, and biodiversity decline. Duly note, Citizen Artist is an approved scientific content provider for ESA's Professional Certified Ecologist Program. In the early 1990s, during my graduate studies in art therapy, I was introduced to art-based research (ABR) methodologies, which originated in the social sciences and were designed to work with a human population. I recognized they could have great potential and application to more than human populations. Simultaneous with pursuing a PhD in environmental studies, I envisioned how I could expand ABR methods and frame them within ecological and biological principles to serve field-based scientific research. Through an iterative process, I developed a new methodology and set of tools responding to applied questions within the natural sciences, specifically in studying environmental change. The evolution of this process resulted in the birth of ABPE research methodologies. Historically, ecologists have discussed the need to include interdisciplinarity within our professional practice. At Citizen Artist, we engage in a transdisciplinary approach to science practices – weaving together multiple disciplines’ perspectives, theories, and methods, transcending their traditional boundaries to generate knowledge and a deeper understanding of the interconnectedness of all life. When creating and collecting data in the field using ABPE methods, the researcher engages in art-based inquiry through a kinesthetic act of “making with the hands” in which neural connections are established and knowledge is constructed. Grounded in multimodal knowledge systems and sensory-based learning, ABPE data communicate through multiple “languages” to tell the land’s story, including the visual language of mark making or drawing, and the languages of metaphor and symbol. Using the visual language at the inquiry stage of scientific thinking has historical ties: consider these connections to important discoveries, such as Galileo's drawings of the moon, or Maria Sibylla Merian's paintings of insect metamorphosis. According to ABPE methods, both the process and the product hold equal value. The art-making process offers a springboard for new insight and questions to arise, ones that had not been considered previously as the information was not yet available. The art product (data) concretizes original knowledge; provides context and frames of reference; and deepens the researcher's understanding of ecological phenomena. The data do not offer finality, but rather a turn in the conversation that often leads to more questions than answers. At Citizen Artist, we are building a diverse, global community trained in best practices within a transdisciplinary research framework, to mutually learn from and explore new pathways to address current ecological challenges. Since the summer of 2020, I have taught an online course, “Introduction to Citizen Artist and Art-Based Research Methods”, offering training in ABPE methodologies for field-based practices and increasing access to science for underserved populations. To date, more than 120 individuals – including researchers, educators, and doctoral and master's students hailing from 14 countries and 22 states – have taken the course, 95% of whom belong to science disciplines (within which 15% are interdisciplinary) and 3% of whom are in the arts. Each of these individuals is working with a larger constituency of stakeholders at universities, government agencies, NGOs, and Indigenous communities. Contrary to our name, there is no need for artistic skills to take the class or engage in ABPE. In conclusion, I believe integrative approaches to science diversify and strengthen the field. Including transdisciplinary approaches to both scientific practices and ways of thinking opens the field to communities of creative co-collaborators, expanding the boundaries of scientific innovation and transformative solutions. For more information on Citizen Artist programs, visit www.citizen-artist.com.

COVID-19 Knowledge, Attitudes, and Vaccine Hesitancy in Ethiopia: A Community-Based Cross-Sectional Study.

The current healthcare system's efforts to reduce the spread of COVID-19 in Ethiopia and limit its effects on human lives are being hampered by hesitancy toward the COVID-19 vaccine. The aim of this study was to assess the knowledge levels, attitudes, and prevention practices of COVID-19, in the context of the level of vaccine hesitancy with other associated factors in Ethiopia. A community-based cross-sectional design with mixed-method data sources was employed. It comprised 1361 study participants for the quantitative survey, with randomly selected study participants from the studied community. This was triangulated by a purposively selected sample of 47 key informant interviews and 12 focus group discussions. The study showed that 53.9%, 55.3%, and 44.5% of participants had comprehensive knowledge, attitudes, and practices regarding COVID-19 prevention and control, respectively. Similarly, 53.9% and 47.1% of study participants had adequate knowledge and favorable attitudes toward the COVID-19 vaccine. Only 29.0% of the total survey participants had been vaccinated with at least one dose of vaccine. Of the total study participants, 64.4% were hesitant about receiving the COVID-19 vaccination. The most frequently reported reasons were a lack of trust in the vaccine (21%), doubts regarding the long-term side effects (18.1%), and refusal on religious grounds (13.6%). After adjusting for other confounding factors, geographical living arrangements, the practices of COVID-19 prevention methods, attitudes about the vaccine, vaccination status, perceived community benefit, perceived barriers toward vaccination, and self-efficacy about receiving the vaccine were significantly associated with vaccine hesitancy. Therefore, to improve vaccine coverage and reduce this high level of hesitancy, there should be specifically designed, culturally tailored health education materials and a high level of engagement from politicians, religious leaders, and other community members.

A Model for Cognitive Personalization of Microtask Design.

The study of data quality in crowdsourcing campaigns is currently a prominent research topic, given the diverse range of participants involved. A potential solution to enhancing data quality processes in crowdsourcing is cognitive personalization, which involves appropriately adapting or assigning tasks based on a crowd worker's cognitive profile. There are two common methods for assessing a crowd worker's cognitive profile: administering online cognitive tests, and inferring behavior from task fingerprinting based on user interaction log events. This article presents the findings of a study that investigated the complementarity of both approaches in a microtask scenario, focusing on personalizing task design. The study involved 134 unique crowd workers recruited from a crowdsourcing marketplace. The main objective was to examine how the administration of cognitive ability tests can be used to allocate crowd workers to microtasks with varying levels of difficulty, including the development of a deep learning model. Another goal was to investigate if task fingerprinting can be used to allocate crowd workers to different microtasks in a personalized manner. The results indicated that both objectives were accomplished, validating the usage of cognitive tests and task fingerprinting as effective mechanisms for microtask personalization, including the development of a deep learning model with 95% accuracy in predicting the accuracy of the microtasks. While we achieved an accuracy of 95%, it is important to note that the small dataset size may have limited the model's performance.

COMMUNICATING ELSI ISSUES IN NEUROBIOBANKING AND STROKE GENOMIC RESEARCH IN AFRICA: PROJECT INTERVENTION TOOLS DEVELOPMENT AND EVALUATION PROCEDURES

Background and aimsActivities about appropriate community engagement and communication of research outcomes with stakeholders have received the attention of scholars in different sub-fields of clinical research. However, given its novel nature, especially in Sub-Saharan Africa, community engagement addressing the ethical, legal, and social implications (ELSI) of neurobiobanking and stroke genomic research has not received much scholarly attention. Therefore, this study was designed to report the procedures for developing and evaluating intervention tools for the community engagement component of the African Neurobiobank for Precision Stroke Medicine ELSI Project. MethodsA community-based participatory research design was adopted to develop three intervention tools. An adapted Doak and Doak's Suitability Assessment Measure (SAM), the Agency for Healthcare Research and Quality (AHRQ) Patient Education Materials Assessment Tool (PEMAT), and semi-structured interview questions based on Willis’ Cognitive Interviewing Techniques were used to evaluate the suitability, actionability, understandability and cultural appropriateness of the tools. ResultsPEMAT mean percentage scores of 71.4% for actionability and 82.4% for understandability, and a SAM suitability score of 67.9% were reported for the videos. Identified weaknesses captured in seven thematic areas after analysis of assessment by experts and community members guided the final refinement of the tools. ConclusionThe overall reviewers’ reports and evaluation scores indicate that, generally, the intervention tools are suitable for community deployment in the sub-Saharan African setting. Clinical researchers must clearly define their policy objectives, partner with key stakeholders, define desired behaviour change, and develop appropriate strategies, tools of engagement and communication channels.

A holistic approach to well-being and neglected tropical diseases: evaluating the impact of community-led support groups in Nigeria using community-based participatory research.

Neglected tropical diseases (NTDs) affect around 1 billion people, many living in the poorest parts of the world. NTDs often lead to serious long-term physical impairments. Stigma, disability, poverty and social isolation interact, resulting in poor quality of life and significant psychosocial impacts. The holistic health and psychosocial needs of persons affected by NTDs are often overlooked in integrated NTD programme design and research. Furthermore, the viewpoints of persons affected are often absent and spaces for empowerment and advocacy are limited. Using a community-based participatory research design, our study partnered with persons affected and caregivers as co-researchers to address this gap. Through the process, we co-designed and implemented community-based support groups in Kaduna and Kwara, Nigeria, where NTDs are endemic. This paper utilises photovoice with support group facilitators (persons affected); participant observation of group meetings; rapid micronarratives with support group members; and key informant interviews with programme implementers at the state and local government area levels to explore the impact of the support groups from the perspective of people affected by NTDs and other health system actors. Perceived impacts of the support groups included a sense of ownership and empowerment, stigma reduction, improved self-esteem, improved health knowledge and health outcomes and capacity strengthening through vocational training. Support groups, as community spaces of healing, offer a low-cost holistic intervention for chronic disease and disability.

‘Help us better understand our changing climate’: Exploring the discourse of Citizen Science

In this article I claim that online Citizen Science projects are exemplars of a digital genre that acts as text and medium. To support this claim I apply a previously proposed two-dimensional genre analytical model and develop empirical procedures to identify how ‘communicative purpose’ is realised by functional units/links, which in turn are realised by rhetorical strategies (verbal and visual) in two dimensions, the reading mode and the navigation mode. Empirical data show that this genre fulfils a set of distinct communicative purposes, namely to build credibility and trust in scientific research, to make specialised contents accessible to audiences with different levels of scientific literacy, to convey emotion and to build and maintain citizen-volunteers’ engagement. Such multifunctionality fulfils the social exigence of the genre, that is, supporting participatory science. The study contributes to the empirical characterisation of non-linear, multimodal genres taking into account the roles of text producer and text receiver.

Citizen science and participatory science communication: an empirically informed discussion connecting research and theory

Citizen Science is believed to contribute significantly to the democratisation of science, engaging non-scientists in scientific research. Participatory approaches to science communication share the same interest through public participation and public engagement. In the attempt to connect these two debates both theoretically and empirically, we provide an analysis of the communication tools and strategies used by 157 Citizen Science projects across the EU, UK, and Switzerland. Our analysis reports that the CS projects surveyed tend to interpret communication as a disseminating activity, rather than as a tool to promote appropriate communication-based encounters with both project participants and other potential target audiences.

Patient public involvement (PPI) in health literacy research: Engagement of adults with literacy needs in the co-creation of a hospital-based health literacy plan.

People with literacy needs can experience many challenges in accessing, understanding and using health services and health information. Such challenges can adversely impact patient-provider interactions and ultimately, health outcomes. Healthcare providers need to be aware of health literacy (HL) to address the demands of healthcare systems, improve their interactions with communities and patients and promote patient engagement for improved health outcomes. This paper reports on a process of patient and public involvement (PPI) with participants in an adult literacy programme acting as PPI contributors to identify priority areas for a local hospital HL action plan and to develop a protocol for a PPI process with other groups. A qualitative community-based participatory research study design informed by principles of PPI was undertaken, drawing on the tools of participatory and visual methods, open discussion and workshop format to facilitate a process of co-creation. Three workshops with six PPI contributors took place to identify issues to be included in the hospital action plan. PPI contributors identified issues and grouped these into priority areas using discussion and ranking procedures. Key areas prioritised for HL action by the PPI contributors were: verbal communication, emphasising the patient's right to understand, and improved understanding of medication use. These were incorporated into the action plan. The workshop format and process were deemed acceptable to the group and input on improvements will be incorporated into further work in this area. PPI acts as a lever in the knowledge translation process. Genuine engagement with service users can meaningfully contribute to relevant and sustainable changes to services as well as foster the empowerment of service users. Members of the public with literacy needs actively participated in the co-creation of a HL action plan for a local hospital and in the development of a protocol for a patient and public process for HL research.

Determinants of Acquisition, Persistence, and Clearance of Oncogenic Cervical Human Papillomavirus Infection in the Philippines Using a Multi-Omics Approach: DEFEAT HPV Study Protocol

HPV infection is one of the most studied risk factors in cervical cancer-the second most common cancer site and cause of death due to cancer in the Philippines. However, there is a lack of population-based epidemiological data on cervical HPV infection in the Philippines. Local reports on co-infections with other lower genital tract pathogens, commonly reported globally, are also lacking, which emphasizes the need to increase efforts in targeting HPV prevalence, genotype, and distribution. Hence, we aim to determine the molecular epidemiology and natural history of HPV infection among reproductive-age Filipino women using a community-based prospective cohort design. Women from rural and urban centers will be screened until the target sample size of 110 HPV-positive women (55 from rural sites and 55 from urban sites) is reached. Cervical and vaginal swabs will be collected from all screened participants. For HPV-positive patients, HPV genotypes will be determined. One hundred ten healthy controls will be selected from previously screened volunteers. The cases and controls will comprise the multi-omics subset of participants and will be followed up after 6 and 12 months for repeat HPV screening. Metagenomic and metabolomic analyses of the vaginal swabs will also be performed at baseline, after 6 months, and after 12 months. The results of this study will update the prevalence and genotypic distribution of cervical HPV infection among Filipino women, determine whether the current vaccines used for HPV vaccination programs capture the most prevalent high-risk HPV genotypes in the country, and identify vaginal community state types and bacterial taxa associated with the natural history of cervical HPV infection. The results of this study will be used as the basis for developing a biomarker that can help predict the risk of developing persistent cervical HPV infection in Filipino women.

On Dynamically Pricing Crowdsourcing Tasks

Crowdsourcing techniques have been extensively explored in the past decade, including task allocation, quality assessment, and so on. Most of professional crowdsourcing platforms adopt the fixed pricing scheme to offer a fixed price for crowd tasks. It is neither incentive for crowd workers to produce good performance, nor profitable for the requester to gain high utility with low budget. In this article, we study the problem of pricing crowdsourcing tasks with optional bonuses. We propose a dynamic pricing mechanism, named CrowdPricer for incentively delivering bonuses to the crowd workers of completing tasks, in addition to offering a base payment for completing a task. We leverage a deep time sequence model to learn the effect of bonuses on workers’ quality for crowd tasks. CrowdPricer makes decisions on whether to provide bonuses on workers, so as to maximize the requester’s utility in expectation. We present an efficient bonus delivery algorithm under the help of beam search technique, in order to efficiently solve the decision making problem. Extensive experiments using both a real crowdsourcing platform and simulations demonstrate that CrowdPricer yields the higher utility for the requester. It also obtains more correct crowd answers than the state-of-the-art pricing methods.

Investigating the challenges and opportunities for sustainable waterfront development in Jeddah City

Abstract The research paper discusses the challenges and opportunities surrounding sustainable waterfront development in Jeddah City, Saudi Arabia. The city faces several challenges such as inadequate infrastructure and hasty urbanization, but its strategic location, cultural heritage, and economic potential provide opportunities for sustainable development. The paper presents a comprehensive literature review of successful case studies from other cities and regions to identify the best practices and innovative methods for sustainable waterfront development in Jeddah City. The paper recommends green infrastructure, public–private partnerships and community-based planning and design to promote sustainable waterfront development. The paper emphasizes the need for an integrated and coordinated approach to waterfront development that focuses on enhancing resilience, preserving cultural heritage and promoting social equity. The paper suggests that further research is necessary to better understand the challenges and opportunities for sustainable waterfront development in Jeddah City. The areas that require attention include stakeholder engagement, the impact of climate change and the potential for alternative financing mechanisms. The findings of this research paper can guide policy makers, planners, developers and community leaders to make informed decisions toward sustainable waterfront development in Jeddah City.

Revisiting transformative WASH: measuring impact

Revisiting transformative WASH: measuring impact

Knowledge equity and Open Science: An attempt to outline the field from a feminist research perspective

Knowledge equity is a broad concept. Although it is linked to the goals of Open Science, it is rarely discussed in the scientific community. The term refers to a variety of aspects such as epistemology, research methods, data analysis, inclusive education, equal representation, participation, and science communication. It is reflected on individual, institutional, and structural levels. In this article, we attempt to outline the field theoretically against the background of a power-theoretical perspective and discuss what knowledge is in the first place. In a second step, we explore the question of what is hidden behind the terms equality and equity and to what extent these concepts can be linked to the underlying concept of knowledge. When can we speak of equity, why, and to what extent? Finally, the article links the overall social development of increasing sensitivity to diversity, which is discussed in conjunction with inclusive education and inclusion in general. Herein we refer to concepts of intersectional feminist research, the principles of Open Science, and a critical perspective on the concept of diversity. For illustration, exemplary projects associated with the Open Science Fellow Program, which address the issue of marginalized groups in the research process, are described. Among others, these relate to the following focal points: Data collection of non-binary gender, awareness of adultism, collaborative interpretation with interviewees, queer narratives, diversity in editorial boards, research in the context of North-South relations, participatory science communication using art, and exclusion factors of science communication. The overarching question we ask in this article is the extent to which knowledge equity is relevant to marginalized groups and exclusive dynamics in terms of an inclusive rationale and how those dynamics can be identified by using critical perspectives and self-reflexive considerations.

Community-Designed Participation: Lessons for Equitable Engagement in Transportation Planning

Despite years of legally mandated public engagement for transportation planning, there is often little evidence that this results in more equitable processes or outcomes. Recently, there has been interest in improving engagement by having community-based or advocacy groups design, lead, and implement public engagement activities. This research examines two separate engagement processes—one led by a public agency, and one designed and carried out by community advocates—to understand the opportunities and barriers for community-led engagement in transportation planning. We assess how these processes differed in: (1) representation of equity-deserving groups in respondents, (2) conceptualization of equity and community needs, and (3) transportation priorities identified in the surveys. While neither process fully reflected city demographics, the community-led process was more representative of equity-deserving groups. We found key differences in priorities between the community- and agency-led surveys, and by respondent identity. Areas that were identified as a high priority in the agency-led survey, such as traffic congestion, were lowly ranked in the community-led survey, as respondents prioritized safety and lower fares. Critically, community- and agency-led processes used substantially different framings of transportation equity, along with different understandings of community needs and experiences, which could have a significant impact on the development of future transportation plans. Community-led strategies require significant resources and capacity to undertake, but meaningful participation in the design and implementation of engagement processes has the potential to better engage a diversity of perspectives and reflect community priorities.

Uncertainty is not sufficient for identifying noisy labels in training data for binary segmentation of building footprints

Obtaining high quality labels is a major challenge for the application of deep neural networks in the remote sensing domain. A common way of acquiring labels is the usage of crowd sourcing which can provide much needed training data sets but also often contains incorrect labels which can affect the training process of a deep neural network significantly. In this paper, we exploit uncertainty to identify a certain type of label noise for semantic segmentation of buildings in satellite imagery. That type of label noise is known as “omission noise,” i.e., missing labels for whole buildings which still appear in the satellite image. Following the literature, uncertainty during training can help in identifying the “sweet spot” between generalizing well and overfitting to label noise, which is further used to differentiate between noisy and clean labels. The differentiation between clean and noisy labels is based on pixel-wise uncertainty estimation and beta distribution fitting to the uncertainty estimates. For our study, we create a data set for building segmentation with different levels of omission noise to evaluate the impact of the noise level on the performance of the deep neural network during training. In doing so, we show that established uncertainty-based methods to identify noisy labels are in general not sufficient enough for our kind of remote sensing data. On the other hand, for some noise levels, we observe some promising differences between noisy and clean data which opens the possibility to refine the state-of-the-art methods further.