Crowdsourcing Method Research Articles

Advantages of AI-Processed Crowdsourcing Method in Map Data Update Mechanism of Navigation System of Autonomous Driving

This research delves into the advantages of the AI-processed crowdsourcing method in the map data update mechanism for the navigation system of autonomous driving. It explores how this innovative approach offers high real-time performance, enabling immediate capture of road changes. The method facilitates large-scale data collection from a wide range of sources, ensuring comprehensive coverage. Cost-effectiveness is achieved by leveraging existing user infrastructure and AI-driven data processing. Data accuracy and reliability are enhanced through crowdsourcing method and AI cleaning algorithms. Additionally, it examines real-world application cases, highlighting successful implementation and outcomes. Looking towards the future, potential developments and challenges are discussed, emphasizing the need for continuous improvement in areas like data security and standardization. The conclusion asserts that with strategic mitigation of challenges, this method holds great promise in advancing autonomous driving technology for safer and more efficient navigation.

3D Pop-Ups: Omnidirectional image visual saliency prediction based on crowdsourced eye-tracking data in VR

The prediction of the visual saliency of omnidirectional images in VR is valuable for understanding visual behaviors. However, the equipment cost, software setups, hardware operation, and other constraints in acquiring eye-tracking data of omnidirectional images for visual saliency prediction would lead to a low training efficiency and prediction performance. Therefore, this paper proposed a crowdsourcing method based on recall fixations, which was used to collect and construct an omnidirectional image with eye-tracking dataset called CrowdSourcing360, which contained 16,200 pieces of data on 180 images. Using this dataset, a visual saliency prediction model CSnet360 was trained. Experiments demonstrated that the visual saliency prediction performance of the CSnet360 outperformed most existing models even without using actual gaze fixations. Finally, a VR interior design assistance prototype system was built and the preliminary study results indicated that the system could help designers to improve the quality of their design solutions.

Active Crowdsourcing for Multilabel Annotation.

Multilabel annotation is a critical step to generate training sets when learning classification models in various application domains, but asking domain experts to provide labels is usually time-consuming and expensive, which cannot meet the current requirement of the fast evolution of the models in the big data era. Although crowdsourcing provides a fast solution to acquire labels for multilabel learning, it faces the risk of high data acquisition cost and low label quality. This article proposes a novel one-coin label-dependent active crowdsourcing (OCLDAC) method to iteratively query noisy labels from crowd workers and learn multilabel classification models. In each iteration of active learning, integrated labels of instances are first inferred by a novel one-coin label-dependent model, which utilizes a mixture of multiple independent Bernoulli distributions to explore and exploit correlations among the labels to increase the accuracy of truth inference. Then, instances, labels, and workers are selected according to the novel strategies that incorporate the distribution of noisy labels, the prediction probability of learning models, label correlations, and the reliability of crowd workers. Simulations on eight multilabel datasets and evaluation on one real-world crowdsourcing dataset consistently show that the proposed OCLDAC significantly outperforms the state-of-the-art methods and their variants.

E-Culture Design in Batubara District by Implementing Web-Based Crowdsourcing Methods

The development of technology today is very rapid and makes it easier for humans to access various information that they want to know by using technology in the form of the internet. E-Cultural is a cultural digitization system that utilizes internet technology to increase efficiency in the field of culture, especially in terms of documentation, dissemination of information as well as knowledge of elements of culture. At this time, web-based information regarding tourism and culture of Batubara Regency is not yet available, the public can only introduce it through social media such as Facebook, Instagram and other media. Lack of legal media at the Batu Bara District Youth, Sports and Culture Service and Tourism to promote tourism and culture in the Batubara District. This prompted the author to create a legal system so that the Batubara Regency Cultural Sports Youth Service can promote tourism and culture by involving the global community. Therefore, this research uses the crowdsourcing method, namely that there is unlimited involvement and regardless of background for everyone who wants to make a contribution, whether paid or free. In the system that I will create, the global community can play a role in updating data or information on the website as well as using the waterfall development method, namely analysis, design, implementation and testing. The results of the development of E-Culture in Batubara Regency using the crowdsourcing method have had a good impact, by making it easier for the local community to promote culture and tourism in their area

The role of open innovation in biomarker discovery

Precision medicine aims to treat diseases with special consideration for the individual biological variability.Novel biomarkers (BM) are needed to predict therapeutic responses and to allow for the selection of suitable patients fortreatment with certain drugs. However, the identification and validation of appropriate BMs is challenging. Close collaboration between different partners seems to be a key success factor. While the importance of partnerships and larger,well-established consortia in BM discovery such as the pharmaceutical industry and academic institutions is well understood and has been investigated in the past, the use of open-innovation models, also known as ‘crowd sourcing for biomarkers’, is still in its infancy. Crowd sourcing comprises of a — usually via internet — request for problem solution to an open group of users in a kind of an ‘open call’. The community (crowd) is asked to provide solutions. Since the application of the crowd sourcing method offers the possibility to collect as many as possible novel ideas from a broad community with different expertise, this approach is particularly promising for BM development. In this article we describe the first examples of open-innovation models, such as the ‘grants for targets’ (G4T) and biomarkers initiative ‘InnoCentive’ (innovation/incentive) platform. They may be a fruitful basis for collaborative BM development in the future.

The advantage of mobile technologies in crowdsourcing landscape preferences: Testing a mobile app to inform planning decisions

Urban green spaces are vital for cities to achieve sustainability. Optimal design and management of green space is necessary to counter pressure from increased use due to urban densification and therefore needs to account for the preferences of its potential users. However, preference is inherently subjective and difficult to measure. This study tested a crowdsourcing method using a mobile app for on-site data collection to study the use of, perception of, and preferences for different green configurations in the Rembrandtpark, a large urban park in Amsterdam. Responses obtained through the mobile app were compared to those obtained through more traditional on-site and postal surveys to identify differences in sampled population and results. We found clear differences in respondent samples and in stated preferences when comparing the mobile app and the traditional survey methods. Mobile app respondents (n = 154) were younger and more willing to become more involved with park management. They were also less satisfied with most aspects of locations and found them more noisy. On-site survey respondents (n = 146) appeared more susceptible to response bias in their answers to certain questions. Employing different recruitment methods is important to engage different park user types and account for multiple perceptions.

Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures

AbstractTopic models, as developed in computer science, are effective tools for exploring and summarizing large document collections. When applied in social science research, however, they are commonly used for measurement, a task that requires careful validation to ensure that the model outputs actually capture the desired concept of interest. In this paper, we review current practices for topic validation in the field and show that extensive model validation is increasingly rare, or at least not systematically reported in papers and appendices. To supplement current practices, we refine an existing crowd-sourcing method by Chang and coauthors for validating topic quality and go on to create new procedures for validating conceptual labels provided by the researcher. We illustrate our method with an analysis of Facebook posts by U.S. Senators and provide software and guidance for researchers wishing to validate their own topic models. While tailored, case-specific validation exercises will always be best, we aim to improve standard practices by providing a general-purpose tool to validate topics as measures.

Kitle Kaynak Yönteminin İstihbarat Üretimindeki Yeri: Ushahidi Platformu Örneği

Bu çalışmanın ana amacı kitle kaynak yönteminin istihbarat üretimine getirdiği yeni bakış açıları ve uygulama alanlarının tespit edilmesidir. Bunun için nitel araştırma yöntemi kullanılmış, kitle kaynağın kavramsal altyapısının ve uygulama alanlarının istihbarat üretimine ve organizasyonel yapısına yeni perspektifler getirip getirmediği sorusu üzerinde durulmuştur. Çalışma bu minvalde iki bölüme ayrılmıştır. İlk bölümde kitle kaynak kavramı irdelenmiştir. Bunun için çalışmanın teorik çerçevesini de oluşturan kitlelerin bilgeliği fikri, kitlenin teknoloji ile etkileşimi ve yeni sosyal sınıflar haline gelen Pro-Amatların ve üreketicilerin ortaya çıkışı işlenmiştir. İkinci bölümünde ise bir haber üretim platformu olan Ushahidi tanımlanmış, 2010’daki Haiti depremi sonrası kullanımı ve ürettiği istihbaratın mahiyeti incelenmiştir.

지식 추출을 위한 효율적인 다중 작업 크라우드소싱 기법

지식 추출이란 자연언어 문장으로부터 컴퓨터가 이해할 수 있는 지식 형태의 정보를 추출하는 작업으로, 지식베이스를 생성 및 확장하고 이를 이용해 질의 응답, 대화 에이전트 등 다양한 응용 분야에 활용된다. 지식 추출을 위해서는 주어진 자연언어 문장 내에 지식베이스에 연결 가능한 개체를 발견하고 그 개체들 간의 의미적 관계를 파악하는 과정이 필수적으로 수반된다. 본 논문에서는 크라우드소싱으로 개체 연결과 관계 추출을 위한 데이터 셋을 효과적으로 제작하는 방법을 제안한다. 지식 추출에는 다중 작업 설계가 필요한데 효율적인 데이터 주석 작업 틀을 설계하고, 또한 양질의 데이터를 수집하기 위해 작업자들을 교육, 데이터 품질을 지속적으로 자동 검수, 그리고 악성 작업자를 걸러내는 장치들을 도입하였다. 본 논문에서 수집한 데이터와 방법의 우수성을 입증하기 위해, 지식 추출에 필요한 여러 가지 모델들을 학습하고 평가한 결과, 본 논문의 크라우드소싱 데이터가 성능 향상에 기여했다는 것을 확인하였다.

Crowd-sourcing as a novel tool to measure financial toxicity in patients with ovarian cancer

Objective: Our objective was to use a novel online crowd-sourcing method to measure financial toxicity; quality of life (QOL); and cost-coping strategies among individuals diagnosed with ovarian cancer.

Visual saliency model based on crowdsourcing eye tracking data and its application in visual design

The visual saliency models based on low-level features of an image have the problem of low accuracy and scalability, while the visual saliency models based on deep neural networks can effectively improve the prediction performance, but require a large amount of training data, e.g., eye tracking data, to achieve good results. However, the traditional eye tracking method is limited by high equipment and time cost, complex operation process, low user experience, etc. Therefore, this paper proposed a visual saliency model based on crowdsourcing eye tracking data, which was collected by gaze recall with self-reporting from crowd workers. Parameter optimization on our crowdsourcing method was explored, and it came out that the accuracy of gaze data reached 1° of visual angle, which was 3.6% higher than other existed crowdsourcing methods. On this basis, we collected a webpage dataset of crowdsourcing gaze data and constructed a visual saliency model based on a fully convolutional neural network (FCN). The evaluation results showed that after trained by crowdsourcing gaze data, the model performed better, such as prediction accuracy increased by 44.8%. Also, our model outperformed the existing visual saliency models. We also applied our model to help webpage designers evaluate and revise their visual designs, and the experimental results showed that the revised design obtained improved ratings by 8.2% compared to the initial design.

Partnership in Weather Observation using the Crowdsourcing Method

Proper weather observation is definitely required to obtain the right and accurate weather information. Currently the weather observation in Indonesia is managed by various parties and coordinated by the Meteorology, Climatology and Geophysics Agency (BMKG). BMKG manage weather observation networks that distributed nationwide. Unfortunately, in terms of quantity and distribution, the current weather observation network is not ideal yet. Increasing the density of the weather observation network is undoubtly needed, therefore BMKG needs to develop a partnership with all parties including general public. This study attempts to identify level of participation of general public and find the ideal partnership model for weather observation in Indonesia. Three partnership variables, namely motivation, consistency and sustainability, examined in this study. As a result of the data analysis it was found that the level of participation of general public is very high especially in the participant group that threat with guidance. And it was also found that the crowdsourcing method can be implemented as an ideal partnership model between BMKG and general public in the field of weather observation.

Capturing Resilience in Context: Development and Validation of a Situational Judgment Test of Resilience

ABSTRACT We developed a 40-item situational judgment test (R-SJT) to capture resilience in context. The scoring key was developed using a crowd-sourcing method (N = 100) and pilot-tested (N = 62). A new sample (N = 208) was used to investigate validity of the R-SJT scores. The R-SJT scores demonstrated acceptable levels of internal consistency, convergent validity, and evidence of better discriminant and criterion-related validity, compared to three Likert-type resilience measures. Scores on the R-SJT moderately correlated with the three resilience scales and personality measures including hardiness, negative affect, agreeableness, conscientiousness, and neuroticism. The R-SJT showed incremental validity beyond traditional resilience measures and personality for criterion measures of psychological adjustment (i.e., depression, anxiety, and stress). The R-SJT appears to be a viable instrument to measure resilience.

A Blockchain-Powered Crowdsourcing Method With Privacy Preservation in Mobile Environment

Crowdsourcing is a booming technique that enables participants to exchange data directly, thus making it possible to answer latency-sensitive service requests and relieve the burden of core networks. With some incentives, providers compete to furnish service requests, thus pledging the quality of experience (QoE) for requestors. However, the decentralized communication in crowdsourcing increases the probability of information tapering. Furthermore, providers’ arbitrary selection of the requests poses great threat to the efficient and profitable service provision for the requestors. To deal with these challenges, we propose a blockchain-powered crowdsourcing method, named BPCM, while considering the privacy preservation in mobile environment. Specifically, a mobile crowdsourcing framework based on blockchain is designed first to preserve the privacy of the participants and keep the integrity of the service request and provision. Then, density-based spatial clustering of applications with noise (DBSCAN) and improved dynamic programming (IDP) are adopted to cluster the requestors and generate service strategies, respectively. Furthermore, simple additive weighting (SAW) and multiple criteria decision making (MCDM) are utilized to select the optimal strategy that achieves the tradeoffs among maximizing the service time, increasing the profits, and reducing the energy consumption for the providers. Finally, comprehensive experiments are conducted to verify the accuracy and effectiveness of BPCM.

Classifying the Category of Workers using Crowd-sourced Job Seekers Data

Crowdsourcing refers to decomposing complex jobs to multiple tasks and solve those task with multiple workers through open call in distributed networking environment. The recruitment of employees for organization has undergone transformation from traditional method to digital domain. Online recruitment facilitates just-in-time hiring to requesters and enables the workers to compete in the global market. This paper proposed an Efficient Machine Learning Crowdsourced(EMLC) method for E-recruitment which uses Crowdsourcing method to collect resumes from the workers and details of work from the requesters. The data is collected from a private job agency through an online recruitment portal which consist of recruiters from companies and job seekers based on qualifications and experience related to their field. The data collected from recruitment portal is analyzed with Machine Learning Approach with decision tree algorithms like ID3, CART and C4.5 for better selection of efficient person to complete the job. Various performance metrics such as Accuracy, Error rate, Recall etc were used to the Crowdsourced Database to categorize the job seekers efficiently. The proposed method gives better result for online recruiting through Crowdsourcing.

Scalable and efficient learning from crowds with Gaussian processes

Over the last few years, multiply-annotated data has become a very popular source of information. Online platforms such as Amazon Mechanical Turk have revolutionized the labelling process needed for any classification task, sharing the effort between a number of annotators (instead of the classical single expert). This crowdsourcing approach has introduced new challenging problems, such as handling disagreements on the annotated samples or combining the unknown expertise of the annotators. Probabilistic methods, such as Gaussian Processes (GP), have proven successful to model this new crowdsourcing scenario. However, GPs do not scale up well with the training set size, which makes them prohibitive for medium-to-large datasets (beyond 10K training instances). This constitutes a serious limitation for current real-world applications. In this work, we introduce two scalable and efficient GP-based crowdsourcing methods that allow for processing previously-prohibitive datasets. The first one is an efficient and fast approximation to GP with squared exponential (SE) kernel. The second allows for learning a more flexible kernel at the expense of a heavier training (but still scalable to large datasets). Since the latter is not a GP-SE approximation, it can be also considered as a whole new scalable and efficient crowdsourcing method, useful for any dataset size. Both methods use Fourier features and variational inference, can predict the class of new samples, and estimate the expertise of the involved annotators. A complete experimentation compares them with state-of-the-art probabilistic approaches in synthetic and real crowdsourcing datasets of different sizes. They stand out as the best performing approach for large scale problems. Moreover, the second method is competitive with the current state-of-the-art for small datasets.

Use of ontology structure and Bayesian models to aid the crowdsourcing of ICD-11 sanctioning rules

The International Classification of Diseases (ICD) is the de facto standard international classification for mortality reporting and for many epidemiological, clinical, and financial use cases. The next version of ICD, ICD-11, will be submitted for approval by the World Health Assembly in 2018. Unlike previous versions of ICD, where coders mostly select single codes from pre-enumerated disease and disorder codes, ICD-11 coding will allow extensive use of multiple codes to give more detailed disease descriptions. For example, “severe malignant neoplasms of left breast” may be coded using the combination of a “stem code” (e.g., code for malignant neoplasms of breast) with a variety of “extension codes” (e.g., codes for laterality and severity). The use of multiple codes (a process called post-coordination), while avoiding the pitfall of having to pre-enumerate vast number of possible disease and qualifier combinations, risks the creation of meaningless expressions that combine stem codes with inappropriate qualifiers. To prevent that from happening, “sanctioning rules” that define legal combinations are necessary. In this work, we developed a crowdsourcing method for obtaining sanctioning rules for the post-coordination of concepts in ICD-11. Our method utilized the hierarchical structures in the domain to improve the accuracy of the sanctioning rules and to lower the crowdsourcing cost. We used Bayesian networks to model crowd workers’ skills, the accuracy of their responses, and our confidence in the acquired sanctioning rules. We applied reinforcement learning to develop an agent that constantly adjusted the confidence cutoffs during the crowdsourcing process to maximize the overall quality of sanctioning rules under a fixed budget. Finally, we performed formative evaluations using a skin-disease branch of the draft ICD-11 and demonstrated that the crowd-sourced sanctioning rules replicated those defined by an expert dermatologist with high precision and recall. This work demonstrated that a crowdsourcing approach could offer a reasonably efficient method for generating a first draft of sanctioning rules that subject matter experts could verify and edit, thus relieving them of the tedium and cost of formulating the initial set of rules.

CLRIC: Collecting Lane-Based Road Information Via Crowdsourcing

Lane-based road network information, such as the number and locations of traffic lanes on a road, has played an important role in intelligent transportation systems. In this paper, we propose a Collecting Lane-based Road Information via Crowdsourcing (CLRIC) method, which can automatically extract detailed lane structure of roads by using crowdsourcing data collected by vehicles. First, CLRIC filters the high-precision GPS data from the raw trajectories based on region growing clustering with prior knowledge. Second, CLRIC mines the number and locations of traffic lanes through optimized constrained Gaussian mixture model. Experiments are conducted with taxi GPS trajectories in Wuhan, China, and the results show that CLRIC is quantified and displays detailed road networks with the number and locations of traffic lanes comparing with the satellite image and human-interpreted situation.

Crowdsourcing Vector Surveillance: Using Community Knowledge and Experiences to Predict Densities and Distribution of Outdoor-Biting Mosquitoes in Rural Tanzania.

Lack of reliable techniques for large-scale monitoring of disease-transmitting mosquitoes is a major public health challenge, especially where advanced geo-information systems are not regularly applicable. We tested an innovative crowd-sourcing approach, which relies simply on knowledge and experiences of residents to rapidly predict areas where disease-transmitting mosquitoes are most abundant. Guided by community-based resource persons, we mapped boundaries and major physical features in three rural Tanzanian villages. We then selected 60 community members, taught them basic map-reading skills, and offered them gridded maps of their own villages (grid size: 200m×200m) so they could identify locations where they believed mosquitoes were most abundant, by ranking the grids from one (highest density) to five (lowest density). The ranks were interpolated in ArcGIS-10 (ESRI-USA) using inverse distance weighting (IDW) method, and re-classified to depict areas people believed had high, medium and low mosquito densities. Finally, we used odor-baited mosquito traps to compare and verify actual outdoor mosquito densities in the same areas. We repeated this process for 12 months, each time with a different group of 60 residents. All entomological surveys depicted similar geographical stratification of mosquito densities in areas classified by community members as having high, medium and low vector abundance. These similarities were observed when all mosquito species were combined, and also when only malaria vectors were considered. Of the 12,412 mosquitoes caught, 60.9% (7,555) were from areas considered by community members as having high mosquito densities, 28% (3,470) from medium density areas, and 11.2% (1,387) from low density areas. This study provides evidence that we can rely on community knowledge and experiences to identify areas where mosquitoes are most abundant or least abundant, even without entomological surveys. This crowd-sourcing method could be further refined and validated to improve community-based planning of mosquito control operations at low-cost.

A Crowdsourcing Method for Obtaining Rephrased Questions

We propose a method for obtaining and ranking paraphrased questions from crowds to be used as a part of instructions in microtask-based crowdsourcing. With our method, we are able to obtain questions that differ in expression yet have the same semantics with respect to the crowdsourcing task. This is done by generating tasks that give hints and elicit instructions from workers. We conducted experiments with data used for a real set of gold standard questions submitted to a commercial crowdsourcing platform and compared the results with those from a direct-rewrite method.