Crowdsourcing Platform Research Articles

Optimal innovation incentive strategies within the health crowdsourcing platform

AbstractLack of incentive strategies within the health crowdsourcing platform has resulted in the loss of patient innovation and the failure in the diffusion of patients' innovations. To solve this problem, we compared four incentive strategies: (i) without incentive, (ii) cost‐sharing incentive, (iii) revenue‐sharing incentive, and (iv) a hybrid incentive strategy combining cost‐sharing and revenue‐sharing. Our analysis indicates that the innovation guidance effect is at its lowest level without incentive strategies, and the hybrid incentive strategy can maximize both innovation enthusiasm and profits. Furthermore, there exists a threshold for revenue‐sharing ratio, making the incentive effect of revenue‐sharing strategy better than that of the cost‐sharing strategy. Finally, short‐sighted thinking among platform participants will reduce the innovation enthusiasm of altruistic patients and the incentive investment of the platform. This study provides guiding principles for the health crowdsourcing platform to regulate online patient innovation solutions.

Empowering Crisis Response Efforts: A Novel Approach to Geolocating Social Media Images for Enhanced Situational Awareness

Social media and online platforms play an important role in assessing the impact of a natural disaster, especially in the immediate aftermath of the event. The information from social media demonstrated the potential to accelerate the response to a crisis. Locating social media images is critically important to help the victims immediately, mobilize community support, and provide enhanced situational awareness. However, finding the location of relevant social media images still challenges humans and computational processes. This study introduces a social media image classifier aimed at enhancing crowdsourced geolocation. The model is trained using data annotated by experts in disaster risk management. Its main goal is to reduce the difficulty of geolocating images by detecting those that are ``easy'' to geolocate. We combined the classifier with an ad-hoc crowdsourcing platform and tested it with pictures posted during a crisis. The experimental results indicate that the proposed approach speeds up the geolocation process of social media images while increasing the level of location precision

Dueling Contests and Platform’s Coordinating Role

Crowdsourcing platforms typically take a passive approach, and they let the competing firms freely design their own contests and allow every solver to self-select and join any of the concurrently running contests. In a model of competing noise-driven contests, we show that the duopoly prize allocation has fewer (but larger) prizes compared with a monopolist contest designer. We also find that contests with firm-chosen budgets and solvers’ endogenous participation create coordination inefficiencies. Thus, platform policies that constrain the competing firms from freely choosing their budgets and offer solvers non-enforceable recommendations toward specific noise-driven contests strictly enhance total welfare. Extending our framework to include arbitrarily correlated ability-driven contests, we highlight the critical role of inter-contest dependence on the efficacy of a platform’s interventions. Specifically, platform nudges to improve solver-contest (mis-)matches are welfare enhancing only when the contests are sufficiently related, and allowing solvers to self-sort is appropriate otherwise. This paper was accepted by Gabriel Weintraub, revenue management and market analytics. Supplemental Material: The online appendix is available at https://doi.org/10.1287/mnsc.2021.03973 .

Participatory Framework for Urban Pluvial Flood Modeling in the Digital Twin Era

The recent advancement in digital twin technology, which creates virtual replicas of real-world processes, offers an interactive testbed for understanding and predicting environmental changes. As pluvial flood damage escalates globally in urban areas, there remains a gap in understanding the most effective collaboration between governments and local residents for sustainable flood risk management. To address this gap, we develop a participatory framework for urban pluvial flood modeling, incorporating open source software, virtual reality, minimum viable product, and gamification components. This framework engages citizens in every phase of the participatory modeling process, from input data preparation, through hydrological model construction, to model verification, and experiments. We present a case study on the recurring pluvial flood damages in South Korea's Gangnam region, demonstrating the practical implications of an interactive, web-based crowdsourcing platform to leverage community engagement and local knowledge. The results underscore the evidence of a proactive role for citizens, not merely as recipients of disaster information but as key contributors collaborating with a range of stakeholders in stormwater management and modeling. Combining digital twin technology with citizen participation can empower informed decision-making and collective actions in the evolving digital era, leading to a disaster-resilient community.

Person-centred data sharing: Empirical studies in private individuals’ attitudes

Background Recognising the power of data analytics, researchers are anxious to gain access to personal data either directly from data subjects or via research data sets. This requires a secure environment, such as a trusted research environment (TRE). However, it is unclear how the data subjects themselves regard sharing their data with TREs, especially if research goals are difficult to specify upfront or data are used for secondary purposes, making informed consent difficult to manage. We review three empirical studies to throw some light on individual attitudes to sharing health data. Methods Three anonymous, online surveys were run. The first involving 800 UK residents aimed at understanding how participants view the health data security. The second involving 500 UK residents aimed at identifying private individual views on privacy. These two surveys used a crowdsourcing platform. The third involved 1086 students at a UK university reporting their engagement with a trial diagnostic method for SARS-CoV-2. Results The first survey demonstrated that private individuals could make security decisions though they usually assume the recipient of their personal data to be responsible for all aspects of keeping the data safe. The second highlighted that individuals were aware of privacy risks but are motivated to share their data based on different contextual assumptions. The third, involving the incidental sharing of sensitive data during the SARS-CoV-2 pilot highlighted that prosocial motivations override potential personal benefit of such testing. Conclusions The three, unconnected surveys make clear that there are tensions between private individual understanding of data security and privacy risk, on the one hand, and how they behave, on the other. Respondents rely on data stewards to keep their data safe, though are likely to share even sensitive data for prosocial benefit. These findings have implications for those offering TRE services for research.

Crowdsourced Assessment of Aesthetic Outcomes of Dorsal Preservation Rhinoplasty

Abstract Goals/Purpose The inherent subjectivity in evaluating aesthetic outcomes presents a unique challenge in assessing rhinoplasty. Crowdsourcing has provided a new metric for objective analysis. The authors designed a study to compare the aesthetic outcomes of dorsal preservation rhinoplasty versus structural rhinoplasty using a reliable and valid crowdsourcing platform. We aim to objectively quantify the relative aesthetic advantages of performing a dorsal preservation technique. Additionally, we aim to demonstrate the efficacy of using crowdsourcing as an efficient and reliable method for evaluating any plastic surgery aesthetic outcome. Methods/Technique This retrospective observational study was approved by the IRB at UCLA. A total of 64 patients who had previously undergone rhinoplasty performed by the senior author were included. All surgeries were performed at the UCLA Ronald Reagan Medical Center. Patients with prior nasal surgery or trauma were excluded. All subjects were photographed using standard rhinoplasty views during the pre-and postoperative visits. Frontal and right profile photographs were then cropped in a standardized fashion (Photos, Version 8.0, Apple Inc.) and used for evaluation. All evaluations and ratings were collected using the HIPPA compliant and encrypted software via a secure online platform (LoveMyDelta, Inc., Philadelphia, PA). All crowd workers were recruited and vetted through this platform, with internal fidelity checks performed. Each crowd worker was shown a profile and frontal view of a patient and asked to score the overall appearance of the nose, the nasal dorsum profile, the symmetry of the dorsal aesthetic lines, and the dorsal contour using a visual analog scale. Scoring was performed using a provided validated visual assessment guide. They were asked whether they believed the subject had undergone rhinoplasty and asked to provide a level of confidence in their guess. Crowdworkers’ responses were aggregated to allow for high-powered intra-rater analysis. Intra-rater reliability and confidence intervals were calculated. Aggregate data from all raters were used to generate an absolute value of aesthetic state for each cohort. A "delta" was then obtained for each value by comparing the preoperative to the postoperative state to obtain a representative value of the improvement after undergoing surgery. Raters were asked if the patient appeared to have had surgery, and each correct and incorrect response was proportionally weighted based on the raters' confidence in their answer. The delta (difference in preoperative to postoperative aesthetic state) for each parameter for the dorsal and non-dorsal preservation cohorts was calculated using non-paired T-tests to determine significance. Results/Complications A total of 64 patients were included for evaluation. The structural rhinoplasty cohort consisted of 34 patients. The dorsal preservation cohort included 30 patients. Both dorsal preservation and non-dorsal preservation rhinoplasty cohorts were associated with improved overall crowdsourced aesthetic outcomes and improved outcomes across all subparameters. Overall, delta (d) improved in both dorsal preservation (0.300, 95% CI +/- 0.047) and structural (0.377, 95% CI +/- 0.055) cohorts. When raters were asked to predict whether a patient had surgery, the correlation coefficient between the confidence in the prediction and that prediction being correct was 0.74 (95% CI +/- 0.47 in the structural group, suggesting that a crowd worker was likely to be correct in identifying whether a patient had structural rhinoplasty. However, the correlation coefficient was -0.0554 (95% CI +/- 0.058) in the dorsal preservation group. This suggests the raters were unable to consistently identify which patients had dorsal preservation rhinoplasty with any confidence. Conclusion In the ongoing challenge to achieve better aesthetic outcomes in plastic surgery, crowdsourcing is a valuable tool for helping to measure these outcomes. We compared the aesthetic outcomes of dorsal preservation rhinoplasty versus structural rhinoplasty using a reliable and valid crowdsourcing platform. We found that significant improvements in overall aesthetic outcomes were achieved with both techniques, while a more natural “unoperated” outcome was achieved when performing a dorsal preservation technique. With this type of study and assessment tool, we helped to demonstrate the efficacy of using crowdsourcing as an efficient and reliable method for evaluating any plastic surgery aesthetic outcome.

Commitment on Volunteer Crowdsourcing Platforms: Implications for Growth and Engagement

Problem definition: Volunteer crowdsourcing platforms match volunteers with tasks that are often recurring. To ensure completion of such tasks, platforms frequently use a lever known as “adoption,” which amounts to a commitment by the volunteer to repeatedly perform the task. Despite reducing match uncertainty, high levels of adoption can decrease the probability of forming new matches, which in turn can suppress growth. We study how platforms should manage this trade-off. Our research is motivated by a collaboration with Food Rescue U.S. (FRUS), a volunteer-based food recovery organization active in more than 30 locations. For platforms such as FRUS, effectively using nonmonetary levers, such as adoption, is critical. Methodology/results: Motivated by the volunteer management literature and our analysis of FRUS data, we develop a model for two-sided markets that repeatedly match volunteers with tasks. We study the platform’s optimal policy for setting the adoption level to maximize the total discounted number of matches. When market participants are homogeneous, we fully characterize the optimal myopic policy and show that it takes a simple extreme form: depending on volunteer characteristics and market thickness, either allow for full adoption or disallow adoption. In the long run, we show that such a policy is either optimal or achieves a constant-factor approximation. We further extend our analysis to settings with heterogeneity and find that the structure of the optimal myopic policy remains the same if volunteers are heterogeneous. However, if tasks are heterogeneous, it can be optimal to only allow adoption for the harder-to-match tasks. Managerial implications: Our work sheds light on how two-sided platforms need to carefully control the double-edged impacts that commitment levers have on growth and engagement. Setting a misguided adoption level may result in marketplace decay. At the same time, a one-size-fits-all solution may not be effective, as the optimal design crucially depends on the characteristics of the volunteer population. Supplemental Material: The online appendices are available at https://doi.org/10.1287/msom.2020.0426 .

Methodological considerations for behavioral studies relying on response time outcomes through online crowdsourcing platforms

This perspective paper explores challenges associated with online crowdsourced data collection, particularly focusing on longitudinal tasks with time-sensitive outcomes like response latencies. Based on our research, we identify two significant sources of bias: technical shortcomings such as low, variable frame rates, and human factors, contributing to high attrition rates. We explored potential solutions to these problems, such as enforcing hardware acceleration and defining study-specific frame rate thresholds, as well as pre-screening participants and monitoring hardware performance and task engagement over each experimental session. With this discussion, we intend to provide recommendations on how to improve the quality and reliability of data collected via online crowdsourced platforms and emphasize the need for researchers to be cognizant of potential pitfalls in online research.

Efficient Online Crowdsourcing with Complex Annotations

Crowdsourcing platforms use various truth discovery algorithms to aggregate annotations from multiple labelers. In an online setting, however, the main challenge is to decide whether to ask for more annotations for each item to efficiently trade off cost (i.e., the number of annotations) for quality of the aggregated annotations. In this paper, we propose a novel approach for general complex annotation (such as bounding boxes and taxonomy paths), that works in an online crowdsourcing setting. We prove that the expected average similarity of a labeler is linear in their accuracy conditional on the reported label. This enables us to infer reported label accuracy in a broad range of scenarios. We conduct extensive evaluations on real-world crowdsourcing data from Meta and show the effectiveness of our proposed online algorithms in improving the cost-quality trade-off.

Introducing the 3MT_French dataset to investigate the timing of public speaking judgements

AbstractIn most public speaking datasets, judgements are given after watching the entire performance, or on thin slices randomly selected from the presentations, without focusing on the temporal location of these slices. This does not allow to investigate how people’s judgements develop over time during presentations. This contrasts with primacy and recency theories, which suggest that some moments of the speech could be more salient than others and contribute disproportionately to the perception of the speaker’s performance. To provide novel insights on this phenomenon, we present the 3MT_French dataset. It contains a set of public speaking annotations collected on a crowd-sourcing platform through a novel annotation scheme and protocol. Global evaluation, persuasiveness, perceived self-confidence of the speaker and audience engagement were annotated on different time windows (i.e., the beginning, middle or end of the presentation, or the full video). This new resource will be useful to researchers working on public speaking assessment and training. It will allow to fine-tune the analysis of presentations under a novel perspective relying on socio-cognitive theories rarely studied before in this context, such as first impressions and primacy and recency theories. An exploratory correlation analysis on the annotations provided in the dataset suggests that the early moments of a presentation have a stronger impact on the judgements.

Online Search Strategies and Results From a Crowdsourced Survey on Asymptomatic Bacteriuria.

Despite the prevalence of asymptomatic bacteriuria (ASB), what proportion of the population is aware of this condition and the quality of internet resources are currently unknown. This study aimed to use an online crowdsourcing platform to explore general knowledge and internet search strategies, along with the quality of information, on ASB. An online survey was administered through a crowdsourcing platform to women 50 years or older via Qualtrics, which is a sophisticated online survey tool. Participants completed a survey on ASB, and participants were asked how they would search the internet for information both on urinary test results and on ASB. Outcomes included survey responses, and qualitative data were coded and analyzed thematically. χ2 Testing and regression modeling were used to look for variables associated with concern for ASB. There were a total of 518 participants who passed attention check qualifications, and only 45 respondents (8.7%) had heard of ASB. Many were concerned about progress to a worsening infection (n = 387 [77.6%]). When controlling for confounders, education beyond a college degree was not associated with a lower concern for ASB when compared with those with a high school education or less (adjusted odds ratio, 0.63; 95% confidence interval, 0.25-1.55; P = 0.31). Medical providers were the target audience for a majority of the websites, and many of the patient-facing results were of poor quality. Our national survey of women demonstrated a prevalent knowledge deficit surrounding ASB. We must seek to create high-quality, readily available, patient-facing information to increase awareness of ASB, allay concerns, and increase antibiotic stewardship.

Do Dynamic Signals Affect High-Quality Solvers’ Participation Behavior? Evidence from the Crowdsourcing Platform

The emergence of the crowdsourcing platform enables seekers to obtain higher-quality services at lower costs. High-quality services are often provided by high-quality solvers, which is the key to the sustainable development of crowdsourcing platforms. Therefore, how to attract more high-quality solvers to participate needs to be focused on. Most previous studies that used stock data to measure crowdsourcing performance failed to describe the contest process of high-quality solvers’ behavior. Different from the previous study, this paper explores the information signals that influence the participation of high-quality solvers in the dynamic process of crowdsourcing contests. Based on the creative projects of the Winvk platform, dynamic models affecting the participation of high-quality solvers are constructed from the perspective of reducing information asymmetry, and the effects of quality signals and intention signals are explored in depth. The results show that for logo design projects, clear information display and monetary mechanisms have a significant impact on alleviating information asymmetry and attracting the participation of high-quality solvers. Interestingly, the effect of market competition on high-quality solvers shows a U-shaped change. The research results provide a reference for enterprises to reduce information asymmetry, obtain high-quality solutions, and enrich the theoretical application in the field of crowdsourcing.

A silver lining for the excluded: Exploring experiences that micro‐task crowdsourcing affords workers with impaired work access

AbstractMicro‐task crowdsourcing (MTCS) platforms offer alternative work settings outside traditional work boundaries and thus increasingly attract crowdworkers who face exclusion from access to other work. However, we know little about these crowdworkers' perspective on MTCS and its implications for their personal life. Building on insights from three qualitative surveys with responses from 538 crowdworkers and 576 forum posts in total, we show that despite the often challenging work conditions, MTCS platforms provide these crowdworkers with a work environment in which they can participate in paid work activities without feeling excluded due to their personal circumstances. As a result, MTCS platform work provides these crowdworkers with a set of positive experiences that were not possible before. These afforded experiences go beyond work‐related experiences but relate directly to crowdworkers' personal situation and life. Our research yields implications for the literature on MTCS and also for policy makers and stakeholders concerned with the creation of more inclusive work settings.

Machine learning based software effort estimation using development-centric features for crowdsourcing platform

Crowd-Sourced software development (CSSD) is getting a good deal of attention from the software and research community in recent times. One of the key challenges faced by CSSD platforms is the task selection mechanism which in practice, contains no intelligent scheme. Rather, rule-of-thumb or intuition strategies are employed, leading to biasness and subjectivity. Effort considerations on crowdsourced tasks can offer good foundation for task selection criteria but are not much investigated. Software development effort estimation (SDEE) is quite prevalent domain in software engineering but only investigated for in-house development. For open-sourced or crowdsourced platforms, it is rarely explored. Moreover, Machine learning (ML) techniques are overpowering SDEE with a claim to provide more accurate estimation results. This work aims to conjoin ML-based SDEE to analyze development effort measures on CSSD platform. The purpose is to discover development-oriented features for crowdsourced tasks and analyze performance of ML techniques to find best estimation model on CSSD dataset. TopCoder is selected as target CSSD platform for the study. TopCoder’s development tasks data with development-centric features are extracted, leading to statistical, regression and correlation analysis to justify features’ significance. For effort estimation, 10 ML families with 2 respective techniques are applied to get broader aspect of estimation. Five performance metrices (MSE, RMSE, MMRE, MdMRE, Pred (25) and Welch’s statistical test are incorporated to judge the worth of effort estimation model’s performance. Data analysis results show that selected features of TopCoder pertain reasonable model significance, regression, and correlation measures. Findings of ML effort estimation depicted that best results for TopCoder dataset can be acquired by linear, non-linear regression and SVM family models. To conclude, the study identified the most relevant development features for CSSD platform, confirmed by in-depth data analysis. This reflects careful selection of effort estimation features to offer good basis of accurate ML estimate.

(Mis)alignment between facial and textual emotions and its effects on donors engagement behavior in online crowdsourcing platforms

(Mis)alignment between facial and textual emotions and its effects on donors engagement behavior in online crowdsourcing platforms

Procedural Motor Memory Deficits in Patients With Long-COVID.

At least 15% of patients who recover from acute severe acute respiratory syndrome coronavirus 2 infection experience lasting symptoms ("Long-COVID") including "brain fog" and deficits in declarative memory. It is not known if Long-COVID affects patients' ability to form and retain procedural motor skill memories. The objective was to determine the ability of patients with Long-COVID to acquire and consolidate a new procedural motor skill over 2 training days. The primary outcome was to determine difference in early learning, measured as the increase in correct sequence typing speed over the initial 11 practice trials of a new skill. The secondary outcomes were initial and final typing speed on days 1 and 2, learning rate, overnight consolidation, and typing accuracy. In this prospective, cross-sectional, online, case-control study, participants learned a sequential motor skill over 2 consecutive days (NCT05746624). Patients with Long-COVID (reporting persistent post-coronavirus disease 2019 [COVID-19] symptoms for more than 4 weeks) were recruited at the NIH. Patients were matched one-to-one by age and sex to controls recruited during the pandemic using a crowd-sourcing platform. Selection criteria included age 18-90 years, English speaking, right-handed, able to type with the left hand, denied active fever or respiratory infection, and no previous task exposure. Data were also compared with an age-matched and sex-matched control group who performed the task online before the COVID-19 pandemic (prepandemic controls). In total, 105 of 236 patients contacted agreed to participate and completed the experiment (mean ± SD age 46 ± 12.8 years, 82% female). Both healthy control groups had 105 participants (mean age 46 ± 13.1 and 46 ± 11.9 years, 82% female). Early learning was comparable across groups (Long-COVID: 0.36 ± 0.24 correct sequences/second, pandemic controls: 0.36 ± 0.53 prepandemic controls: 0.38 ± 0.57, patients vs pandemic controls [CI -0.068 to 0.067], vs prepandemic controls [CI -0.084 to 0.052], and between controls [CI -0.083 to 0.053], p = 0.82). Initial and final typing speeds on days 1 and 2 were slower in patients than controls. Patients with Long-COVID showed a significantly reduced overnight consolidation and a nonsignificant trend to reduced learning rates. Early learning was comparable in patients with Long-COVID and controls. Anomalous initial performance is consistent with executive dysfunction. Reduction in overnight consolidation may relate to deficits in procedural memory formation.

Taking on Grand Challenges Through Collaborative Crowdsourcing: The Importance of Common Ground for Knowledge Integration

Tackling grand challenges calls for collaborative innovation approaches that can gather various stakeholders around a common goal. One such approach receiving growing scholarly attention is collaborative crowdsourcing. By enabling crowds to collaboratively share and integrate their knowledge, perspectives, and partial solutions, collaborative crowdsourcing changes the role of ordinary people from passive receivers to active creators of innovative solutions for addressing grand challenges. However, our understanding of the knowledge sharing and integrating processes in temporary online crowds is limited and calls for further research. Through a netnographic analysis of an online crowdsourcing platform tackling problems around the Covid-19 pandemic, we provide a detailed account of the knowledge integration processes as they unfold on the platform over time. We identify fourteen activities that individuals engage in and relate these activities to six subprocesses that constitute the knowledge integration process on a collaborative crowdsourcing platform for addressing grand challenges. Our insights contribute to the research on collaborative crowdsourcing by showing the importance of crowds establishing common ground, not only around solutions but also around the different aspects of the problem. Furthermore, when addressing grand challenges, connecting contributions emerges as an important intermediate step between the diverging and converging cycles previously identified in crowdsourcing studies. Finally, our article highlights the importance of (collaborative) crowdsourcing initiatives in making participants feel part of something greater by “building a collective.”

DIGITAL LOGISTICS AS A TOOL FOR IMPROVING THE LEVEL AND QUALITY OF CONSUMER SERVICE

In the modern conditions of the digital economy and many force majeure events, both of an economic and social nature, the issue of providing production with the necessary material and informational resources and their effective use is gaining particular relevance; improvement of operational and strategic management of warehousing, product stocks, and differentiated transport flows, sales activities and customer experience. In order to ensure effective operations, enterprises should organize their activities in such a way as to achieve levelling of risks, minimization of losses and costs associated with the organization of the processes of material and technical support, production of products, transportation, logistic service and sales, as well as to maximize profitability from implementation. First, the above-mentioned tasks can be carried out by increasing the efficiency of the management of logistics activities of enterprises, taking into account the analysis of its components, their interconnection and the presence of an accelerated and uneven process of digitalization. With this in mind, today the problems of implementing digital logistics as a new paradigm of logistics management are being actualized. As a result of the study, it was established that to increase the level and quality of logistics service, it is advisable to implement a “physical” Internet based on the Internet of Things; to automate the processes of product delivery to consumers; use digital crowdsourcing platforms when delivering goods. This will allow obtaining a synergistic (economic, social and environmental) effect by ensuring the automation of logistics processes; significant increase in the level of cooperation between logistics companies; improving the quality of logistics service as a result of reducing delivery times; increase in the level of employment; reducing the load on transport and logistics infrastructure and the level of natural pollution by optimizing transport flows. Keywords: digital economy, digitalization of business processes, digital transformation, digital logistics, customer service, logistics service, information and communication technologies, crowdsourcing platform, efficiency.

Data Processing on Crowdsourcing Platforms: A Comparison between Germany and the United States

Data Processing on Crowdsourcing Platforms: A Comparison between Germany and the United States

Towards Temporal Event Detection: A Dataset, Benchmarks and Challenges

The availability of datasets annotated with verified events by the public is a necessary prerequisite for unleashing the potential of multimodal deep learning for news event detection. Publicly available datasets are either incompletely annotated due to expensive cost, or ignore the verifiability of event labels, which are susceptible to bias and errors introduced by a limited number of annotators. In this paper, we provide a YouTube dataset labelled by real-world news events that can be verified by Wikipedia-like crowd sourcing platforms, with the target of advancing temporal event detection. The events in our dataset cover a wide range of event topics including public security, natural disasters, elections, sports, and entertainment events, etc. In the dataset, each sample is labelled with real-world event that is verifiable by the public. We extensively evaluate the performance of 13 state-of-the-art algorithms on our dataset in a temporal manner, involving the multiple relationships between training and testing event labels, and provide a thorough analysis of the findings. The dataset is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhengyang5/TED</uri> .