Challenge Workshop Research Articles

Current Progress and Challenges in Large-Scale 3D Mitochondria Instance Segmentation.

In this paper, we present the results of the MitoEM challenge on mitochondria 3D instance segmentation from electron microscopy images, organized in conjunction with the IEEE-ISBI 2021 conference. Our benchmark dataset consists of two large-scale 3D volumes, one from human and one from rat cortex tissue, which are 1,986 times larger than previously used datasets. At the time of paper submission, 257 participants had registered for the challenge, 14 teams had submitted their results, and six teams participated in the challenge workshop. Here, we present eight top-performing approaches from the challenge participants, along with our own baseline strategies. Posterior to the challenge, annotation errors in the ground truth were corrected without altering the final ranking. Additionally, we present a retrospective evaluation of the scoring system which revealed that: 1) challenge metric was permissive with the false positive predictions; and 2) size-based grouping of instances did not correctly categorize mitochondria of interest. Thus, we propose a new scoring system that better reflects the correctness of the segmentation results. Although several of the top methods are compared favorably to our own baselines, substantial errors remain unsolved for mitochondria with challenging morphologies. Thus, the challenge remains open for submission and automatic evaluation, with all volumes available for download.

Exploring the brain epitranscriptome: perspectives from the NSAS summit.

Increasing evidence reinforces the essential function of RNA modifications in development and diseases, especially in the nervous system. RNA modifications impact various processes in the brain, including neurodevelopment, neurogenesis, neuroplasticity, learning and memory, neural regeneration, neurodegeneration, and brain tumorigenesis, leading to the emergence of a new field termed neuroepitranscriptomics. Deficiency in machineries modulating RNA modifications has been implicated in a range of brain disorders from microcephaly, intellectual disability, seizures, and psychiatric disorders to brain cancers such as glioblastoma. The inaugural NSAS Challenge Workshop on Brain Epitranscriptomics hosted in Crans-Montana, Switzerland in 2023 assembled a group of experts from the field, to discuss the current state of the field and provide novel translational perspectives. A summary of the discussions at the workshop is presented here to simulate broader engagement from the general neuroscience field.

Towards job screening and personality traits estimation from video transcriptions

In recent years, natural language processing (NLP) has gained new territory beyond its traditional use in text mining applications. This paper shows the effectiveness of NLP techniques in assessing the apparent human personality from his/her video transcript, building a bridge between NLP and computer vision-based reasoning. In this paper, a new deep learning model using attention mechanism and bidirectional LSTM layers for estimating the Big-five personality traits is provided and then tested on ChatLearn video dataset. The robustness of the approach is then tested by generalizing the method to two other datasets (B5 corpus and Mypersonality datasets). Several empirical evaluations taking into account the various inputs of the data processing pipeline have been performed to yield optimal model parameters. The developed model is tested on the APA’2016 competition dataset from Chalearn V2 Challenge Workshop for both Big-five personality trait estimation and interview score estimation. We achieved an average of 89.27% in personality trait recognition rate and 89.16% score in the Job Interview challenge. Similar trends of high accuracy score estimation is held for B5-corpus and MyPersonality datasets as well. The results outperformed several state-of-art approaches, demonstrating our approach’s feasibility in extending the computer vision approach of first impression personality to natural language processing. This opens up a new direction in multimedia analysis.

Mitigation insights from emergency managers on working with stakeholders.

Emergency managers have the important responsibility of planning and implementing mitigation policies and programs to reduce losses to life and property. To accomplish these goals, they must use limited time and resources to ensure the communities they serve have adequately mitigated against potential disasters. As a result, it is common to collaborate and coordinate with a wide variety of partner agencies and community organizations. While it is well established that strengthening relationships and increasing familiarity improve coordination, this article advances that narrative by providing direct insights on the ways a select group of local, state, and federal emergency managers view relationships with other mitigation stakeholders. Using insights from a 1-day workshop hosted at the University of Delaware to gather information from mitigation stakeholders, this article provides a discussion of commonalities and challenges workshop participants identified with other stakeholder groups. These insights can inform other emergency managers about potential collaborators and coordination opportunities with similar stakeholders in their own communities.

Improved HardNet and Stricter Outlier Filtering to Guide Reliable Matching

As the fundamental problem in the computer vision area, image matching has wide applications in pose estimation, 3D reconstruction, image retrieval, etc. Suffering from the influence of external factors, the process of image matching using classical local detectors, e.g., scale-invariant feature transform (SIFT), and the outlier filtering approaches, e.g., Random sample consensus (RANSAC), show high computation speed and pool robustness under changing illumination and viewpoints conditions, while image matching approaches with deep learning strategy (such as HardNet, OANet) display reliable achievements in large-scale datasets with challenging scenes. However, the past learning-based approaches are limited to the distinction and quality of the dataset and the training strategy in the image-matching approaches. As an extension of the previous conference paper, this paper proposes an accurate and robust image matching approach using fewer training data in an end-to-end manner, which could be used to estimate the pose error This research first proposes a novel dataset cleaning and construction strategy to eliminate the noise and improve the training efficiency; Secondly, a novel loss named quadratic hinge triplet loss (QHT) is proposed to gather more effective and stable feature matching; Thirdly, in the outlier filtering process, the stricter OANet and bundle adjustment are applied for judging samples by adding the epipolar distance constraint and triangulation constraint to generate more outstanding matches; Finally, to recall the matching pairs, dynamic guided matching is used and then submit the inliers after the PyRANSAC process. Multiple evaluation metrics are used and reported in the 1st place in the Track1 of CVPR Image-Matching Challenge Workshop. The results show that the proposed method has advanced performance in large-scale and challenging Phototourism benchmark.

Detection and Tracking Meet Drones Challenge.

Drones, or general UAVs, equipped with cameras have been fast deployed with a wide range of applications, including agriculture, aerial photography, and surveillance. Consequently, automatic understanding of visual data collected from drones becomes highly demanding, bringing computer vision and drones more and more closely. To promote and track the developments of object detection and tracking algorithms, we have organized three challenge workshops in conjunction with ECCV 2018, ICCV 2019 and ECCV 2020, attracting more than 100 teams around the world. We provide a large-scale drone captured dataset, VisDrone, which includes four tracks, i.e., (1) image object detection, (2) video object detection, (3) single object tracking, and (4) multi-object tracking. In this paper, we first present a thorough review of object detection and tracking datasets and benchmarks, and discuss the challenges of collecting large-scale drone-based object detection and tracking datasets with fully manual annotations. After that, we describe our VisDrone dataset, which is captured over various urban/suburban areas of 14 different cities across China from North to South. Being the largest such dataset ever published, VisDrone enables extensive evaluation and investigation of visual analysis algorithms for the drone platform. We provide a detailed analysis of the current state of the field of large-scale object detection and tracking on drones, and conclude the challenge as well as propose future directions. We expect the benchmark largely boost the research and development in video analysis on drone platforms. All the datasets and experimental results can be downloaded from https://github.com/VisDrone/VisDrone-Dataset.

VizWiz grand challenge workshop at CVPR 2022

Our goal is to educate a broader population about the technological needs and interests of people with vision impairments while encouraging artificial intelligence (AI) researchers to develop new algorithms that can help eliminate accessibility barriers. Towards this goal, we organised the VizWiz Grand Challenge Workshop at the IEEE/CVF Computer Vision and Pattern Recognition conference (CVPR 2022). This workshop's scope included charting and celebrating progress on accessibility-related AI challenges as well as engaging invited speakers and stakeholders to discuss challenges and opportunities related to designing next-generation assistive technologies. A total of 72 teams participated in our three AI challenges and the winners received awards sponsored by Microsoft. The facilitated discussions highlighted insights from ten invited speakers who provided a range of expertise spanning the cutting edge of computer vision research, development of industry products and services for assisting people with vision impairments, and perspectives of people with vision impairments who use visual assistance technologies. Finally, nine teams who submitted extended abstracts about their research related to the AI challenges and assistive technologies for people with visual impairments gave spotlight and poster presentations about their research. Links to the content shared at the event can be found at VizWiz Workshop

Cross-Project Support of Registries in Development, Implementation and Operation

The German Federal Ministry of Education and Research funded a project accompanying a funding initiative for registries in health services research. The aim was to provide cross-registry support initially for 16 and later 6 projects with regard to methodological, technical and structural standards. The 16 projects were initially guided in concept development, e. g., providing a template for a registry protocol. Furthermore, an expert consultation was organized and carried out. To assist in the selection of an IT solution, a challenge workshop was hosted where different vendors presented their software for registries. The catalogs of data elements of the projects were migrated into a metadata catalog and transferred to the standard model of ISO/IEC 11179. A set of quality indicators was defined for a cross-registry quality management approach to be implemented during the operational phase. To improve data quality, the indicators were to be transmitted and evaluated on a regular basis. The template for a registry protocol was used by the majority of projects when applying for funding of their operational phase. At the workshop on IT solutions, 12 products for registry software were presented; however, the projects opted for other solutions for different reasons. Transferring the catalogs of data elements into a standard model enabled a comparison of attributes and value sets, which in turn enabled formulation of recommendations for important elements. A set of five quality indicators was defined for quality management, for which an initial evaluation was carried out for 2020. The template of a registry protocol serves a systematic development of a concept. The use of a uniformly structured catalog of data elements supports compliance with the FAIR principles. Monitoring of data quality can be achieved by regularly identifying quality indicators across registries.

The design and delivery of a workshop to support curriculum development, education for sustainability and students as partners: sustainability in your curriculum – identify, improve, inspire!

Embedding citizenship and sustainability into higher education curricula is vital for ensuring that curricula remain up-to-date and support students with the skills and knowledge they need for our ever-changing world. But the conceptualisation of the term “sustainability” radically affects its perceived relevance for curriculum design, and hence the recognition of where education for sustainability is already embedded within a curriculum. Here we present a student-designed, freely accessible workshop, which can be used by colleagues off-the-shelf to challenge workshop participants to reconsider their understanding of sustainability and recognise its vast scope. The workshop is provocative yet encourages collaboration, drawing on participants' prior experiences to identify sustainability concepts already embedded within their course, and opportunities to further enhance the inclusion of sustainability in the curriculum. The workshop is also fully supportive of the increasing recognition of the value of engaging students, and others, as partners in curriculum development.

The design and delivery of a workshop to support curriculum development, education for sustainability and students as partners: Sustainability in your curriculum – identify, improve, inspire!

Embedding citizenship and sustainability into higher education curricula is vital for ensuring that curricula remain up-to-date and support students with the skills and knowledge they need for our ever-changing world. But the conceptualisation of the term ‘sustainability’ radically affects its perceived relevance for curriculum design, and hence the recognition of where education for sustainability is already embedded within a curriculum. Here we present a student-designed, freely accessible workshop, which can be used by colleagues off-the-shelf to challenge workshop participants to reconsider their understanding of sustainability and recognise its vast scope. The workshop is provocative yet encourages collaboration, drawing on participants’ prior experiences to identify sustainability concepts already embedded within their course, and opportunities to further enhance the inclusion of sustainability in the curriculum. The workshop is also fully supportive of the increasing recognition of the value of engaging students, and others, as partners in curriculum development.

66 Piloting a respectful challenges workshop in pediatric residency training

Abstract Background Unaddressed conflict has been shown to negatively impact resident wellness and patient safety, and development of communication and conflict resolution skills are key competencies in medical education. While there are a number of existing communication training programs directed at healthcare providers, these can be costly and difficult to accommodate in an already packed curriculum. We developed a conflict resolution workshop to address local conflict scenarios facing our residents that could be integrated into our program curriculum at no cost. Objectives Evaluate resident satisfaction with the addition of a “respectful challenges” workshop to our training program’s academic curriculum. Design/Methods We designed a two- hour workshop for pediatric residents at our institution focusing on addressing conflict during residency training. A didactic lecture outlining different approaches was followed by three role-playing scenarios with coaching (conflict with another resident, allied health and faculty). The workshop was facilitated by a senior resident and faculty identified by residents as “approachable”. Residents (PGY1-4) were asked to evaluate the workshop’s content and effectiveness as well as their own comfort addressing conflict using pre and post-surveys with Likert scales. Results of the survey were analyzed using descriptive statistics. Results A total of 15 residents were involved in the workshop and completed the pre and post surveys. Eleven were female. Six indicated that they had some form of previous training in conflict management. All residents reported having experienced workplace conflict at least a few times per year, but most said they didn’t always address it (73%). Twelve (80%) thought it was either “quite” or “extremely important” that conflict resolution skills be taught in residency. All 15 residents felt that the scenarios represented real life and that the format was an appropriate way to discuss conflict in the workplace. The majority of residents felt the workshop helped them better understand their approach to conflict and allowed them to feel better prepared to face challenges in the future (14/15 and 11/15 rated as “quite” or “extremely”). Eighty percent of residents reported that they would feel more comfortable addressing workplace conflict after the workshop. Conclusion This concise workshop provided a foundation on conflict resolution for our residents. The use of role play and coaching during simulated scenarios was well received, and residents reported increased comfort approaching workplace conflict in the future. Further evaluation will be important to determine if residents translate these skills into practice.

Bridging the Gap Between Computational Photography and Visual Recognition.

What is the current state-of-the-art for image restoration and enhancement applied to degraded images acquired under less than ideal circumstances? Can the application of such algorithms as a pre-processing step improve image interpretability for manual analysis or automatic visual recognition to classify scene content? While there have been important advances in the area of computational photography to restore or enhance the visual quality of an image, the capabilities of such techniques have not always translated in a useful way to visual recognition tasks. Consequently, there is a pressing need for the development of algorithms that are designed for the joint problem of improving visual appearance and recognition, which will be an enabling factor for the deployment of visual recognition tools in many real-world scenarios. To address this, we introduce the UG 2 dataset as a large-scale benchmark composed of video imagery captured under challenging conditions, and two enhancement tasks designed to test algorithmic impact on visual quality and automatic object recognition. Furthermore, we propose a set of metrics to evaluate the joint improvement of such tasks as well as individual algorithmic advances, including a novel psychophysics-based evaluation regime for human assessment and a realistic set of quantitative measures for object recognition performance. We introduce six new algorithms for image restoration or enhancement, which were created as part of the IARPA sponsored UG 2 Challenge workshop held at CVPR 2018. Under the proposed evaluation regime, we present an in-depth analysis of these algorithms and a host of deep learning-based and classic baseline approaches. From the observed results, it is evident that we are in the early days of building a bridge between computational photography and visual recognition, leaving many opportunities for innovation in this area.

Design practices for strategic innovation in start-ups

This paper looks at the use of design practices in start-up firms for the creation of strategic advantage through product/service innovation. Start-ups face non-negligible challenges during the early-stage of development. The research questions examined to what extent design practices can provide the leverage needed to face these challenges. A 4-day Innovation by Design Challenge workshop provided the field for the research carried out. Participants were start-up firms each working together with two designers to form six teams. Methods used included: observation for the mapping of team activities; a short self-report questionnaire and; pre- and post-workshop semi-directed interviews with the start-ups. The findings support the idea that design practice integration into the initial development of a start-up can indeed provide a lever for success and provide the start-up with the strategic vision needed to go through the early-stage and bring their products/services to market successfully.

Medical knowledge infused convolutional neural networks for cohort selection in clinical trials.

In this era of digitized health records, there has been a marked interest in using de-identified patient records for conducting various health related surveys. To assist in this research effort, we developed a novel clinical data representation model entitled medical knowledge-infused convolutional neural network (MKCNN), which is used for learning the clinical trial criteria eligibility status of patients to participate in cohort studies. In this study, we propose a clinical text representation infused with medical knowledge (MK). First, we isolate the noise from the relevant data using a medically relevant description extractor; then we utilize log-likelihood ratio based weights from selected sentences to highlight "met" and "not-met" knowledge-infused representations in bichannel setting for each instance. The combined medical knowledge-infused representation (MK) from these modules helps identify significant clinical criteria semantics, which in turn renders effective learning when used with a convolutional neural network architecture. MKCNN outperforms other Medical Knowledge (MK) relevant learning architectures by approximately 3%; notably SVM and XGBoost implementations developed in this study. MKCNN scored 86.1% on F1metric, a gain of 6% above the average performance assessed from the submissions for n2c2 task. Although pattern/rule-based methods show a higher average performance for the n2c2 clinical data set, MKCNN significantly improves performance of machine learning implementations for clinical datasets. MKCNN scored 86.1% on the F1 score metric. In contrast to many of the rule-based systems introduced during the n2c2 challenge workshop, our system presents a model that heavily draws on machine-based learning. In addition, the MK representations add more value to clinical comprehension and interpretation of natural texts.

Grand challenges for aerosol science and technology

The Grand Challenges Workshop for Aerosol Science and Technology was organized for the International Aerosol Conference (IAC), in St Louis, September 2–7, 2018. The purpose of the workshop was to i...

Model validation based on probability boxes under mixed uncertainties

A non-probabilistic validation framework based on probability boxes (p-boxes) is developed for estimating predictive uncertainty involving input parameter, model form, and experimental uncertainties. These uncertainties are treated as aleatory and epistemic uncertainties. In order to characterize model form uncertainty, an interval-valued area validation metric is introduced to characterize the disagreement between the quantitative predictions from simulation model and experimental data when either or both of predictions and experimental data are represented by probability boxes, whose scales are the same as the original physical units. When experimental data belong to different distributions or physical observations, an extended u-pooling method is used to pool all the comparisons at different sites together, and it offers a global metric for model validation and is interval-valued too. The proposed method is examined by the thermal challenge problem from the Sandia Validation Challenge Workshop.

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

The problem of visual question answering (VQA) is of significant importance both as a challenging research question and for the rich set of applications it enables. In this context, however, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in VQA models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of VQA and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset (Antol et al., in: ICCV, 2015) by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at http://visualqa.org/ as part of the 2nd iteration of the VQA Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. We also present interesting insights from analysis of the participant entries in VQA Challenge 2017, organized by us on the proposed VQA v2.0 dataset. The results of the challenge were announced in the 2nd VQA Challenge Workshop at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.

Data challenges and opportunities for environmental management of North Sea oil and gas decommissioning in an era of blue growth

Maritime industries routinely collect critical environmental data needed for sustainable management of marine ecosystems, supporting both the blue economy and future growth. Collating this information would provide a valuable resource for all stakeholders. For the North Sea, the oil and gas industry has been a dominant presence for over 50 years that has contributed to a wealth of knowledge about the environment. As the industry begins to decommission its offshore structures, this information will be critical for avoiding duplication of effort in data collection and ensuring best environmental management of offshore activities. This paper summarises the outcomes of a Blue Growth Data Challenge Workshop held in 2017 with participants from: the oil and gas industry; the key UK regulatory and management bodies for oil and gas decommissioning; open access data facilitators; and academic and research institutes. Here, environmental data collection and archiving by oil and gas operators in the North Sea are described, alongside how this compares to other offshore industries; what the barriers and opportunities surrounding environmental data sharing are; and how wider data sharing from offshore industries could be achieved. Five primary barriers to data sharing were identified: 1) Incentives, 2) Risk Perception, 3) Working Cultures, 4) Financial Models, and 5) Data Ownership. Active and transparent communication and collaboration between stakeholders including industry, regulatory bodies, data portals and academic institutions will be key to unlocking the data that will be critical to informing responsible decommissioning decisions for offshore oil and gas structures in the North Sea.

Calibrating a Stochastic, Agent-Based Model Using Quantile-Based Emulation

In a number of cases, the quantile Gaussian process has proven effective in emulating stochastic, univariate computer model output [M. Plumlee and R. Tuo, Technometrics, 56 (2014), pp. 466--473]. In this paper, we develop an approach that uses this emulation approach within a Bayesian model calibration framework to calibrate an agent-based model of an epidemic. In addition, this approach is extended to handle the multivariate nature of the model output, which gives a time series of the count of infected individuals. The basic modeling approach is adapted from [D. Higdon et al., J. Amer. Statist. Assoc., 103 (2008), pp. 570--583], using a basis representation to capture the multivariate model output. The approach is motivated with an example taken from the 2015 Ebola challenge workshop which simulated an Ebola epidemic to evaluate methodology.

An End-to-End Deep Learning Approach to Simultaneous Speech Dereverberation and Acoustic Modeling for Robust Speech Recognition

We propose an integrated end-to-end automatic speech recognition (ASR) paradigm by joint learning of the front-end speech signal processing and back-end acoustic modeling. We believe that “only good signal processing can lead to top ASR performance” in challenging acoustic environments. This notion leads to a unified deep neural network (DNN) framework for distant speech processing that can achieve both high-quality enhanced speech and high-accuracy ASR simultaneously. Our goal is accomplished by two techniques, namely: (i) a reverberation-time-aware DNN based speech dereverberation architecture that can handle a wide range of reverberation times to enhance speech quality of reverberant and noisy speech, followed by (ii) DNN-based multicondition training that takes both clean-condition and multicondition speech into consideration, leveraging upon an exploitation of the data acquired and processed with multichannel microphone arrays, to improve ASR performance. The final end-to-end system is established by a joint optimization of the speech enhancement and recognition DNNs. The recent REverberant Voice Enhancement and Recognition Benchmark (REVERB) Challenge task is used as a test bed for evaluating our proposed framework. We first report on superior objective measures in enhanced speech to those listed in the 2014 REVERB Challenge Workshop on the simulated data test set. Moreover, we obtain the best single-system word error rate (WER) of 13.28% on the 1-channel REVERB simulated data with the proposed DNN-based pre-processing algorithm and clean-condition training. Leveraging upon joint training with more discriminative ASR features and improved neural network based language models, a low single-system WER of 4.46% is attained. Next, a new multi-channel-condition joint learning and testing scheme delivers a state-of-the-art WER of 3.76% on the 8-channel simulated data with a single ASR system. Finally, we also report on a preliminary yet promising experimentation with the REVERB real test data.