Shift In Distribution Research Articles

Analysis of representation and generalization capabilities of pre-trained audio models in urban environments

In the last decade, urban noise pollution has become a significant environmental concern that can be mitigated with the help of audio detection algorithms for classifying different sources of noise and creating more informative noise maps. In this context, machine learning, particularly transfer learning, is an essential technology that enables accurate analysis of urban noise sources. However, the choice of the pre-trained model used to compute audio embeddings can significantly influence the performance of downstream classification tasks. This paper aims to compare the embeddings of various pre-trained models on different data collection campaigns in the context of the Sons al balcó project and quantify the robustness of audio representations. To achieve this, we develop metrics and statistically test the presence of distribution shifts in learned latent features. To evaluate the quality of the embeddings, we perform both qualitative and quantitative analysis using dimensionality reduction methods and assess the performance on downstream tasks using data from different collection campaigns. Results highlight major differences between general purpose and specific models. Our findings suggest the need for careful consideration during the choice of the pre-trained model to use in audio event detection applications.

Cascaded Cross-modal Alignment for Visible-Infrared Person Re-Identification

Visible-Infrared Person Re-Identification faces significant challenges due to cross-modal and intra-modal variations. Although existing methods explore semantic alignment from various angles, severe distribution shifts in heterogeneous data limit the effectiveness of single-level alignment approaches. To address this issue, we propose a Cascaded Cross-modal Alignment (CCA) framework that gradually eliminates distribution discrepancies and aligns semantic features from three complementary perspectives in a cascaded manner. First, at the input-level, we propose a Channel-Spatial Recombination (CSR) strategy that strategically reorganizes and preserves crucial details from channel and spatial dimensions to diminish visual discrepancies between modalities, thereby narrowing the modality gap in input images. Second, at the frequency-level, we introduce a Low Frequency Masking (LFM) module to emphasize global details that CSR might overlook by randomly masking low-frequency information, thus driving comprehensive alignment of identity semantics. Third, at the part-level, we design a Prototype-based Semantic Refinement (PSR) module to refine fine-grained features and mitigate the impact of irrelevant areas in LFM. It accurately aligns body parts and enhances semantic consistency guided by global discriminative clues from LFM and flipped views with pose variations. Comprehensive experimental results on the SYSU-MM01 and RegDB datasets demonstrate the superiority of our proposed CCA.

Superpixel-ComBat modeling: A joint approach for harmonization and characterization of inter-scanner variability in T1-weighted images

Abstract T1-weighted imaging holds wide applications in clinical and research settings; however, the challenge of inter-scanner variability arises when combining data across scanners, which impedes multi-site research. To address this, post-acquisition harmonization methods such as statistical or deep learning approaches have been proposed to unify cross-scanner images. Nevertheless, how inter-scanner variability manifests in images and derived measures, and how to harmonize it in an interpretable manner, remains underexplored. To broaden our knowledge of inter-scanner variability and leverage it to develop a new harmonization strategy, we devised a pipeline to assess the interpretable inter-scanner variability in matched T1-weighted images across four 3T MRI scanners. The pipeline incorporates ComBat modeling with 3D superpixel parcellation algorithm (namely SP-ComBat), which estimates location and scale effects to quantify the shift and spread in relative signal distributions, respectively, concerning brain tissues in the image domain. The estimated parametric maps revealed significant contrast deviations compared to the joint signal distribution across scanners (p < 0.001), and the identified deviations in signal intensities may relate to differences in the inversion time acquisition parameter. To reduce the inter-scanner variability, we implemented a harmonization strategy involving proper image preprocessing and site effect removal by ComBat-derived parameters, achieving substantial improvement in image quality and significant reduction in variation of volumetric measures of brain tissues (p < 0.001). We also applied SP-ComBat to evaluate and characterize the performance of various image harmonization techniques, demonstrating a new way to assess image harmonization. In addition, we reported various metrics of T1-weighted images to quantify the impact of inter-scanner variation, including signal-to-noise ratio, contrast-to-noise ratio, signal inhomogeneity index, and structural similarity index. This study demonstrates a pipeline that extends the implementation of statistical ComBat method to the image domain in a practical manner for characterizing and harmonizing the inter-scanner variability in T1-weighted images, providing further insight for the studies focusing on the development of image harmonization methodologies and their applications.

Mycorrhizal fungi as critical biotic filters for tree seedling establishment during species range expansions

AbstractGlobal warming has been shifting climatic envelopes of many tree species to higher latitudes and elevations across the globe; however, unsuitable soil biota may inhibit tree migrations into these areas of suitable climate. Specifically, the role of mycorrhizal fungi in facilitating tree seedling establishment beyond natural species range limits has not been fully explored within forest ecosystems. We used three experiments to isolate and quantify the effects of mycorrhizal colonization and common mycorrhizal networks (CMN) on tree seedling survival and growth across (within and beyond) the elevational ranges of two dominant tree species in northeastern North America, which were associated with either arbuscular mycorrhiza (AMF, Acer saccharum) or ectomycorrhiza (EMF, Fagus grandifolia). In order to quantify the influence of mycorrhiza on seedling establishment independent of soil chemistry and climate, we grew seedlings in soils from within and beyond our study species ranges in a greenhouse experiment (GE) as well as in the field using a soil translocation experiment (STE) and another field experiment manipulating seedling connections to potential CMNs (CMNE). Root length colonized, seedling survival and growth, foliar nutrients, and the presence of potential root pathogens were examined as metrics influencing plant performance across species' ranges. Mycorrhizal inoculum from within species ranges, but not from outside, increased seedling survival and growth in a greenhouse setting; however, only seedling survival, and not growth, was significantly improved in field studies. Sustained potential connectivity to AMF networks increased seedling survival across the entire elevational range of A. saccharum. Although seedlings disconnected from a potential CMN did not suffer decreased foliar nutrient levels compared with connected seedlings, disconnected AM seedlings, but not EM seedlings, had significantly higher aluminum concentrations and more potential pathogens present. Our results indicate that mycorrhizal fungi may facilitate tree seedling establishment beyond species range boundaries in this forested ecosystem and that the magnitude of this effect is modulated by the dominant mycorrhizal type present (i.e., AM vs. EM). Thus, despite changing climate conditions beyond species ranges, a lack of suitable mutualists can still limit successful seedling establishment and stall adaptive climate‐induced shifts in tree species distributions.

Rethinking self-supervised learning for time series forecasting: A temporal perspective

Self-supervised learning has garnered significant attention for its ability to learn meaningful representations. Recent advancements have introduced self-supervised methods for time series forecasting. However, these efforts have faced limitations due to two primary drawbacks. Firstly, these approaches often borrow techniques from vision and language domains without adequately addressing the unique temporal dependencies inherent in time series data. Secondly, time series often show that the distribution shifts over time, which makes accurate forecasting challenging. In response to these issues, we propose TempSSL, a self-supervised learning framework designed for time series forecasting. TempSSL divides the time series data into context (history data) and target (future data), employing two pre-training strategies: (1) Temporal Masked Modeling (TMM) designed to capture temporal dependencies by reconstructing future time series based on historical context; (2) Temporal Contrastive Learning (TCL) employs context and target as positive samples to enhance discriminative representations and mitigate distribution shifts within the time series. TempSSL’s innovation lies in two key aspects. Firstly, it underscores the importance of temporal dependencies for time series forecasting by designing specific pre-training tasks. Secondly, it effectively integrates contrastive learning and masked modeling, leveraging their respective strengths to develop time series representation with strong instance discriminability and local perceptibility. Extensive experiments across seven widely used benchmark datasets demonstrate that TempSSL consistently outperforms existing self-supervised and end-to-end forecasting methods, achieving improvements ranging from 1.92% ∼ 78.12%. Additionally, TempSSL’s practical effectiveness is further demonstrated through successful application in natural gas demand forecasting.

Evaluating species distribution model predictions through time against paleozoological records.

Species distribution models (SDMs) are widely used to project how species distributions may vary over time, particularly in response climate change. Although the fit of such models to current distributions is regularly enumerated, SDMs are rarely tested across longer time spans to gauge their actual performance under environmental change. Here, we utilise paleozoological presence/absence records to independently assess the predictive accuracy of SDMs through time. To illustrate the approach, we focused on modelling the Holocene distribution of the hartebeest, Alcelaphus buselaphus, a widespread savannah-adapted African antelope. We applied various modelling algorithms to three occurrence datasets, including a point dataset from online repositories and two range maps representing current and 'natural' (i.e. hypothetical assuming no human impact) distributions. We compared conventional model evaluation metrics which assess fit to current distributions (i.e. True Skill Statistic, TSSc, and Area Under the Curve, AUCc) to analogous 'paleometrics' for past distributions (i.e. TSSp, AUCp, and in addition Boycep, F2-scorep and Sorensenp). Our findings reveal only a weak correlation between the ranking of conventional metrics and paleometrics, suggesting that the models most effectively capturing present-day distributions may not be the most reliable to hindcast historical distributions, and that the choice of input data and modelling algorithm both significantly influences environmental suitability predictions and SDM performance. We thus advocate assessment of model performance using paleometrics, particularly those capturing the correct prediction of presences, such as F2-scorep or Sorensenp, due to the potential unreliability of absence data in paleozoological records. By integrating archaeological and paleontological records into the assessment of alternative models' ability to project shifts in species distributions over time, we are likely to enhance our understanding of environmental constraints on species distributions.

Spontaneous asymmetry effect induced by uniform magnetic fields in capacitively coupled plasmas under perfectly symmetric conditions

Abstract Introducing asymmetry in capacitively coupled plasmas (CCPs) is a common strategy for achieving independent control of ion mean energy and flux. Our 1d3v particle-in-cell/Monte Carlo collision simulations reveal that a uniform magnetic field within a specific range can induce spatial asymmetry in low-pressure CCPs, even under perfectly symmetric conditions. This asymmetry, characterized by a shift in the plasma density distribution and significant differences in electron kinetics between the two sides of the plasma, leads to strong ionization and most electron losses on the low-density side, while the high-density side experiences weak ionization and minimal electron losses. The underlying mechanism triggering this spontaneous asymmetry is the differential influence of the magnetic field on low-energy (local) and high-energy (relatively nonlocal) electrons. Under conditions of low pressure and an appropriate magnetic field, this disparity in electron kinetic behavior leads to a spontaneous amplification of the asymmetry induced by random fluctuations until a steady state is reached, culminating in a spontaneous asymmetric effect.

Climate Futures for Lizards and Snakes in Western North America May Result in New Species Management Issues.

We assessed changes in fundamental climate-niche space for lizard and snake species in western North America under modeled climate scenarios to inform natural resource managers of possible shifts in species distributions. We generated eight distribution models for each of 130 snake and lizard species in western North America under six time-by-climate scenarios. We combined the highest-performing models per species into a single ensemble model for each scenario. Maps were generated from the ensemble models to depict climate-niche space for each species and scenario. Patterns of species richness based on climate suitability and niche shifts were calculated from the projections at the scale of the entire study area and individual states and provinces, from Canada to Mexico. Squamate species' climate-niche space for the recent-time climate scenario and published known ranges were highly correlated (r = 0.81). Overall, reptile climate-niche space was projected to move northward in the future. Sixty-eight percent of species were projected to expand their current climate-niche space rather than to shift, contract, or remain stable. Only 8.5% of species were projected to lose climate-niche space in the future, and these species primarily occurred in Mexico and the southwestern U.S. We found few species were projected to lose all suitable climate-niche space at the state or province level, although species were often predicted to occupy novel areas, such as at higher elevations. Most squamate species were projected to increase their climate-niche space in future climate scenarios. As climate niches move northward, species are predicted to cross administrative borders, resulting in novel conservation issues for local landowners and natural resource agencies. However, information on species dispersal abilities, landscape connectivity, biophysical tolerances, and habitat suitability is needed to contextualize predictions relative to realized future niche expansions.

Vertical Distribution of Rocky Intertidal Organisms Shifts With Sea-Level Variability on the Northeast Pacific Coast.

Disentangling the effects of cyclical variability in environmental forcing and long-term climate change on natural communities is a major challenge for ecologists, managers, and policy makers across ecosystems. Here we examined whether the vertical distribution of rocky intertidal taxa has shifted with sea-level variability occurring at multiple temporal scales and/or long-term anthropogenic sea-level rise (SLR). Because of the distinct zonation characteristic of intertidal communities, any shift in tidal dynamics or average sea level is expected to have large impacts on community structure and function. We found that across the Northeast Pacific Coast (NPC), sea level exhibits cyclical seasonal variability, tidal amplitude exhibits ecologically significant variability coherent with the 18.6-year periodicity of lunar declination, and long-term sea-level rise is occurring. Intertidal taxa largely do not exhibit significant vertical distribution shifts coherent with short-term (monthly to annual) sea-level variability but do exhibit taxa-specific vertical distribution shifts coherent with cyclical changes in lunar declination and long-term SLR at decadal timescales. Finally, our results show that responses to cyclical celestial mechanics and SLR vary among taxa, primarily according to their vertical distribution. Long-term SLR is occurring on ecologically relevant scales, but the confounding effects of cyclical celestial mechanics make interpreting shifts in zonation or community structure challenging. Such cyclical dynamics alternatingly amplify and dampen long-term SLR impacts and may modify the impacts of other global change related stressors, such as extreme heat waves and swell events, on intertidal organisms living at the edge of their physiological tolerances. As a result, intertidal communities will likely experience cyclical periods of environmental stress and concomitant nonlinear shifts in structure and function as long-term climate change continues. Our results demonstrate that consistent, large-scale monitoring of marine ecosystems is critical for understanding natural variability in communities and documenting long-term change.

Denoising for balanced representation: A diffusion-informed approach to causal effect estimation

Estimating individual treatment effects with observational data is crucial in causal inference. While it faces two major challenges: the absence of counterfactual outcomes and selection bias due to non-random assignment and intervention of confounding variables. This paper specifically addresses these by quantifying the distribution shift between the treated and control groups from a generative view to achieve balanced representation and then predict causal effects. We propose a novel method called Denoising for a Balanced Representation in Treatment Effect Estimation (DBRT). Motivated by the aim to generate analogous distributions in the Diffusion model, which aligns with our goal of achieving similar distributions in the latent space, our approach considers the divergence between the treated and control groups as’noise’ that needs to be removed incrementally. We then construct a network to fit and then denoise it step by step, ensuring a consistent reduction of discrepancies between the two groups. Additionally, to enhance the representation of input features, we incorporate memory vectors into the attention mechanism to capture prior information and combined correlations between covariates. Furthermore, we utilize the Hilbert–Schmidt Independence Criterion (HSIC) within the loss function to constrain the learned representation, ensuring its relevance to the original data. DBRT has demonstrated superior performance on different datasets compared to previous classic works, highlighting its effectiveness in treatment effect estimation.

Meta Koopman decomposition for time series forecasting under temporal distribution shifts

Time series forecasting facilitates various real-world applications and has attracted great research interests. In real-world scenarios, time series forecasting models confront a fundamental issue of temporal distribution shifts (TDS), i.e., the statistical properties of time series are evolving over time. In this paper, we utilize Koopman theory to address temporal distribution shifts. Koopman theory states any time series can be mapped into a Koopman space by proper measurement functions with their dynamics modeled by the infinite-dimensional linear Koopman operator. Therefore, time series under different distributions can be modeled by different Koopman operators. Considering the linearity of Koopman operators, the Koopman operators for representing time series under different distributions can be decomposed as a linear combination of a set of Koopman operators, which we termed as meta Koopman operators. We further theoretically show the infinite-dimensional Koopman operators can be approximated by finite matrix multiplications and the meta Koopman operators are equivalent to a set of matrices. Based on the analysis, we propose an auto-encoder framework for implementing the meta Koopman decomposition of time series, which is theoretically capable of handling TDS. Extensive experiments conducted on four real-world time series datasets demonstrate the superiority of the proposed model in tackling temporal distribution shifts.

DCNFYOLO: Dual-Convolution Network and Feature Fusion for High-Precision Smoke Detection

Fast, real-time, and accurate detection of smoke characteristics in the early stage of a fire is crucial for reducing fire losses. Existing smoke detection methods mainly rely on traditional algorithms and smoke sensors, and these approaches have limitations in false detection rates, accuracy, and real-time performance. Therefore, a novel DCNFYOLO network for smoke detection is proposed in this paper. Firstly, Switchable Atrous Convolution (SAConv) is introduced in the YOLOv5 backbone network to enhance the fusion extraction of smoke features by the Convolutional Neural Network (CNN). Secondly, both Distribution Shifts Convolution (DSConv) operator and Efficient Channel Attention (ECA) mechanisms are considered in the neck part to reduce the computational load of the model, and better capture the relationship between channels to improve the detection performance. Finally, to make low-quality examples less harmful to the gradients, the Wise-IoU (WIoU) loss function in the prediction part is used for reducing the competitiveness of high-quality anchor frames during model training, allowing the model to converge more quickly and stably. Experimental results show that the DCNFYOLO network can achieve a remarkable detection accuracy of 96.6%, which has a substantial improvement of 7.7% compared with the original YOLOv5 network performance, thereby validating the effectiveness of the proposed network.

SimLVSeg: Simplifying Left Ventricular Segmentation in 2-D+Time Echocardiograms With Self- and Weakly Supervised Learning

ObjectiveAchieving reliable automatic left ventricle (LV) segmentation from echocardiograms is challenging due to the inherent sparsity of annotations in the dataset, as clinicians typically only annotate two specific frames for diagnostic purposes. Here we aim to address this challenge by introducing simplified LV segmentation (SimLVSeg), a novel paradigm that enables video-based networks for consistent LV segmentation from sparsely annotated echocardiogram videos. MethodsSimLVSeg consists of two training stages: (i) self-supervised pre-training with temporal masking, which involves pre-training a video segmentation network by capturing the cyclic patterns of echocardiograms from largely unannotated echocardiogram frames, and (ii) weakly supervised learning tailored for LV segmentation from sparse annotations. ResultsWe extensively evaluated SimLVSeg using EchoNet-Dynamic, the largest echocardiography dataset. SimLVSeg outperformed state-of-the-art solutions by achieving a 93.32% (95% confidence interval: 93.21–93.43%) dice score while being more efficient. We further conducted an out-of-distribution test to showcase SimLVSeg's generalizability on distribution shifts (CAM US dataset). ConclusionOur findings show that SimLVSeg exhibits excellent performance on LV segmentation with a relatively cheaper computational cost. This suggests that adopting video-based networks for LV segmentation is a promising research direction to achieve reliable LV segmentation. Our code is publicly available at https://github.com/BioMedIA-MBZUAI/SimLVSeg.

Uncertainty guided test-time training for face forgery detection

The rapid development of generative image modeling poses security risks of spreading unreal visual information, even though those techniques make a lot of applications possible in positive aspects. To provide alerts and maintain a secure social environment, forgery detection has been an urgent and crucial solution to deal with this situation and try to avoid any negative effects, especially for human faces, owing to potential severe results when malicious creators spread disinformation widely. In spite of the success of recent works w.r.t. model design and feature engineering, detecting face forgery from novel image creation methods or data distributions remains unresolved, because well-trained models are typically not robust to the distribution shift during test-time. In this work, we aim to alleviate the sensitivity of an existing face forgery detector to new domains, and then boost real-world detection under unknown test situations. In specific, we leverage test examples, selected by uncertainty values, to fine-tune the model before making a final prediction. Therefore, it leads to a test-time training based approach for face forgery detection, that our framework incorporates an uncertainty-driven test sample selection with self-training to adapt a classifier onto target domains. To demonstrate the effectiveness of our framework and compare with previous methods, we conduct extensive experiments on public datasets, including FaceForensics++, Celeb-DF-v2, ForgeryNet and DFDC. Our results clearly show that the proposed framework successfully improves many state-of-the-art methods in terms of better overall performance as well as stronger robustness to novel data distributions.

ASAP: Automatic Synthesis of Attack Prototypes, an online-learning, end-to-end approach

Zero-day attack detection and categorization is an open-research field where four main context factors need to be taken into account: novel or zero-day attacks (i) are unlabeled by definition, (ii) may correspond to out-of-distribution data, (iii) can arise concurrently, and (iv) distribution shifts in the feature space need online-learning. Given such constraints, the online detection and categorization of new cyber threats can be modeled as a heterogeneous collective anomaly detection problem, for which no online-learning solutions exist purely based on back-propagation. To this respect, this paper presents an online-learning, end-to-end back-propagation strategy for Automatically Synthesizing the potential signatures or Attack Prototypes of novel cyber threats (asap). The presented framework incorporates automatic feature engineering, operating over raw data from the OpenFlow monitoring API and raw bytes of traffic captures. In asap, specialized inductive biases enhance the training data efficiency and accommodate the inference machinery to resource-constrained scenarios such as the Internet of Things. Finally, the validity of this framework is demonstrated in a live training experiment comprising IoT traffic emulation 33To foster research in the field, the source-code of asap alongside instructions to reproduce the experiments are made publicly available among the set of SmartVille NID models in [1]..

Pressurized hydrogasification and cobalt-catalyzed hydrogasification behaviors of naphthalene as a coal-based model compound

The pressurized hydrogasification/catalytic hydrogasification behaviors of naphthalene as a coal-based model compound were investigated for the first time in a batch reactor. The composition of products was roundly analyzed by gas chromatography (GC), gas chromatography–mass spectrometer (GC-MS) and laser desorption time-of-flight mass spectrometry (TOF-MS). Based on the product analysis results, the detailed reaction pathways for naphthalene hydrogasification and the effects of cobalt on reaction pathways were elucidated. Naphthalene first destabilized during hydrogasification. Subsequently, the destabilized naphthalene either underwent stepwise hydrocracking by active hydrogen atoms to ultimately produce benzene and methane, or formed naphthalene free radicals to initiate condensation. Cobalt can regulate products distribution to boost methane, benzene and toluene yield by facilitating active hydrogen generation, despite it had a limited ability to facilitate the naphthalene destabilization at temperature below 700 °C. Whereas above 750 °C, cobalt can promote naphthalene destabilization, thereby remarkably enhancing the conversion of naphthalene. Furthermore, cobalt intensified condensation leading to a shift of molecular mass distribution of condensation products from 252 ∼ 500 Da to 750 ∼ 2000 Da. These phenomena supported similar findings in coal catalytic hydrogasification. The rise in temperature, initial H2 pressure (P0), and cobalt content all facilitated the cobalt catalyzed naphthalene hydrocracking to gaseous product, with temperature exerting a particularly significant effect. This trend was similar with cobalt catalyzed coal hydrogasification. For example, when temperature increased from 650 ℃ to 750 ℃, naphthalene conversion improved from 21.2 % to 49.6 %, and gas yield rose from 2.6 % to 29.4 % at 1 % Co and 1.3 MPa P₀. The investigation serves to shed light on the molecular-level understanding of the mechanism underpinning coal hydrogasification.

Effects of water limitation and competition on tree carbon allocation in an Earth system modelling framework

Abstract Earth system models (ESMs) have a limited capacity to represent plant functional diversity and shifts in trait distributions. Approaches to improving the representation of this complexity in ESMs include (i) optimality‐based approaches that predict trait–environment responses and (ii) explicitly modelling coexistence and community assembly. These approaches are expected to converge only when optimality‐based approaches identify competitively dominant strategies, which often differ from strategies that maximize ecosystem functioning or fitness components in monoculture. We used two models, LM3‐PPA (a vegetation demographic model designed as an ESM component) and BiomeE (a computationally efficient analog for LM3‐PPA), to explore how water limitation affects carbon allocation strategies of canopy trees. We compared competitive allocation strategies and those that maximize biomass or productivity in monoculture. We did not explicitly model coexistence or community assembly. Rather, we used model experiments to identify competitive and maximizing strategies in a two‐dimensional trait space under different precipitation and mortality scenarios. At 10 eastern US locations, we simulated historical, wet and dry climate scenarios, novel drought and three different mortality scenarios (low, medium or high sensitivity to water deficit). For each site and scenario, we identified the competitive strategy and three maximizing strategies (maximum biomass, productivity or drought‐tolerance). Root: leaf ratios tended to increase and leaf area tended to decrease with increasing water stress (increasing water limitation and its effects on mortality). However, relative to maximizing strategies, competitive strategies shifted towards greater allocation to roots and leaves with increasing water stress. Competitive overinvestments (greater allocation to roots and leaves by competitive strategies compared with maximizing strategies) were robust across different modelling contexts, including vegetation parameter sets (Acer vs. Populus), models (LM3‐PPA vs. BiomeE) and uncalibrated vsersus calibrated BiomeE versions. Synthesis: The theoretical prediction that competitive and maximizing allocation strategies differ under water limitation is confirmed for a demographic model designed as an ESM component. Optimality‐based trait predictions can simplify representing trait diversity in ESMs but do not always correspond to competitive outcomes. Explicitly modelling coexistence and community assembly in ESMs is challenging but is likely the most general approach to representing trait diversity.

Towards robust visual understanding: A paradigm shift in computer vision from recognition to reasoning

AbstractModels that learn from data are widely and rapidly being deployed today for real‐world use, but they suffer from unforeseen failures that limit their reliability. These failures often have several causes such as distribution shift; adversarial attacks; calibration errors; scarcity of data and/or ground‐truth labels; noisy, corrupted, or partial data; and limitations of evaluation metrics. But many failures also occur because many modern AI tasks require reasoning beyond pattern matching and such reasoning abilities are difficult to formulate as data‐based input–output function fitting. The reliability problem has become increasingly important under the new paradigm of semantic “multimodal” learning. In this article, I will discuss findings from our work to provide avenues for the development of robust and reliable computer vision systems, particularly by leveraging the interactions between vision and language. This article expands upon the invited talk at AAAI 2024 and covers three thematic areas: robustness of visual recognition systems, open‐domain reliability for visual reasoning, and challenges and opportunities associated with generative models in vision.

Efficient and robust sequential decision making algorithms

AbstractSequential decision‐making involves making informed decisions based on continuous interactions with a complex environment. This process is ubiquitous in various applications, including recommendation systems and clinical treatment design. My research has concentrated on addressing two pivotal challenges in sequential decision‐making: (1) How can we design algorithms that efficiently learn the optimal decision strategy with minimal interactions and limited sample data? (2) How can we ensure robustness in decision‐making algorithms when faced with distributional shifts due to environmental changes and the sim‐to‐real gap? This paper summarizes and expands upon the talk I presented at the AAAI 2024 New Faculty Highlights program, detailing how my research aims to tackle these challenges.

A rapid increase in tropical species of grouper (Perciformes: Serranidae) in the temperate waters, the Goto Islands, Japan.

Global warming has resulted in rapid poleward shifts in the geographical distributions of many tropical fish species. This study conducted daily market surveys from 2008 to 2013 to investigate catch trends of seven commercially important grouper species in the temperate Goto Islands, Japan. Our results revealed that the catch numbers of tropical grouper species increased rapidly by an average of 5.9-fold (12.3-fold at maximum) within six years, whereas the temperate and subtropical species did not exhibit substantial changes. Based on the findings of several previous studies, the rapid increase in the number of tropical groupers in temperate waters was most likely caused by the successful settlement of larvae transported from tropical waters. Large-scale ocean currents may facilitate larval transport from tropical waters because the Goto Islands face the Tsushima Warm Current, which branches from the Kuroshio Current. Meanwhile, the transition processes of size distribution in tropical groupers suggest a possible hypothesis that adults migrating from tropical waters first settle in temperate waters and then enhance their populations by reproduction. Further studies are required to determine how tropical grouper species settle and how their populations increase in temperate waters.