Visual Task Research Articles

A novel image dehazing algorithm for complex natural environments

As an essential preprocessing step in computer vision tasks, image dehazing is preparatory work in many practical applications. To ensure the navigation safety of intelligent ships in complex haze weather, this paper proposes a lightweight real-time dehazing method for the natural environment based on an all-in-one dehazing network. Specifically, a hybrid dilated convolution is used to construct a dehazing network, which effectively expands the receptive field and improves the ability to extract characteristic information without increasing the calculation amount of the model. Secondly, a hybrid attention module is developed to enhance the model's ability to remove haze while retaining more detailed features by giving weights to light haze and heavy haze areas in the image. Finally, considering the subjective perception of human eyes, a mixed loss function is designed to address image dimness and color distortion after dehazing. Through experiments on both synthetic and real-world haze datasets, the proposed method attains ideal dehazing results and has high real-time performance. It can dehaze under different haze conditions, address the critical challenge of degraded visual perception in severe weather conditions of intelligent ship navigation on inland rivers in complex environments, and promote intelligent ships' development.

The visual statistical learning overcomes scene dissimilarity through an independent clustering process.

Contextual cueing is a phenomenon of visual statistical learning observed in visual search tasks. Previous research has found that the degree of deviation of items from its centroid, known as variability, determines the extent of generalization for that repeated scene. Introducing variability increases dissimilarity between multiple occurrences of the same repeated layout significantly. However, current theories do not explain the mechanisms that help to overcome this dissimilarity during contextual cue learning. We propose that the cognitive system initially abstracts specific scenes into scene layouts through an automatic clustering unrelated to specific repeated scenes, and subsequently uses these abstracted scene layouts for contextual cue learning. Experiment 1 indicates that introducing greater variability in search scenes leads to a hindering in the contextual cue learning. Experiment 2 further establishes that conducting extensive visual searches involving spatial variability in entirely novel scenes facilitates subsequent contextual cue learning involving corresponding scene variability, confirming that learning clustering knowledge precedes the contextual cue learning and is independent of specific repeated scenes. Overall, this study demonstrates the existence of multiple levels of learning in visual statistical learning, where item-level learning can serve as material for layout-level learning, and the generalization reflects the constraining role of item-level knowledge on layout-level knowledge.

Transcranial focused ultrasound modulates visual thalamus in a nonhuman primate model.

The thalamus plays a pivotal role as a neural hub, integrating and distributing visual information to cortical regions responsible for visual processing. Transcranial focused ultrasound (tFUS) has emerged as a promising non-invasive brain stimulation technology, enabling modulation of neural circuits with high spatial precision. This study investigates the tFUS neuromodulation at visual thalamus and characterizes the resultant effects on interconnected visual areas in a nonhuman primate model. Experiments were conducted on a rhesus macaque trained in a visual fixation task, combining tFUS stimulation with simultaneous scalp electroencephalography (EEG) and intracranial recordings from area V4, a region closely linked to the thalamus. Ultrasound was delivered through a 128-element random array ultrasound transducer operating at 700 kHz, with the focus steered onto the pulvinar of the thalamus based on neuroanatomical atlas and individual brain model. EEG source imaging revealed localized tFUS-induced activities in the thalamus, midbrain, and visual cortical regions. Critically, tFUS stimulation of the pulvinar can elicit robust neural responses in V4 without visual input, manifested as significant modulations in local field potentials, elevated alpha and gamma power, corroborating the functional thalamocortical connectivity. Furthermore, the tFUS neuromodulatory effects on visually-evoked V4 activities were region-specific within the thalamus and dependent on ultrasound pulse repetition frequency. This work provides direct electrophysiological evidence demonstrating the capability of tFUS in modulating the visual thalamus and its functional impact on interconnected cortical regions in a large mammalian model, paving the way for potential investigations for tFUS treating visual, sensory, and cognitive impairments.

Clinically established early Parkinson's disease patients do not show impaired use of priors in conditions of perceptual uncertainty

The ability to use past learned experiences to guide decisions is an important component of adaptive behavior, especially when decision-making is performed under time pressure or when perceptual information is unreliable. Previous studies using visual discrimination tasks have shown that this prior-informed decision-making ability is impaired in Parkinson's disease (PD), but the mechanisms underlying this deficit and the precise impact of dopaminergic denervation within cortico-basal circuits remain unclear. To shed light on this problem, we evaluated prior-informed decision-making under various conditions of perceptual uncertainty in a sample of 13 clinically established early PD patients, and compared behavioral performance with healthy control (HC) subjects matched in age, sex and education. PD patients and HC subjects performed a random dot motion task in which they had to decide the net direction (leftward vs. rightward) of a field of moving dots and communicate their choices through manual button presses. We manipulated prior knowledge by modulating the probability of occurrence of leftward vs. rightward motion stimuli between blocks of trials, and by explicitly giving these probabilities to subjects at the beginning of each block. We further manipulated stimulus discriminability by varying the proportion of dots moving coherently in the signal direction and speed-accuracy instructions. PD patients used choice probabilities to guide perceptual decisions in both speed and accuracy conditions, and their performance did not significantly differ from that of HC subjects. An additional analysis of the data with the diffusion decision model confirmed this conclusion. These results suggest that the impaired use of priors during visual discrimination observed at more advanced stages of PD is independent of dopaminergic denervation, though additional studies with larger sample sizes are needed to more firmly establish this conclusion.

Dual-Task Interference in the Assessment of Listening Effort: Results of Normal-Hearing Adults, Cochlear Implant Users, and Hearing Aid Users.

The purpose of the current study was to assess dual-task interference (i.e., changes between the dual-task and baseline condition) in a listening effort dual-task paradigm in normal-hearing (NH) adults, hearing aid (HA) users, and cochlear implant (CI) users. Three groups of 31 participants were included: (a) NH adults, (b) HA users, and (c) CI users. The dual-task paradigm consisted of a primary speech understanding task in a quiet condition, and a favorable and unfavorable noise condition, and a secondary visual memory task. Dual-task interference was calculated for both tasks, and participants were classified based on their patterns of interference. Descriptive analyses were established and differences between the three groups were examined. The descriptive results showed varying patterns of dual-task interference between the three listening conditions. Most participants showed the pattern of visual memory interference (i.e., worse results for the secondary task in the dual-task condition and no difference for the primary task) in the quiet condition, whereas the pattern of speech understanding priority trade-off (i.e., worse results for the secondary task in the dual-task condition and better results for the primary task) was most prominent in the unfavorable noise condition. Particularly, in HA and CI users, this shift was seen. However, the patterns of dual-task interference were not statistically different between the three groups. Results of this study may provide additional insight into the interpretation of dual-task paradigms for measuring listening effort in diverse participant groups. It highlights the importance of considering both the primary and secondary tasks for accurate interpretation of results. https://doi.org/10.23641/asha.26409088.

G-Net: Implementing an enhanced brain tumor segmentation framework using semantic segmentation design.

A fundamental computer vision task called semantic segmentation has significant uses in the understanding of medical pictures, including the segmentation of tumors in the brain. The G-Shaped Net architecture appears in this context as an innovative and promising design that combines components from many models to attain improved accuracy and efficiency. In order to improve efficiency, the G-Shaped Net architecture synergistically incorporates four fundamental components: the Self-Attention, Squeeze Excitation, Fusion, and Spatial Pyramid Pooling block structures. These factors work together to improve the precision and effectiveness of brain tumor segmentation. Self-Attention, a crucial component of G-Shaped architecture, gives the model the ability to concentrate on the image's most informative areas, enabling accurate localization of tumor boundaries. By adjusting channel-wise feature maps, Squeeze Excitation completes this by improving the model's capacity to capture fine-grained information in the medical pictures. Since the G-Shaped model's Spatial Pyramid Pooling component provides multi-scale contextual information, the model is capable of handling tumors of various sizes and complexity levels. Additionally, the Fusion block architectures combine characteristics from many sources, enabling a thorough comprehension of the image and improving the segmentation outcomes. The G-Shaped Net architecture is an asset for medical imaging and diagnostics and represents a substantial development in semantic segmentation, which is needed more and more for accurate brain tumor segmentation.

Human Visual Pathways for Action Recognition Versus Deep Convolutional Neural Networks: Representation Correspondence in Late But Not Early Layers.

Deep convolutional neural networks (DCNNs) have attained human-level performance for object categorization and exhibited representation alignment between network layers and brain regions. Does such representation alignment naturally extend to other visual tasks beyond recognizing objects in static images? In this study, we expanded the exploration to the recognition of human actions from videos and assessed the representation capabilities and alignment of two-stream DCNNs in comparison with brain regions situated along ventral and dorsal pathways. Using decoding analysis and representational similarity analysis, we show that DCNN models do not show hierarchical representation alignment to human brain across visual regions when processing action videos. Instead, later layers of DCNN models demonstrate greater representation similarities to the human visual cortex. These findings were revealed for two display formats: photorealistic avatars with full-body information and simplified stimuli in the point-light display. The discrepancies in representation alignment suggest fundamental differences in how DCNNs and the human brain represent dynamic visual information related to actions.

Subword Representations Successfully Decode Brain Responses to Morphologically Complex Written Words

Abstract This study extends the idea of decoding word-evoked brain activations using a corpus-semantic vector space to multimorphemic words in the agglutinative Finnish language. The corpus-semantic models are trained on word segments, and decoding is carried out with word vectors that are composed of these segments. We tested several alternative vector-space models using different segmentations: no segmentation (whole word), linguistic morphemes, statistical morphemes, random segmentation, and character-level 1-, 2- and 3-grams, and paired them with recorded MEG responses to multimorphemic words in a visual word recognition task. For all variants, the decoding accuracy exceeded the standard word-label permutation-based significance thresholds at 350–500 ms after stimulus onset. However, the critical segment-label permutation test revealed that only those segmentations that were morphologically aware reached significance in the brain decoding task. The results suggest that both whole-word forms and morphemes are represented in the brain and show that neural decoding using corpus-semantic word representations derived from compositional subword segments is applicable also for multimorphemic word forms. This is especially relevant for languages with complex morphology, because a large proportion of word forms are rare and it can be difficult to find statistically reliable surface representations for them in any large corpus.

Distinct neural plasticity enhancing visual perception.

The developed human brain shows remarkable plasticity following perceptual learning, resulting in improved visual sensitivity. However, such improvements commonly require extensive stimuli exposure. Here we show that efficiently enhancing visual perception with minimal stimuli exposure recruits distinct neural mechanisms relative to standard repetition-based learning. Participants (n=20, 12 women, 8 men) encoded a visual discrimination task, followed by brief memory reactivations of only five trials each performed on separate days, demonstrating improvements comparable to standard repetition-based learning (n=20, 12 women, 8 men). Reactivation-induced learning engaged increased bilateral intra-parietal sulcus activity relative to repetition-based learning. Complementary evidence for differential learning processes was further provided by temporal-parietal resting functional connectivity changes, which correlated with behavioral improvements. The results suggest that efficiently enhancing visual perception with minimal stimuli exposure recruits distinct neural processes, engaging higher-order control and attentional resources, while leading to similar perceptual gains. These unique brain mechanisms underlying improved perceptual learning efficiency may have important implications for daily life and in clinical conditions requiring re-learning following brain damage.Significance Statement The adult human brain shows remarkable plasticity resulting in improved visual perception following practice. Here, we document a distinct neural pathway in the human brain, supporting enhanced perceptual learning efficiency. These unique neural mechanisms are triggered by brief memory reactivations, which replace prolonged repetition-based stimuli exposure to enable enhanced visual perception. The results suggest that efficiently enhancing visual perception with minimal stimuli exposure distinctively engages higher-order control and attentional resources, while leading to similar behavioral gains. Evidence for differential offline learning processes was further provided by resting functional connectivity changes. The findings shed light on unique brain mechanisms underlying improved perceptual learning efficiency, and may have important implications for daily life and in clinical conditions.

The effects of N-acetyl cysteine on intrinsic functional connectivity and neural alcohol cue reactivity in treatment-seeking individuals with alcohol use disorder: a preliminary study.

N-acetyl cysteine (NAC) is a potential pharmacotherapy for alcohol use disorder (AUD), but it is not known whether it modulates neural activation to alcohol cues or intrinsic functional connectivity. We investigated whether NAC attenuates (i) alcohol cue-elicited activation, and (ii) intrinsic functional connectivity compared to placebo in patients with AUD. In this preliminary study, twenty-three individuals (7 females) with moderate-severe AUD received daily NAC (2400mg/day, n = 9), or a placebo (n = 14) for at least 2 weeks. Participants completed a pre-treatment functional magnetic resonance imaging session (T0) and a post-treatment session (T1) comprising resting-state and visual alcohol cue reactivity task acquisitions. Activation differences between sessions, treatment, and session-by-treatment interaction were assessed. Resting-state functional connectivity examined using 377 node ROI-to-ROIs evaluated whether NAC reduced intrinsic functional connectivity after treatment. There were no differences in alcohol cue reactivity for brain activation or subjective craving between NAC and placebo during treatment or across sessions, or significant interaction. A significant treatment-by-time interaction, with reduced intrinsic connectivity was observed after treatment (T1) for NAC-treated compared to placebo-treated patients in the posterior cingulate node (9, left hemisphere) of the dorsal attentional network and connections to salience, ventral-attentional, somatosensory, and visual-peripheral networks implicated in AUD. NAC reduced intrinsic functional connectivity in patients with moderate-severe AUD after treatment compared to placebo, but did not attenuate alcohol cue-elicited activation. However, the absence of cue reactivity findings may result from low power, rather than the absence of cue reactivity findings associated with NAC. These results provide preliminary evidence that NAC treatment may modulate intrinsic functional connectivity brain activation in patients with alcohol use disorder, but replication in larger studies are required to determine the strength of this effect and any associations with clinical outcomes. Clinical Trials Registration: ClinicalTrials.gov Identifier: NCT03879759.

Neural tracking of speech acoustics in noise is coupled with lexical predictability as estimated by large language models.

Adults heard recordings of two spatially separated speakers reading newspaper and magazine articles. They were asked to listen to one of them and ignore the other, and EEG was recorded to assess their neural processing. Machine learning extracted neural sources that tracked the target and distractor speakers at three levels: the acoustic envelope of speech (delta- and theta-band modulations), lexical frequency for individual words, and the contextual predictability of individual words estimated by GPT-4 and earlier lexical models. To provide a broader view of speech perception, half of the subjects completed a simultaneous visual task, and the listeners included both native and non-native English speakers. Distinct neural components were extracted for these levels of auditory and lexical processing, demonstrating that native English speakers had greater target-distractor separation compared to non-native English speakers on most measures, and that lexical processing was reduced by the visual task. Moreover, there was a novel interaction of lexical predictability and frequency with auditory processing; acoustic tracking was stronger for lexically harder words, suggesting that people listened harder to the acoustics when needed for lexical selection. This demonstrates that speech perception is not simply a feedforward process from acoustic processing to the lexicon. Rather, the adaptable context-sensitive processing long known to occur at a lexical level has broader consequences for perception, coupling with the acoustic tracking of individual speakers in noise.Significance Statement In challenging listening conditions, people use focused attention to help understand individual talkers and ignore others, which changes their neural processing for speech at auditory through lexical levels. However, lexical processing for natural materials (e.g., conversations, audiobooks) has been difficult to measure, because of limitations of tools to estimate the predictability of individual words in longer discourses. The present investigation uses a contemporary large language model, GPT-4, to estimate word predictability, and demonstrates that listeners make online adaptations to their auditory neural processing in accord with these predictions; neural activity more closely tracks the acoustics of the target talker when words are less predictable from the context.

The impact of fine-tuning paradigms on unknown plant diseases recognition

Plant diseases pose significant threats to agriculture, impacting both food safety and public health. Traditional plant disease detection systems are typically limited to recognizing disease categories included in the training dataset, rendering them ineffective against new disease types. Although out-of-distribution (OOD) detection methods have been proposed to address this issue, the impact of fine-tuning paradigms on these methods has been overlooked. This paper focuses on studying the impact of fine-tuning paradigms on the performance of detecting unknown plant diseases. Currently, fine-tuning on visual tasks is mainly divided into visual-based models and visual-language-based models. We first discuss the limitations of large-scale visual language models in this task: textual prompts are difficult to design. To avoid the side effects of textual prompts, we futher explore the effectiveness of purely visual pre-trained models for OOD detection in plant disease tasks. Specifically, we employed five publicly accessible datasets to establish benchmarks for open-set recognition, OOD detection, and few-shot learning in plant disease recognition. Additionally, we comprehensively compared various OOD detection methods, fine-tuning paradigms, and factors affecting OOD detection performance, such as sample quantity. The results show that visual prompt tuning outperforms fully fine-tuning and linear probe tuning in out-of-distribution detection performance, especially in the few-shot scenarios. Notably, the max-logit-based on visual prompt tuning achieves an AUROC score of 94.8%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} in the 8-shot setting, which is nearly comparable to the method of fully fine-tuning on the full dataset (95.2%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}), which implies that an appropriate fine-tuning paradigm can directly improve OOD detection performance. Finally, we visualized the prediction distributions of different OOD detection methods and discussed the selection of thresholds. Overall, this work lays the foundation for unknown plant disease recognition, providing strong support for the security and reliability of plant disease recognition systems. We will release our code at https://github.com/JiuqingDong/PDOOD to further advance this field.

Multi-task prompt tuning with soft context sharing for vision–language models

Vision–language models have recently shown great potential on many tasks in computer vision. Meanwhile, prior work demonstrates prompt tuning designed for vision–language models could acquire superior performance on few-shot image recognition compared to linear probe, a strong baseline. In practice, many few-shot tasks are inherently correlated, particularly within specialized domains. However, such information is overlooked previously. Inspired by the fact that modeling task relationship by multi-task learning can usually boost performance, we propose a novel method SoftCPT (Soft Context Sharing for Prompt Tuning) to tune pre-trained vision–language models on multiple target few-shot tasks jointly. Specifically, we design a task-shared meta network to generate prompt context for each task using task name together with a learnable task context as input. The parameters of this meta network as well as the task context are tuned on the joint training set of all tasks. As such, the prompt context of all tasks will be shared in a soft manner. Extensive experiments across four multi-task few-shot datasets covering 44 tasks and 1593 categories demonstrate that SoftCPT significantly outperforms single-task prompt tuning methods, highlighting the effectiveness of multi-task learning for vision–language prompt tuning.

Immersive analytics with augmented reality in meteorology: an exploratory study on ontology and linked data

Although Augmented Reality (AR) has been extensively studied in supporting Immersive Analytics (IA), there are still many challenges in visualising and interacting with big and complex datasets. To deal with these datasets, most AR applications utilise NoSQL databases for storing and querying data, especially for managing large volumes of unstructured or semi-structured data. However, NoSQL databases have limitations in their reasoning and inference capabilities, which can result in insufficient support for certain types of queries. To fill this gap, we aim to explore and evaluate whether an intelligent approach based on ontology and linked data can facilitate visual analytics tasks with big datasets on AR interface. We designed and implemented a prototype of this method for meteorological data analytics. An experiment was conducted to evaluate the use of a semantic database with linked data compared to a conventional approach in an AR-based immersive analytics system. The results significantly highlight the performance of semantic approach in helping the users analysing meteorological datasets and their subjective appreciation in working with the AR interface, which is enhanced with ontology and linked data.

Recurrent inhibition refines mental templates to optimize perceptual decisions.

Translating sensory inputs to perceptual decisions relies on building internal representations of features critical for solving complex tasks. Yet, we still lack a mechanistic account of how the brain forms these mental templates of task-relevant features to optimize decision-making. Here, we provide evidence for recurrent inhibition: an experience-dependent plasticity mechanism that refines mental templates by enhancing γ-aminobutyric acid (GABA)-mediated (GABAergic) inhibition and recurrent processing in superficial visual cortex layers. We combine ultrahigh-field (7 T) functional magnetic resonance imaging at submillimeter resolution with magnetic resonance spectroscopy to investigate the fine-scale functional and neurochemical plasticity mechanisms for optimized perceptual decisions. We demonstrate that GABAergic inhibition increases following training on a visual (i.e., fine orientation) discrimination task, enhancing the discriminability of orientation representations in superficial visual cortex layers that are known to support recurrent processing. Modeling functional and neurochemical plasticity interactions reveals that recurrent inhibitory processing optimizes brain computations for perpetual decisions and adaptive behavior.

Vision-Language Models for Vision Tasks: A Survey.

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition.

Tiny Object Tracking: A Large-Scale Dataset and a Baseline.

Tiny objects, frequently appearing in practical applications, have weak appearance and features, and receive increasing interests in many vision tasks, such as object detection and segmentation. To promote the research and development of tiny object tracking, we create a large-scale video dataset, which contains 434 sequences with a total of more than 217K frames. Each frame is carefully annotated with a high-quality bounding box. In data creation, we take 12 challenge attributes into account to cover a broad range of viewpoints and scene complexities, and annotate these attributes for facilitating the attribute-based performance analysis. To provide a strong baseline in tiny object tracking, we propose a novel multilevel knowledge distillation network (MKDNet), which pursues three-level knowledge distillations in a unified framework to effectively enhance the feature representation, discrimination, and localization abilities in tracking tiny objects. Extensive experiments are performed on the proposed dataset, and the results prove the superiority and effectiveness of MKDNet compared with state-of-the-art methods. The dataset, the algorithm code, and the evaluation code are available at https://github.com/mmic-lcl/Datasets-and-benchmark-code.

Coarse-to-fine mechanisms mitigate diffusion limitations on image restoration

Recent years have witnessed the remarkable performance of diffusion models in various vision tasks. However, for image restoration that aims to recover clear images with sharper details from given degraded observations, diffusion-based methods may fail to recover promising results due to inaccurate noise estimation. Moreover, simple constraining noises cannot effectively learn complex degradation information, which subsequently hinders the model capacity. To solve the above problems, we propose a coarse-to-fine diffusion Transformer (C2F-DFT) to mitigate diffusion limitations mentioned before on image restoration. Specifically, the proposed C2F-DFT contains diffusion self-attention (DFSA) and diffusion feed-forward network (DFN) within a new coarse-to-fine training mechanism. The DFSA and DFN with embedded diffusion steps respectively capture the long-range diffusion dependencies and learn hierarchy diffusion representation to guide the restoration process in different time steps. In the coarse training stage, our C2F-DFT estimates noises and then generates the final clean image by a sampling algorithm. To further improve the restoration quality, we propose a simple yet effective fine training pipeline. It first exploits the coarse-trained diffusion model with fixed steps to generate restoration results, which then would be constrained with corresponding ground-truth ones to optimize the models to remedy the unsatisfactory results affected by inaccurate noise estimation. Extensive experiments show that C2F-DFT significantly outperforms diffusion-based restoration method IR-SDE and achieves competitive performance compared with Transformer-based state-of-the-art methods on 3 tasks, including image deraining, image deblurring, and real image denoising. The source codes and visual results are available at https://github.com/wlydlut/C2F-DFT.

Position-Aware Relational Transformer for Knowledge Graph Embedding.

Although Transformer has achieved success in language and vision tasks, its capacity for knowledge graph (KG) embedding has not been fully exploited. Using the self-attention (SA) mechanism in Transformer to model the subject-relation-object triples in KGs suffers from training inconsistency as SA is invariant to the order of input tokens. As a result, it is unable to distinguish a (real) relation triple from its shuffled (fake) variants (e.g., object-relation-subject) and, thus, fails to capture the correct semantics. To cope with this issue, we propose a novel Transformer architecture, namely, Knowformer, for KG embedding. It incorporates relational compositions in entity representations to explicitly inject semantics and capture the role of an entity based on its position (subject or object) in a relation triple. The relational composition for a subject (or object) entity of a relation triple refers to an operator on the relation and the object (or subject). We borrow ideas from the typical translational and semantic-matching embedding techniques to design relational compositions. We carefully design a residual block to integrate relational compositions into SA and efficiently propagate the composed relational semantics layer by layer. We formally prove that the SA with relational compositions is able to distinguish the entity roles in different positions and correctly capture relational semantics. Extensive experiments and analyses on six benchmark datasets show that Knowformer achieves state-of-the-art performance on both link prediction and entity alignment.

SpectralGPT: Spectral Remote Sensing Foundation Model.

The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS Big Data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; and 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS Big Data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.