Visual Sequences Research Articles

Synthesis of Arts by means of Sculpture Forms

The purpose of the work is to characterise the syncretism of sculptural forms in the development of monumental Ukrainian sculpture; emphasise the national features of the development of national sculpture on the example of the works of modern Ukrainian sculptors. The methodology of the work consists in the principles of holistic and systemic approaches to the study of the synthesis of arts in the sculptural forms of Ukrainian artists. An analytical method was applied to study scientific approaches to the synthesis of arts, as well as an artistic-stylistic method, as well as a system-functional method to the structuring of artistic works as a phenomenon of modern monumental sculpture. The scientific novelty of the study consists in the discourse on the synthesis of arts in the art history environment, characterising some works of art that belong to the category of monumental sculpture. Conclusions. It should be noted that in the Ukrainian art space there is a synthesis between classical forms and intermedia installations, which creates a complex and multi-layered context for modern artists. The perception of time, space and material becomes plastic and changeable, and this, in turn, stimulates the development of new aesthetic forms. Sculptural images have a significant influence on the formation of styles of young artists, where the emphasis is not only on silhouettes, but also on specific elements originating from various forms. Artists build associative visual sequences, embodying unrealistic, yet deeply symbolic images that resonate with viewers on an emotional level. In the examples from the workshops you mentioned, we see how a variety of materials and art forms and execution methods are used to create coherent and complete images that reflect the synthesis of different arts. The theme of the synthesis of arts in different sculptural forms, which sometimes seem completely incompatible, is indeed one of the most relevant for research.

Different subregions of monkey lateral prefrontal cortex respond to abstract sequences and their components.

Sequential information permeates daily activities, such as when watching for the correct series of buildings to determine when to get off the bus or train. These sequences include periodicity (the spacing of the buildings), the identity of the stimuli (the kind of house), and higher-order more abstract rules that may not depend on the exact stimulus (e.g. house, house, house, business). Previously, we found that the posterior fundus of area 46 in the monkey lateral prefrontal cortex (LPFC) responds to rule changes in such abstract visual sequences. However, it is unknown if this region responds to other components of the sequence, i.e., image periodicity and identity, in isolation. Further, it is unknown if this region dissociates from other, more ventral LPFC subregions that have been associated with sequences and their components. To address these questions, we used awake functional magnetic resonance imaging in three male macaque monkeys during two no-report visual tasks. One task contained abstract visual sequences, and the other contained no visual sequences but maintained the same image periodicity and identities. We found the fundus of area 46 responded only to abstract sequence rule violations. In contrast, the ventral bank of area 46 responded to changes in image periodicity and identity, but not changes in the abstract sequence. These results suggest a functional specialization within anatomical substructures of LPFC to signal different kinds of stimulus regularities. This specialization may provide key scaffolding to identify abstract patterns and construct complex models of the world for daily living.Significance Statement Daily tasks, such as a bus commute, require tracking or monitoring your place (same, same, same, different building) until your stop. Sequence components such as rule, periodicity (timing), and item identity are involved in this process. While prior work located responses to sequence rule changes to area 46 of monkey lateral prefrontal cortex (LPFC) using awake monkey fMRI, less was known about other components. We found that LPFC subregions differentiated between sequence components. Area 46 posterior fundus responded to abstract visual sequence rule changes, but not to changes in image periodicity or identity. The converse was true for the more ventral, adjacent shoulder region. These results suggest that interactions between adjacent LPFC subregions provide key scaffolding for complex daily behaviors.

REVOLUTIONIZING ONCOLOGY IMAGING: A PARADIGM SHIFT THROUGH STANDARDIZED MRI PROTOCOLS: VOLUME 2

Introduction: Oncology imaging is always developing, thus rapid and precise diagnosis are essential for cancer therapy. Recognizing the importance of Magnetic Resonance Imaging, our Mahamana Pandit Madan Mohan Malaviya Cancer Centre is standardizing MRI protocols to improve inspection and interpretation. This project aims to standardize, eliminate variance, and speed up patient care choices. Methods: Our standardized MRI procedures summarize scan parameters, sequences, modes, matrix and other data by body part. A 3 Tesla MRI machine was used to study how scanner differences affect parameters. Detailing appropriate pathological visualization sequences allowed diagnostic versatility. Localiser or scanogram was emphasized for exact scan plane alignment, ensuring uniform pictures for smooth comparisons. Results:Specific procedures were established for certain anatomical locations, with a focus on precise patient placement. Optimizing the localizer setup and selecting the appropriate coil can enhance the quality of images and increase diagnostic accuracy. Discussion: The conversation emphasizes the crucial need of standardizing MRI methods to ensure prompt and precise cancer diagnosis. The benefits of using this technology, such as maintaining consistency, saving time, and improving picture quality, help speed up the diagnostic process and lead to better outcomes for patients. Conclusion: Standardizing MRI procedures is a major step toward efficient and patient-centered cancer imaging, addressing varied oncological issues and improving consistency, time efficiency, and picture quality. Staff training and cooperation put our institution at the forefront of technology advances to improve cancer patient outcomes. We aim to enhance cancer care and treatment via cooperation, improvement, and a patient-centric approach.

Different subregions of monkey lateral prefrontal cortex respond to abstract sequences and their components.

Daily tasks, such as a bus commute, require tracking or monitoring your place (same, same, same, different building) until your stop. Sequence components such as rule, periodicity (timing), and item identity are involved in this process. While prior work located responses to sequence rule changes to area 46 of monkey lateral prefrontal cortex (LPFC) using awake monkey fMRI, less was known about other components. We found that LPFC subregions differentiated between sequence components. Area 46 posterior fundus responded to abstract visual sequence rule changes, but not to changes in image periodicity or identity. The converse was true for the more ventral, adjacent shoulder region. These results suggest that interactions between adjacent LPFC subregions provide key scaffolding for complex daily behaviors.

Visual sequence encoding is modulated by music schematic structure and familiarity.

Music is omnipresent in daily life and may interact with critical cognitive processes including memory. Despite music's presence during diverse daily activities including studying, commuting, or working, existing literature has yielded mixed results as to whether music improves or impairs memory for information experienced in parallel. To elucidate how music memory and its predictive structure modulate the encoding of novel information, we developed a cross-modal sequence learning task during which participants acquired sequences of abstract shapes accompanied with paired music. Our goal was to investigate whether familiar and structurally regular music could provide a "temporal schema" (rooted in the organized and hierarchical structure of music) to enhance the acquisition of parallel temporally-ordered visual information. Results revealed a complex interplay between music familiarity and music structural regularity in learning paired visual sequences. Notably, compared to a control condition, listening to well-learned, regularly-structured music (music with high predictability) significantly facilitated visual sequence encoding, yielding quicker learning and retrieval speed. Conversely, learned but irregular music (where music memory violated musical syntax) significantly impaired sequence encoding. While those findings supported our mechanistic framework, intriguingly, unlearned irregular music-characterized by the lowest predictability-also demonstrated memory enhancement. In conclusion, this study demonstrates that concurrent music can modulate visual sequence learning, and the effect varies depending on the interaction between both music familiarity and regularity, offering insights into potential applications for enhancing human memory.

Semantic-driven diffusion for sign language production with gloss-pose latent spaces alignment

Sign Language Production (SLP) aims to translate spoken language into visual sign language sequences. The most challenging process in SLP is the transformation of a sequence of sign glosses into corresponding sign poses (G2P). Existing approaches on G2P mainly focus on constructing mappings of sign language glosses to frame-level sign pose representations, while neglecting gloss is just a weak annotation of the sequence of sign poses. To address this problem, this paper proposes the semantic-driven diffusion model with gloss-pose latent spaces alignment (SDD-GPLA) for G2P. G2P is divided into two phases. In the first phase, we design the gloss-pose latent spaces alignment (GPLA) to model the sign pose latent representations with glosses dependency. In the second phase, we propose semantic-driven diffusion (SDD) with supervised pose reconstruction guidance as a mapping between the gloss and sign poses latent features. In addition, we propose the sign pose decoder (Decoderp) to progressively generate high-resolution sign poses from latent sign pose features and to guide the SDD training process. We evaluated SDD-GPLA on a self-collected dataset of Daily Chinese Sign Language (DCSL) and a public dataset called RWTH-Phoenix-Weather-2014T. Compared with the state-of-the-art G2P methods, we obtain at least 22.9% and 2.3% improvement in WER scores on the above two datasets, respectively.

Cross-modal interaction and multi-source visual fusion for video generation in fetal cardiac screening

To address the limitation of preserving data for dynamic visualization in fetal ultrasound screening, a novel framework is proposed to facilitate the generation of fetal four-chamber echocardiogram videos, incorporating multi-source visual fusion and understanding. The framework utilizes an effective spectrogram-ultrasound synchronizer to align the ultrasound images with time, ensuring the generated video matches the actual heartbeat rhythm. It further employs effective frame interpolation techniques to synthesize a video by incorporating a nonlinear bidirectional motion prediction. By integrating a Transformer model for the autoregressive generation of visual semantic sequence, the proposed framework demonstrates its capability to generate high-resolution frames. Experimental outcomes show the Clip-Similarity of 96.23% and DINOv2-Similarity of 99.77%. Furthermore, a multimodal dataset of fetal echocardiogram examinations has been constructed.

First experience with a vendor-neutral three-dimensional mapping system for cardiac magnetic resonance-guided electrophysiological procedures: a case report

Abstract Background Fluoroscopy-guided catheter ablation has become the gold standard for treatment of cardiac arrhythmias. High resolution electro-anatomical mapping systems have become fundamental to perform these procedures. Recently, interventional cardiac magnetic resonance (iCMR) has been proposed as an alternative for fluoroscopy to guide atrial flutter ablations. The clinical experience with iCMR and dedicated three-dimensional mapping systems is growing. NorthStar is currently the first available vendor-neutral mapping system. Case summary We performed a real-time CMR-guided cavotricuspid isthmus (CTI) catheter ablation (CA) on a 69-year-old man using a novel mapping system (NorthStar Mapping System, Imricor Medical Systems, MN, USA). Starting from the CMR imaging, a pre-rendered segmentation model was loaded on NorthStar and used to guide the catheters, display voltage and activation maps, show mapping and ablation points. NorthStar can also take full control of the CMR scanner (i.e. start/stop sequences for anatomical information, tissue characterization, and catheter visualization) and communicate with the recorder/stimulator system (Advantage-MR EP, Imricor Medical Systems, MN, USA). With comparable procedural time to standard fluoroscopy-guided CA, CTI bidirectional block was achieved, without any complication. Discussion Using the NorthStar Mapping System, we managed to achieve a successful CMR-guided CTI ablation without any complication. Its further use should be explored, especially in more complex arrhythmias where a substrate-guided ablation is critical, as it could significantly improve results in terms of arrhythmia recurrence.

Making acute ischemic stroke thrombi visible in MRI imaging

Knowledge of thrombus behavior and visualization on MRI in acute ischemic stroke is less than optimal. However, MRI sequences could be enhanced based on the typical T1 and T2 relaxation times of the target tissues, which mainly determine their signal intensities on imaging. We studied the relaxation times of a broad spectrum of clot analogs along with their image characteristics of three sequences analyzed: a T1-weighted turbo inversion-recovery sequence (T1w Turbo IR), a T1-weighted turbo spin echo with fat suppression (T1w TSE SPIR), and a T2-weighted 3D TSE with magnetization refocusing to remove T1 dependence (T2w TSE DRIVE). We compared their imaging behavior with the intensity values of normal brain tissue using the same imaging protocols as for clots. Each histological and biochemical clot component contributed to each of the relaxation times. Overall, histological composition correlated strongly with T1 times, and iron content, specifically, with T2 relaxation time. Using decision trees, fibrin content was selected as the primary biomarker for T1 relaxation times, inducing an increase. Up to four clot subgroups could be defined based on its distinctive T1 relaxation time. Clot signal intensity in the T1 and T2-weighted images varied significantly according to T1 and T2 relaxation times. Moreover, in comparison with normal brain tissue intensity values, T2w DRIVE images depict thrombi according to the principle of the more fibrin, the higher the intensity, and in T1w TSE, the more erythrocytes, the higher the intensity. These findings could facilitate improvements in MRI sequences for clot visualization and indicate that T2w DRIVE and T1w TSE sequences should depict the vast majority of acute ischemic stroke thrombi as more hyperintense than surrounding tissues.

Effects of sensory cues on consumers’ wine taste perceptions and behavior: evidence from a wine-tasting experiment

PurposeThis study aims to investigate how sensory cue order, wine knowledge and visual–olfactory (V–O) congruence affect consumer’s taste perceptions of wine and their subsequent behavior.Design/methodology/approachAn experiment was performed to identify the effects of sensory cue congruence and sensory cue order on wine consumers’ perceptions of wine, affective evaluations, cognitive evaluations and purchase intentions.FindingsWine experts exhibited positive emotional responses to congruent sensory cues in the V–O order. Experts’ enjoyment of wine’s aroma, their emotional responses, their cognitive evaluations and their purchase intentions were lower in the incongruent condition. Consumers’ negative emotions elicited by the V–O sequence were also less intense than those triggered by the olfactory–visual (O–V) sequence. Wine experts demonstrated more positive emotional responses in the V–O sensory congruent condition.Research limitations/implicationsThis study highlights how visual and olfactory sensory cue order, wine knowledge and sensory cue congruence interact to clarify wine-related behavioral intention. Findings reveal the roles of these factors in shaping sensory perceptions, cognitive evaluations, affective evaluations and behavior related to wine consumption.Practical implicationsThis study holds implications for various stakeholders, including winemakers, wine businesses, restaurants and the broader hospitality industry. Wine businesses can enhance advertising effectiveness by tailoring their marketing efforts to customers’ knowledge levels and emphasizing the inherent attributes that align with individuals’ preferences. Winemakers can improve consumers’ sensory experiences by enhancing the natural color of wines. Restaurants can strive to ignite diners’ positive emotions and experiences by providing congruent information. Furthermore, sensory-driven strategies can be used in the hospitality sector to elevate customers’ positive emotions.Originality/valueThis study fills gaps in wine research by delineating how wine knowledge and related sensory cues can influence consumers’ sensory perceptions, cognitive evaluations, affective evaluations and behavior. These aspects have been largely overlooked in previous work.

Investigating the Influence of Varied Light-Emitting Diode (LED) Wavelengths on Phototactic Behavior and Opsin Genes in Vespinae.

The phototactic behavior of insects is commonly used to manage pest populations in practical production. However, this elusive behavior is not yet fully understood. Investigating whether the opsin genes play a crucial role in phototaxis is an intriguing topic. Vespinae (Hymenoptera: Vespidae) are a common group of social wasps that are closely associated with human activities. Efficiently controlling wasp populations while maintaining ecological balance is a pressing global challenge that still has to be resolved. This research aims to explore the phototactic behavior and key opsin genes associated with Vespinae. We found significant differences in the photophilic rates of Vespula germanica and Vespa analis under 14 different light conditions, indicating that their phototactic behavior is rhythmic. The results also showed that the two species exhibited varying photophilic rates under different wavelengths of light, suggesting that light wavelength significantly affects their phototactic behavior. Additionally, the opsin genes of the most aggressive hornet, Vespa basalis, have been sequenced. There are only two opsin genes, one for UV light and the other for blue light, and Vespa basalis lacks long-wavelength visual proteins. However, they exhibit peak phototaxis for long-wavelength light and instead have the lowest phototaxis for UV light. This suggests that the visual protein genes have a complex regulatory mechanism for phototactic behavior in Vespinae. Additionally, visual protein sequences have a high degree of homology among Hymenoptera. Despite the hypotheses put forward by some scholars regarding phototaxis, a clear and complete explanation of insect phototaxis is still lacking to date. Our findings provide a strong theoretical basis for further investigation of visual expression patterns and phototactic mechanisms in Vespinae.

Visual statistical learning in preverbal infants at a higher likelihood of autism and its association with later social communication skills.

Visual statistical Learning (SL) allows infants to extract the statistical relationships embedded in a sequence of elements. SL plays a crucial role in language and communication competencies and has been found to be impacted in Autism Spectrum Disorder (ASD). This study aims to investigate visual SL in infants at higher likelihood of developing ASD (HL-ASD) and its predictive value on autistic-related traits at 24-36 months. At 6 months of age, SL was tested using a visual habituation task in HL-ASD and neurotypical (NT) infants. All infants were habituated to a visual sequence of shapes containing statistically predictable patterns. In the test phase, infants viewed the statistically structured, familiar sequence in alternation with a novel sequence that did not contain any statistical information. HL-ASD infants were then evaluated at 24-36 months to investigate the associations between visual SL and ASD-related traits. Our results showed that NT infants were able to learn the statistical structure embedded in the visual sequences, while HL-ASD infants showed different learning patterns. A regression analysis revealed that SL ability in 6-month-old HL-ASD infants was related to social communication and interaction abilities at 24-36 months of age. These findings indicate that early differences in learning visual statistical patterns might contribute to later social communication skills.

Tactile adaptation to orientation produces a robust tilt aftereffect and exhibits crossmodal transfer when tested in vision

Orientation processing is one of the most fundamental functions in both visual and somatosensory perception. Converging findings suggest that orientation processing in both modalities is closely linked: somatosensory neurons share a similar orientation organisation as visual neurons, and the visual cortex has been found to be heavily involved in tactile orientation perception. Hence, we hypothesized that somatosensation would exhibit a similar orientation adaptation effect, and this adaptation effect would be transferable between the two modalities, considering the above-mentioned connection. The tilt aftereffect (TAE) is a demonstration of orientation adaptation and is used widely in behavioural experiments to investigate orientation mechanisms in vision. By testing the classic TAE paradigm in both tactile and crossmodal orientation tasks between vision and touch, we were able to show that tactile perception of orientation shows a very robust TAE, similar to its visual counterpart. We further show that orientation adaptation in touch transfers to produce a TAE when tested in vision, but not vice versa. Additionally, when examining the test sequence following adaptation for serial effects, we observed another asymmetry between the two conditions where the visual test sequence displayed a repulsive intramodal serial dependence effect while the tactile test sequence exhibited an attractive serial dependence. These findings provide concrete evidence that vision and touch engage a similar orientation processing mechanism. However, the asymmetry in the crossmodal transfer of TAE and serial dependence points to a non-reciprocal connection between the two modalities, providing further insights into the underlying processing mechanism.

Investigating emotional design of the intelligent cockpit based on visual sequence data and improved LSTM

To enhance affective experience and customer satisfaction in the intelligent cockpit of new energy vehicle (NEV-IC), this article proposes a novel method that combines the visual sequence data of eye movements with the sentiment prediction using improved Long Short-Term Memory (LSTM). Specifically, we used eye-tracking technology to capture users' visual sequence of design morphology for NEV-IC. We then adopted entropy-TOPSIS to compute the ranking of morphological components based on experts’ opinions, establishing the coupling between users' visual perception and experts' opinion to obtain the key morphological dataset of NEV-IC based on user visual sequence. To tackle the shortcomings of LSTM, meanwhile, we employed the sparrow search algorithm (SSA) to optimize the hyperparameters of the LSTM model. Moreover, an attention mechanism has been introduced to address LSTM's difficulty in preserving key information when processing the sequential data, enabling a stronger focus on critical sequential features within the user's visual path. To assess the efficacy of the proposed SSA-LSTM-Attention model, a dataset incorporating user emotional imagery was constructed, within the research framework of Kansei engineering (KE). This dataset, in conjunction with the morphological dataset of visual sequential features, was applied to our model. The study results indicated that compared to traditional machine learning models like BP neural network (BPNN), support vector regression (SVR), and LSTM, our model performed better in capturing the nonlinear relationship between user sentiment and design features. Additionally, it exhibited higher predictive accuracy, better generalization ability and stronger robustness.

Three-dimensional magnetic resonance cholangiography is superior to two-dimensional single-shot magnetic resonance cholangiography for visualization and image quality of the feline and canine biliary tract: A postmortem study.

Magnetic resonance cholangiography (MRC) is an established diagnostic tool for noninvasive assessment of the biliary tract in humans. It has also been found to be feasible in companion animals, but no published studies have compared MRC sequences in veterinary medicine. The present study is part of a prospective, observational, analytical investigation on MR cholangiopancreatography performed on the donated bodies of 12 cats and eight dogs. The main aim of this study was to compare the images of 2D-SSh-TSE-MRC and 3D-TSE-MRC sequences for visualization and image quality of the feline and canine biliary tract. Both sequences are T2-weighted and noncontrast. Three independent readers scored the visibility of four segments of the biliary tract, namely the gallbladder (GB), cystic duct, common bile duct (CBD), and extrahepatic ducts, and the image quality of the two MRC sequences using five-point Likert scales. Wilcoxon signed-rank test was used to compare the scores between the MRC sequences separately for cats and dogs. Inter- and intraobserver agreements were measured using Gwet's AC2 with linear weighting. The 3D-TSE-MRC images were scored significantly higher than the 2D-SSh-TSE-MRC for both visibility and image quality (P<.001-.016 for cats, P=.008-.031 for dogs); the only exception was GB in dogs. In both cats and dogs, interobserver agreement for segment visibility and image quality ranged from slight to substantial in 2D-SSh-TSE-MRC and from poor to almost perfect in 3D-TSE-MRC. Most of the assessments (73% for segment visibility and 66% for image quality) had substantial to almost perfect intraobserver agreement. Findings from the current study support the use of 3D-TSE-MRC over 2D-SSh-TSE-MRC for evaluation of the feline and canine biliary tract, but further studies on live animals are warranted.

Audiovisual integration of rhythm in musicians and dancers.

Music training is associated with better beat processing in the auditory modality. However, it is unknown how rhythmic training that emphasizes visual rhythms, such as dance training, might affect beat processing, nor whether training effects in general are modality specific. Here we examined how music and dance training interacted with modality during audiovisual integration and synchronization to auditory and visual isochronous sequences. In two experiments, musicians, dancers, and controls completed an audiovisual integration task and an audiovisual target-distractor synchronization task using dynamic visual stimuli (a bouncing figure). The groups performed similarly on the audiovisual integration tasks (Experiments 1 and 2). However, in the finger-tapping synchronization task (Experiment 1), musicians were more influenced by auditory distractors when synchronizing to visual sequences, while dancers were more influenced by visual distractors when synchronizing to auditory sequences. When participants synchronized with whole-body movements instead of finger-tapping (Experiment 2), all groups were more influenced by the visual distractor than the auditory distractor. Taken together, this study highlights how training is associated with audiovisual processing, and how different types of visual rhythmic stimuli and different movements alter beat perception and production outcome measures. Implications for the modality appropriateness hypothesis are discussed.

Moving stimuli enhance beat timing and sensorimotor coupling in vision.

Vision has long been known for its inefficiency in beat perception and synchronization. However, this has been challenged by the finding that moving stimuli (bouncing ball or moving bar) can significantly improve visual beat synchronization. The present study examined two possible mechanisms for this phenomenon: visual motion facilitates temporal processing or promotes sensorimotor coupling. Instead of a single visual object (such as a ball or bar), random-dot kinematograms (RDKs) were used to construct visual motion sequences to avoid confounding factors, such as changes in trajectory and velocity. Experiment 1 showed that RDKs improved beat-timing discrimination compared with visual flashes, but auditory tones were still superior to RDKs. In Experiment 2, synchronized movements improved auditory-tone beat timing but impaired visual-flash beat timing, with no effect on RDK beat timing. Experiment 3 indicated that the regression slope of the phase correction response in RDKs was higher than that in visual flashes but still lower than that in auditory tones. The results showed that moving stimuli enhances both temporal processing (Experiment 1) and sensorimotor coupling (Experiments 2 and 3) in vision, but to a lesser degree, with audition retaining an advantage. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Transformer-Augmented Network With Online Label Correction for Facial Expression Recognition

Facial expression recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses under unconstrained conditions and incorrect annotations (e.g., label noise). In this paper, we aim to improve the performance of in-the-wild FER with Transformers and online label correction. Different from pure CNNs based methods, we propose a Transformer-augmented network (TAN) to dynamically capture the relationships within each facial patch and across the facial patches. Specifically, the TAN translates a number of facial patch images into a set of visual feature sequences by a backbone convolutional neural network. The intra-patch Transformer is subsequently utilized to capture the most discriminative features within each visual feature sequence. The position-disentangled attention mechanism of the intra-patch Transformer is proposed to better incorporate the positional information for feature sequences. Furthermore, we propose the inter-patch Transformer to model the dependencies across these feature sequences. More importantly, we present the online label correction (OLC) framework to correct suspicious hard labels and accumulate soft labels based on the predictions of the model, which strengthens the robustness of our model against label noise. We validate our method on several widely-used datasets (RAF-DB, FERPlus, AffectNet), realistic occlusion and pose variation datasets, and synthetic noisy datasets. Extensive experiments on these benchmarks demonstrate that the proposed method performs favorably against state-of-the-art methods. The source code will be made publicly available.

Multi-tailed vision transformer for efficient inference

Recently, Vision Transformer (ViT) has achieved promising performance in image recognition and gradually serves as a powerful backbone in various vision tasks. To satisfy the sequential input of Transformer, the tail of ViT first splits each image into a sequence of visual tokens with a fixed length. Then, the following self-attention layers construct the global relationship between tokens to produce useful representation for the downstream tasks. Empirically, representing the image with more tokens leads to better performance, yet the quadratic computational complexity of self-attention layer to the number of tokens could seriously influence the efficiency of ViT’s inference. For computational reduction, a few pruning methods progressively prune uninformative tokens in the Transformer encoder, while leaving the number of tokens before the Transformer untouched. In fact, fewer tokens as the input for the Transformer encoder can directly reduce the following computational cost. In this spirit, we propose a Multi-Tailed Vision Transformer (MT-ViT) in the paper. MT-ViT adopts multiple tails to produce visual sequences of different lengths for the following Transformer encoder. A tail predictor is introduced to decide which tail is the most efficient for the image to produce accurate prediction. Both modules are optimized in an end-to-end fashion, with the Gumbel-Softmax trick. Experiments on ImageNet-1K demonstrate that MT-ViT can achieve a significant reduction on FLOPs with no degradation of the accuracy and outperform compared methods in both accuracy and FLOPs.

Visual Clue Guidance and Consistency Matching Framework for Multimodal Named Entity Recognition

The goal of multimodal named entity recognition (MNER) is to detect entity spans in given image–text pairs and classify them into corresponding entity types. Despite the success of existing works that leverage cross-modal attention mechanisms to integrate textual and visual representations, we observe three key issues. Firstly, models are prone to misguidance when fusing unrelated text and images. Secondly, most existing visual features are not enhanced or filtered. Finally, due to the independent encoding strategies employed for text and images, a noticeable semantic gap exists between them. To address these challenges, we propose a framework called visual clue guidance and consistency matching (GMF). To tackle the first issue, we introduce a visual clue guidance (VCG) module designed to hierarchically extract visual information from multiple scales. This information is utilized as an injectable visual clue guidance sequence to steer text representations for error-insensitive prediction decisions. Furthermore, by incorporating a cross-scale attention (CSA) module, we successfully mitigate interference across scales, enhancing the image’s capability to capture details. To address the third issue of semantic disparity between text and images, we employ a consistency matching (CM) module based on the idea of multimodal contrastive learning, facilitating the collaborative learning of multimodal data. To validate the effectiveness of our proposed framework, we conducted comprehensive experimental studies, including extensive comparative experiments, ablation studies, and case studies, on two widely used benchmark datasets, demonstrating the efficacy of the framework.