Retrieval Framework Research Articles

Enhancing suicidal behavior detection in EHRs: A multi-label NLP framework with transformer models and semantic retrieval-based annotation

Background:Suicide is a leading cause of death worldwide, making early identification of suicidal behaviors crucial for clinicians. Current Natural Language Processing (NLP) approaches for identifying suicidal behaviors in Electronic Health Records (EHRs) rely on keyword searches, rule-based methods, and binary classification, which may not fully capture the complexity and spectrum of suicidal behaviors. This study aims to create a multi-class labeled dataset with annotation guidelines and develop a novel NLP approach for fine-grained, multi-label classification of suicidal behaviors, improving the efficiency of the annotation process and accuracy of the NLP methods. Methods:We develop a multi-class labeling system based on guidelines from FDA, CDC, and WHO, distinguishing between six categories of suicidal behaviors and allowing for multiple labels per data sample. To efficiently create an annotated dataset, we use an MPNet-based semantic retrieval framework to extract relevant sentences from a large EHR dataset, reducing annotation space while capturing diverse expressions. Experts annotate the extracted sentences using the multi-class system. We then formulate the task as a multi-label classification problem and fine-tune transformer-based models on the curated dataset to accurately classify suicidal behaviors in EHRs. Results:Lexical analysis revealed key themes in assessing suicide risk, considering an individual’s history, mental health, substance use, and family background. Fine-tuned transformer-based models effectively identified suicidal behaviors from EHRs, with Bio_ClinicalBERT, BioBERT, and XLNet achieving the F1 scores (0.81), outperforming BERT and RoBERTa. The proposed approach, based on a multi-label classification system, captures the complexity of suicidal behaviors effectively particularly “Suicide Attempt” and “Family History” instances. The proposed approach, using task-specific NLP models and a multi-label classification system, captures the complexity of suicidal behaviors more effectively than traditional binary classification. However, direct comparisons with existing studies are difficult due to varying metrics and label definitions. Conclusion:This study presents a robust NLP framework for detecting suicidal behaviors in EHRs, leveraging task-specific fine-tuning of transformer-based models and a semi-automated pipeline. Despite limitations, the approach demonstrates the potential of advanced NLP techniques in enhancing the identification of suicidal behaviors. Future work should focus on model expansion and integration to further improve patient care and clinical decision-making.

Implementation of Aerosol Mie Scattering in POSEIDON with Application to the Hot Jupiter HD 189733 b’s Transmission, Emission, and Reflected Light Spectrum

Aerosols are a ubiquitous feature of planetary atmospheres and leave clear spectral imprints in exoplanet spectra. Pre-JWST, exoplanet retrieval frameworks mostly adopted simple parametric approximations. With JWST, we now have access to mid-infrared wavelengths where aerosols have detectable composition-specific resonance features. Here, we implement new features into the open-source atmospheric retrieval code POSEIDON to account for the complex scattering, reflection, and absorption properties of Mie-scattering aerosols. We provide an open-source database of these Mie-scattering cross sections and optical properties. We also extend the radiative transfer and retrieval functionality in POSEIDON to include multiple scattering reflection and emission spectroscopy. We demonstrate these new retrieval capabilities on archival Hubble and Spitzer transmission and secondary-eclipse spectra of the hot Jupiter HD 189733 b. We find that a high-altitude, low-density, and thin slab composed of submicron particles is necessary to fit HD 189733 b’s transmission spectrum, with multiple aerosol species providing a good fit. We additionally retrieve a subsolar H2O abundance, a subsolar K abundance, and do not detect CO2. Our joint thermal and reflection retrievals of HD 189733 b’s secondary-eclipse spectrum, however, finds no evidence of dayside aerosols, a subsolar dayside H2O abundance, enhanced CO2, or slightly subsolar alkali abundances. We additionally explore how retrieval model choices such as cloud parameterization, aerosol species and properties, and thermal structure parameterization affect retrieved atmospheric properties. Upcoming JWST data for hot Jupiters such as HD 189733 b will be well suited to enable deeper exploration of aerosol properties, allowing the formulation of a self-consistent, multidimensional picture of cloud formation processes.

Deep ensemble architectures with heterogeneous approach for an efficient content-based image retrieval

<span lang="EN-US">In the field of digital image processing, content-based image retrieval (CBIR) has become essential for searching images based on visual content characteristics like color, shape, and texture, rather than relying on text-based annotations. To address the increasing demands for efficiency and precision in CBIR systems, we introduce the HybridEnsembleNet methodology. HybridEnsembleNet combines deep learning algorithms with an asymmetric retrieval framework to optimize feature extraction and comparison in extensive image databases. This novel approach, specifically custom-made for CBIR, employs a lightweight query structure skilled at handling large-scale data under resource-constrained environments. The experiments were performed on the ROxford and RParis datasets. The deep learning component of HybridEnsembleNet significantly refines the accuracy of image matching and retrieval. RParis The ROxford dataset, specifically in the medium and hard difficulty benchmarks, demonstrates an enhancement of 5.53% and 10.44%, respectively. Similarly, the RParis dataset, under medium and hard benchmarks, exhibits improvements of 3.01% and 5.83%, showcasing superior performance compared to existing models. By overcoming the traditional limitations of CBIR systems in mean average precision (mAP) metrics, HybridEnsembleNet provides a scalable, efficient, and more accurate solution for retrieving relevant images from vast digital libraries.</span>

Improving solution availability and temporal consistency of an optimal-estimation physical retrieval for ground-based thermodynamic boundary layer profiling

Abstract. Thermodynamic profiles in the atmospheric boundary layer can be retrieved from ground-based passive remote sensing instruments like infrared spectrometers and microwave radiometers with optimal-estimation physical retrievals. With a high temporal resolution on the order of minutes, these thermodynamic profiles are a powerful tool to study the evolution of the boundary layer and to evaluate numerical models. In this study, we describe three recent modifications to the Tropospheric Remotely Observed Profiling via Optimal Estimation (TROPoe) retrieval framework, which improve the availability of valid solutions for different atmospheric conditions and increase the temporal consistency of the retrieved profiles. We present methods to enhance the availability of valid solutions retrieved from infrared spectrometers by preventing overfitting and by adding information from an additional spectral band in high-moisture environments. We show that the characterization of the uncertainty of the input and the choice of spectral infrared bands are crucial for retrieval performance. Since each profile is retrieved independently from the previous one, the time series of the thermodynamic variables contain random uncorrelated noise, which may hinder the study of diurnal cycles and temporal tendencies. By including a previous retrieved profile as input to the retrieval, we increase the temporal consistency between subsequent profiles without suppressing real mesoscale atmospheric variability. We demonstrate that these modifications work well at midlatitudes, polar and tropical sites, and for retrievals based on infrared spectrometer and microwave radiometer measurements.

Retrieve–Revise–Refine: A novel framework for retrieval of concise entailing legal article set

The retrieval of entailing legal article sets aims to identify a concise set of legal articles that holds an entailment relationship with a legal query or its negation. Unlike traditional information retrieval that focuses on relevance ranking, this task demands conciseness. However, prior research has inadequately addressed this need by employing traditional methods. To bridge this gap, we propose a three-stage Retrieve–Revise–Refine framework which explicitly addresses the need for conciseness by utilizing both small and large language models (LMs) in distinct yet complementary roles. Empirical evaluations on the COLIEE 2022 and 2023 datasets demonstrate that our framework significantly enhances performance, achieving absolute increases in the macro F2 score by 3.17% and 4.24% over previous state-of-the-art methods, respectively. Specifically, our Retrieve stage, employing various tailored fine-tuning strategies for small LMs, achieved a recall rate exceeding 0.90 in the top-5 results alone—ensuring comprehensive coverage of entailing articles. In the subsequent Revise stage, large LMs narrow this set, improving precision while sacrificing minimal coverage. The Refine stage further enhances precision by leveraging specialized insights from small LMs, resulting in a relative improvement of up to 19.15% in the number of concise article sets retrieved compared to previous methods. Our framework offers a promising direction for further research on specialized methods for retrieving concise sets of entailing legal articles, thereby more effectively meeting the task’s demands.

An integrated blockchain and fractional DCT based highly secured framework for storage and retrieval of retinal images

The digital image capability of storing large amount of data has resulted in it’s increased popularity. Images are used to transmit large amount of information across different geographical locations using different cloud services. Securing these digitally stored images has remained a challenging task for researchers as they are prone to cyber- attacks. A potential solution to this problem can be blockchain, which provide secure and unchangeable storage. However, securing images on blockchain has another challenge as the image size increases the associated cost involved in blockchain. Fractional Discrete Cosine Transform(fctDCT) has the capability to minimizes the amount of data necessary for expressing an image in a secure way. This paper presents a novel framework for securely storing and retrieving medical images by extracting feature maps from medical images by fctDCT, followed by encoding and storing the feature map on decentralized cloud and linking them on blockchain. The integration has been implemented by using four different α angles which are stored on blockchain and are needed to be same at storage and retrieval stage as only the authentic user would have access to unique α angles and number of coefficients that have been used in storing their medical images. The performance of proposed framework has been evaluated by employing image quality metric such as Mean Square Error (MSE), Peak Signal to Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) and multi-SSIM by comparing it with correct and incorrect α values on four different values of α.

Text-Enhanced Graph Attention Hashing for Cross-Modal Retrieval.

Deep hashing technology, known for its low-cost storage and rapid retrieval, has become a focal point in cross-modal retrieval research as multimodal data continue to grow. However, existing supervised methods often overlook noisy labels and multiscale features in different modal datasets, leading to higher information entropy in the generated hash codes and features, which reduces retrieval performance. The variation in text annotation information across datasets further increases the information entropy during text feature extraction, resulting in suboptimal outcomes. Consequently, reducing the information entropy in text feature extraction, supplementing text feature information, and enhancing the retrieval efficiency of large-scale media data are critical challenges in cross-modal retrieval research. To tackle these, this paper introduces the Text-Enhanced Graph Attention Hashing for Cross-Modal Retrieval (TEGAH) framework. TEGAH incorporates a deep text feature extraction network and a multiscale label region fusion network to minimize information entropy and optimize feature extraction. Additionally, a Graph-Attention-based modal feature fusion network is designed to efficiently integrate multimodal information, enhance the affinity of the network for different modes, and retain more semantic information. Extensive experiments on three multilabel datasets demonstrate that the TEGAH framework significantly outperforms state-of-the-art cross-modal hashing methods.

Pursuing Truth: Improving Retrievals on Mid-infrared Exo-Earth Spectra with Physically Motivated Water Abundance Profiles and Cloud Models

Atmospheric retrievals are widely used to constrain exoplanet properties from observed spectra. We investigate how the common nonphysical retrieval assumptions of vertically constant molecule abundances and cloud-free atmospheres affect our characterization of an exo-Earth (an Earth-twin orbiting a Sun-like star). Specifically, we use a state-of-the-art retrieval framework to explore how assumptions for the H2O profile and clouds affect retrievals. In the first step, we validate different retrieval models on a low-noise simulated 1D mid-infrared (MIR) spectrum of Earth. Thereafter, we study how these assumptions affect the characterization of Earth with the Large Interferometer For Exoplanets (LIFE). We run retrievals on LIFE mock observations based on real disk-integrated MIR Earth spectra. The performance of different retrieval models is benchmarked against ground truths derived from remote sensing data. We show that assumptions for the H2O abundance and clouds directly affect our characterization. Overall, retrievals that use physically motivated models for the H2O profile and clouds perform better on the empirical Earth data. For observations of Earth with LIFE, they yield accurate estimates for the radius, pressure–temperature structure, and the abundances of CO2, H2O, and O3. Further, at R = 100, a reliable and bias-free detection of the biosignature CH4 becomes feasible. We conclude that the community must use a diverse range of models for temperate exoplanet atmospheres to build an understanding of how different retrieval assumptions can affect the interpretation of exoplanet spectra. This will enable the characterization of distant habitable worlds and the search for life with future space-based instruments.

Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval

Artificial intelligence (AI) hiring tools have revolutionized resume screening, and large language models (LLMs) have the potential to do the same. However, given the biases which are embedded within LLMs, it is unclear whether they can be used in this scenario without disadvantaging groups based on their protected attributes. In this work, we investigate the possibilities of using LLMs in a resume screening setting via a document retrieval framework that simulates job candidate selection. Using that framework, we then perform a resume audit study to determine whether a selection of Massive Text Embedding (MTE) models are biased in resume screening scenarios. We simulate this for nine occupations, using a collection of over 500 publicly available resumes and 500 job descriptions. We find that the MTEs are biased, significantly favoring White-associated names in 85.1% of cases and female-associated names in only 11.1% of cases, with a minority of cases showing no statistically significant differences. Further analyses show that Black males are disadvantaged in up to 100% of cases, replicating real-world patterns of bias in employment settings, and validate three hypotheses of intersectionality. We also find an impact of document length as well as the corpus frequency of names in the selection of resumes. These findings have implications for widely used AI tools that are automating employment, fairness, and tech policy.

Dilated multilevel fused network for virus classification using transmission electron microscopy images

Previous studies have demonstrated significant performance in the field of virus classification; however, they focused on the classification of a small number of virus classes, with a maximum of 16 classes. To address this limitation, this study aims to create a deep learning-based network that outperforms the state-of-the-art (SOTA) models for the classification of 22 different virus classes with the fewest possible trainable parameters. We introduce an automatic identification system for virus classes based on our classification-driven retrieval framework. The proposed dilated multilevel fused network (DMLF-Net) utilizes the multilevel feature fusion concept within a network to exploit more abstract features for microscopic data analysis. A multi-stage training strategy was applied to achieve optimal model convergence without overfitting the training data.We evaluated the performance of the DMLF-Net on three open databases including two virus datasets and one bacteria species dataset. The results demonstrated an accuracy of 89.89%, a weighted harmonic mean of precision and recall (F1-score) of 83.39%, and an area under the curve (AUC) of 92.50% for the 1st virus dataset. For the 2nd virus dataset, the accuracy was 80.70%, the F1-score was 81.20%, and the AUC was 86.20%. For the 3rd bacteria species dataset, the accuracy was 95.93% and the F1-score was 96.24%. DMLF-Net outperforms SOTA methods in terms of classification accuracy while utilizing nearly 5.3 times fewer trainable parameters (25.5 million) compared to the second-best model, visual geometry group (VGG)16 (134.3 million).

SMedIR: secure medical image retrieval framework with ConvNeXt-based indexing and searchable encryption in the cloud

The security and privacy of medical images are crucial due to their sensitive nature and the potential for severe consequences from unauthorized modifications, including data breaches and inaccurate diagnoses. This paper introduces a method for lossless medical image retrieval from encrypted images stored on third-party clouds. The proposed approach employs a symmetric integrity-centric image encryption scheme, leveraging multiple chaotic maps and cryptographic hash techniques, to ensure lossless image reconstruction. Medical images are first encrypted by the image owners and converted into hashcodes encapsulating essential features using a deep hashing technique with the ConvNeXt network as the backbone in parallel. To ensure index privacy, these hashcodes are encrypted in a searchable manner. The encrypted medical images, along with a secure index, are subsequently uploaded to cloud storage. Authorized medical image users can request similar medical images for diagnostic purposes by submitting a query image, from which a search trapdoor is generated and sent to the cloud. The retrieval process involves a secure similar image search over the encrypted indexes, followed by decryption along with integrity verification of the retrieved images. The proposed method has been rigorously tested on three standard medical datasets, demonstrating an improvement of 5-20% in retrieval accuracy compared to standard baselines. Formal security analysis and experimental results indicate that the proposed scheme offers enhanced security and retrieval accuracy, making it an effective solution for the encrypted storage and secure retrieval of medical image data.

Retrieving Continuous Time Event Sequences using Neural Temporal Point Processes with Learnable Hashing

Temporal sequences have become pervasive in various real-world applications such as finance, spatial mobility, health records, etc. Consequently, the volume of data generated in the form of continuous time-event sequence(s) or CTES(s) has increased exponentially in the past few years. Thus, a significant fraction of the ongoing research on CTES datasets involves designing models to address downstream tasks such as next-event prediction, long-term forecasting, sequence classification etc. The recent developments in predictive modeling using marked temporal point processes (MTPP) have enabled an accurate characterization of several real-world applications involving the CTESs. However, due to the complex nature of these CTES datasets, the task of large-scale retrieval of temporal sequences has been overlooked by the past literature. In detail, by CTES retrieval we mean that for an input query sequence, a retrieval system must return a ranked list of relevant sequences from a large corpus. To tackle this, we propose NeuroSeqRet , a first-of-its-kind framework designed specifically for end-to-end CTES retrieval. Specifically, NeuroSeqRet introduces multiple enhancements over standard retrieval frameworks and first applies a trainable unwarping function on the query sequence which makes it comparable with corpus sequences, especially when a relevant query-corpus pair has individually different attributes. Next, it feeds the unwarped query sequence and the corpus sequence into MTPP-guided neural relevance models. We develop four variants of the relevance model for different kinds of applications based on the trade-off between accuracy and efficiency. We also propose an optimization framework to learn binary sequence embeddings from the relevance scores, suitable for the locality-sensitive hashing leading to a significant speedup in returning top-K results for a given query sequence. Our experiments with several datasets show the significant accuracy boost of NeuroSeqRet beyond several baselines, as well as the efficacy of our hashing mechanism.

Advancing continual lifelong learning in neural information retrieval: Definition, dataset, framework, and empirical evaluation

Continual learning refers to the capability of a machine learning model to learn and adapt to new information, without compromising its performance on previously learned tasks. Although several studies have investigated continual learning methods for neural information retrieval (NIR) tasks, a well-defined task definition is still lacking, and it is unclear how typical learning strategies perform in this context. To address this challenge, a systematic task definition of continual NIR is presented, along with a multiple-topic dataset that simulates continuous information retrieval. A comprehensive continual neural information retrieval framework consisting of typical retrieval models and continual learning strategies is then proposed. Empirical evaluations illustrate that the proposed framework can successfully prevent catastrophic forgetting in neural information retrieval and enhance performance on previously learned tasks. The results also indicate that embedding-based retrieval models experience a decline in their continual learning performance as the topic shift distance and dataset volume of new tasks increase. In contrast, pretraining-based models do not show any such correlation. Adopting suitable learning strategies can mitigate the effects of topic shift and data augmentation in continual neural information retrieval.

A Blockchain-Based Supervision Data Security Sharing Framework

Ensuring trust, security, and privacy among all participating parties in the process of sharing supervision data is crucial for engineering quality and safety. However, the current centralized architecture platforms that are commonly used for engineering supervision data have problems such as low data sharing and high centralization. A blockchain-based framework for the secure sharing of engineering supervision data is proposed by utilizing the tamper-proof, decentralized, and traceable characteristics of blockchain. The secure storage of supervision data is achieved by combining it with the IPFS (InterPlanetary File System), reducing the storage pressure of on-chain data. Additionally, a fast data retrieval framework is designed based on the storage characteristics of supervision data. Then, CP-ABE (Ciphertext Policy Attribute Based Encryption) is combined with a data storage framework to ensure the privacy, security, and reliability of supervisory data during the sharing process. Finally, smart contracts are designed under the designed framework to ensure the automatic and trustworthy execution of access control processes. The analysis and evaluation results of the security, encryption and decryption, and cost performance of the proposed blockchain framework show that the encryption and decryption time is completed within 0.1 s, the Gas cost is within the normal consumption range, and the time cost of smart contract invocation does not exceed 5 s, demonstrating good availability and reusability of the method proposed in this article.

An Interactive Multi-Modal Query Answering System with Retrieval-Augmented Large Language Models

Retrieval-augmented Large Language Models (LLMs) have reshaped traditional query-answering systems, offering unparalleled user experiences. However, existing retrieval techniques often struggle to handle multi-modal query contexts. In this paper, we present an interactive M ulti-modal Q uery A nswering (MQA) system, empowered by our newly developed multi-modal retrieval framework and navigation graph index, integrated with cutting-edge LLMs. It comprises five core components: Data Preprocessing, Vector Representation, Index Construction, Query Execution, and Answer Generation, all orchestrated by a dedicated coordinator to ensure smooth data flow from input to answer generation. One notable aspect of MQA is its utilization of contrastive learning to assess the significance of different modalities, facilitating precise measurement of multimodal information similarity. Furthermore, the system achieves efficient retrieval through our advanced navigation graph index, refined using computational pruning techniques. Another highlight of our system is its pluggable processing framework, allowing seamless integration of embedding models, graph indexes, and LLMs. This flexibility provides users diverse options for gaining insights from their multi-modal knowledge base. A preliminary video introduction of MQA is available at https://youtu.be/xvUuo2ZIqWk.

The ESO SupJup Survey

Context. It has been proposed that the distinct formation and evolutionary pathways of exoplanets and brown dwarfs may affect the chemical and isotopic content of their atmospheres. Recent work has indeed shown differences in the 12C/13C isotope ratio, which have provisionally been attributed to the top-down formation of brown dwarfs and the core accretion pathway of super-Jupiters. Aims. The ESO SupJup Survey is aimed at disentangling the formation pathways of isolated brown dwarfs and planetary-mass companions using chemical and isotopic tracers. The survey utilises high-resolution spectroscopy with the recently upgraded CRyogenic high-resolution InfraRed Echelle Spectrograph (CRIRES+) at the Very Large Telescope, covering a total of 49 targets. Here, we present the first results of this survey: an atmospheric characterisation of DENIS J0255-4700, an isolated brown dwarf near the L-T transition. Methods. We analysed its observed CRIRES+ K-band spectrum using an atmospheric retrieval framework in which the radiative transfer code petitRADTRANS was coupled with the PyMultiNest sampling algorithm. Gaussian processes were employed to model inter-pixel correlations. In addition, we adopted an updated parameterisation of the pressure-temperature profile. Results. Abundances of CO, H2O, CH4, and NH3 were retrieved for this fast-rotating L-dwarf. The ExoMol H2O line list provides a significantly better fit than that of HITEMP. A free-chemistry retrieval is strongly favoured over equilibrium chemistry, caused by an under-abundance of CH4. The free-chemistry retrieval constrains a super-solar C/O-ratio of ~0.68 and a solar metallicity. We find tentative evidence (~3σ) for the presence of 13CO, with a constraint on the isotopologue ratio of 12CO/13CO = 184−40+61 and a lower limit of ≳97, which suggests a depletion of 13C compared to the local interstellar medium (12C/13C ~ 68). Conclusions. High-resolution, high signal-to-noise K-band spectra provide an excellent means of constraining the chemistry and isotopic content of sub-stellar objects, which is the main objective of the ESO SupJup Survey.

Exploring the ultra-hot Jupiter WASP-178b

Despite recent progress in the spectroscopic characterization of individual exoplanets, the atmospheres of key ultra-hot Jupiters (UHJs) still lack comprehensive investigations. These include WASP-178b, one of the most irradiated UHJs known to date. We observed the dayside emission signal of this planet with CRIRES+ in the spectral K band. By applying the cross-correlation technique and a Bayesian retrieval framework to the high-resolution spectra, we identified the emission signature of 12CO (S/N = 8.9) and H2O (S/N = 4.9), and a strong atmospheric thermal inversion. A joint retrieval with space-based secondary eclipse measurements from TESS and CHEOPS allowed us to refine our results on the thermal profile and thus to constrain the atmospheric chemistry, yielding a solar to super-solar metallicity (1.4 ± 1.6 dex) and a solar C/O ratio (0.6 ± 0.2). We infer a significant excess of spectral line broadening and identify a slight Doppler-shift between the 12CO and H2O signals. These findings provide strong evidence for a super-rotating atmospheric flow pattern and suggest the possible existence of chemical inhomogeneities across the planetary dayside hemisphere. In addition, the inclusion of photometric data in our retrieval allows us to account for stellar light reflected by the planetary atmosphere, resulting in an upper limit on the geometric albedo (0.23). The successful characterization of WASP-178b’s atmosphere through a joint analysis of CRIRES+, TESS, and CHEOPS observations highlights the potential of combined studies with space- and ground-based instruments and represents a promising avenue for advancing our understanding of exoplanet atmospheres.

Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis

In this work, we present Semantic Gesticulator , a novel framework designed to synthesize realistic gestures accompanying speech with strong semantic correspondence. Semantically meaningful gestures are crucial for effective non-verbal communication, but such gestures often fall within the long tail of the distribution of natural human motion. The sparsity of these movements makes it challenging for deep learning-based systems, trained on moderately sized datasets, to capture the relationship between the movements and the corresponding speech semantics. To address this challenge, we develop a generative retrieval framework based on a large language model. This framework efficiently retrieves suitable semantic gesture candidates from a motion library in response to the input speech. To construct this motion library, we summarize a comprehensive list of commonly used semantic gestures based on findings in linguistics, and we collect a high-quality motion dataset encompassing both body and hand movements. We also design a novel GPT-based model with strong generalization capabilities to audio, capable of generating high-quality gestures that match the rhythm of speech. Furthermore, we propose a semantic alignment mechanism to efficiently align the retrieved semantic gestures with the GPT's output, ensuring the naturalness of the final animation. Our system demonstrates robustness in generating gestures that are rhythmically coherent and semantically explicit, as evidenced by a comprehensive collection of examples. User studies confirm the quality and human-likeness of our results, and show that our system outperforms state-of-the-art systems in terms of semantic appropriateness by a clear margin. We will release the code and dataset for academic research.

A Framework for Extensive Content-Based Image Retrieval System Incorporating Relevance Feedback and Query Suggestion

Over the last decade, there is a huge amount of images in which visual information has become increasingly important. If the images are to be utilized, they must be organized into databases, from there it can be searched based on different criteria. Content-based Image retrieval (CBIR) frameworks have turned into a mainstream subject of exploration; for their ability to retrieve images based on the real visual content rather than by manually connected textual descriptions. In the CBIR framework, analysis and interpretation of image information in large and diverse image databases is evidently complex because there is no prior information on the size or scale of individual structures within the images to be analyzed. In CBIR, retrieval is based on visual image features, which can be extracted automatically from the images with the help of human intervention, namely a technique called relevance feedback. Nonetheless, an efficient way of differentiating the visual content of images is complex to produce. Therefore, rather than a perfect solution CBIR systems must be able to exploit a partial solution to the problem of image understanding. In this paper, there is implementation of a CBIR framework is introduced that not only tries to efficiently capture the user intent based on the feedback but also provide query suggestions that can help its users to pose better queries in order to retrieve desired results in an efficient manner. The proposed technique is straightforward to implement and scopes efficiently to huge datasets. Extensive experiments on diverse real datasets with image similarity measures have revealed the dominance of the proposed method over original algorithms.

Rethink video retrieval representation for video captioning

Video captioning, a challenging task targeting the automatic generation of accurate and comprehensive descriptions based on video content, has witnessed substantial success recently driven by bridging video representations and textual semantics. Inspired by the nature of the video retrieval task, which learns visual features strongly related to text queries, we propose to take advantage of visual representation learning from the video retrieval framework to tackle video captioning tasks and construct adequate multi-grained cross-modal matching while extracting visual features. However, a simple direct application of recent video retrieval models fails to capture sufficient temporal details and the rich visual features of local patch tokens of video frames lack semantic information essential for captioning tasks. These deficiencies are primarily due to these models lack fine-grained interactions between video frames and offer only weak textual supervision over frame patch tokens. To increase the attention on temporal details, we propose a learnable token shift module, which flexibly captures subtle movements in local regions across the temporal sequence. Furthermore, we devise a Refineformer, which learns to integrate local video patch tokens strongly related to desired captions via a cross-attention mechanism. Extensive experiments on MSVD, MSR-VTT and VATEX demonstrate the favorable performance of our method. Code will be available at https://github.com/tiesanguaixia/IVRC.