Improve Retrieval Accuracy Research Articles

Improving Scene Text Retrieval via Stylized Middle Modality

Scene text retrieval addresses the challenge of localizing and searching for all text instances within scene images based on a query text. This cross-modal task has significant applications in various domains, such as intelligent transportation systems and social media analysis. In practice, ensuring consistency of the same content between two modalities is crucial in improving retrieval accuracy. This paper addresses the issue by introducing a stylized middle modality, which fuses the graphical query text with the style of the extracted text proposal. To this end, we propose a stylized middle modality learning (SM \({}^{2}\) L) framework. The proposed stylized middle modality enables the network to jointly enforce constraints on visual feature coherence and text semantic feature consistency in the optimization phase, thereby minimizing the modality gap in the retrieval space. This brings in two major advantages: 1) SM \({}^{2}\) L will pave the way to seamlessly benefit the scene text retrieval and 2) the proposed learning paradigm enables the machine to avoid adding redundant computing resources in the inference phase. Substantial experiments demonstrate that the proposed method outperforms the state-of-the-art retrieval performance considerably.

Development of soil water content retrieving method for irrigation agriculture areas using the red-edge band of Gaofen-6 satellite

Soil water content (SWC) is an essential index that reflects water conditions in the soil surface during drought, and changes in SWC are critical in guiding agricultural irrigation management. Remote sensing is a crucial means of monitoring SWC, and the red and near-infrared bands of optical remote sensing data are often used to monitor SWC through remote sensing retrieval. With the development of remote sensing technology, red-edge band has also been used to retrieve SWC. It exhibits higher sensitivity to changes in vegetation and SWC, relatively more precise differentiation between vegetation and soil, and more accurate SWC detection. However, there has been relatively little research regarding the use of the red-edge band for retrieving SWC. In this study, we developed a novel method, based on the principle of spectral space and the calculation of modified perpendicular drought index, to calculate the soil water index using the red-edge band of the Gaofen-6 (GF-6) satellite by comparing the retrieval accuracy of GF-6 using different combinations of red, near-infrared, and red-edge bands. Based on the measured SWC in the Shijin Irrigation District of Hebei Province, an SWC retrieval model was established using linear regression. Moreover, the new method was used to retrieve SWC from GF-6 and Sentinel-2 with the same central wavelength, and the reliability of the new method was compared and analyzed. The results revealed that the combination of red band and red-edge band 2 with a central wavelength of 0.74 μm substantial. improved the retrieval accuracy of SWC. Our study provides a new method for retrieving SWC using optical remote sensing, which can effectively improve retrieval accuracy and provide data support for agricultural irrigation monitoring and water management.

Cross media knowledge information retrieval model based on D-S evidence theory

Cross media knowledge information retrieval provides strong support for information processing and utilization in the information society, but there are problems such as heterogeneity in cross media knowledge information. Therefore, a cross media knowledge information retrieval model using D-S evidence theory is proposed, which involves using approximate calculation methods to improve this theory for information fusion, reducing computational complexity, and using deep networks for fine-grained information retrieval to improve retrieval accuracy. The results showed that the improved theory enhanced computational efficiency by about 27.23 %. The memory usage was <60 %, and the average accuracy of information fusion reached 93.14 %. It also exhibited high recall and low false alarm rates. The cross media knowledge information retrieval model proposed in the study achieved accuracy of 92.64 %, 96.49 %, and 97.46 % on the three datasets used in the experiment, respectively. The study provides an effective, computationally efficient, and highly accurate model for cross media knowledge information retrieval, which is expected to promote research and application in this field. The combination of improved D-S evidence theory and deep networks provides a powerful approach to solving the problem of cross media heterogeneous information retrieval, which has a positive promoting effect on the processing and utilization of information in the information society.

Leveraging Large Language Models for Information Retrieval from NEPA Documents

This paper explores the application of large language models (LLMs) to efficiently and accurately extract relevant information from National Environmental Policy Act (NEPA) documents, specifically focusing on environmental impact statements (EIS). NEPA mandates federal agencies to evaluate the environmental effects of their proposed actions, and EIS documents are essential for this process. However, these documents are often lengthy and complex, making manual information extraction time-consuming and error-prone. We address this challenge by leveraging advanced natural language processing techniques and the newly introduced NEPAQuAD1.0 dataset, which contains 1,450 question-answer pairs generated under human supervision. Our approach involves fine-tuning the Meta-Llama-3.1-8B-Instruct model on this dataset. The results demonstrate significant improvements in retrieval accuracy and efficiency compared to baseline models, highlighting the potential of LLMs to streamline the environmental review process and provide valuable insights for environmental policy analysis. This work contributes to the broader field of natural language processing by offering a robust method for handling complex, domain-specific information retrieval tasks.

Deep Learning with Pretrained Framework Unleashes the Power of Satellite-Based Global Fine-Mode Aerosol Retrieval.

Fine-mode aerosol optical depth (fAOD) is a vital proxy for the concentration of anthropogenic aerosols in the atmosphere. Currently, the limited data length and high uncertainty of the satellite-based data diminish the applicability of fAOD for climate research. Here, we propose a novel pretrained deep learning framework that can extract information underlying each satellite pixel and use it to create new latent features that can be employed for improving retrieval accuracy in regions without in situ data. With the proposed model, we developed a new global fAOD (at 0.5 μm) data from 2001 to 2020, resulting in a 10% improvement in the overall correlation coefficient (R) during site-based independent validation and a 15% enhancement in non-AERONET site areas validation. Over the past two decades, there has been a noticeable downward trend in global fAOD (-1.39 × 10-3/year). Compared to the general deep-learning model, our method reduces the global trend's previously overestimated magnitude by 7% per year. China has experienced the most significant decline (-5.07 × 10-3/year), which is 3 times greater than the global trend. Conversely, India has shown a significant increase (7.86 × 10-4/year). This study bridges the gap between sparse in situ observations and abundant satellite measurements, thereby improving predictive models for global patterns of fAOD and other climate factors.

Advanced Embedding Techniques in Multimodal Retrieval Augmented Generation A Comprehensive Study on Cross Modal AI Applications

Our study presents significant advancements in the handling of multimodal data through an extended Retrieval-Augmented Generation (RAG) model. By integrating advanced embedding techniques, efficient retrieval mechanisms, and robust generative capabilities, our model demonstrates notable improvements in retrieval accuracy, real-time efficiency, generative quality, and scalability. The retrieval accuracy of our model reached 85%, showing a 10% improvement over existing benchmarks. Furthermore, the retrieval time was reduced by 40%, enhancing real-time application performance. The model's generative quality was also significantly improved, with BLEU and ROUGE scores increasing by 15% and 12%, respectively. These results validate the effectiveness of our approach and its applicability to various AI applications, including information retrieval, recommendation systems, and content creation. Future research directions include the integration of additional modalities and further optimization of retrieval mechanisms to broaden the applicability of our model.

Video and audio are images: A cross-modal mixer for original data on video–audio retrieval

Cross-modal retrieval has become popular in recent years, particularly with the rise of multimedia. Generally, the information from each modality exhibits distinct representations and semantic information, which makes feature tends to be in separate latent spaces encoded with dual-tower architecture and makes it challenging to establish semantic relationships between modalities, resulting in poor retrieval performance. To address this issue, we propose a novel framework for cross-modal retrieval consisting of a cross-modal mixer, a masked autoencoder for pre-training, and a cross-modal retriever for downstream tasks. Specifically, we first adopt cross-modal mixer and mask modeling to fuse the original modality and eliminate redundancy. Then, an encoder–decoder architecture is applied to achieve a fuse-then-separate task in the pre-training phase. We feed masked fused representations into the encoder and reconstruct them with the decoder, ultimately separating the original data of two modalities. We use the pre-trained encoder in downstream tasks to build the cross-modal retrieval method. Extensive experiments on 2 real-world datasets show that our approach outperforms previous state-of-the-art methods in video–audio matching tasks, improving retrieval accuracy by up to 2×. Furthermore, we prove our model performance by transferring it to other downstream tasks as a universal model.

Visual Question Answering based Educational Tool for Medical Students using Cross-ViT

Abstract: This paper introduces an advanced approach to medical visual question answering (VQA) using the Cross-ViT architecture. The model employs a dual-branch method to extract multi-scale feature representations from images, utilizing cross-attention mechanisms to enhance visual features. By integrating Stacked Attention Networks (SAN) and leveraging semantic extraction from LSTM for textual data, the model shows significant performance improvements. Experiments on various biomedical VQA tasks demonstrate notable improvements in retrieval accuracy and image-text correlation. The study highlights the potential of medical VQA systems to transform healthcare delivery, improve diagnostic accuracy, and facilitate patient engagement and education, with promising future applications in telemedicine, surgery assistance, and integration with electronic health records

Retrieval of Atmospheric Temperature Profiles from FY-4A/GIIRS Hyperspectral Data Based on TPE-MLP: Analysis of Retrieval Accuracy and Influencing Factors

In this study, a novel method for retrieving atmospheric temperature profiles with tree-structured Parzen estimator (TPE) and multilayer perceptron (MLP) algorithms was proposed, using FY-4A/GIIRS (Geosynchronous Interferometric Infrared Sounder) and ERA5 data. Firstly, by adding solar altitude angle, satellite zenith angle, 2m temperature, and surface temperature to the input layer of MLP, there is an improvement in retrieval accuracy. Secondly, TPE is effective in optimizing the hyper-parameters of MLP, and a set of optimized hyper-parameters is obtained through iterative optimization. Thirdly, comparing the retrieved temperature profiles with ERA5 data, we found that retrieval accuracy is influenced by detector, signal-to-noise ratio, terrain, solar altitude angle, satellite zenith angle, and the horizontal temperature gradient. The mean biases of the two adjacent detectors show significant differences, and the retrieval accuracy of the center detectors is greater than that of the north and south sides. The retrieval accuracy is relatively poor in areas with high terrain and large satellite zenith angle. There is a monthly variation in the retrieval accuracy due to the horizontal temperature gradient and signal-to-noise ratio and a significant diurnal variation due to solar altitude angle and signal-to-noise ratio. Compared to in situ sounding data, the mean biases vary from −0.56 K to 0.60 K, and the standard deviations vary from 1.26 K to 2.17 K. The analysis of factors influencing retrieval accuracy provides important insights into improving the ability to retrieve atmospheric temperatures from geostationary hyperspectral IR sounder observations for near real-time (NRT) applications.

Deep Learning Model for Retrieving Color Logo Images in Content Based Image Retrieval

Content-Based Image retrieval (CBIR) has gained a magnificent deal of consideration because of the digital image data's epidemic growth. The advancement of deep learning has enabled Convolutional Neural Networks to become an influential technique for extraction of discriminative image features. In recent years, convolutional neural networks (CNNs) have proven extremely effective at extracting unique information from images. In contrast to text-based image retrieval, CBIR gathers comparable images based primarily on their visual content. The use of deep learning, especially CNNs, for feature extraction and image processing has been shown to perform better than other techniques. In the proposed study, we investigate CNNs for CBIR focusing on how well they extract discriminative visual features and facilitate accurate image retrieval. Also Principal Component Analysis and Linear Discriminant Analysis are combined for optimization of features resulting in boosting the retrieval results. Using hierarchical representations learned by CNNs, we aim to improve retrieval accuracy and efficiency. In comparison with conventional retrieval techniques, our proposed CBIR system shows superior performance on a benchmark dataset.

Web Page Ranking Based on Text Content and Link Information Using Data Mining Techniques

Thanks to the rapid expansion of the Internet, anyone can now access a vast array of information online. However, as the volume of web content continues to grow exponentially, search engines face challenges in delivering relevant results. Early search engines primarily relied on the words or phrases found within web pages to index and rank them. While this approach had its merits, it often resulted in irrelevant or inaccurate results. To address this issue, more advanced search engines began incorporating the hyperlink structures of web pages to help determine their relevance. While this method improved retrieval accuracy to some extent, it still had limitations, as it did not consider the actual content of web pages. The objective of the work is to enhance Web Information Retrieval methods by leveraging three key components: text content analysis, link analysis, and log file analysis. By integrating insights from these multiple data sources, the goal is to achieve a more accurate and effective ranking of relevant web pages in the retrieved document set, ultimately enhancing the user experience and delivering more precise search results the proposed system was tested with both multi-word and single-word queries, and the results were evaluated using metrics such as relative recall, precision, and F-measure. When compared to Google’s PageRank algorithm, the proposed system demonstrated superior performance, achieving an 81% mean average precision, 56% average relative recall, and a 66% F-measure.

Commonsense Knowledge-Driven Joint Reasoning Approach for Object Retrieval in Virtual Reality

National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), China Retrieving out-of-reach objects is a crucial task in virtual reality (VR). One of the most commonly used approaches for this task is the gesture-based approach, which allows for bare-hand, eyes-free, and direct retrieval. However, previous work has primarily focused on assigned gesture design, neglecting the context. This can make it challenging to accurately retrieve an object from a large number of objects due to the one-to-one mapping metaphor, limitations of finger poses, and memory burdens. There is a general consensus that objects and contexts are related, which suggests that the object expected to be retrieved is related to the context, including the scene and the objects with which users interact. As such, we propose a commonsense knowledge-driven joint reasoning approach for object retrieval, where human grasping gestures and context are modeled using an And-Or graph (AOG). This approach enables users to accurately retrieve objects from a large number of candidate objects by using natural grasping gestures based on their experience of grasping physical objects. Experimental results demonstrate that our proposed approach improves retrieval accuracy. We also propose an object retrieval system based on the proposed approach. Two user studies show that our system enables efficient object retrieval in virtual environments (VEs).

Towards Human–Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework

The multimodality nature of web data has necessitated complex multimodal information retrieval for a wide range of web applications. Deep neural networks (DNNs) have been widely employed to extract semantic features from raw samples to improve retrieval accuracy. In addition, hashing is widely used to improve computational and storage efficiency. As such, deep hashing frameworks have been applied for multimodal retrieval tasks. However, there is still a great recognitive gap between primate brain structure-inspired DNNs and humans. On computer vision tasks, well-crafted DNN models can be easily defeated by invisible small attacks, and this phenomenon indicates a large recognition gap between DNN models and humans. Recently, adversarial defense methods have been shown to improve the human&#x2013;machine recognition alignment in several classification tasks. However, the robustness problem on the retrieval tasks, especially on the deep hashing-based multimodal retrieval models, is still not well studied. Therefore, in this article, we present an adversarially robust training mechanism to improve model robustness for the purpose of human&#x2013;machine recognition alignment on retrieval tasks. Through extensive experimental results on several social multimodal retrieval benchmarks, we show that the robust training hashing framework proposed can mitigate the recognition gap on retrieval tasks. Our study highlights the necessity of robustness enhancement on deep hashing models.

SE‐Swin: An improved Swin‐Transfomer network of self‐ensemble feature extraction framework for image retrieval

AbstractThe Swin‐Transformer is a variant of the Vision Transformer, which constructs a hierarchical Transformer that computes representations with shifted windows and window multi‐head self‐attention. This method can handle the scale invariance problem and performs well in many computer vision tasks. In image retrieval, high‐quality feature descriptors are necessary to improve retrieval accuracy. This paper proposes a self‐ensemble Swin‐Transformer network structure to fuse the features of different layers of the Swin‐Transformer network, eliminating noise points present in a single layer, and improving the retrieval effect. Two experiments were conducted, one on the In‐shop Clothes Retrieval dataset and another on the Stanford Online Product dataset. The experiments showed that the proposed method significantly increased the retrieval effect of features extracted using Vision Transformer, surpassing previous state‐of‐the‐art image retrieval methods. In the second experiment, the feature map of the trained model was visualized, revealing that the improved network significantly reduces focus on some noise points and enhances focus on image features compared to the original network.

An improved image processing algorithm for visual characteristics in graphic design.

Drawing the clothing plan is an essential part of the clothing industry. However, the irregular shape of clothing, strong deformability and sensitivity to light make the fast and accurate realization of clothing image retrieval a very challenging problem. The successful application of the Transformer in image recognition shows the application potential of the Transformer in the image field. This article proposes an efficient and improved clothing plan based on ResNet-50. Firstly, in the feature extraction section, the ResNet-50 network structure embedded in the Transformer module is used to improve the network's receptive field range and feature extraction ability. Secondly, dense jump connections are added to the ResNet-50 upsampling process, making full use of feature extraction information at each stage, further improving the quality of the generated image. The network consists of three steps: the sketch stage, which aims to predict the color distribution of clothing and obtain watercolor images without gradients and shadows. The second is the thinning stage, which refines the watercolor image into a clothing image with light and shadow effect; The third is the optimization stage, which combines the outputs of the first two stages to optimize the generation quality further. The experimental results show that the improved network's IS and first input delay (FID) scores are 4.592 and 1.506, respectively. High-quality clothing images can be generated only by inputting line drawings and a few color points. Compared with the existing methods, the image generated by this network has excellent advantages in realism and accuracy. This method can combine various feature information of images, improve retrieval accuracy, has strong robustness and practicability, and can provide a reference for the daily work of fashion designers.

Efficient text-image semantic search: A multi-modal vision-language approach for fashion retrieval

In this paper, we address the problem of multi-modal retrieval of fashion products. State-of-the-art (SOTA) works proposed in literature use vision-and-language transformers to assign similarity scores to joint text-image pairs, then used for sorting the results during a retrieval phase. However, this approach is inefficient since it requires coupling a query with every record in the dataset and computing a forward pass for each sample at runtime, precluding scalability to large-scale datasets. We thus propose a solution that overcomes the above limitation by combining transformers and deep metric learning to create a latent space where texts and images are separately embedded, and their spatial proximity translates into semantic similarity. Our architecture does not use convolutional neural networks to process images, allowing us to test different levels of image-processing details and metric learning losses. We vastly improve retrieval accuracy results on the FashionGen benchmark (+18.71% and +9.22% Rank@1 on Image-to-Text and Text-to-Image, respectively) while being up to 512x faster. Finally, we analyze the speed-up obtainable by different approximate nearest neighbor retrieval strategies—an optimization precluded to current SOTA contributions. We release our solution as a web application available at https://disi-unibo-nlp.github.io/projects/fashion_retrieval/.

Temperature and Relative Humidity Profile Retrieval from Fengyun-3D/VASS in the Arctic Region Using Neural Networks

In this study, a new technique is proposed to retrieve temperature and relative humidity profiles under clear sky conditions in the Arctic region based on the artificial neural network (ANN) algorithm using Fengyun-3D (FY-3D) vertical atmospheric sounder suit (VASS: HIRAS, MWTS-II, and MWHS-II) observations. This technology combines infrared (IR) and microwave (MW) observations to improve retrieval accuracy in the middle and low troposphere by reducing the sensitivity of the neural networks (NNs) to cloud coverage. The approach was compared against other methods available in the literature on retrieving profiles only from FY-3D/HIRAS data. Furthermore, its retrieval performance was tested by comparing the NNs’ prediction accuracy versus the corresponding FY-3D/VASS and Aqua/AIRS L2 products. The results showed that: (1) NNs retrieval accuracy is higher during the warm season and over the ocean; (2) the retrieval accuracy of NNs has been significantly improved compared with satellite L2 products; (3) referring to radiosonde observations, the retrieval accuracy of NNs below 600 hPa is effectively improved by adding the information of the MW channel, especially on land where cloud clearing is more difficult. The root mean square error (RMSE) of temperature and relative humidity in the cold season were reduced by 0.3 K and 2%, respectively. The advanced NNs proposed herein offer a more stable retrieval performance compared with NNs built only by FY-3D/HIRAS data. The study results indicated the potential value in time and space domain of the NN algorithm in retrieving temperature and relative humidity profiles of the Arctic region from FY-3D/VASS observations under clear-sky conditions. All in all, this work enhances our knowledge towards improving operational use of FY-3D satellite data in the Arctic region.

Using cased based reasoning for automated safety risk management in construction industry

The purpose of this study is to develop a CBR-based platform that integrates all steps of safety risk management (SRM) for the realization of automation in construction safety risk management. To realize this purpose, accident data from 2015 to 2020 in China has been collected and analyzed. Based on the collected data, this study adopts system thinking and stakeholder theory to establish an accident attribute system for accurate and comprehensive case representation, which was usually ignored in the previous study. Following case representation, the development process of case-based reasoning (CBR) is introduced in detail. The core of CBR—case retrieval is designed with the k-Nearest Neighbor (k-NN) algorithm and rough set theory to improve retrieval accuracy. In terms of case reuse and revise, a hybrid risk response model based on historical experience and risk factor control is proposed. Finally, the practical application of this study is illustrated with a specific construction project. The findings of this study confirm the feasibility of machine learning in improving safety performance at construction sites. Construction accidents can be effectively prevented by identifying the potential safety risks, and employing on-site management.

DCRL-KG: Distributed Multi-Modal Knowledge Graph Retrieval Platform Based on Collaborative Representation Learning

The knowledge graph with relational abundant information has been widely used as the basic data support for the retrieval platforms. Image and text descriptions added to the knowledge graph enrich the node information, which accounts for the advantage of the multi-modal knowledge graph. In the field of cross-modal retrieval platforms, multi-modal knowledge graphs can help to improve retrieval accuracy and efficiency because of the abundant relational information provided by knowledge graphs. The representation learning method is significant to the application of multi-modal knowledge graphs. This paper proposes a distributed collaborative vector retrieval platform (DCRL-KG) using the multimodal knowledge graph VisualSem as the foundation to achieve efficient and high-precision multimodal data retrieval. Firstly, use distributed technology to classify and store the data in the knowledge graph to improve retrieval efficiency. Secondly, this paper uses BabelNet to expand the knowledge graph through multiple filtering processes and increase the diversification of information. Finally, this paper builds a variety of retrieval models to achieve the fusion of retrieval results through linear combination methods to achieve high-precision language retrieval and image retrieval. The paper uses sentence retrieval and image retrieval experiments to prove that the platform can optimize the storage structure of the multi-modal knowledge graph and have good performance in multi-modal space.

Retrieval of Subsurface Soil Moisture and Vegetation Water Content From Multifrequency SoOp Reflectometry: Sensitivity Analysis

Signals of opportunity reflectometry (SoOp-R), the re-utilization of non-cooperative satellite transmissions for communication and navigation, is a promising approach to remote sensing of root-zone soil moisture (RZSM). Satellite transmissions in the frequency ranges of 137–138, 240–270, and 360–380 MHz are of interest due to the increased penetration depth. These can be combined with Global Navigation Satellite System Reflectometry (GNSS-R) in L-band (1575.42 MHz) to estimate the subsurface SM profile. The objective is to define requirements (e.g. frequency and polarization combinations, observation error, and temporal coincidence of multi-source observations) for satellite-based remote sensing of RZSM. Our approach is to use synthetic observations generated from multi-year time series of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-situ</i> SM measurements from seven United States Climate Reference Network (USCRN) sites and dynamic vegetation structure based on a simple scaling method. A multi-frequency/polarimetric retrieval algorithm is developed and applied to these synthetic observations and used to predict retrieval errors for a range of changes in system parameters. We found that the use of both high and low frequencies improves retrieval accuracy by limiting uncertainties from vegetation and surface SM and providing sensitivity to deeper layers. Moreover, the retrieval errors were found to increase linearly with the reflectivity error and inter-frequency time delays. A bivariate model derived from this linear relationship will be useful for developing requirements on reflectivity precision based upon science requirements for SM/VWC retrievals. Although orbits of specific transmitter constellations were used to generate realistic distributions of incidence angle combinations, the method and results could be applied more generally.