Regions Of Interest In Images Research Articles

Abstract 6245: Missed opportunity at the subcellular level: Enhancing the utility of cellular imaging modality in spatial transcriptomic profiling of tumor tissues

Abstract Mini-bulk or single-spot spatial transcriptomic technologies, such as GeoMX Digital Spatial Profiler (DSP), have revolutionized our ability to probe spatial heterogeneity and examine transcripts at a subcellular level. In cancer research, this level of granularity is crucial for uncovering the distinct gene expression signatures within specific compartments or structures of interest such as tumor, immune, stroma, and tertiary lymphoid structures. However, in most GeoMX DSP studies, the spatial information obtained from multiplex immunofluorescence imaging has been primarily used for identifying the regions of interest (ROI), rather than as an integral part of the downstream transcriptomic data interpretation. To fix this missed opportunity, we developed a machine learning based analytical framework to fully leverage the rich spatial context provided by the in situ imaging modality. The framework has two main functionalities. (1) ROI cropping and cell segmentation: our framework enables the automated cropping of ROI images, followed by a cell segmentation process utilizing a multi-color-optimized deep learning model pre-trained based on the TissueNet. (2) Cell typing and integrative spatial analysis: the pipeline extracts cell-level morphological features (e.g. cell size and circularity), and AI-driven characteristics. These features are subsequently employed for cell typing and the quantification of cell mixture. The framework is equipped to perform a series of advanced spatial analyses, such as spatial metrics calculation and consensus clustering analysis with the matched gene expression data. To validate the framework, we conducted GeoMX DSP analyses on samples obtained from bladder cancer and upper tract urothelial carcinoma, encompassing 56 ROIs annotated by pathologists. Our results demonstrated high accuracy, with the calculated tumor-immune proportions aligning precisely with the original annotations. We further compared the cell deconvolution results of our framework with those obtained using SpatialDecon. Consistency was noted across both methods, with a high correlation reaching 0.90. However, we observed that SpatialDecon tends to underestimate tumor purity in tumor core regions, with discrepancies as large as 0.30, and overestimating tumor purity in stromal regions. Additionally, analysis of ROIs in tumor margins revealed that SpatialDecon consistently overestimates the overall proportions of immune cells. In conclusion, our results underscore the importance of integrating in situ imaging with subcellular-level spatial transcriptomics for a more accurate and reliable analysis of tumor tissues. Our approach provides critical insights into the tumor microenvironment and cellular interactions, with significant implications for both research and clinical applications in oncology. Citation Format: Xiaofei Song, Xiaoqing Yu, Carlos M. Moran-Segura, G Daniel Grass, Roger Li, Xuefeng Wang. Missed opportunity at the subcellular level: Enhancing the utility of cellular imaging modality in spatial transcriptomic profiling of tumor tissues [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 6245.

Application Exploration of Medical Image-aided Diagnosis of Breast Tumour Based on Deep Learning.

Nowadays, people attach increasing importance to accurate and timely disease diagnosis and personalized treatment. Because of the uncertainty and latency of the pathogenesis, it is difficult to detect breast tumour early. With higher resolution, magnetic resonance imaging (MRI) has become an important method for early detection of cancer in recent years. At present, DL technology can automatically study imaging features of different depths. This work aimed to use DL to study medical image-assisted diagnosis. The image data were collected from the patients. ROI (region of interest) containing the complete tumor area in the medical image was generated. The ROI image was extracted, and the extracted feature data were expanded. By constructing a three-dimensional (3D) CNN model, the evaluation indicators of breast tumour diagnosis results have been proposed. In the experiment part, 3D CNN model and other models have been used to diagnose the medical image of breast tumour. The 3D CNN model exhibited good ROI region extraction effect and breast tumor image diagnosis effect, and the average diagnostic accuracy of breast tumor image diagnosis was 0.736, which has been found to be much higher than other models and could be applied to breast tumor medical image-aided diagnosis. The 3D CNN model has been trained by combining the two-dimensional CNN training mode, and the evaluation index of diagnostic results has been established. The experimental part verified the medical image diagnosis effect of the 3D CNN model. The model had exhibited a high ROI region extraction effect and breast tumor image diagnosis effect.

A Realistic Hand Image Composition Method for Palmprint ROI Embedding Attack

Palmprint recognition (PPR) has recently garnered attention due to its robustness and accuracy. Many PPR methods rely on preprocessing the region of interest (ROI). However, the emergence of ROI attacks capable of generating synthetic ROI images poses a significant threat to PPR systems. Despite this, ROI attacks are less practical since PPR systems typically take hand images as input rather than just the ROI. Therefore, there is a pressing need for a method that specifically targets the system by composing hand images. The intuitive approach involves embedding an ROI into a hand image, a comparatively simpler process requiring less data than generating entirely synthetic images. However, embedding faces challenges, as the composited hand image must maintain a consistent color and texture. To overcome these challenges, we propose a training-free, end-to-end hand image composition method incorporating ROI harmonization and palm blending. The ROI harmonization process iteratively adjusts the ROI to seamlessly integrate with the hand using a modified style transfer method. Simultaneously, palm blending employs a pretrained inpainting model to composite a hand image with a continuous transition. Our results demonstrate that the proposed method achieves a high attack performance on the IITD and Tongji datasets, with the composited hand images exhibiting realistic visual quality.

Low-cost single-shot complex optical field imaging with a simplified aperture

We report a new computational imaging methodology for complex optical field based on a single-frame diffraction intensity pattern acquisition with a simplified aperture and an extremely low-cost industrial machine vision camera. The current design of the apertures for the recovery of complex optical field was suffering from specific coding associated with the use of spatial light modulator or digital micromirror device, where it is difficult to make the optical implementations compact and cost-effective. In this work, we demonstrate that a simplified aperture without any pattern-like coding is efficient to establish the physical constraint for computational reconstruction. The aperture attached to the object defines the imaging region of interest (ROI) with coherent trans-illumination, and only one snapshot of the diffraction intensity pattern is required for recovering the ROI's complex field. Proof-of-concept numerical simulations and optical experiments have been carried out to validate the feasibility, noise robustness, resolution, and potential biological imaging applications of the proposed computational method.

LEACS: a learnable and efficient active contour model with space-frequency pooling for medical image segmentation.

Diseases can be diagnosed and monitored by extracting regions of interest (ROIs) from medical images. However, accurate and efficient delineation and segmentation of ROIs in medical images remain challenging due to unrefined boundaries, inhomogeneous intensity and limited image acquisition. To overcome these problems, we propose an end-to-end learnable and efficient active contour segmentation model, which integrates a global convex segmentation (GCS) module into a light-weighted encoder-decoder convolutional segmentation network with a multiscale attention module (ED-MSA). The GCS automatically obtains the initialization and corresponding parameters of the curve deformation according to the prediction map generated by the ED-MSA, while provides the refined object boundary prediction for ED-MSA optimization. To provide precise and reliable initial contour for the GCS, we design the space-frequency pooling operation layers in the encoder stage of ED-MSA, which can effectively reduce the number of iterations of the GCS. Beside, we construct ED-MSA using the depth-wise separable convolutional residual module to mitigate the overfitting of the model. The effectiveness of our method is validated on four challenging medical image datasets. Code is here:https://github.com/Yang-fashion/ED-MSA_GCS.

Object Detection and Regression Based Visible Spectrophotometric Analysis: A Demonstration Using Methylene Blue Solution

This study investigates the estimation of the concentration of methylene blue solutions to understand if visible spectrophotometry could be performed using a smartphone and machine learning. The presented procedure consists of taking photos, detecting test tubes and sampling region of interest (ROI) with YOLOv5, finding the hue, saturation, value (HSV) code of the dominant color in the ROI, and regression. 257 photos were taken for the procedure. The YOLOv5 object detection architecture was trained on 928 images and the highest mAP@05 values were detected as 0.915 in 300 epochs. For automatic ROI sampling, the YOLOv5 detect.py file was edited. The trained YOLOv5 detected 254 out of 257 test tubes and extracted ROIs. The HSV code of the dominant color in the exported ROI images was determined and stored in a csv file together with the concentration values. Subsequently, 25 different regression algorithms were applied to the generated data set. The extra trees regressor was the most generalizing model with 99.5% training and 99.4% validation R2 values. A hyperparameter tuning process was performed on the extra trees regressor and a mixed model was created using the best 3 regression algorithms to improve the R2 value. Finally, all three models were tested on unseen data and the lowest MSE value was found in the untuned extra trees regressor and blended model with values of 0.10564 and 0.16586, respectively. These results prove that visible spectrophotometric analysis can be performed using the presented procedure and that a mobile application can be developed for this purpose.

Color image ROI encryption algorithm based on a novel 4D hyperchaotic system

The significance of safeguarding the security of image information has escalated significantly, owing to the exponential proliferation of digital images containing sensitive information being disseminated on the Internet. In this paper, we first propose a novel 4D hyperchaotic system and design a new image encryption algorithm in conjunction with the hyperchaotic system. The algorithm uses a split random swap permutation method to permute the image and combines the S-box to diffuse the image. To improve the diffusivity of this encryption algorithm, a cross-random diffusion method is designed to diffuse the image again. Then, we propose a region of interest (ROI) encryption scheme for images. This scheme can automatically identify irregular privacy targets in images and encrypt them. To ensure the security of the region of interest location information during transmission, the scheme compresses the location information of the privacy target using a run-length encoding technique and then embeds the compressed data into the ciphertext image using reversible steganography based on histogram shift. The experimental results and security analysis unequivocally demonstrate that the image encryption algorithm proposed in this paper exhibits robust resistance against a wide array of attacks, thereby ensuring a high level of security. Additionally, the devised image ROI encryption scheme effectively safeguards diverse privacy targets.

Analysis of the Slanted-Edge Measurement Method for the Modulation Transfer Function of Remote Sensing Cameras

The modulation transfer function (MTF) serves as a crucial technical index for assessing the imaging quality of remote sensing cameras, which is integral throughout their entire operational cycle. Currently, the MTF evaluation of remote sensing cameras primarily relies on the slanted-edge method. The factors influencing the slanted-edge method’s effectiveness are broadly classified into two categories: algorithmic factors and image factors. This paper innovatively comprehensively analyzes the influencing factors of the slanted-edge method and proposes an improved slanted-edge method to calculate the MTF testing method of remote sensing cameras, which is applied to the MTF testing of remote sensing cameras. Since the traditional algorithm can only be applied in the small angle situation, this paper proposes a new method of slanted-edge method test calculation based on the optimal oversampling rate (OSR) adaptive model of the slanted edge and uses simulation experiments to verify the reliability of the algorithm model through the deviation of the slanted-edge angle calculation and MTF measurement, and the results show that the algorithm improves the accuracy of the MTF measurement compared with the ISO-cos and OMINI-sine methods. Then, the effects of the slanted-edge angle, image region of interest (ROI), as well as image contrast and signal-to-noise ratio (SNR) on the accuracy of the MTF calculation by the slanted-edge method were quantitatively analyzed as the constraints of the slanted-edge method test. Based on the laboratory target experiment, the algorithm flow and various influencing factors obtained in the simulation stage are verified, and the experimental results are more consistent with the various test results obtained in the simulation stage. Consequently, the slanted-edge method introduced in this paper is applicable for future remote sensing camera MTF testing. This approach offers a valuable reference for on-orbit focusing, satellite operational condition monitoring, lifespan estimation, and image restoration.

Region of interest (ROI) extraction and crack detection for UAV-based bridge inspection using point cloud segmentation and 3D-to-2D projection

For digital-image-based bridge inspection tasks, images captured by camera-carrying unmanned aircraft vehicles (UAVs) usually contain both the region of interest (ROI) and the background. However, accurately detecting cracks in concrete surface images containing background information is challenging. To improve UAV-based bridge inspection, an image extraction and crack detection methodology is presented in this paper. First, a deep-learning-based semantic segmentation network RandLA-BridgeNet for large-scale bridge point clouds, which can facilitate 3D ROI extraction, is trained and tested. Second, an image ROI extraction method based on 3D-to-2D projection is presented to generate images containing only the ROI. Finally, a data-driven deep learning convolutional neural network (CNN) called the grid-based classification and box-based detection fusion model (GCBD) is utilized to identify cracks in the processed images. An experiment is conducted on highway bridge images to validate the presented methodology. The overall semantic segmentation and image ROI extraction accuracies are 97.0% and 98.9%, respectively. After ROI extraction, 47.9% of the grid cells, which represent background misrecognition, are filtered, greatly improving the crack identification accuracy.

Computed tomography and guidelines-based human-machine fusion model for predicting resectability of the pancreatic cancer.

The study aims to develop a hybrid machine learning model for predicting resectability of the pancreatic cancer, which is based on computed tomography (CT) and National Comprehensive Cancer Network (NCCN) guidelines. We retrospectively studied 349 patients. One hundred seventy-one cases from Center 1 and 92 cases from Center 2 were used as the primary training cohort, and 66 cases from Center 3 and 20 cases from Center 4 were used as the independent test dataset. Semi-automatic module of ITK-SNAP software was used to assist CT image segmentation to obtain three-dimensional (3D) imaging region of interest (ROI). There were 788 handcrafted features extracted for 3D ROI using PyRadiomics. The optimal feature subset consists of three features screened by three feature selection methods as the input of the SVM to construct the conventional radiomics-based predictive model (cRad). 3D ROI was used to unify the resolution by 3D spline interpolation method for constructing the 3D tumor imaging tensor. Using 3D tumor image tensor as input, 3D kernelled support tensor machine-based predictive model (KSTM), and 3D ResNet-based deep learning predictive model (ResNet) were constructed. Multi-classifier fusion ML model is constructed by fusing cRad, KSTM, and ResNet using multi-classifier fusion strategy. Two experts with more than 10years of clinical experience were invited to reevaluate each patient based on their CECT following the NCCN guidelines to obtain resectable, unresectable, and borderline resectable diagnoses. The three results were converted into probability values of 0.25, 0.75, and 0.50, respectively, according to the traditional empirical method. Then it is used as an independent classifier and integrated with multi-classifier fusion machine learning (ML) model to obtain the human-machine fusion ML model (HMfML). Multi-classifier fusion ML model's area under receiver operating characteristic curve (AUC; 0.8610), predictive accuracy (ACC: 80.23%), sensitivity (SEN: 78.95%), and specificity (SPE: 80.60%) is better than cRad, KSTM, and ResNet-based single-classifier models and their two-classifier fusion models. This means that three different models have mined complementary CECT feature expression from different perspectives and can be integrated through CFS-ER, so that the fusion model has better performance. HMfML's AUC (0.8845), ACC (82.56%), SEN (84.21%), SPE (82.09%). This means that ML models might learn extra information from CECT that experts cannot distinguish, thus complementing expert experience and improving the performance of hybrid ML models. HMfML can predict PC resectability with high accuracy.

A Segmentation Method of Serialized Human Body Slices Based on Matting Strategy and Skeleton Extraction.

In this paper, a semiautomatic image segmentation method for the serialized body slices of the Visible Human Project (VHP) is proposed. In our method, we first verified the effectiveness of the shared matting method for the VHP slices and utilized it to segment a single image. Then, to meet the need for the automatic segmentation of serialized slice images, a method based on the parallel refinement method and flood-fill method was designed. The ROI (region of interest) image of the next slice can be extracted by using the skeleton image of the ROI in the current slice. Utilizing this strategy, the color slice images of the Visible Human body can be continuously and serially segmented. This method is not complex but is rapid and automatic with less manual participation. The experimental results show that the primary organs of the Visible Human body can be accurately extracted.

MODAN: Multifocal Object Detection Associative Network for Maritime Horizon Surveillance

In maritime surveillance systems, object detection plays a crucial role in ensuring the security of nearby waters by tracking the movement of various objects, such as ships and aircrafts, that are found at sea, detecting illegal activities and preemptively countering or predicting potential risks. Using vision sensors such as cameras to monitor the sea can help to identify the shape, size, and color of objects, enabling the precise analysis of maritime situations. Additionally, vision sensors can monitor or track small ships that may escape radar detection. However, objects located at considerable distances from vision sensors have low resolution and are small in size, rendering their detection difficult. This paper proposes a multifocal object detection associative network (MODAN) to overcome these vulnerabilities and provide stable maritime surveillance. First, it searches for the horizon using color quantization based on K-means; then, it selects and partitions the region of interest (ROI) around the horizon using the ROI selector. The original image and ROI image, converted to high resolution through the Super-Resolution Convolutional Neural Network (SRCNN), are then passed to the near-field and far-field detectors, respectively, for object detection. The weighted box fusion removes duplicate detected objects and estimates the optimal object. The proposed network is more stable and efficient in detecting distant objects than existing single-object detection models. Through performance evaluations, the proposed network exhibited an average precision surpassing that of the existing single-object detection models by more than 7%, and the false detection rate was reduced by 59% compared to similar multifocal-based state-of-the-art detection methods.

Projection mapping system for laser speckle contrast image: feasibility study for clinical application.

Laser speckle contrast images (LSCIs) have been utilized to monitor blood flow perfusion. However, they have conventionally been observed on monitor screens, resulting in potential spatial mismatching between the imaging region of interest (IROI) and monitor screen. This study proposes a projection mapping (PM) system for LSCIs (PMS_LSCI) that projects LSCIs to directly observe the blood flow perfusion in the IROI. The PMS_LSCI consists of a camera, imaging optics, a laser projector, and graphic user interface software. The spatial matching in the regions of interest was performed by adjusting the software screen of the LSCI in the IROI and evaluated by conducting in-vitro and in-vivo studies. An additional in-vivo study was performed to investigate the feasibility of real-time PM of the LSCI. The spatial mismatching in the regions of interest was ranged from 2.74% to 6.47% depending on the surface curvature. The PMS_LSCI could enable real-time PM of LSCI at four different blood flow states depending on blood pressure. The PMS_LSCI projects the LSCI in the IROI by interacting with a projector instead of the monitor screen. The PMS_LSCI presented clinical feasibility in the in-vitro and in-vivo studies.

Visualized Multiple Image Selection Encryption Based on Log Chaos System and Multilayer Cellular Automata Saliency Detection

For the security of multiple image regions of interest, this paper designs a multi-image visual encryption algorithm based on region of interest (ROI). Firstly, a new one-dimensional chaotic system (1-DLCP) is proposed. The performance analysis shows that the system has better dynamic behavior. Then, the salient parts of multiple images are extracted using multilayer cellular automata saliency detection. The measured matrix generated by the proposed chaotic system compresses the extracted plaintext images respectively. The compressed image and the extracted salient parts are recombined into one image, and a scrambling-diffusion synchronization operation is performed on the recombined image to obtain a secret image. The final embedding stage uses integer wavelet transform (IWT) to embed the secret image into a color carrier image. It is worth mentioning that this design scheme not only reconstructs the significant part without damage, but also reduces the storage space.

Head and neck cancer treatment outcome prediction: a comparison between machine learning with conventional radiomics features and deep learning radiomics.

Radiomics can provide in-depth characterization of cancers for treatment outcome prediction. Conventional radiomics rely on extraction of image features within a pre-defined image region of interest (ROI) which are typically fed to a classification algorithm for prediction of a clinical endpoint. Deep learning radiomics allows for a simpler workflow where images can be used directly as input to a convolutional neural network (CNN) with or without a pre-defined ROI. The purpose of this study was to evaluate (i) conventional radiomics and (ii) deep learning radiomics for predicting overall survival (OS) and disease-free survival (DFS) for patients with head and neck squamous cell carcinoma (HNSCC) using pre-treatment 18F-fluorodeoxuglucose positron emission tomography (FDG PET) and computed tomography (CT) images. FDG PET/CT images and clinical data of patients with HNSCC treated with radio(chemo)therapy at Oslo University Hospital (OUS; n = 139) and Maastricht University Medical Center (MAASTRO; n = 99) were collected retrospectively. OUS data was used for model training and initial evaluation. MAASTRO data was used for external testing to assess cross-institutional generalizability. Models trained on clinical and/or conventional radiomics features, with or without feature selection, were compared to CNNs trained on PET/CT images without or with the gross tumor volume (GTV) included. Model performance was measured using accuracy, area under the receiver operating characteristic curve (AUC), Matthew's correlation coefficient (MCC), and the F1 score calculated for both classes separately. CNNs trained directly on images achieved the highest performance on external data for both endpoints. Adding both clinical and radiomics features to these image-based models increased performance further. Conventional radiomics including clinical data could achieve competitive performance. However, feature selection on clinical and radiomics data lead to overfitting and poor cross-institutional generalizability. CNNs without tumor and node contours achieved close to on-par performance with CNNs including contours. High performance and cross-institutional generalizability can be achieved by combining clinical data, radiomics features and medical images together with deep learning models. However, deep learning models trained on images without contours can achieve competitive performance and could see potential use as an initial screening tool for high-risk patients.

YOLO-based ROI selection for joint encryption and compression of medical images with reconstruction through super-resolution network

Medical images contain significant patient information, and this confidential data should not be accessed without proper authorisation. Concurrently, due to the high redundancy of image data, compression is necessary to minimise image size and efficiently utilise network resources. This paper presents an effective joint encryption and compression method for medical images that prevent critical data leakage while reducing redundancy. Initially, a powerful real-time object detection method, You Only Look Once v7, is employed to accurately and swiftly detect the region of interest (ROI) within the medical images. Subsequently, a joint three-dimensional chaotic map and Huffman encoding are applied to secure medical images without compromising the compression ratio or increasing the time cost. Lastly, a super-resolution network is established at the receiver end to better reconstruct the ROI image for precise diagnostic purposes. The comprehensive experimental analysis demonstrates that our method delivers high levels of security, compression, and visual quality performance on standard datasets used in smart healthcare applications, at a minimum. Furthermore, our approach outperforms other competitive state-of-the-art schemes when compared. We hope this study will inspire further research within the healthcare community.

The effect of ultrasound wave path estimation to defect characterization capability in half-skip total focusing method

The total focusing method (TFM) is a post-processing imaging technique applied on full matrix capture (FMC) ultrasonic inspection (UT) dataset. In TFM the ultrasonic wave energy is synthetically focused on every pixel in the image region of interest (ROI). In terms of half-skip TFM (HSTFM), wave mode conversion happens when the wave rebounds at interface, such as specimen backwall. This paper aims to propose and evaluate a method that involves Snell’s law to address accurate estimation of distanceof-flight (DOF) of wave propagation when wave mode conversion appears in HSTFM. This HSTFM algorithm is applied to both experimental and simulated FMC dataset that inspects a surface-breaking notch for notch image reconstruction. Comparisons between images with and without considering Snell’s law in wave mode conversion show visible difference that could lead to misinterpretations in characterizing the defect. The sensitivity of TFM to varying defect features such as defect tilt angle is also studied using simulated FMC datasets.

A comprehensive multi-domain dataset for mitotic figure detection

The prognostic value of mitotic figures in tumor tissue is well-established for many tumor types and automating this task is of high research interest. However, especially deep learning-based methods face performance deterioration in the presence of domain shifts, which may arise from different tumor types, slide preparation and digitization devices. We introduce the MIDOG++ dataset, an extension of the MIDOG 2021 and 2022 challenge datasets. We provide region of interest images from 503 histological specimens of seven different tumor types with variable morphology with in total labels for 11,937 mitotic figures: breast carcinoma, lung carcinoma, lymphosarcoma, neuroendocrine tumor, cutaneous mast cell tumor, cutaneous melanoma, and (sub)cutaneous soft tissue sarcoma. The specimens were processed in several laboratories utilizing diverse scanners. We evaluated the extent of the domain shift by using state-of-the-art approaches, observing notable differences in single-domain training. In a leave-one-domain-out setting, generalizability improved considerably. This mitotic figure dataset is the first that incorporates a wide domain shift based on different tumor types, laboratories, whole slide image scanners, and species.

A DCT probability histogram-based ROI features for content-based natural and medical image retrieval applications

The effectiveness of a Content-based Image Retrieval (CBIR) system depends on selecting representative and salient visual features. Capturing the salient image features for forming a feature vector is an essential part of the CBIR process, as it can significantly affect retrieval efficiency. Most images contain a visually prominent region that perceives maximum image semantic information, known as Regions-of-Interest (ROIs). This paper suggests a new CBIR scheme that employs a Graph-based Visual Saliency (GBVS) map to extract the ROI of the image. Further, block-level Discrete Cosine Transforms (DCT) and subsequent histograms are computed for the DC and significant AC coefficients of the extracted ROI image. In addition, a low-dimensional feature vector is formed from the modified histograms. Subsequently, six statistical features from the background scene have also been computed for the final feature vector construction. As a result, the final feature vector contains ROI and background information in their proper proportion to improve retrieval performance. Moreover, the performance of the proposed CBIR scheme has been evaluated using one medical, three natural, two objects, one texture, and one natural heterogeneous image database. All these databases collectively contain 53732 different images from 200 image classes. The outcomes of the experiments demonstrate substantial improvement in the retrieval process with some existing related schemes. At the same time, it also shows that the performance of the proposed scheme is robust concerning all kinds of image databases.

Target-Free Text-Guided Image Manipulation

We tackle the problem of target-free text-guided image manipulation, which requires one to modify the input reference image based on the given text instruction, while no ground truth target image is observed during training. To address this challenging task, we propose a Cyclic-Manipulation GAN (cManiGAN) in this paper, which is able to realize where and how to edit the image regions of interest. Specifically, the image editor in cManiGAN learns to identify and complete the input image, while cross-modal interpreter and reasoner are deployed to verify the semantic correctness of the output image based on the input instruction. While the former utilizes factual/counterfactual description learning for authenticating the image semantics, the latter predicts the "undo" instruction and provides pixel-level supervision for the training of cManiGAN. With the above operational cycle-consistency, our cManiGAN can be trained in the above weakly supervised setting. We conduct extensive experiments on the datasets of CLEVR and COCO datasets, and the effectiveness and generalizability of our proposed method can be successfully verified. Project page: sites.google.com/view/wancyuanfan/projects/cmanigan.