Image Retargeting Research Articles

Directorial Editing: A Hybrid Deep-Learning Approach to Content-Aware Image Retargeting and Resizing

Directorial Editing: A Hybrid Deep-Learning Approach to Content-Aware Image Retargeting and Resizing

Detection and classification of breast cancer in mammographic images with fine-tuned convolutional neural networks

ABSTRACT Breast cancer is cancer that forms in the cells of the breasts and is a severe health issue that affects many people around the world, especially since it is the most deadly cancer in women. By finding it early and using new treatments, patients can overcome this challenge and get back to a healthier life. This study proposed a procedure to fine-tune the Convolutional Neural Networks (CNN) model with data preprocessing and augmentation in classifying mammogram images called the Hybrid Mammogram Classification and Detection Pipeline (HMCaD). After using CNN for classification because it brings higher confidence in classifying tasks, the YOLOv8 has been applied for localization subtask to detect abnormal positions with predicted bounding boxes. The database is provided by the Mammographic Image Analysis Society (MIAS) and is protected by the United Kingdom. It comprises 330 samples, including 79 benign, 54 malignant, and 207 normal images. As a result, the classification in our model based on the custom EfficientNetB3 model and seam carving technique received a great validation accuracy, test accuracy, and F1 score throughout three scenarios at 96.73%, 97.59%, and 97.58%, respectively. Furthermore, the area under the Receiver Operating Characteristic (ROC) curve also has a surprise result of 0.96 (i.e. AUC = 0.96 ). Moreover, YOLOv8 for detecting abnormal positions in our study achieved 83.22% in Intersection over Union (IoU). This led to the research reaching good results in classifying and detecting breast cancer by considering several performance metrics.

Retargeting Video With an End-to-End Framework.

Video holds significance in computer graphics applications. Because of the heterogeneous of digital devices, retargeting videos becomes an essential function to enhance user viewing experience in such applications. In the research of video retargeting, preserving the relevant visual content in videos, avoiding flicking, and processing time are the vital challenges. Extending image retargeting techniques to the video domain is challenging due to the high running time. Prior work of video retargeting mainly utilizes time-consuming preprocessing to analyze frames. Plus, being tolerant of different video content, avoiding important objects from shrinking, and the ability to play with arbitrary ratios are the limitations that need to be resolved in these systems requiring investigation. In this paper, we present an end-to-end RETVI method to retarget videos to arbitrary aspect ratios. We eliminate the computational bottleneck in the conventional approaches by designing RETVI with two modules, content feature analyzer (CFA) and adaptive deforming estimator (ADE). The extensive experiments and evaluations show that our system outperforms previous work in quality and running time.

SMART: Supervised multi-class image retargeting generative model based on a long-range sampling strategy

Content-aware image retargeting (CAIR) techniques are crucial in multimedia processing for displaying images on various devices while preserving visually salient contents with desirable visual effects. There are discrete and continuous algorithms. For the former, the artefacts happen when the foreground proportion is larger than the retargeting ratio; for the latter, the salient regions are prone to be squeezed. In this paper, we reformulate the retargeting process into sampling the salient signal and reconstruction under aesthetic supervision, the supervised multi-class image retargeting reconstruction (SMART) framework. The target images can be represented into complementary parts, the masked and unmasked ones, according to the saliency influences in the encoder phrase. The long-range sampling algorithm is proposed to calculate similarities through an 8-connected planar path while considering spatial distance and feature correlation. The sampled embeddings in latent space reconstruct the retargeted images under supervised signals for aesthetic quality. The semantic loss Lsem from the pretrained CLIP model can maintain consistency for both content and semantics. The supervised loss, Lir, is introduced to ensure the retargeted qualities are close to the preferred labels. Then, we release a new retargeting dataset comprising seven image classes (animal, building, car, flower, indoor, landscape and people) with supervised labels collected from designers for further aesthetic retargeting study. The ablation studies are conducted to confirm the effectiveness of the new dataset, and comparative experiments with state-of-the-art baselines demonstrate the advantages of the proposed method.

A convolutional neural network based on noise residual for seam carving detection

Seam carving is a method of resizing images based on content awareness. It can realize image retargeting while retaining the main content of the image. However, it may also be maliciously used to tamper with images, such as changing image semantic content by object removal. Therefore, seam carving detection has become important in image forensics. In this paper, a noise residual-based deep learning method is proposed to detect seam carving images. We try to learn the local noise in-consistency of images to recognize them. Firstly, a noise residual extraction segment is used to learn local noise features in the image, and a noise augmentation module is designed to enrich the features, which leverages the noise features extracted from a steganalysis rich model filter to discover the noise in-consistency between authentic and tampered regions. Then, through the feature dimensionality reduction section, the features are further learned and the size of feature maps are reduced. Finally, the output is obtained through global average pooling and a fully-connected layer. A careful testing strategy is further proposed, which greatly improves the detection performance, especially for seam carving with small scaling ratios. The experimental results demonstrate that our method achieves state-of-the-art performance at various scales, and has good robustness and generalization compared with other methods.

Supervised deep learning for content-aware image retargeting with Fourier Convolutions

Supervised deep learning for content-aware image retargeting with Fourier Convolutions

A comprehensive review of image retargeting

With the development of display technologies, image retargeting plays a significant role in computer vision and pattern recognition communities currently. Image retargeting aims to display an image on a series of appliances with different resolutions and target aspect ratios. During the last decade, representative algorithms for image retargeting have been presented in the literature and achieved state-of-the-art performance. In this survey, we provide a comprehensive review of image retargeting, covering a wide variety of pioneering works for 2D image retargeting and stereoscopic image retargeting. 2D image retargeting focuses on preserving interesting regions when modifying the original image with arbitrary resolutions appropriately. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and the depth consistency of 3D scenes simultaneously. In this survey, we start the first attempt to analyze the trends of 2D image retargeting and then summarize different types of stereoscopic image retargeting. Secondly, image retargeting quality assessment metrics for 2D images and stereoscopic images are introduced to evaluate retargeted images. Thirdly, we also investigate the evaluation datasets, and give the comparison results and analysis between different representative methods. Finally, the promising future research is thoroughly discussed to further improve the performance of 2D and stereoscopic image retargeting.

Integration of local and global features for image retargeting quality assessment

Integration of local and global features for image retargeting quality assessment

A Tacholess Order Tracking Method Based on the STFTSC Algorithm for Rotor Unbalance Fault Diagnosis Under Variable-Speed Conditions

Abstract Due to the fact that rotors usually operate in a non-stationary mode with changing speeds, the conventional rotor unbalance detection method based on the stationary signal will produce a major “spectrum ambiguity issue” and affect the accuracy of rotor unbalance detection. To this end, a tacholess order tracking method based on the STFTSC algorithm is suggested in this study, where the STFTSC algorithm is developed by combining the short-time Fourier transform and the seam carving algorithm. First, the STFTSC algorithm is utilized to accurately extract the instantaneous frequency (IF) of the rotor and calculate the instantaneous phase under variable-speed conditions. Subsequently, the original signal is resampled in the angular domain to transform the non-stationary time domain signal into a stable angle domain signal, eliminating the effect of the speed variations. Finally, the angular domain signal is transformed into the order domain signal, which uses the discrete Fourier transform and the discrete spectrum correction method to identify the amplitude and phase corresponding to the fundamental frequency component of the signal. The simulation results show that the IF extracted by the STFTSC algorithm has higher extraction accuracy compared with the traditional STFT spectral peak detection method and effectively eliminates the effect of speed fluctuations. A rotor dynamic-balancing experiment shows that the unbalance correction effect based on the STFTSC algorithm is remarkable, with the average unbalance amount decrease rate on the left and right sides being 90.02% and 92.56%, respectively, after a single correction.

Aesthetic-aware image retargeting based on foreground–background separation and PSO optimization

Aesthetic-aware image retargeting based on foreground–background separation and PSO optimization

Multi-operator Image Retargeting based on Saliency Object Ranking and Similarity Evaluation Metric

Image Retargeting (IR) technology is proposed to flexibly display images on various display devices while protecting the important content of the images undistorted. IR methods mainly use Salient Object Detection (SOD) to obtain important content, however, most existing SOD methods treat multiple salient objects with the same saliency degrees, which makes IR methods assign the same retargeting ratios for different objects and leads to producing information-loss retargeted results. Multi-operator IR demonstrates better generalization than single operator by using multiple operators to find the optimal sequence of operators. Meanwhile, the tremendous processing time limits its practical use. To address these problems, we propose a multi-operator IR method based on Salient Object Ranking (SOR) and Similarity Evaluation Metric (SORSEM-IR), which includes two stages: importance map generation and multi-operator IR. In the first stage, a SOR module with Context-aware Semantic Refinement (SORCSR) is proposed, which extracts the salient instances and infers their saliency ranks with a context-aware semantic refinement module, then the SOR map, face map, and gradient map are fused as the importance map. In the second stage, to speed up multiple operations, a similarity evaluation metric is proposed to measure the similarity between the original image and the seam-removal image by Seam Carving (SC) operation, and switch SC to uniform scaling to meet the aspect ratio when distortion caused by SC arrives at a certain extent. Experimental results show that the SORCSR network achieves state-of-the-art performance on the ASSR dataset subjectively and objectively, and the SORSEM-IR guided by SORCSR can not only protect the salient objects with minimum deformation but also meet human aesthetic perception.

A shadow preservation framework for effective content-aware image retargeting process

<p>In the discipline of image retargeting, the structure of shadows in the image is required to be preserved since it gives structural information about the objects. Many traditional image retargeting techniques do not pay attention to preserve the shadow objects present in the image. In this paper a shadow-preservation framework is proposed in which the saliency map is derived using segmentation-based gPb-owt-ucm approach to emphasize the salient regions. To assess the quality of saliency map a fixation prediction analysis is conducted on each section of the image using the eye tracker technology. Second, to pay more attention on the shadow objects a shadow map is developed after executing image pre-processing, shadow detection, region growing, and region filling operations. To rectify the issue of misclassification of shadow pixels erosion and dilation operations are performed. To obtain an efficient importance map, a gradient map is prepared using canny operator which is further combined with the saliency map and shadow map. The obtained importance map is supplied to the seam diversion based image retargeting (SDIR) technique to resize the image in horizontal and vertical directions. To justify the efficiency of the proposed framework’s the obtained results are compared with the existing state-of-the-art. By preserving shadows, the proposed framework enhances the overall visual quality and maintains the integrity of objects in the retargeted images. The framework’s effectiveness is validated through quantitative evaluations and visual comparisons with existing methods. The results demonstrate its potential for improving content-aware image retargeting applications, paving the way for more realistic and visually appealing image resizing techniques in various domains.<em></em></p>

Text line extraction strategy for palm leaf manuscripts

Text line segmentation is an important step in the historical document image analysis pipeline to supply useful information for recognition, keyword spotting and indexing. Many handcrafted-based and learning-based approaches have been developed to cope with text line extraction challenges. In this work we present a hybrid technique which combines a convolutional-based denoising task and heuristic Seam Carving framework. We propose the following changes to the original Seam Carving: (1) We applied adaptive slice to anticipate miss-extraction on a short text during medial seam computation. (2) We applied a triple smoothing to find the best local maxima of the smoothed horizontal projection profile which represents candidate medial seams. (3) We utilized five post-processing steps aimed at reconstructing a more precise medial seam. Three different palm leaf data sets: old Sundanese, old Balinese, and old Khmer, have been used to compare Convolutional Seam Carving (CSC) to several baseline methods, including the original Seam Carving. Experimental results show that the proposed method outperforms other current handcrafted-based baselines on all three palm leaf manuscript (PLM) data sets. On the old Sundanese data sets, CSC can produce a significant improvement of the performance rate compared to all other baseline approaches, and it also enhance the measurement on old Balinese and old Khmer datasets. In the ablation study, we discovered that a foreground extraction step is not only able to reduce noises and color degradation but also provide better separation of text region. Following that, an adaptive slice and triple smoothing approach contribute to solve the text length variation problem. Finally, post-processing steps were effective in connecting discontinuous medial seam. The code has been published in https://github.com/erickpaulus/text-line-segmentation.

CAMID: An assuasive approach to reveal source camera through inconspicuous evidence

Identifying the source camera of digital images is an important problem due to the abundance of such images nowadays. Although photo response non-uniformities (PRNUs) can accurately identify the source camera, seam-carving techniques have been introduced to hide the cameras’ identity by invalidating the positional correspondence assumption in the PRNU. Especially, the seam carving techniques create irregular carving patterns, which makes source camera identification from seam-carved images challenging. In this paper, we propose a feature-based method called CAM1D, to identify the source camera of seam-carved images. CAM1D constructs a series of camera signatures to enhance robustness and reliability, rather than relying on a single PRNU for identification. Useful features are extracted from the correlation between a series of camera signatures and seam-carved photos, giving high source camera identification accuracy. Experimental results using the confusion matrix and F1 score confirm the effectiveness of CAM1D.

Content-aware image retargeting technique and iterated function system: frameworks, applications, and possible future advancements

Content-aware image retargeting technique and iterated function system: frameworks, applications, and possible future advancements

Gaze-guided CT image retargeting by multi-attribute binary hashing

Computed tomography (CT) imaging is pervasively utilized for detecting tumors and internal body injuries. CT image retargeting means to horizontally/vertically shrink the semantically non-salient regions (e.g, the normal organs) while preserving the salient ones (e.g., the diseased organs) inside a CT image, as exemplified in Fig. 1. In practice, retargeting can substantially facilitate CT image displaying, which can benefit the subsequent medical treatment. In this work, we propose a bio-inspired CT image retargeting pipeline by mimicking human gaze behavior. More specifically, for each CT image, we extract the gaze shifting path (GSP) to capture human gaze distribution during the visual perception toward each CT image. Afterward, a multi-attribute binary hashing (MABH) is formulated to exploit the semantics of these GSPs. Thereby, each graphlet can be converted into the binary hash codes. Finally, the hash codes corresponding to GSP from each CT image are quantized into a feature vector, which is leveraged to learn a Gaussian mixture model (GMM) that guides CT image shrinking. In the experiments, to evaluate how gaze allocation influencing CT image retargeting, a user study is designed to compare the GSPs produced by normal observers and Alzheimer’s patients respectively. Besides, a comparative study has verified the superiority of our method.

Content-Aware Image Resizing Technology Based on Composition Detection and Composition Rules

A novel content-aware image resizing mechanism based on composition detection and composition rules is proposed to address the lack of esthetic perception in current content-aware resizing algorithms. A composition detection module is introduced for the detection of the composition of the input image types in the proposed algorithm. According to the classification results, the corresponding composition rules in computational esthetics are selected. Finally, the algorithm performs the operations of seam carving using the corresponding esthetic rules. The resized image not only protects the important content of the image, but also meets the composition rules to optimize the overall visual effect of the image. The simulation results show that the proposed algorithm achieves a better visual effect. Compared with the existing algorithms, the proposed algorithm not only effectively protects important image content, but also protects important structures and improves the overall beauty of the image.

Image Retargeting in Cyber-Physical Systems with IoT and Network Security using Fractal Images

Image Retargeting in Cyber-Physical Systems with IoT and Network Security using Fractal Images

IMGCAT: An approach to dismantle the anonymity of a source camera using correlative features and an integrated 1D convolutional neural network

With the proliferation of smartphones, digital data collection has become trivial. The ability to analyze images has increased, but source authentication has stagnated. Editing and tampering of images has become more common with advancements in signal processing technology. Recent developments have introduced the use of seam carving (insertion and deletion) techniques to disguise the identity of the camera, specifically in the child pornography market. In this article, we focus on the available features in the image based on PRNU (photo response nonuniformity). The forced-seam sculpting technique is a well-known method to create occlusion for camera attribution by injecting seams into each 50 × 50 pixel block. To counter this, we perform camera identification using a 1D CNN integrated with feature extractions on 20 × 20 pixel blocks. We achieve state-of-the-art performance for our proposed IMGCAT (image categorization) in three-class classification over the baselines (original, seam removed, seam inserted). Based on our experimental findings, our model is robust when dealing with blind facts related to the questionable camera.

Robust detection of seam carving with low ratio via pixel adjacency subtraction and CNN-based transfer learning

Seam carving is an outstanding content-aware image resizing technique. When the resizing ratio is relatively lower, seam carving can achieve image resizing without leaving any visual obvious artifacts. However, it may also be used for malicious purposes, such as changing image semantic content by object removal. Seam caving changes the relationship between neighboring pixels, thus seam carved images should be different from natural ones. Inspired by this, an end-to-end seam carving detection network, namely SCDNet, is proposed to detect seam carved images. To highlight seam carving artifacts, a pre-processing layer, namely the subtractive pixel adjacency module, is designed as the first layer of SCDNet. Then, 16 residual blocks with shortcut learn features from the input, and a spatial pyramid pooling module is embedded at the end of the residual block to generate fixed-length features, which can eliminate the requirement of fixed-resolution input images for existing deep models. Finally, the features are fed into the fully-connected layer for final classification. In addition, to expose seam carved images with a small carving ratio, a transfer learning strategy is exploited to gradually clarify the artifacts. Extensive experiments demonstrate that SCDNet obtains better detection accuracy than state-of-the-art methods, especially for seam carving with small scaling ratios.