Input RGB Research Articles

Image splicing detection using integrated LBP and DCT features

Abstract. Image splicing is one of the most common techniques used for picture manipulation and forgery. With the advent of user-friendly photo editing software, image splicing has become more prevalent and increasingly difficult to detect. This paper proposes a passive photo splicing detection approach based on Local Binary Patterns (LBP) and Discrete Cosine Transform (DCT) to identify splicing forgeries. The input RGB images are first converted to the YCbCr color space. Subsequently, the chrominance channels, Cb and Cr, are divided into overlapping blocks. Each block's LBP code is then transformed into the DCT domain. For each block, the standard deviation of each DCT coefficient is computed and used as a feature. Support Vector Machine (SVM) is employed as the classifier in a predictive model to determine whether the images have been spliced. To evaluate the proposed approach, two benchmark datasets for photo tampering were utilized. Experimental results indicate that the proposed method outperforms traditional splicing detection techniques in terms of detection accuracy and performance. This enhanced detection capability underscores the potential of combining LBP and DCT features with SVM classification for robust image splicing detection, paving the way for improved digital forensics tools in combating image manipulation.

Semantic-aware frame-event fusion based pattern recognition via large vision–language models

Pattern recognition through the fusion of RGB frames and Event streams has emerged as a novel research area in recent years. Current methods typically employ backbone networks to individually extract the features of RGB frames and event streams, and subsequently fuse these features for pattern recognition. However, we posit that these methods may suffer from two key issues: (1). They attempt to directly learn a mapping from the input vision modality to the semantic labels. This approach often leads to sub-optimal results due to the disparity between the input and semantic labels; (2). They utilize small-scale backbone networks for the extraction of RGB and Event input features, thus these models fail to harness the recent performance advancements of large-scale visual-language models. In this study, we introduce a novel pattern recognition framework that consolidates the semantic labels, RGB frames, and event streams, leveraging pre-trained large-scale vision–language models. Specifically, given the input RGB frames, event streams, and all the predefined semantic labels, we employ a pre-trained large-scale vision model (CLIP vision encoder) to extract the RGB and event features. To handle the semantic labels, we initially convert them into language descriptions through prompt engineering and polish using ChatGPT, and then obtain the semantic features using the pre-trained large-scale language model (CLIP text encoder). Subsequently, we integrate the RGB/Event features and semantic features using multimodal Transformer networks. The resulting frame and event tokens are further amplified using self-attention layers. Concurrently, we propose to enhance the interactions between text tokens and RGB/Event tokens via cross-attention. Finally, we consolidate all three modalities using self-attention and feed-forward layers for recognition. Comprehensive experiments on the HARDVS and PokerEvent datasets fully substantiate the efficacy of our proposed SAFE model. The source code has been released at https://github.com/Event-AHU/SAFE_LargeVLM.

Image manipulation localization via dynamic cross-modality fusion and progressive integration

Image manipulation localization aims at distinguishing forged regions from the whole test image. Though many outstanding prior arts have been proposed for this task, there are still two issues that need to be further studied: (1) how to fuse diverse types of features with forgery clues; (2) how to progressively integrate multistage features for better localization performance. In this paper, we propose a tripartite progressive integration network (TriPINet) for end-to-end image manipulation localization. First, we extract both visual perception information, e.g., RGB input images, and visual imperceptible features, e.g., frequency and noise traces for forensic feature learning. Second, we develop a guided cross-modality dual-attention (gCMDA) module to fuse different types of forged clues. Third, we design a set of progressive integration squeeze-and-excitation (PI-SE) modules to improve localization performance by appropriately incorporating multiscale features in the decoder. Extensive experiments are conducted to compare our method with state-of-the-art image forensics approaches. The proposed TriPINet obtains competitive results on several benchmark datasets.

Scalable video transformer for full-frame video prediction

Vision Transformers (ViTs) have shown success in many low-level computer vision tasks. However, existing ViT models are limited by their high computation and memory cost when generating high-resolution videos for tasks like video prediction. This paper presents a scalable video transformer for full-frame video predication. Specifically, we design a backbone transformer block for our video transformer. This transformer block decouples the temporal and channel features to reduce the computation cost when processing large-scale spatial–temporal video features. We use transposed attention to focus on the channel dimension instead of the spatial window to further reduce the computation cost. We also design a Global Shifted Multi-Dconv Head Transposed Attention module (GSMDTA) for our transformer block. This module is built upon two key ideas. First, we design a depth shift module to better incorporate the cross-channel or temporal information from video features. Second, we introduce a global query mechanism to capture global information to handle large motion for video prediction. This new transformer block enables our video transformer to predict a full frame from multiple past frames at the resolution of 1024 × 512 with 12 GB VRAM. Experiments on various video prediction benchmarks demonstrate that our method with only RGB input outperforms state-of-the-art methods that require additional data, like segmentation maps and optical flows. Our method exceeds the state-of-the-art RGB-only methods by a large margin (1.2 dB) in PSNR. Our method is also faster than state-of-the-art video prediction transformers.

Experimental Comparison of Two 6D Pose Estimation Algorithms in Robotic Fruit-Picking Tasks

This paper presents an experimental comparison between two existing methods representative of two categories of 6D pose estimation algorithms nowadays commonly used in the robotics community. The first category includes purely deep learning methods, while the second one includes hybrid approaches combining learning pipelines and geometric reasoning. The hybrid method considered in this paper is a pipeline of an instance-level deep neural network based on RGB data only and a geometric pose refinement algorithm based on the availability of the depth map and the CAD model of the target object. Such a method can handle objects whose dimensions differ from those of the CAD. The pure learning method considered in this comparison is DenseFusion, a consolidated state-of-the-art pose estimation algorithm selected because it uses the same input data, namely, RGB image and depth map. The comparison is carried out by testing the success rate of fresh food pick-and-place operations. The fruit-picking scenario has been selected for the comparison because it is challenging due to the high variability of object instances in appearance and dimensions. The experiments carried out with apples and limes show that the hybrid method outperforms the pure learning one in terms of accuracy, thus allowing the pick-and-place operation of fruits with a higher success rate. An extensive discussion is also presented to help the robotics community select the category of 6D pose estimation algorithms most suitable to the specific application.

P‐52: Scalable Multi‐layered Real‐time Holography Processor Architecture with High Bandwidth Memory (HBM)

In this paper, we present a fast and efficiently scalable 3D holographic video processor using a layer‐based method using a modified inverse Fresnel transform. In our previous paper, we designed a single‐layer holographic core using a fixed‐point model and tested its operation on an FPGA. This paper implements an 8‐layer, 15FPS real‐time hologram processor by receiving RGB and depth input. For fast CGH processing, we utilized HBM memory, which is faster than DDR4, to store FFT results. After applying 2× linear interpolation, the implemented real‐time holography processor converts into real‐time holograms using a 4K color space light modulator

A non-contact colour correction system on mobile display under various light sources

Human recognize perceived color on the display device of a smart phone under various types of light sources and illuminance differently even though the display represents the same color. This study investigated the relationship between input RGB code and output color tristimulus (XYZ) on two types of smartphone displays under various types of light sources and illuminances. A non-contact color correction system to transfer similar color with actual object was then proposed under various types of light sources and illuminances. Furthermore, the color correction model was validated by input color RGB predicted for representing the desired color and a survey test on displays of two types of smartphone, iPhone 13 Pro and Galaxy S22.

Convolutional Neural Networks Based Paddy Leaf Disease Detection

Abstract: This study proposes a novel approach for detecting paddy leaf diseases through the integration of Convolutional Neural Networks (CNNs). The process begins by converting input RGB images into HSV color space, where the saturation component is extracted. Subsequently, a binary image is generated, followed by background removal to enhance the accuracy of segmentation. The background-removed image is then converted back to the HSV color space for further processing. Kmeans segmentation is applied to segment the leaf regions effectively. The CNN architecture is employed for classification, categorizing the detected regions into various disease classes such as Healthy, Sheath Rot, Bacterial Leaf Blight, Brown Spot, and Leaf Blast. Additionally, a voice output feature is incorporated to provide audible feedback on the detected diseases. The study focuses on evaluating the accuracy of disease classification, aiming to demonstrate the effectiveness of the CNN approach in improving performance and reducing misclassifications compared to traditional methods. Overall, the proposed methodology offers a promising solution for accurate and efficient paddy leaf disease detection, with potential applications in agricultural management and crop protection

Learning to Detect Attended Objects in Cultural Sites with Gaze Signals and Weak Object Supervision

Cultural sites such as museums and monuments are popular tourist destinations worldwide. Visitors come to these places to learn about the cultures, histories, and arts of a particular region or country. However, for many cultural sites, traditional visiting approaches are limited and may fail to engage visitors. To enhance visitors’ experiences, previous works have explored how wearable devices can be exploited in this context. Among the many functions that these devices can offer, understanding which artwork or detail the user is attending to is fundamental to provide additional information on the observed artworks, understand the visitor’s tastes, and provide recommendations. This motivates the development of algorithms for understanding visitor attention from egocentric images. We considered the attended object detection task, which involves detecting and recognizing the object observed by the camera wearer, from an input RGB image and gaze signals. To study the problem, we collect a dataset of egocentric images collected by subjects visiting a museum. Since collecting and labeling data in cultural sites for real applications is a time-consuming problem, we present a study comparing unsupervised, weakly supervised, and fully supervised approaches for attended object detection. We evaluate the considered approaches on the collected dataset, assessing also the impact of training models on external datasets such as COCO and EGO-CH. The experiments show that weakly supervised approaches requiring only a 2D point label related to the gaze can be an effective alternative to fully supervised approaches for attended object detection. To encourage research on the topic, we publicly release the code and the dataset at the following url: https://iplab.dmi.unict.it/EGO-CH-Gaze/ .

Emerging Vanadium-Doped Cobalt Chloride Carbonate Hydroxide for Flexible Electrochromic Micro-Supercapacitor: Charged-State Prediction from RGB Input by ANN Model.

Multifunctional devices integrated with electrochromic and supercapacitance properties are fascinating because of their extensive usage in modern electronic applications. In this work, vanadium-doped cobalt chloride carbonate hydroxide hydrate nanostructures (V-C3H NSs) are successfully synthesized and show unique electrochromic and supercapacitor properties. The V-C3H NSs material exhibits a high specific capacitance of 1219.9Fg-1 at 1mVs-1 with a capacitance retention of 100% over 30000 CV cycles. The electrochromic performance of the V-C3H NSs material is confirmed through in situ spectroelectrochemical measurements, where the switching time, coloration efficiency (CE), and optical modulation (∆T) are found to be 15.7 and 18.8s, 65.85cm2C-1 and 69%, respectively. A coupled multilayer artificial neural network (ANN) model is framed to predict potential and current from red (R), green (G), and blue (B) color values. The optimized V-C3H NSs are used as the active materials in the fabrication of flexible/wearable electrochromic micro-supercapacitor devices (FEMSDs) through a cost-effective mask-assisted vacuum filtration method. The fabricated FEMSD exhibits an areal capacitance of 47.15mFcm-2 at 1mVs-1 and offers a maximum areal energy and power density of 104.78Whcm-2 and 0.04mWcm-2, respectively. This material's interesting energy storage and electrochromic properties are promising in multifunctional electrochromic energy storage applications.

Image Color Balance with Laplacian and Gaussian Pyramid (CBLGP) Algorithm

In recent times, technology has rapidly and significantly evolved across all sectors. Digital image processing stands out as a modern technology aimed at achieving clear images. However, digitized images often encounter issues of low quality, such as unclear or underwater images that require enhancement for better visibility. These problems stem from factors like deficient focusing, lighting, and various constraints leading to low contrast, shading, and artifacts. Underwater and satellite images consistently face less-than-ideal conditions due to environmental factors like light refraction in water, particle scattering, and dust in aquatic environments. Similarly, challenges in space, such as poor illumination and lack of contrast, further complicate image analysis. Overcoming these obstacles is crucial for extracting valuable information, necessitating advanced processing techniques. This paper introduces an enhanced Gaussian/Laplace color balance-fusion algorithm designed to improve image visibility. Modifications to certain equations result in sharper and clearer images. The algorithm begins by determining the white balance of the input RGB color image and subsequently enhances its intensity. Edge improvement is carried out separately using a depth filter. The weights for each image are then determined and combined to form a Laplace Pyramid. A color restoration technique is applied to process the resulting image, producing the final enhanced image. While existing methods for image contrast enhancement typically focus on image features, they often neglect user characteristics. This paper explores the application of image sharpening, a prominent image enhancement technique, in clearing underwater or low-quality images using the proposed algorithm. Keywords: EDSHE, High pass filter, White patch ratix, Laplacian pyramid, Color restoration.

US2Mask: Image-to-mask generation learning via a conditional GAN for cardiac ultrasound image segmentation

Cardiac ultrasound (US) image segmentation is vital for evaluating clinical indices, but it often demands a large dataset and expert annotations, resulting in high costs for deep learning algorithms. To address this, our study presents a framework utilizing artificial intelligence generation technology to produce multi-class RGB masks for cardiac US image segmentation. The proposed approach directly performs semantic segmentation of the heart’s main structures in US images from various scanning modes. Additionally, we introduce a novel learning approach based on conditional generative adversarial networks (CGAN) for cardiac US image segmentation, incorporating a conditional input and paired RGB masks. Experimental results from three cardiac US image datasets with diverse scan modes demonstrate that our approach outperforms several state-of-the-art models, showcasing improvements in five commonly used segmentation metrics, with lower noise sensitivity. Source code is available at https://github.com/energy588/US2mask.

HAR-Net: An Hourglass Attention ResNet Network for Dangerous Driving Behavior Detection

Ensuring safety while driving relies heavily on normal driving behavior, making the timely detection of dangerous driving patterns crucial. In this paper, an Hourglass Attention ResNet Network (HAR-Net) is proposed to detect dangerous driving behavior. Uniquely, we separately input optical flow data, RGB data, and RGBD data into the network for spatial–temporal fusion. In the spatial fusion part, we combine ResNet-50 and the hourglass network as the backbone of CenterNet. To improve the accuracy, we add the attention mechanism to the network and integrate center loss into the original Softmax loss. Additionally, a dangerous driving behavior dataset is constructed to evaluate the proposed model. Through ablation and comparative studies, we demonstrate the efficacy of each HAR-Net component. Notably, HAR-Net achieves a mean average precision of 98.84% on our dataset, surpassing other state-of-the-art networks for detecting distracted driving behaviors.

Tensor-Based Sparse Representation for Hyperspectral Image Reconstruction Using RGB Inputs

Hyperspectral image (HSI) reconstruction from RGB input has drawn much attention recently and plays a crucial role in further vision tasks. However, current sparse coding algorithms often take each single pixel as the basic processing unit during the reconstruction process, which ignores the strong similarity and relation between adjacent pixels within an image or scene, leading to an inadequate learning of spectral and spatial features in the target hyperspectral domain. In this paper, a novel tensor-based sparse coding method is proposed to integrate both spectral and spatial information represented in tensor forms, which is capable of taking all the neighboring pixels into account during the spectral super-resolution (SSR) process without breaking the semantic structures, thus improving the accuracy of the final results. Specifically, the proposed method recovers the unknown HSI signals using sparse coding on the learned dictionary pairs. Firstly, the spatial information of pixels is used to constrain the sparse reconstruction process, which effectively improves the spectral reconstruction accuracy of pixels. In addition, the traditional two-dimensional dictionary learning is further extended to the tensor domain, by which the structure of inputs can be processed in a more flexible way, thus enhancing the spatial contextual relations. To this end, a rudimentary HSI estimation acquired in the sparse reconstruction stage is further enhanced by introducing the regression method, aiming to eliminate the spectral distortion to some extent. Abundant experiments are conducted on two public datasets, indicating the considerable availability of the proposed framework.

An Effective Video Transformer With Synchronized Spatiotemporal and Spatial Self-Attention for Action Recognition.

Convolutional neural networks (CNNs) have come to dominate vision-based deep neural network structures in both image and video models over the past decade. However, convolution-free vision Transformers (ViTs) have recently outperformed CNN-based models in image recognition. Despite this progress, building and designing video Transformers have not yet obtained the same attention in research as image-based Transformers. While there have been attempts to build video Transformers by adapting image-based Transformers for video understanding, these Transformers still lack efficiency due to the large gap between CNN-based models and Transformers regarding the number of parameters and the training settings. In this work, we propose three techniques to improve video understanding with video Transformers. First, to derive better spatiotemporal feature representation, we propose a new spatiotemporal attention scheme, termed synchronized spatiotemporal and spatial attention (SSTSA), which derives the spatiotemporal features with temporal and spatial multiheaded self-attention (MSA) modules. It also preserves the best spatial attention by another spatial self-attention module in parallel, thereby resulting in an effective Transformer encoder. Second, a motion spotlighting module is proposed to embed the short-term motion of the consecutive input frames to the regular RGB input, which is then processed with a single-stream video Transformer. Third, a simple intraclass frame interlacing method of the input clips is proposed that serves as an effective video augmentation method. Finally, our proposed techniques have been evaluated and validated with a set of extensive experiments in this study. Our video Transformer outperforms its previous counterparts on two well-known datasets, Kinetics400 and Something-Something-v2.

Low Intricacy‎ Multistage Algorithm for Underwater Image Enhancement

Humanity currently lives in a technological era that witnesses rapid progress in multiple fields. Digital image processing is one of the modern technologies that has provided practical answers to many challenges including image enhancement, analysis, reconstruction, recovery, compression, processing, and understanding. One of these notable challenges relates to underwater photography. Underwater images are always exposed to less-than-ideal conditions due to environmental and physical factors. These include refraction of light in water, scattering of particles and dust in the aquatic medium, lack of illumination in deep water, and poor contrast. These challenges make it extremely difficult to analyze and extract valuable information without advanced processing. In this study, an improved color balance-fusion algorithm is provided by improving the image visuality and modifying some equations to obtain sharper and clearer images. The proposed algorithm begins by finding the white balance of the input RGB color image, after that, it improves the intensity. Next, the edges are improved using Gamma separately. The weights are then found for each image and combined to find naive fusion. The resulting image is processed using a color retrieval algorithm to produce the final image. along with comparisons to eleven other algorithms with various processing methods. Experimental results showed that this algorithm can significantly improve underwater images, increasing image clarity and making colors clearer. The improvement rates reached 5.8389 and 2.6778 for UISM and UICM metrics, respectively.

Hyperspectral Image Reconstruction From RGB Input Through Highlighting Intrinsic Properties

Hyperspectral Image Reconstruction From RGB Input Through Highlighting Intrinsic Properties

Hyperspectral Image Super Resolution With Real Unaligned RGB Guidance.

Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

FocalPose++: Focal Length and Object Pose Estimation Via Render and Compare.

We introduce FocalPose++, a neural render-and-compare method for jointly estimating the camera-object 6D pose and camera focal length given a single RGB input image depicting a known object. The contributions of this work are threefold. First, we derive a focal length update rule that extends an existing state-of-the-art render-and-compare 6D pose estimator to address the joint estimation task. Second, we investigate several different loss functions for jointly estimating the object pose and focal length. We find that a combination of direct focal length regression with a reprojection loss disentangling the contribution of translation, rotation, and focal length leads to improved results. Third, we explore the effect of different synthetic training data on the performance of our method. Specifically, we investigate different distributions used for sampling object's 6D pose and camera's focal length when rendering the synthetic images, and show that parametric distribution fitted on real training data works the best. We show results on three challenging benchmark datasets that depict known 3D models in uncontrolled settings. We demonstrate that our focal length and 6D pose estimates have lower error than the existing state-of-the-art methods.

Recognition and detection of unusual activities in ATM using dual-channel capsule generative adversarial network

Real-time detection of human activity has become essential for monitoring and security of public spaces such as bank ATMs and workplaces, due to daily rise in criminal activities. Currently, monocular CCTV cameras that only record RGB video are used for monitoring such restricted areas. In addition to RGB data, the RGB + D sensor also offers depth information about the scene. To overcome the issue of online recognition of anomalous activities in Bank ATMs, a supervised deep learning method utilizing Dual-Channel Capsule Generative Adversarial Network (DCCGAN) and RGB + D sensor is proposed. The input RGB + D Dataset is given to super pixel motion detection method to find the region of interest (ROI). Motion detection is a significant stage examination of wide scene for background subtraction and foreground detection. After arranging the motion detection, its region is positioned frame by frame. Then, the detected ROI is given to fast discrete curvelet transform with the wrapping (FDCT-WRP) method. FDCT-WRP feature extraction method. These extracting features are supplied to Deep Convolutional Spiking neural network (DCSNN), which realize the object. RGB and depth video segments are used to create motion templates from the RGB + D data online video stream. These templates are trained on DCCGANs to identify distrustful events in current activity and categorized as normal and abnormal. Additionally, a unique RGB + D dataset is employed because there was no existing dataset available for analyzing human activity in ATMs. The proposed DUA-DCCGAN-ATM approach is assessed on qualitative with quantitative statistical evaluation parameters and identify suspect occurrence with 0.932 precision and 98.2 % accuracy. The detailed statistical analysis exemplifies that the proposed technique can identify distrustful events in a real-time online manner prior completing the anomalous activity.