Compact Image Research Articles

MixFuse: An iterative mix-attention transformer for multi-modal image fusion

Multi-modal image fusion plays a crucial role in various visual systems. However, existing methods typically involve a multi-stage pipeline, i.e., feature extraction, integration, and reconstruction, which limits the effectiveness and efficiency of feature interaction and aggregation. In this paper, we propose MixFuse, a compact multi-modal image fusion framework based on Transformers. It smoothly unifies the process of feature extraction and integration, As its core, the Mix Attention Transformer Block (MATB) integrates the Cross-Attention Transformer Module (CATM) and the Self-Attention Transformer Module (SATM). The CATM introduces a symmetrical cross-attention mechanism to identify modality-specific and general features, filtering out irrelevant and redundant information. Meanwhile, the SATM is designed to refine the combined features via a self-attention mechanism, enhancing the internal consistency and proper preservation of the features. This successive cross and self-attention modules work together to enhance the generation of more accurate and refined feature maps, which are essential for later reconstruction. Extensive evaluation of MixFuse on five public datasets shows its superior performance and adaptability over state-of-the-art methods. The code and model will be released at https://github.com/Bitlijinfu/MixFuse.

An efficient segment anything model for the segmentation of medical images

This paper introduces the efficient medical-images-aimed segment anything model (EMedSAM), addressing the high computational demands and limited adaptability of using SAM for medical image segmentation tasks. We present a novel, compact image encoder, DD-TinyViT, designed to enhance segmentation efficiency through an innovative parameter tuning method called med-adapter. The lightweight DD-TinyViT encoder is derived from the well-known ViT-H using a decoupled distillation approach.The segmentation and recognition capabilities of EMedSAM for specific structures are improved by med-adapter, which dynamically adjusts the model parameters specifically for medical imaging. We conducted extensive testing on EMedSAM using the public FLARE 2022 dataset and datasets from the First Hospital of Zhejiang University School of Medicine. The results demonstrate that our model outperforms existing state-of-the-art models in both multi-organ and lung segmentation tasks.

Perspectives on 2D materials for hybrid and beyond-Si image sensor applications

Abstract The complementary metal–oxide–semiconductor (CMOS) image sensor has become essential and ubiquitous in our daily lives as it is present in almost every pocket. As demand for compact, multifunction, and high-efficiency Internet of Things applications continues to rise, novel configuration designs and manufacturing methods, such as neural network integration and 3D stacking have been implemented to enhance the CMOS image sensor’s (CIS) performance. However, the progress of image sensors based on silicon CMOS technology would eventually be limited by the intrinsic optical, electrical, and mechanical properties of silicon material. This has led to the exploration of two-dimensional materials (2DMs) and the emergence of 2DMs as promising candidates for the next generation of optoelectronic devices. In this article, we discuss the current advancements and challenges associated with silicon CISs and the potential benefits of incorporating 2DMs in the image sensor. We highlight three critical opportunities for 2DMs, including Si CMOS/2DMs hybrid structure and direct growth techniques of 2DMs on Si for back-end-of-line integration, 2DMs-based neuromorphic photodetectors (PDs) and optical neural networks for in-image-sensor-processing, and curved image sensor based on 2DMs PDs for bionic detection. With the growing maturity of 2DM technologies, we anticipate that the device scaling and the increase of integration density of 2DM electronics in the image sensor will continue, leading to the development of highly efficient, compact, intelligent, and versatile 2DM image sensors in the near future.

Freeform metasurface color router for deep submicron pixel image sensors.

Advances in imaging technologies have led to a high demand for ultracompact, high-resolution image sensors. However, color filter-based image sensors, now miniaturized to deep submicron pixel sizes, face challenges such as low signal-to-noise ratio due to fewer photons per pixel and inherent efficiency limitations from color filter arrays. Here, we demonstrate a freeform metasurface color router that achieves ultracompact pixel sizes while overcoming the efficiency limitations of conventional architectures by splitting and focusing visible light instead of filtering. This development is enabled by a fully differentiable topology optimization framework to maximize the use of the design space while ensuring fabrication feasibility and robustness to fabrication errors. The metasurface can distribute an average of 85% of incident visible light according to the Bayer pattern with a pixel size of 0.6 μm. The device and design methodology enable the compact, high-sensitivity, and high-resolution image sensors for various modern technologies and pave the way for the advanced photonic device design.

Brain MR Image Classification Using Superpixel-Based Deep Transfer Learning.

Nowadays, brain MR (Magnetic Resonance) images are widely used by clinicians to examine the brain's anatomy to look into various pathological conditions like cerebrovascular incidents and neuro-degenerative diseases. Generally, these diseases can be identified with the MR images as "normal" and "abnormal" brains in a two-class classification problem or as disease-specific classes in a multi-class problem. This article presents an ensemble transfer learning-inspired deep architecture that uses the simple linear iterative clustering (SLIC)-based superpixel algorithm along with convolutional neural network (CNN) to classify the MR images as normal or abnormal. Superpixel algorithm segments the input MR images into clusters of regions defined by similarity measures using perceptual feature space. These superpixel images are beneficial as they can provide a compact and meaningful role in computationally demanding applications. The superpixel images are then fed to the deep convolutional neural network (CNN) to classify the images. Three brain MR image datasets, NITR-DHH, DS-75, and DS-160, are used to conduct the experimentation. Through the use of deep transfer learning, the model achieves performance accuracy of 88.15% (NITR-DHH), 98.15% (DS-160), and 98.33% (DS-75) even with the small-scale medical image dataset. The experimentally obtained results demonstrate that the proposed method is promising and efficient for clinical applications for diagnosing different brain diseases via MR images.

Compact and effective photon-resolved image scanning microscope

Compact and effective photon-resolved image scanning microscope

Simultaneous two-plane flame front detection using PIV based on defocusing.

This Letter presents a simultaneous two-plane flame front detection method. It is based on a standard single-camera single-plane particle image velocimetry (PIV) system in combination with an inexpensive and compact image splitting device. The image splitting optics places images from two depth-offset planes onto the two halves of a camera sensor. A shallow depth of field ensures only one plane is in focus on each half of the sensor. By using a high-pass filter and a novel two-step filter we have devised, the out-of-focus particle images are effectively removed, while the in-focus particle images remain, allowing the turbulent flame fronts on two planes to be detected simultaneously. Our approach could be combined with conventional polarization/wavelength discrimination methods to achieve simultaneous multi-plane flame front reconstruction with similarly high in-plane spatial resolution to single-plane measurement and is suitable for practical combustion devices with limited optical access.

Microplate-based 3D-printed image box for urea determination in milk by digital image colorimetry.

In this study, we describe a rapid and high-throughput smartphone-based digital colorimetric method for determining urea in milk. A compact and cost-effective 3D-printed image box microplate-based system was designed to measure multiple samples simultaneously, using minimal sample and reagent volumes. The apparatus was applied for the quantification of urea in milk based on its reaction with p-dimethylaminobenzaldehyde (DMAB). The predictive performance of calibration was evaluated using RGB and different colour models (CMYK, HSV, and CIELAB), with the average blue (B) values of the RGB selected as the analytical signal for urea quantification. Under optimized conditions, a urea concentration linear range from 50 to 400 mg L-1 was observed, with a limit of detection (LOD) of 15 mg L-1. The values found with the smartphone-based DIC procedure are in good agreement with spectrophotometric (spectrophotometer and microplate treader) and reference method (mid-infrared spectroscopy) values. This proposed approach offers an accessible and efficient solution for digital image colorimetry, with potential applications for various target analytes in milk and other fields requiring high-throughput colorimetric analysis.

Privacy preserving image sensor based visible light positioning receiver utilizing an ultra-flat singlet lens with high field-of-view and high aperture

Visible Light Positioning (VLP) is a promising candidate to enable widespread availability of location-based services in indoor environments. Existing VLP receiver designs achieve satisfactory accuracy, but often neglect cost-effectiveness and size constraints. To address this, we designed an ultra-flat singlet lens with a height of only 20 μm to be utilized in a compact image sensor-based VLP receiver. This specially designed lens brings the advantage of cost-effective manufacturing, high aperture, high field-of-view and short focal length. Compared to conventional cameras used in VLP systems, the presented optical receiver can be used in scenarios where privacy concerns play a critical role, like in wearable devices or in privacy-sensitive environments, as no perceivable image of the scene or the user is formed on the image sensor. With two separate VLP approaches, image processing based triangulation and machine learning assisted fingerprinting, we demonstrate positioning accuracy under 10 cm in an experimental setup. Furthermore, the demonstrator is integrated in an autonomously driving robot that has to avoid a geofence, exemplifying a location-based service.

Learning image representations for anomaly detection: Application to discovery of histological alterations in drug development

We present a system for anomaly detection in histopathological images. In histology, normal samples are usually abundant, whereas anomalous (pathological) cases are scarce or not available. Under such settings, one-class classifiers trained on healthy data can detect out-of-distribution anomalous samples. Such approaches combined with pre-trained Convolutional Neural Network (CNN) representations of images were previously employed for anomaly detection (AD). However, pre-trained off-the-shelf CNN representations may not be sensitive to abnormal conditions in tissues, while natural variations of healthy tissue may result in distant representations. To adapt representations to relevant details in healthy tissue we propose training a CNN on an auxiliary task that discriminates healthy tissue of different species, organs, and staining reagents. Almost no additional labeling workload is required, since healthy samples come automatically with aforementioned labels. During training we enforce compact image representations with a center-loss term, which further improves representations for AD. The proposed system outperforms established AD methods on a published dataset of liver anomalies. Moreover, it provided comparable results to conventional methods specifically tailored for quantification of liver anomalies. We show that our approach can be used for toxicity assessment of candidate drugs at early development stages and thereby may reduce expensive late-stage drug attrition.

Charge Management Enables Efficient Spontaneous Chromatic Adaptation Bipolar Photodetector.

Solution-processed photodetectors have emerged as promising candidates for next-generation of visible-near infrared (vis-NIR) photodetectors. This is attributed to their ease of processing, compatibility with flexible substrates, and the ability to tune their detection properties by integrating complementary photoresponsive semiconductors. However, the limited performance continues to hinder their further development, primarily influenced by the difference of charge transport properties between perovskite and organic semiconductors. In this work, a perovskite-organic bipolar photodetectors (PDs) is introduced with multispectral responsivity, achieved by effectively managing charges in perovskite and a ternary organic heterojunction. The ternary heterojunction, incorporating a designed NIR guest acceptor, exhibits a faster charge transfer rate and longer carrier diffusion length than the binary heterojunction. By achieving a more balanced carrier dynamic between the perovskite and organic components, the PD achieves a low dark current of 3.74nAcm-2 at -0.2V, a fast response speed of <10 µs, and a detectivity of exceeding 1012 Jones. Furthermore, a bioinspired retinotopic system for spontaneous chromatic adaptation is achieved without any optical filter. This charge management strategy opens up possibilities for surpassing the limitations of photodetection and enables the realization of high-purity, compact image sensors with exceptional spatial resolution and accurate color reproduction.

A 0.5 MP, 3D-Stacked, Voltage-Domain Global Shutter Image Sensor with NIR QE Enhancement, Event Detection Modes, and 90 dB Dynamic Range

We introduce a compact voltage-domain global shutter CMOS image sensor for a wide range of applications including consumer, IoT, and industrial applications. With 0.5 MP and 2.79 µm pixels packed in a die size of only 2.3 mm × 2.8 mm, the sensor achieves more than 92% quantum efficiency (QE) in the visible wavelength and more than 36% at near-infrared (940 nm) all while drawing a mere 20 mW at a 10-bit, 30-frame-per-second operational mode. In this article, we focus on the architecture of the sensor and the design challenges encountered to fit all the necessary circuitry in such a limited footprint. Moreover, we detail the new solutions we have developed to meet the demanding specifications of low-power operation and high dynamic range (HDR).

Evolutionary multi-objective design of autoencoders for compact representation of histopathology whole slide images

The recent success of deep learning coincides with the surge of digital pathology as an effect of the pandemic. However, processing a massive volume of gigapixel histopathology images is daunting. Hence, establishing compact image representations for efficient computational pathology becomes paramount. Deep autoencoders have been frequently used to compress feature vectors extracted from pre-trained networks. However, the topology of autoencoders is generally set heuristically. In this paper, we propose a multi-objective evolutionary framework as a bi-level optimization scheme in which the outer level evolves autoencoders for the goal of dimension reduction while the inner level is to optimize their weights for training. Throughout the evolution process, multiple minimization objectives such as complexity, classification error, and code size are considered. These objectives can obtain compressed models to generate a very small set of deep features with the highest classification accuracy. We use images from The Cancer Genome Atlas (TCGA) repository provided by the National Cancer Institute to train and validate our optimization framework. We increased the classification accuracy by 8% while the image representation is 46,000 times shorter. We apply the framework to demonstrate higher accuracy, substantial memory reduction and faster processing for image retrieval compared to SOTA.

On polylinear differential realization of the determined dynamic chaos in the class of higher order equations with delay

The investigation has defined the characteristic criterion (and its modification) of solvability of the problem of differential realization of the bundle of controlled trajectory curves of determined chaotic dynamic processes in the class of bilinear non-autonomous ordinary second- and higher-order differential equations (with and without delay) in the separable Hilbert space. The problem statement under consideration belongs to the type of converse problems for the additive combination of nonstationary linear and bilinear operators of the evolution equation in the Hilbert space. The constructions of tensor products of the Hilbert spaces, structures of lattices with an orthocomplement, the theory of extension of M2 -operators and the functional apparatus of the entropy Relay Ritz operator represent the basis of this theory. It has been shown that in the case of the finite bundle of the controlled trajectory curves the existence of the property of sub-linearity of the given operator allows one to obtain sufficient conditions of existence of such realizations. Side by side with solving the main problems, grounded are topological-group conditions of continuity of projectivization of the Relay Ritz operator with computing the fundamental group (Poincare group) of its compact image. The results obtained give incentives for the development of the quantitative theory of converse problems of higher-order multilinear evolution equations with the operators of generalized delay describing, for example, differential modeling of nonlinear Van der Pol oscillators or Lorentz strange attractors.

Software-defined meta-optics

Rapid advancements in autonomous systems and the Internet of Things have necessitated the development of compact and low-power image sensors to bridge the gap between the digital and physical world. To that end, sub-wavelength diffractive optics, commonly known as meta-optics, have garnered significant interest from the optics and photonics community due to their ability to achieve multiple functionalities within a small form factor. Despite years of research, however, the performance of meta-optics has often remained inferior compared to that of traditional refractive optics. In parallel, computational imaging techniques have emerged as a promising path to miniaturize optical systems, albeit often at the expense of higher power and latency. The lack of desired performance from either meta-optical or computational solutions has motivated researchers to look into a jointly optimized meta-optical–digital solution. While the meta-optical front end can preprocess the scene to reduce the computational load on the digital back end, the computational back end can in turn relax requirements on the meta-optics. In this Perspective, we provide an overview of this up-and-coming field, termed here as “software-defined meta-optics.” We highlight recent contributions that have advanced the current state of the art and point out directions toward which future research efforts should be directed to leverage the full potential of subwavelength photonic platforms in imaging and sensing applications. Synergistic technology transfer and commercialization of meta-optic technologies will pave the way for highly efficient, compact, and low-power imaging systems of the future.

Fuzzy-based cross-image pixel contrastive learning for compact medical image segmentation

Fuzzy-based cross-image pixel contrastive learning for compact medical image segmentation

Compact imaging system and deep learning based segmentation for objective longissimus muscle area in Korean beef carcass

With growing consumer interest in meat quality, the need for accurate quality assessment becomes increasingly important. One crucial factor of Korean beef quality is the longissimus muscle area, which is closely associated with both quality and yield grade. Currently, the measurement is visually assessed, introducing subjectivity and placing a substantial burden on inspectors in terms of labor. To address these challenges, we have developed a compact image acquisition system designed to acquire accurate grading assessment images of beef carcasses. Several preprocessing steps after image acquisition were conducted, including radial distortion correction and color calibration. We have employed conventional image-processing techniques and four deep-learning models to segment the longissimus muscle area using the calibrated images. Among the segmentation models, DeepLab model based on ResNet50 achieved the highest accuracy. It demonstrated a Global Accuracy, Weighted IoU, and Mean BF Score of approximately 99.26%, 98.54%, and 95.70%, respectively. The results of our study are expected to contribute to the development of objective criteria for loin area assessment. By enabling precise and consistent determination of beef carcass quality, our research has the potential to reduce labor requirements for inspectors and provide a standardized approach to assessing loin area.

Towards Compact Single Image Dehazing via Task-related Contrastive Network

Single image dehazing is a challenging vision task that recovers haze-free images from observed hazy images. Recently, numerous learning-based dehazing methods have been proposed and achieved promising performance. However, most of them suffer from a heavy computation burden, and even worse, they cannot leverage negative-orient supervision information well in the training stage. To address the above issues, we propose a novel dehazing method called Task-related Contrastive Network (TC-Net). For a better trade-off between performance and parameters, we design a compact dehazing network based on autoencoder-like architecture. It mainly includes two key modules: the feature enhanced module and the attention fusion module, which can improve the feature representation capability of the network and preserve details information, respectively. More importantly, we propose a task-related contrastive learning framework to fully exploit the negative-orient supervision information. To be specific, we utilize various task-specific data augmentation approaches (e.g., blur, sharpening, color, and light enhancement) to generate informative positive samples and hard negative samples, respectively. Furthermore, we employ an efficient and task-friendly feature embedding network, i.e., the encoder of the dehazing pipeline instead of the pre-trained model, to encode augmented samples into the latent space where negative-orient supervision information can be fully leveraged by contrastive constraint. Extensive experiments demonstrate that our TC-Net can obtain remarkable performance compared with other state-of-the-art dehazing methods, by extreme PSNR gains of 1.75 dB and 0.52 dB, on SOTS and Dense-Haze datasets, respectively.

Image reconstruction using superpixel clustering and tensor completion

This paper presents a pixel selection method for compact image representation based on superpixel segmentation and tensor completion. Our method divides the image into several regions that capture important textures or semantics and selects a representative pixel from each region to store. We experiment with different criteria for choosing the representative pixel and find that the centroid pixel performs the best. We also propose two smooth tensor completion algorithms that can effectively reconstruct different types of images from the selected pixels. Our experiments show that our superpixel-based method achieves better results than uniform sampling for various missing ratios.

Compact Image Transformer Based on Convolutional Variational Autoencoder with Augmented Attention Backbone for Target Recognition in Infrared Images

Compact Image Transformer Based on Convolutional Variational Autoencoder with Augmented Attention Backbone for Target Recognition in Infrared Images