Image Aspect Ratio Research Articles

Robust Estimation of Crowd Density Using Vision Transformers

Estimating and predicting crowd density at large events or during disasters is of paramount importance for enhancing emergency management systems. This includes planning effective evacuation routes, optimizing rescue operations, and ensuring efficient deployment of emergency services. Traditionally, surveillance systems that rely on cameras have been employed to monitor crowd movements. However, accurately estimating crowd density using such systems presents several challenges. These challenges stem primarily from the interaction between large crowds and the limitations of two-dimensional cameras in capturing the full scope of three-dimensional spaces. Optical distortions, environmental factors, and variations in camera angles further complicate the task, making accurate estimations difficult to achieve. To address these challenges, this paper introduces a robust method for calculating crowd density that leverages advanced vision transformers. By combining the output of these transformers with a two-stage neural network, the method effectively mitigates the limitations of traditional approaches. One of the key advantages of the proposed system is its robustness, which allows it to perform well across different camera specifications, installation locations, and image aspect ratios. The method applies and evaluates various deep learning techniques, introducing improvements to existing network structures that are better suited for the problem at hand. Extensive experimental verification demonstrates that the proposed method consistently produces accurate crowd density estimates, even in diverse and complex crowd environments. This robust performance underscores its potential for improving emergency management and crowd control in real-world situations.

Elderly Fall Detection in Complex Environment Based on Improved YOLOv5s and LSTM

This work was conducted mainly to provide a healthy and safe monitoring system for the elderly living in the home environment. In this paper, two different target fall detection schemes are proposed based on whether the target is visible or not. When the target is visible, a vision-based fall detection algorithm is proposed, where an image of the target captured by a camera is transmitted to the improved You Only Look Once version 5s (YOLOv5s) model for posture detection. In contrast, when the target is invisible, a WiFi-based fall detection algorithm is proposed, where channel state information (CSI) signals are used to estimate the target’s posture with an improved long short-term memory (LSTM) model. In the improved YOLOv5s model, adaptive picture scaling technology named Letterbox is used to maintain consistency in the aspect ratio of images in the dataset, and the weighted bidirectional feature pyramid (BiFPN) and the attention mechanisms of squeeze-and-excitation (SE) and coordinate attention (CA) modules are added to the Backbone network and Neck network, respectively. In the improved LSTM model, the Hampel filter is used to eliminate the noise from CSI signals and the convolutional neural network (CNN) model is combined with the LSTM to process the image made from CSI signals, and thus the object of the improved LSTM model at a point in time is the analysis of the amplitude of 90 CSI signals. The final monitoring result of the health status of the target is the result of combining the fall detection of the improved YOLOv5s and LSTM models with the physiological information of the target. Experimental results show the following: (1) the detection precision, recall rate, and average precision of the improved YOLOv5s model are increased by 7.2%, 9%, and 7.6%, respectively, compared with the original model, and there is almost no missed detection of the target; (2) the detection accuracy of the improved LSTM model is improved by 15.61%, 29.36%, and 52.39% compared with the original LSTM, CNN, and neural network (NN) models, respectively, while the convergence speed is improved by 90% compared with the original LSTM model; and (3) the proposed algorithm can meet the requirements of accurate, real-time, and stable applications of health monitoring.

Machine/deep learning-assisted hemoglobin level prediction using palpebral conjunctival images.

Palpebral conjunctival hue alteration is used in non-invasive screening for anaemia, whereas it is a qualitative measure. This study constructed machine/deep learning models for predicting haemoglobin values using 150 palpebral conjunctival images taken by a smartphone. The median haemoglobin value was 13.1 g/dL, including 10 patients with <11 g/dL. A segmentation model using U-net was successfully constructed. The segmented images were subjected to non-convolutional neural network (CNN)-based and CNN-based regression models for predicting haemoglobin values. The correlation coefficients between the actual and predicted haemoglobin values were 0.38 and 0.44 in the non-CNN-based and CNN-based models, respectively. The sensitivity and specificity for anaemia detection were 13% and 98% for the non-CNN-based model and 20% and 99% for the CNN-based model. The performance of the CNN-based model did not improve with a mask layer guiding the model's attention towards the conjunctival regions, however, slightly improved with correction by the aspect ratio and exposure time of input images. The gradient-weighted class activation mapping heatmap indicated that the lower half area of the conjunctiva was crucial for haemoglobin value prediction. In conclusion, the CNN-based model had better results than the non-CNN-based model. The prediction accuracy would improve by using more input data with anaemia.

Theme-Aware Aesthetic Distribution Prediction With Full-Resolution Photographs.

Aesthetic quality assessment (AQA) is a challenging task due to complex aesthetic factors. Currently, it is common to conduct AQA using deep neural networks (DNNs) that require fixed-size inputs. The existing methods mainly transform images by resizing, cropping, and padding or use adaptive pooling to alternately capture the aesthetic features from fixed-size inputs. However, these transformations potentially damage aesthetic features. To address this issue, we propose a simple but effective method to accomplish full-resolution image AQA by combining image padding with region of image (RoM) pooling. Padding turns inputs into the same size. RoM pooling pools image features and discards extra padded features to eliminate the side effects of padding. In addition, the image aspect ratios are encoded and fused with visual features to remedy the shape information loss of RoM pooling. Furthermore, we observe that the same image may receive different aesthetic evaluations under different themes, which we call the theme criterion bias. Hence, a theme-aware model that uses theme information to guide model predictions is proposed. Finally, we design an attention-based feature fusion module to effectively use both the shape and theme information. Extensive experiments prove the effectiveness of the proposed method over state-of-the-art methods.

Distance estimation technique from 360-degree images in built-in environments

The present study introduces a novel approach for quantifying distances within constructed environments. A mathematical model was developed for distance estimation in image processing using width and height estimation. In order to determine distance, the study employed the use of visual angle and sky view factor (SVF). Additionally, a camera with capabilities similar to the human eye was utilized to capture 360-degree photographs from a fixed position within a virtual reality corridor. The technique of Sky View Factor (SVF) is employed in indoor environments with ceilings by eliminating windows, doors, and roofs, thereby simulating a virtual sky. This enables the calculation of various parameters such as the image's area, area fraction, and aspect ratio through the utilization of image processing methods. Distance estimation can be predicted through the utilization of the sky view factor and visual angle, employing a linear regression analysis. The method of virtual sky view factor (VSVF) has potential applications in the fields of Engineering, robotics, and architecture for the estimation of indoor distances.

Pre-Activating Semantic Information for Image Aesthetic Assessment

Automatic image aesthetic evaluation is an attractive and challenging visual task. Recently, methods based on convolutional neural networks have achieved remarkable performance. However, semantic information, an intuitive prerequisite for evaluating image aesthetics, has not received enough attention regarding its importance in previous methods. How to efficiently extract semantic information and make better use of it to assist the aesthetic evaluation task remains unsolved. In this article, we propose to utilize the self-supervised model Auto-Encoder to extract semantic information in the form of multi-task learning. Then, a fusing module is prepended at the bottleneck layer to explicitly combine semantic information with aesthetic information in a pre-activated manner. Specifically, we implement a customized pooling operation to pool the semantic features extracted by Auto-Encoder and apply a weak constraint between the pooled semantic features and aesthetic information to realize the combination. The following regressor can complete aesthetic evaluation based on the semantic–aesthetic combined features. In addition, to enable our model to adapt to arbitrary aspect ratios of images, another pooling strategy called spatial pyramid pooling is adopted to obtain the image features of a fixed length. Our method achieves competitive performance on the public image aesthetic evaluation benchmark. Especially on the most commonly used metric Spearman rank-order correlation coefficient, the proposed model achieved the best performance compared with some state-of-the-art methods. Extensive ablation studies and visualization experiments were conducted to demonstrate the effectiveness of our method.

Image retargeting quality assessment: A survey

With the increasing variety of display devices, image retargeting has become an indispensable technology for adjusting the aspect ratio of images to adapt to different display terminals. Since the retargeting operation would cause geometric distortion and content loss of the image, the image retargeting quality assessment (IRQA) is necessary to guide the retargeting algorithm’s optimization, selection, and design. Our paper mainly works for systematically reviewing the state-of-the-art technologies in IRQA. And then, this paper further discusses image registration algorithms for matching the original image and the retargeted image. Next, we investigate the feature measurement methods for image retargeting quality evaluation. To facilitate the quantitative assessment of the IRQA methods, this paper gives a list of publicly open datasets and the performance of the mainstream methods. Finally, some promising research directions towards IRQA are pointed out. From this survey, engineers from the industry may find skills to improve their image retargeting systems, and researchers from academia may find ideas to conduct some innovative work.

ARET-IQA: An Aspect-Ratio-Embedded Transformer for Image Quality Assessment

Image quality assessment (IQA) aims to automatically evaluate image perceptual quality by simulating the human visual system, which is an important research topic in the field of image processing and computer vision. Although existing deep-learning-based IQA models have achieved significant success, these IQA models usually require input images with a fixed size, which varies the perceptual quality of images. To this end, this paper proposes an aspect-ratio-embedded Transformer-based image quality assessment method, which can implant the adaptive aspect ratios of input images into the multihead self-attention module of the Swin Transformer. In this way, the proposed IQA model can not only relieve the variety of perceptual quality caused by size changes in input images but also leverage more global content correlations to infer image perceptual quality. Furthermore, to comprehensively capture the impact of low-level and high-level features on image quality, the proposed IQA model combines the output features of multistage Transformer blocks for jointly inferring image quality. Experimental results on multiple IQA databases show that the proposed IQA method is superior to state-of-the-art methods for assessing image technical and aesthetic quality.

Image aesthetics prediction using multiple patches preserving the original aspect ratio of contents

The spread of social networking services has created an increasing demand for selecting, editing, and generating impressive images. This trend increases the importance of evaluating image aesthetics as a fundamental function of automatic image processing. However, most existing methods for aesthetics score prediction require image rescaling for input, which can affect the prediction, especially for images with unusual aspect ratios. We propose a multi-patch method, called a multi-patch aggregation network (MPA-Net), to predict image aesthetics scores by maintaining the original aspect ratios of the contents in the images. One of our key contributions is the adoption of an equal-interval multi-patch selection approach for the prediction of the aesthetics score. The effectiveness of our strategy is shown through experiments involving the large-scale AVA dataset. Our MPA-Net outperformed the reported scores of the baseline methods and achieved a better performance in terms of the mean square error (MSE) than the state-of-the-art end-to-end continuous aesthetics score prediction methods. Most notably, MPA-Net yields a significantly lower MSE particularly for images with aspect ratios far from 1.0, indicating that MPA-Net is useful for a wide range of image aspect ratios. Moreover, MPA-Net has several benefits for training and evaluation procedures. MPA-Net meets the conditions of end-to-end learning and mini-batch learning simultaneously, and MPA-Net uses only images that do not require external information during the training or prediction stages. Thus, our easy-to-handle method improves the prediction of image aesthetics scores, outstandingly for images with extraordinary aspect ratios.

Towards More Efficient Security Inspection via Deep Learning: A Task-Driven X-ray Image Cropping Scheme.

X-ray imaging machines are widely used in border control checkpoints or public transportation, for luggage scanning and inspection. Recent advances in deep learning enabled automatic object detection of X-ray imaging results to largely reduce labor costs. Compared to tasks on natural images, object detection for X-ray inspection are typically more challenging, due to the varied sizes and aspect ratios of X-ray images, random locations of the small target objects within the redundant background region, etc. In practice, we show that directly applying off-the-shelf deep learning-based detection algorithms for X-ray imagery can be highly time-consuming and ineffective. To this end, we propose a Task-Driven Cropping scheme, dubbed TDC, for improving the deep image detection algorithms towards efficient and effective luggage inspection via X-ray images. Instead of processing the whole X-ray images for object detection, we propose a two-stage strategy, which first adaptively crops X-ray images and only preserves the task-related regions, i.e., the luggage regions for security inspection. A task-specific deep feature extractor is used to rapidly identify the importance of each X-ray image pixel. Only the regions that are useful and related to the detection tasks are kept and passed to the follow-up deep detector. The varied-scale X-ray images are thus reduced to the same size and aspect ratio, which enables a more efficient deep detection pipeline. Besides, to benchmark the effectiveness of X-ray image detection algorithms, we propose a novel dataset for X-ray image detection, dubbed SIXray-D, based on the popular SIXray dataset. In SIXray-D, we provide the complete and more accurate annotations of both object classes and bounding boxes, which enables model training for supervised X-ray detection methods. Our results show that our proposed TDC algorithm can effectively boost popular detection algorithms, by achieving better detection mAPs or reducing the run time.

Automatic Liver Segmentation in CT Images with Enhanced GAN and Mask Region-Based CNN Architectures.

Liver image segmentation has been increasingly employed for key medical purposes, including liver functional assessment, disease diagnosis, and treatment. In this work, we introduce a liver image segmentation method based on generative adversarial networks (GANs) and mask region-based convolutional neural networks (Mask R-CNN). Firstly, since most resulting images have noisy features, we further explored the combination of Mask R-CNN and GANs in order to enhance the pixel-wise classification. Secondly, k-means clustering was used to lock the image aspect ratio, in order to get more essential anchors which can help boost the segmentation performance. Finally, we proposed a GAN Mask R-CNN algorithm which achieved superior performance in comparison with the conventional Mask R-CNN, Mask-CNN, and k-means algorithms in terms of the Dice similarity coefficient (DSC) and the MICCAI metrics. The proposed algorithm also achieved superior performance in comparison with ten state-of-the-art algorithms in terms of six Boolean indicators. We hope that our work can be effectively used to optimize the segmentation and classification of liver anomalies.

Ratio-and-Scale-Aware YOLO for Pedestrian Detection.

Current deep learning methods seldom consider the effects of small pedestrian ratios and considerable differences in the aspect ratio of input images, which results in low pedestrian detection performance. This study proposes the ratio-and-scale-aware YOLO (RSA-YOLO) method to solve the aforementioned problems. The following procedure is adopted in this method. First, ratio-aware mechanisms are introduced to dynamically adjust the input layer length and width hyperparameters of YOLOv3, thereby solving the problem of considerable differences in the aspect ratio. Second, intelligent splits are used to automatically and appropriately divide the original images into two local images. Ratio-aware YOLO (RA-YOLO) is iteratively performed on the two local images. Because the original and local images produce low- and high-resolution pedestrian detection information after RA-YOLO, respectively, this study proposes new scale-aware mechanisms in which multiresolution fusion is used to solve the problem of misdetection of remarkably small pedestrians in images. The experimental results indicate that the proposed method produces favorable results for images with extremely small objects and those with considerable differences in the aspect ratio. Compared with the original YOLOs (i.e., YOLOv2 and YOLOv3) and several state-of-the-art approaches, the proposed method demonstrated a superior performance for the VOC 2012 comp4, INRIA, and ETH databases in terms of the average precision, intersection over union, and lowest log-average miss rate.

Electrical equipment identification in infrared images based on ROI-selected CNN method

The thermographic evaluation method is widely used in substations to detect potential faults in advance. However, thousands of infrared images are expected for each substation. Almost all the data is collected, sorted and analyzed by humans. Therefore, to automate the data management and diagnosis, this paper proposes a method based on the deep convolutional neural network (CNN) for infrared image recognition of electrical equipment. Firstly, a deep CNN identification model is built based on MobileNet with initial weights in ImageNet. In addition, combined with the dataset augmentation including cropping, flipping, rotation and zoom, 3547 images of 500 kV substation equipment are used for training. Besides, a fine-tuning method is adopted to optimize the training. Finally, a fast region of interest (ROI) selection method based on hotspot sensitivity in infrared images is used to improve the identification accuracy. The original image aspect ratio is fixed throughout the process. Results demonstrate that the prediction accuracy of the proposed method reaches 97.72% in validation, and the ROI selection method improves the confidence by 8% in the test. As a result, this proposed method can promote the calculation efficiency, and has good application prospects in embedded devices such as cameras and substation robots.

Mining discriminative patches for script identification in natural scene images

This paper focuses on script identification in natural scene images. Traditional CNNs (Convolution Neural Networks) cannot solve this problem perfectly for two reasons: one is the arbitrary aspect ratios of scene images which bring much difficulty to traditional CNNs with a fixed size image as the input. And the other is that some scripts with minor differences are easily confused because they share a subset of characters with the same shapes. We propose a novel approach combing Score CNN, Attention CNN and patches. Attention CNN is utilized to determine whether a patch is a discriminative patch and calculate the contribution weight of the discriminative patch to script identification of the whole image. Score CNN uses a discriminative patch as input and predict the score of each script type. Firstly patches with the same size are extracted from the scene images. Secondly these patches are used as inputs to Score CNN and Attention CNN to train two patch-level classifiers. Finally, the results of multiple discriminative patches extracted from the same image via the above two classifiers are fused to obtain the script type of this image. Using patches with the same size as inputs to CNN can avoid the problems caused by arbitrary aspect ratios of scene images. The trained classifiers can mine discriminative patches to accurately identify some confusing scripts. The experimental results show the good performance of our approach on four public datasets.

Perception-guided multi-channel visual feature fusion for image retargeting

Image retargeting aims to arbitrarily change the aspect ratio of images with minimal visual artifacts, that is, preserving salient regions within the image. Conventional approaches have achieved impressive performance, however, only low-level features are exploited. Besides, these approaches do not take human visual system into account. To solve these problems, we propose a perception-guided multi-channel visual feature fusion method for image retargeting, where human visual perception is well encoded. Specifically, we first construct a series of graphlets which represent salient regions that might attract human attention, where both low-level and high-level features are utilized to describe graphlets. Considering that only a few regions can attract human attention when observing the image, we engineer a sparsity-constraint algorithm to select these regions, which will be further concatenated to form human gaze shifting paths (GSPs). Subsequently, we design a statistic-based CNN architecture to extract the deep representation of GSPs. Afterward, a probabilistic model is built to learn the priors of GSPs and the learned probabilistic model will guide image retargeting. A test image whose distribution is similar to the learned probabilistic model is deemed as the aesthetic image and thus should be shrunk slightly. Comprehensive experiments demonstrate the effectiveness and robustness of the proposed method.

Aberration-corrected holographic projection on a two-dimensionally highly tilted spatial light modulator.

Image projection by holographic allows efficient and compact optical setups; nevertheless, the limited throw angle and 1:1 image aspect ratio are impractical. We present the method to increase the diffractive angle of a spatial light modulator in one and two directions by introducing the highly 2-dimensionally tilted illuminating beam. The inevitable image aberrations, such as astigmatism, for off-axis imaging are corrected by proper modifications of the phase patterns on the modulator. Experimental results show the image aspect ratio of 2.4:1 suitable for human vision, with sustained image contrast and noise level. Study of the experimental diffractive efficiency is also presented.

Quantitative comparisons of ultra-widefield images of model eye obtained with Optos\xae 200Tx and Optos\xae California

BackgroundTo compare the quality of the ultra-widefield images acquired by Optos® 200Tx to those acquired by Optos® California.MethodsImages of the posterior surface of a Gullstrand’s model eye obtained by Optos® 200Tx were compared to those obtained by Optos® California in terms of the angular field of view and the symmetry of the image, i.e., vertical and horizontal aspect ratios at the center (0°) and at the periphery (40° and 80°) in each direction. In addition, we compared the enlargement of the image on the posterior surface as the square ratio, and the differences in the contrasts.ResultsNo significant differences were detected in the angular field of view between the two instruments. The aspect ratios showed that the Optos® California had more symmetrical images than the Optos® 200Tx at the center (0.98 vs 0.89, P < 0.01) and at the 40° periphery (0.93–1.04, △0.11 vs 0.79–1.01, △0.22) and 80° periphery (0.81–1.25, △0.44 vs 0.42–1.12, △0.70) in each direction. The amplitude of the square ratio of the Optos® California was smaller at 40° periphery (1.16–1.28, △0.12 vs 1.06–1.37, △0.31) and 80° periphery (2.12–2.46, △0.34 vs 1.14–3.29, △2.15). The contrast of the Optos® California images was significantly higher in the posterior pole (0.09 vs 0.12, P < 0.01), upper (0.07 vs 0.03, P < 0.01), and right (0.12 vs 0.07, P < 0.01) peripheries.ConclusionOptos® California can record equal angular widefield images to Optos® 200Tx and more symmetrical images with higher contrast in the posterior pole, upper and right peripheries.

A Content-Aware Image Retargeting Quality Assessment Method Using Foreground and Global Measurement

Image retargeting methods aim to minimize the perceptual loss while changing sizes and aspect ratios of images. Since optimal retargeting methods for different images are generally not the same, the image retargeting quality assessment (IRQA) becomes a meaningful task. This paper proposes a content-aware image retargeting quality assessment method using foreground and global measurement to achieve better performance. In our proposed method, images are first divided into two categories according to the foreground object detection result, and then different corresponding measurements are designed for them. For those with obvious foreground object, both foreground and global measurement are applied. For others, only global measurement is conducted. Foreground measurement includes two complementary features: the high-level semantic similarity feature and the low-level size ratio feature. Global measurement includes another two features: an improved aspect ratio similarity (ARS) feature and edge group similarity (EGS) feature. Two public databases, i.e., the RetargetMe and CUHK, have been evaluated, and experimental results demonstrate that our method is quite effective, and it also provides state-of-the-art performance in the IRQA. aa Our code are available at https://github.com/SCUT-ML-GUO/IRQA.

Redox-Magnetohydrodynamically Controlled Fluid Flow with Poly(3,4-ethylenedioxythiophene) Coupled to an Epitaxial Light Sheet Confocal Microscope for Image Cytometry Applications.

We present the merging of two technologies to perform continuous high-resolution fluorescence imaging of cellular suspensions in a deep microfluidics chamber with no moving parts. An epitaxial light sheet confocal microscope (e-LSCM) was used to image suspensions enabled by fluid transport via redox-magnetohydrodynamics (R-MHD). The e-LSCM features a linear solid state sensor, oriented perpendicular to the direction of flow, that can bin the emission across different numbers of pixels, yielding electronically adjustable optical sectioning. This, in addition to intensity thresholding, defines the axial resolution, which was validated with an optical phantom of polystyrene microspheres suspended in agarose. The linear fluid speed within the microfluidics chamber was uniform (0.16-2.9%) across the 0.5-1.0 mm lateral field of view (dependent upon the chosen magnification) with continuous acquisition. Also, the camera's linear exposure periods were controlled to ensure an accurate image aspect ratio across this span. Poly(3,4-ethylenedioxythiophene) (PEDOT) was electrodeposited as an immobilized redox film on electrodes of a chip for R-MHD, and the fluid flow was calibrated to specific linear speeds as a function of applied current. Images of leukocytes stained with acridine orange, a fluorescent, amphipathic vital dye that intercalates DNA, were acquired in the R-MHD microfluidics chamber with the e-LSCM to demonstrate imaging of biological samples. The combination of these technologies provides a miniaturizable platform for large sample volumes and high-throughput, image-based analysis without the requirement of moving parts, enabling development of robust, point-of-care image cytometry.

3D Object Classification Based on Multi Convolutional Neural Networks

With the development of 3D sensing technology in recent years, making it possible to obtain high quality color image as well as its depth information, the combination of both can improve the accuracy of image classification. Traditional methods are mostly based on well-designed features separately for color image or depth image, which does not consider relation-features between color and depth information. A 3D object classification approach based on multi convolutional neural networks is presented in this paper. Our approach consists of three parts: (1) Two resizing methods for keeping image aspect ratio were proposed, which can keep the aspect ratio information of the original image, besides, data was augmented by applying both methods. (2) Different from traditional methods, CNN (Convolutional Neural Network) was used to extract features from both color image and depth image. (3) Consider on the relation-features about color and depth, we designed a multi CNN which focuses on relation-features and trained it successful, that is, we could extract relation-features using this network. Comparing with current approaches, our experiment results show better performance on 3D object classification.