Rectilinear Images Research Articles

Omnidirectional image quality assessment with local–global vision transformers

With the rising popularity of omnidirectional images (ODIs) in virtual reality applications, the need for specialized image quality assessment (IQA) methods becomes increasingly critical. Traditional IQA approaches, designed for rectilinear images, often fail to evaluate ODIs accurately due to their 360-degree scene representation. Addressing this, we introduce the Local–Global Transformer for 360-degree Image Quality Assessment (LGT360IQ). This novel framework features dual branches tailored to mimic top-down and bottom-up visual attention mechanisms, adapted for the spherical characteristics of ODIs. The local branch processes tangent viewports from salient regions within the equirectangular projection image, extracting detailed features for granular quality assessment. In parallel, the global branch utilizes a task-dependent token sampling strategy for holistic image feature processing and quality score prediction. This integrated approach combines local and global information, offering an effective IQA method for ODIs. Our extensive evaluation across three benchmark ODI datasets, CVIQ, OIQA, and ODI, demonstrates LGT360IQ superior performance and establishes its role in advancing the field of IQA for omnidirectional images.

Intra- and Inter-Reasoning Graph Convolutional Network for Saliency Prediction on 360° Images

Cubic projection can be utilized to divide 360° images into multiple rectilinear images, with little distortion. However, the existing saliency prediction models fail to integrate semantic information of these images. In this paper, we address this by proposing an intra- and inter-reasoning graph convolutional network for saliency prediction on 360° images (SalReGCN360). The whole framework contains six sub-networks, each of which contains two branches. In the training phase, after utilizing Multiple Cubic Projection (MCP), six rectilinear images are simultaneously put into corresponding sub-networks. In one of the branches, the global features of a single rectilinear image are extracted by the intra-graph inference module to finely predict local saliency of 360° images. In the other branch, the contextual features are extracted by the inter-graph inference module to effectively integrate semantic information of six rectilinear images. Finally, the feature maps are generated by the two branches fusion, and six corresponding rectilinear saliency maps are predicted. Extensive experiments on two popular saliency datasets illustrate the superiority of the proposed model, especially the improvement in KLD metric.

Metric Rectification of Spherical Images

This paper describes a method for metric recording based on spherical images, which are rectified to document planar surfaces. The proposed method is a multistep workflow in which multiple rectilinear images are (i) extracted from a single spherical projection and (ii) used to recover metric properties. The workflow is suitable for documenting buildings with small and narrow rooms, i.e., documentation projects where the acquisition of 360 images is faster than the traditional acquisition of several photographs. Two different rectification procedures were integrated into the current implementation: (i) an analytical method based on control points and (ii) a geometric procedure based on two sets of parallel lines. Constraints based on line parallelism can be coupled with the focal length of the rectified image to estimate the rectifying transformation. The calculation of the focal length does not require specific calibrations projects. It can be derived from the spherical image used during the documentation project, obtaining a rectified image with just an overall scale ambiguity. Examples and accuracy evaluation are illustrated and discussed to show the pros and cons of the proposed method.

Deep Learning and Bidirectional Optical Flow Based Viewport Predictions for 360° Video Coding

The rapid development of virtual reality applications continues to urge better compression of 360° videos owing to the large volume of content. These videos are typically converted to 2-D formats using various projection techniques in order to benefit from ad-hoc coding tools designed to support conventional 2-D video compression. Although recently emerged video coding standard, Versatile Video Coding (VVC) introduces 360° video specific coding tools, it fails to prioritize the user observed regions in 360° videos, represented by the rectilinear images called the viewports. This leads to the encoding of redundant regions in the video frames, escalating the bit rate cost of the videos. In response to this issue, this paper proposes a novel 360° video coding framework for VVC which exploits user observed viewport information to alleviate pixel redundancy in 360° videos. In this regard, bidirectional optical flow, Gaussian filter and Spherical Convolutional Neural Networks (Spherical CNN) are deployed to extract perceptual features and predict user observed viewports. By appropriately fusing the predicted viewports on the 2-D projected 360° video frames, a novel Regions of Interest (ROI) aware weightmap is developed which can be used to mask the source video and introduce adaptive changes to the Lagrange and quantization parameters in VVC. Comprehensive experiments conducted in the context of VVC Test Model (VTM) 7.0 show that the proposed framework can improve bitrate reduction, achieving an average bitrate saving of 5.85% and up to 17.15% at the same perceptual quality which is measured using Viewport Peak Signal-To-Noise Ratio (VPSNR).

3D Object Detection Algorithm for Panoramic Images With Multi-Scale Convolutional Neural Network

This paper addresses the challenge of 3D object detection from a single panoramic image under severe deformation. The advent of the two-stage approach has impelled significant progress in 3D object detection. However, most available methods only can localize region proposals by a single-scale architecture network, which are sensitive to deformation and distortion. To address this issue, we propose a multi-scale convolutional neural network (MSCNN) to estimate the 3D pose of an object. To be specific, the proposed MSCNN consists of three steps for effectively detecting the distorted object on the panoramic images. The MSCNN contains the CycleGAN network that converts rectilinear images into panoramas, a fused framework that improves both accuracy and speed for object detection, and an adversarial spatial transformer network (ASTN) that extracts the deformation features of the object on panoramic images. Additionally, we recover the 3D pose of the object using a coordinate projection and a 3D bounding box. Extensive experiments demonstrate that the proposed method can achieve a 3D detection accuracy of 38.7% in high-resolution panoramic images, which is higher than the current state-of-the-art algorithm of 5.2%. Moreover, the speed of detection is only about 0.6 seconds per image, which is six times faster than Faster R-CNN (COCO). The code will be available at https://github.com/Yanhui-He.

Curved features are critical for animate/inanimate categorization in macaques

In an earlier fMRI study, we showed that multivoxel activity patterns, measured with support vector machine classification, encoded animate vs. inanimate categories in the macaque inferior temporal cortex. However, the classification accuracy was reduced to chance after removing the variance in the fMRI activity patterns that was explained by the curved and rectilinear image features, as quantified using curved and rectilinear Gabor filters. These results indicate that categorization in the macaque inferior temporal cortex might not stem from acquired semantic knowledge of the characteristics that distinguish animate from inanimate object categories, but rather from the unique image-based features. The current experiment was designed to directly examine those two possibilities using behavioral tests. First, one rhesus macaque was trained to categorize images of animate and inanimate objects. Then, the same monkey was tested on a large number of trial-unique images of animate and inanimate objects across five days to assess whether this training generalized to unfamiliar objects. We found that the animal's classification accuracy for these unfamiliar objects averaged 84.55%, supporting our fMRI conclusion that animate/inanimate categorization does not stem from acquired semantic knowledge of animate vs. inanimate categories. We also tested whether image features that differ substantially between the two object categories, such as curvilinear and rectilinear information, contribute to the monkey's classification accuracy. The same animal was tested across five days on sets of synthetic animate and inanimate images created using an algorithm that significantly distorted the global shape of the original images, while maintaining the original images' intermediate features (e.g. curvilinear and rectilinear information). We found that the animal's classification accuracy was significantly above chance (63.57%), suggesting that unique image-based features, such as curvilinear features, distinguish animate from inanimate objects and give rise to the formation of animate/inanimate categorization in macaques to some extent. Meeting abstract presented at VSS 2018

GBVS360, BMS360, ProSal: Extending existing saliency prediction models from 2D to omnidirectional images

In this paper it is studied how existing visual saliency models designed for 2D images can be extended and applied to Omnidirectional images in the equirectangular format. Three different frameworks, BMS360, GBVS360 and Projected Saliency were designed to address this task. In the particular case of BMS360 and GBVS360, the 2D models Boolean Map Saliency (BMS) and Graph-Based Visual Saliency (GBVS) were rigorously extended to integrate the specific properties of equirectangular images to model visual attention in omnidirectional images watched using Head Mounted Displays (HMDs). With the proposed extensions, the saliency prediction performance in comparison to their original design could significantly be improved. Another key result of this paper comes from the design of a framework called projected saliency which allows applying existing saliency models on equirectangular images without requiring in-depth adaptation of the prediction algorithms. The projected saliency framework enabled to study the interaction of features in the feature activation process of computational saliency models. It shows that the activation should not be done per-viewport, but should account for neighboring regions. As a consequence, we argue that the feature activation process should be performed at a global level instead of a local one. Here, the use of a local approach with a large FOV was found to be a good compromise between global activation and appropriate feature computation to directly port existing saliency models designed for rectilinear images to the domain of equirectangular images. In this paper an adaptive equatorial prior is also described. This equatorial prior allows to account for pitch shift of the horizon line. Finally, results show that the newly designed BMS360 and GVBS360 outperform even recent state-of-the-art models at head motion prediction and are competitive at head/eye saliency-map prediction. Moreover, all models and their code are made publicly available on GitHub allowing to further iterate on this work.

Part-based vehicle detection in side-rectilinear images for blind-spot detection

The blind-spot detection (BSD) system is designed to prevent accidents during lane changing and overtaking scenarios. Current BSD systems that use side- or rear-view cameras suffer from limited performance because of the severe distortion in the appearance of nearby vehicles depending on their positions relative to the host vehicle. To overcome such limitations, this manuscript introduces a side-rectilinear image to detect and use the side parts of the vehicles. In the side-rectilinear image, the side parts of the vehicles do not contain radial or perspective distortions; consequently, the appearance of the tires remains identical from different positions on the vehicle. By utilizing this rectilinear image, a rear-camera-based BSD system that detects both vehicles and motorcycles is constructed to prevent possible accidents occurring in blind spots. The proposed BSD system detects the vehicles in three stages: tire hypothesis generation/verification, front-rear tire classification, and vehicle hypothesis generation/verification. For motorcycle detection, the proposed system detects the lower parts of the motorcycle, which are not affected by the appearance of the drivers and cargos. Using the property of the side-rectilinear image, the detection procedures of the proposed system are straight-forward and resemble the object detection/recognition rules of humans. Based on the detection results, the system tracks nearby vehicles and gives a warning signal to the driver when there are obstacles in blind spots. An evaluation of the system performance demonstrates that the warning rate of the proposed system outperforms that of radar-based systems.

A GENERALIZED NON-LINEAR METHOD FOR DISTORTION CORRECTION AND TOP-DOWN VIEW CONVERSION OF FISH EYE IMAGES

Advanced driver assistance systems (ADAS) have been developed to automate and modify vehicles for safety and better driving experience. Among all computer vision modules in ADAS, 360-degree surround view generation of immediate surroundings of the vehicle is very important, due to application in on-road traffic assistance, parking assistance etc. This paper presents a novel algorithm for fast and computationally efficient transformation of input fisheye images into required top down view. This paper also presents a generalized framework for generating top down view of images captured by cameras with fish-eye lenses mounted on vehicles, irrespective of pitch or tilt angle. The proposed approach comprises of two major steps, viz. correcting the fish-eye lens images to rectilinear images, and generating top-view perspective of the corrected images. The images captured by the fish-eye lens possess barrel distortion, for which a nonlinear and non-iterative method is used. Thereafter, homography is used to obtain top-down view of corrected images. This paper also targets to develop surroundings of the vehicle for wider distortion less field of view and camera perspective independent top down view, with minimum computation cost which is essential due to limited computation power on vehicles.

Affine Covariant Features for Fisheye Distortion Local Modeling.

Perspective cameras are the most popular imaging sensors used in computer vision. However, many application fields, including automotive, surveillance, and robotics, require the use of wide angle cameras (e.g., fisheye), which allow to acquire a larger portion of the scene using a single device at the cost of the introduction of noticeable radial distortion in the images. Affine covariant feature detectors have proved successful in a variety of computer vision applications, including object recognition, image registration, and visual search. Moreover, their robustness to a series of variabilities related to both the scene and the image acquisition process has been thoroughly studied in the literature. In this paper, we investigate their effectiveness on fisheye images providing both theoretical and experimental analyses. As theoretical outcome, we show that the inherently non-linear radial distortion can be locally approximated by linear functions with a reasonably small error. The experimental analysis builds on Mikolajczyk's benchmark to assess the robustness of three popular affine region detectors (i.e., maximally stable extremal regions, and Harris and Hessian affine region detectors), with respect to different variabilities as well as to radial distortion. To support the evaluations, we rely on the Oxford data set and introduce a novel benchmark data set comprising 50 images depicting different scene categories. Experiments are carried out on rectilinear images to which radial distortion is artificially added, and on real-world images acquired using fisheye lenses. Our analysis points out that affine region detectors can be effectively employed directly on fisheye images and that the radial distortion is locally modeled as an additional affine variability.

Reconstruction of Wide FOV Image from Hyperbolic Cylinder Mirror Camera

Abstract In order to contain as much information as possible in a single image, a wide FOV(Field-Of-View) imaging system is required. The catadioptric imaging system with hyperbolic cylinder mirror can acquire over 180 degree horizontal FOV realtime panorama image by using a conventional camera. Because the hyperbolic cylinder mirror has a curved surface in horizontal axis, the original image acquired from the imaging system has the geometrical distortion, which requires the image processing algorithm for reconstruction. In this paper, the image reconstruction algorithms for two cases are studied: (1) to obtain an image with uniform angular resolution and (2) to obtain horizontally rectilinear image. The image acquisition model of the hyperbolic cylinder mirror imaging system is analyzed by the geometrical optics and the image reconstruction algorithms are proposed based on the image acquisition model. To show the validity of the proposed algorithms, experiments are carried out and presented in this paper. The experimental results show that the reconstructed images have a uniform angular resolution and a rectilinear form in horizontal axis, which are natural to human.

Image Based Modeling from Spherical Photogrammetry and Structure for Motion. The Case of the Treasury, Nabatean Architecture in Petra

This research deals with an efficient and low cost methodology to obtain a metric and photorealstic survey of a complex architecture. Photomodeling is an already tested interactive approach to produce a detailed and quick 3D model reconstruction. Photomodeling goes along with the creation of a rough surface over which oriented images can be back-projected in real time. Lastly the model can be enhanced checking the coincidence between the surface and the projected texture. The challenge of this research is to combine the advantages of two technologies already set up and used in many projects: spherical photogrammetry (Fangi, 2007,2008,2009,2010) and structure for motion (Photosynth web service and Bundler + CMVS2 + PMVS2). The input images are taken from the same points of view to form the set of panoramic photos paying attention to use well-suited projections: equirectangular for spherical photogrammetry and rectilinear for Photosynth web service. The performance of the spherical photogrammetry is already known in terms of its metric accuracy and acquisition quickness but time is required in the restitution step because of the manual homologous point recognition from different panoramas. In Photosynth instead the restitution is quick and automated: the provided point clouds are useful benchmarks to start with the model reconstruction even if lacking in details and scale. The proposed workflow needs of ad-hoc tools to capture high resolution rectilinear panoramic images and visualize Photosynth point clouds and orientation camera parameters. All of them are developed in VVVV programming environment. 3DStudio Max environment is then chosen because of its performance in terms of interactive modeling, UV mapping parameters handling and real time visualization of projected texture on the model surface. Experimental results show how is possible to obtain a 3D photorealistic model using the scale of the spherical photogrammetry restitution to orient web provided point clouds. Moreover the proposed research highlights how is possible to speed up the model reconstruction without losing metric and photometric accuracy. In the same time, using the same panorama dataset, it picks out a useful chance to compare the orientations coming from the two mentioned technologies (Spherical Photogrammetry and Structure for Motion).

Comparison of Brain MR Images at 1.5T Using BLADE and Rectilinear Techniques for Patients Who Move during Data Acquisition

MR imaging of moving patients can be challenging and motion correction techniques have been proposed though some have associated new artifacts. The objective of this study was to semiquantitatively compare brain MR images of moving patients obtained at 1.5T by using partially radial and rectilinear acquisition techniques. FLAIR, T2-, T1-, and contrast-enhanced T1-weighted image sets of 25 patients (14-94 years) obtained by using BLADE (like PROPELLER, a partially radial acquisition) and rectilinear techniques in the same imaging session were compared by 2 neuroradiologists in terms of extent of the motion artifact, image quality, and lesion visibility. ICC between opinions of the evaluators was calculated. Of the total of 70 image sets, the motion artifact was small in the partially radial images in 43 and in the rectilinear images in 13, and the opinions of the evaluators were discordant in the remaining 14 sets (ICC = 0.63, P < .05). The quality of partially radial images was higher for 36 sets versus 9 rectilinear sets, with disagreement between the 2 evaluators in the remaining 25 (ICC = 0.15, P < .05). Pathologic lesions were better characterized on 37 sets of partially radial images versus 13 sets of rectilinear images, and opinions of the evaluators differed in 20 sets (ICC = 0.90, P < .05). The neuroradiologists deemed 4 sets of rectilinear images nondiagnostic compared with only 1 set of radial images. The data demonstrate that our application of BLADE sequences reduces the extent of motion artifacts in brain images of moving patients, improving image quality and lesion characterization.

7.5 MHz Dual-Layer Transducer Array for 3-D Rectilinear Imaging

The difficulties associated with fabrication and interconnection have limited the development of 2-D ultrasound transducer arrays with a large number ofelements (>5000). In previous work, we described a 5 MHz center frequency PZT-P[VDF-TrFE] dual-layer transducer that used two perpendicular 1-D arrays for 3-D rectilinear imaging. This design substantially reduces the channel count as well as fabrication complexity, which makes 3-D imaging more realizable. Higher frequencies (>5 MHz) are more commonly used in clinical applications or imaging targets near transducers, such as the breast, carotid and musculoskeletal tissue. In this paper, we present a 7.5 MHz dual-layer transducer array for 3-D rectilinear imaging. A modified acoustic stack model was designed and fabricated. PZT elements were sub-diced to eliminate lateral coupling. This sub-dicing process made the PZT into a 2-2 composite material, which could help improve transducer sensitivity and bandwidth. Full synthetic-aperture 3-D data sets were acquired by interfacing the transducer with a Verasonics data-acquisition system (VDAS). Offline 3-D beamforming was then performed to obtain volumes of a multiwire phantom and a cyst phantom. The generalized coherence factor (GCF) was applied to improve the contrast of cyst images. The measured -6 dB fractional bandwidth of the transducer was 71% with a center frequency of 7.5 MHz. The measured lateral beamwidths were 0.521 mm and 0.482 mm in azimuth and elevation, respectively, compared with a simulated beamwidth of 0.43 mm.

Panoramic Image Composed of Multiple Rectilinear Images Generated from a Single Fisheye Image

We have developed mathematically precise image-processing algorithms for extracting rectilinear images from fisheye images as well as digital pan/tilt/zoom technology. Using this technology, vertical lines always appear as vertical lines in the panned and/or tilted images. Furthermore, polygonal panoramic images composed of multiple rectilinear images have been obtained using the developed digital pan/tilt technology.

A 256 x 256 2-D array transducer with row-column addressing for 3-D rectilinear imaging.

We present simulation and experimental results from a 5-MHz, 256 x 256 2-D (65,536 elements, 38.4 x 38.4 mm) 2-D array transducer with row-column addressing. The main benefits of this design are a reduced number of interconnects, a modified transmit/receive switching scheme with a simple diode circuit, and an ability to perform volumetric imaging of targets near the transducer with transmit beamforming in azimuth and receive beamforming in elevation. The final dimensions of the transducer were 38.4 mm x 38.4 mm x 300 microm. After a row-column transducer was prototyped, the series resonance impedance was 104 Omega at 5.4 MHz. The measured -6 dB fractional bandwidth was 53% with a center frequency of 5.3 MHz. The SNR at the transmit focus was measured to be 30 dB. At 5 MHz, the average nearest neighbor crosstalk was -25 dB. In this paper, we present 3-D images of both 5 pairs of nylon wires embedded in a clear gelatin phantom and an 8 mm diameter cylindrical anechoic cyst phantom acquired from a 256 x 256 2-D array transducer made from a 1-3 composite. We display the azimuth and elevation B-scans as well as the C-scan for each image. The cross-section of the wires is visible in the azimuth B-scan, and the long axes can be seen in the elevation B-scan and C-scans. The pair of wires with 1-mm axial separation is discernible in the elevational B-scan. When a single wire from the wire target phantom was used, the measured lateral beamwidth was 0.68 mm and 0.70 mm at 30 mm depth in transmit beamforming and receive beamforming, respectively, compared with the simulated beamwidth of 0.55 mm. The cross-section of the cyst is visible in the azimuth B-scan whereas the long axes can be seen as a rectangle in the elevation B-scan and C-scans.

A dual-layer transducer array for 3-D rectilinear imaging.

Very large element counts (16,000-65,000) are required for 2-D arrays for 3-D rectilinear imaging. The difficulties in fabricating and interconnecting 2-D arrays with a large number of elements (>5,000) have limited the development of suitable transducers for 3-D rectilinear imaging. In this paper, we propose an alternative solution to this problem by using a dual-layer transducer array design. This design consists of 2 perpendicular 1-D arrays for clinical 3-D imaging of targets near the transducer. These targets include the breast, carotid artery, and musculoskeletal system. This transducer design reduces the fabrication complexity and the channel count, making 3-D rectilinear imaging more realizable. With this design, an effective N x N 2-D array can be developed using only N transmitters and N receivers. This benefit becomes very significant when N becomes greater than 128, for example. To demonstrate feasibility, we constructed a 4 x 4 cm prototype dual-layer array. The transmit array uses diced PZT-5H elements, and the receive array is a single sheet of undiced P[VDF-TrFE] copolymer. The receive elements are defined by the copper traces on the flexible interconnect circuit. The measured -6 dB fractional bandwidth was 80% with a center frequency of 4.8 MHz. At 5 MHz, the nearest neighbor crosstalk of the PZT array and PVDF array was -30.4 +/- 3.1 dB and -28.8 +/- 3.7 dB, respectively. This dual-layer transducer was interfaced with an Ultrasonix Sonix RP system, and a synthetic aperture 3-D data set was acquired. We then performed offline 3-D beamforming to obtain volumes of nylon wire targets. The theoretical lateral beamwidth was 0.52 mm compared with measured beamwidths of 0.65 mm and 0.67 mm in azimuth and elevation, respectively. Then, 3-D images of an 8 mm diameter anechoic cyst phantom were also acquired.

Lane Tracking with Omnidirectional Cameras: Algorithms and Evaluation

With a panoramic view of the scene, a single omnidirectional camera can monitor the 360-degree surround of the vehicle or monitor the interior and exterior of the vehicle at the same time. We investigate problems associated with integrating driver assistance functionalities that have been designed for rectilinear cameras with a single omnidirectional camera instead. Specifically, omnidirectional cameras have been shown effective in determining head gaze orientation from within a vehicle. We examine the issues involved in integrating lane tracking functions using the same omnidirectional camera, which provide a view of both the driver and the road ahead of the vehicle. We present analysis on the impact of the omnidirectional camera's reduced image resolution on lane tracking accuracy, as a consequence of gaining the expansive view. And to do so, we present Omni-VioLET, a modified implementation of the vision-based lane estimation and tracking system (VioLET), and conduct a systematic performance evaluation of both lane-trackers operating on monocular rectilinear images and omnidirectional images. We are able to show a performance comparison of the lane tracking from Omni-VioLET and Recti-VioLET with ground truth using images captured along the same freeway road in a specified course. The results are surprising: with 1/10th the number of pixels representing the same space and about 1/3rd the horizontal image resolution as a rectilinear image of the same road, the omnidirectional camera implementation results in only three times the amount the mean absolute error in tracking the left lane boundary position.

The residual distortion of phase-corrected autofocused SAR images

Abstract This paper investigates the residual distortion in SAR images produced by the RSRE linear SAR processing technique. This has been used at RSRE to produce focused, rectilinear images for many years. This article outlines the autofocus and phase-correction techniques on which the method is based and quantifies its effectiveness and accuracy in producing linear focused undistorted SAR imagery.

Change detection in SAR imagery

Abstract The two major difficulties associated with SAR image change detection are addressed. These are the removal of speckle noise and the registration of information between images. The problem of image registration is severe in airborne SAR imagery due to the unpredictable nature of the aircraft track. Although inertial navigation systems may be employed to measure this motion, the accuracy obtained is insufficient to allow the creation of large rectilinear images. However, autofocus techniques are used here to measure residual aircraft motions thus allowing the production of large geometrically accurate images. The second problem of speckle reduction has been approached in two ways. The first technique applies an intensity segmentation algorithm to each image and the regions generated by this segmentation are then compared using the change detection algorithm. An alternative approach to the problem of speckle removal is to use neural network methods to learn the speckle removal and region generation ...