Reference View Research Articles

End-to-End Residual Network for Light Field Reconstruction on Raw Images and View Image Stacks

Light field (LF) technology has become a focus of great interest (due to its use in many applications), especially since the introduction of the consumer LF camera, which facilitated the acquisition of dense LF images. Obtaining densely sampled LF images is costly due to the trade-off between spatial and angular resolutions. Accordingly, in this research, we suggest a learning-based solution to this challenging problem, reconstructing dense, high-quality LF images. Instead of training our model with several images of the same scene, we used raw LF images (lenslet images). The raw LF format enables the encoding of several images of the same scene into one image. Consequently, it helps the network to understand and simulate the relationship between different images, resulting in higher quality images. We divided our model into two successive modules: LFR and LF augmentation (LFA). Each module is represented using a convolutional neural network-based residual network (CNN). We trained our network to lessen the absolute error between the novel and reference views. Experimental findings on real-world datasets show that our suggested method has excellent performance and superiority over state-of-the-art approaches.

Analysis of COVID-19 Collective Irrationalities Based on Epidemic Psychology.

As the SARS-CoV-2 virus swept the world in late 2019, it has brought widespread fear, some suspicion, and degrees of stigma. In the shadow of the COVID-19 pandemics, a series of collective irrationalities such as panic buying, protest marches against vaccines, and pandemic stigma occurred. This phenomenon is inseparable from the spread of rumors about the epidemic. The advent of social media has radically changed the way we consume information and form opinions and made a flood of digital misinformation becoming ubiquitous. The diffusion of false rumors affects the public’s perception of reality and disrupts the prevention of the epidemic. This paper analyzes the COVID-19 collective irrationalities from epidemic psychology to provide a new reference view for overcoming psychological problems related to COVID-19.

Validation of a deep learning algorithm identifying diagnostic quality cardiac ultrasound reference views during search phase

Abstract Funding Acknowledgements Type of funding sources: Private company. Main funding source(s): DESKi Background Transthoracic echocardiography requires long and demanding training. The use of deep learning algorithms for the identification of diagnostic quality reference views opens the way for assisting low-experienced operators. However, such algorithms have only been trained and validated on retrospective data extracted from daily hospital practice. Those retrospective data are the best clips selected by experts during each exam. Thus, the performance of those algorithms has not been evaluated on complete acquisitions, including the search phase, i.e. when the operator is moving the probe to reach the optimal position. Those search phases are not recorded in clinical practice. Purpose The objective of this study was to evaluate the performance of a deep learning algorithm that identifies diagnostic quality images for 7 echocardiographic reference views on a prospective dataset including search phases. Methods A retrospective dataset containing acquisitions extracted from daily hospital practice was created. The retrospective dataset was manually annotated by experts following quality criteria that were defined for 7 main reference views: parasternal long and short axis, apical 4, 2, and 3-chamber, subcostal 4-chamber and inferior vena cava. Each frame of the acquisitions was annotated by an expert and reviewed by a second expert. A deep learning model was trained on the retrospective dataset and evaluated with 5-fold cross-validation. In a prospective dataset, for each included patient and each reference view, operators recorded the entire acquisition, including the search phase. Acquisitions were manually annotated by 2 experts following the same quality criteria. The model trained on retrospective data was evaluated on this prospective dataset against manual experts’ annotations. Additionally, 3 experts annotated a subset of the prospective dataset to evaluate the inter-observer variability. Results 481,111 frames from 1,325 patients were annotated for the retrospective dataset. On the 5-fold cross-validation, the algorithm reached an average accuracy of 89.2 ± 0.7% per frame and an average F1-score of 87.9 ± 0.9% per frame. For the prospective dataset, 70 patients were included and 143,804 frames annotated. The average accuracy was 84.8 ± 5.0% per patient, and the average F1-score was 83.1 ± 8.7% per patient. The average inter-observer agreement rates, performed on 10 patients (23,390 frames), were between 82.8 ± 4.9% and 86.3 ± 6.2%. Conclusions We propose a deep learning algorithm that automatically identifies 7 of the most common reference views with diagnostic quality while achieving performance within the range of the inter-observer variability in acquisitions including search phases.

Towards real-time 3D visualization with multiview RGB camera array.

A real-time 3D visualization (RT3DV) system using a multiview RGB camera array is presented. RT3DV can process multiple synchronized video streams to produce a stereo video of a dynamic scene from a chosen view angle. Its design objective is to facilitate 3D visualization at the video frame rate with good viewing quality. To facilitate 3D vision, RT3DV estimates and updates a surface mesh model formed directly from a set of sparse key points. The 3D coordinates of these key points are estimated from matching 2D key points across multiview video streams with the aid of epipolar geometry and trifocal tensor. To capture the scene dynamics, 2D key points in individual video streams are tracked between successive frames. We implemented a proof of concept RT3DV system tasked to process five synchronous video streams acquired by an RGB camera array. It achieves a processing speed of 44 milliseconds per frame and a peak signal to noise ratio (PSNR) of 15.9 dB from a viewpoint coinciding with a reference view. As a comparison, an image-based MVS algorithm utilizing a dense point cloud model and frame by frame feature detection and matching will require 7 seconds to render a frame and yield a reference view PSNR of 16.3 dB.

Novel view synthesis in embedded virtual reality devices

Virtual Reality and Free Viewpoint navigation require high-quality rendered images to be realistic. Current hardware assisted raytracing methods cannot reach the expected quality in real-time and are also limited by the 3D mesh quality. An alternative is Depth Image Based Rendering (DIBR) where the input only consists of images and their associated depth maps for synthesizing virtual views to the Head Mounted Display (HMD). The MPEG Immersive Video (MIV) standard uses such DIBR algorithm called the Reference View Synthesizer (RVS). We have first implemented a GPU version, called the Realtime accelerated View Synthesizer (RaViS), that synthesizes two virtual views in real-time for the HMD. In the present paper, we explore the differences between desktop and embedded GPU platforms, porting RaViS to an embedded HMD without the need for a separate, discrete desktop GPU. The proposed solution gives a first insight into DIBR View Synthesis techniques in embedded HMDs using OpenGL and Vulkan, a cross-platform 3D rendering library with support for embedded devices.

Inference-Reconstruction Variational Autoencoder for Light Field Image Reconstruction.

Light field cameras can capture the radiance and direction of light rays by a single exposure, providing a new perspective to photography and 3D geometry perception. However, existing sub-aperture based light field cameras are limited by their sensor resolution to obtain high spatial and angular resolution images simultaneously. In this paper, we propose an inference-reconstruction variational autoencoder (IR-VAE) to reconstruct a dense light field image out of four corner reference views in a light field image. The proposed IR-VAE is comprised of one inference network and one reconstruction network, where the inference network infers novel views from existing reference views and viewpoint conditions, and the reconstruction network reconstructs novel views from a latent variable that contains the information of reference views, novel views, and viewpoints. The conditional latent variable in the inference network is regularized by the latent variable in the reconstruction network to facilitate information flow between the conditional latent variable and novel views. We also propose a statistic distance measurement dubbed the mean local maximum mean discrepancy (MLMMD) to enable the measurement of the statistic distance between two distributions with high-resolution latent variables, which can capture richer information than their low-resolution counterparts. Finally, we propose a viewpoint-dependent indirect view synthesis method to synthesize novel views more efficiently by leveraging adaptive convolution. Experimental results show that our proposed methods outperform state-of-the-art methods on different light field datasets.

An Efficient Algorithm to Select Reference Views for Virtual View Synthesis

An Efficient Algorithm to Select Reference Views for Virtual View Synthesis

SEMANTIC NEGOTIATION AND RELIGIOUS TRANSLATION

The translation of religious texts has always been a perilous undertaking, especially when it concerns a sacred text. Indeed, seeing translation as an act of communication requires a knowledge of the communicative parameters that constitute it. The translator-speaker, who plays the role of intermediary, must create an interaction at the level of meaning and its semantic components. From this emerges the role of the addressee as the main protagonist and the only interpretative force of the message. Thus, the process of negotiation manages the translating activity, since the transfer of a message is not only reduced to a process of reformulation of the source text in another language, but, largely exceeds this compartmentalization. We will first discuss the challenges and stakes of the translation of religious texts, which we have chosen to illustrate with examples that make this aspect clear. Then, we will insist on the question of the translation as a semantic negotiation, considering that there is a confrontation between transmitter and receiver from the point of view of knowledge, historical and linguistic references. Indeed, this facet of the pyramid where the cultural dimension of translation and interpretation is played out remains unknown and implicit. It is quite easy to reveal what is related to the cultural, social and historical reality of a particular receiver. But that is not enough to communicate, it is also necessary to question the cultural references of this one, the representations conveyed by the language. Indeed, we have chosen to present examples of the Holy Qur'an translated by translators of different religions, and given that some translations have deviated from the noble mission of translation, in this case, the faithful transmission of the meaning, without any objectivity whatsoever, especially when it is a sacred text whose inimitability is not to be proven or tested. Thus, the difficulty in our choice stems, on the one hand, from the sacred character of the chosen writing, and on the other hand, from the nature of research in this field which is condemned to remain always relative. Since several elements control the translation of the sacred text, namely, the language, the tradition and the personal factors of the translator. Finally, the field of religious translation requires a great deal of precision and neutrality, and a constant rethinking of the fact that the slightest subjective or cultural interpretation could call into question our research work. Thus, when it comes to cultural transfer, the translator's task is to take into consideration the knowledge that already exists in the target culture about the source culture. Indeed, it is necessary to know how to relate the knowledge concerning the target culture to the knowledge via the source culture. The translator is the first receiver who receives and interprets the message; he will therefore try to understand the source culture with his own knowledge and value judgments. The translator is indeed a mediator because he assumes two roles, that of receiver and reader of the source message, however, the translator having a task of reader which must be different from that of the normal reader, since he must undoubtedly appropriate the competence of mediator in communication which crosses with the function of transmitter of the translated message. Moreover, this same message undergoes a second transfer; the target reader receives it and interprets it in turn according to his own ideological and cultural schemes. Thus, the process of the translated text is not limited to a single phase, but, it enchains back and forth during which the text acquires the imprint of the translator who makes the transfer. Without forgetting to negotiate the meaning by respecting all the circumstances of the original text. So, to what extent can this semantic negotiation communicate the said and the unsaid of the text? .

Unsupervised virtual view synthesis from monocular video with generative adversarial warping

Virtual view synthesis from monocular video is challenging, as it aims to infer photorealistic views from single reference view. Previous work have achieved acceptable visual quality, however, are heavily relied on supervision information, such as depth or pristine virtual view, which are inadequate in practical application. In this paper, an unsupervised virtual view synthesis method is proposed to get rid of the supervision information. Firstly, it embed a spatiotemporal generative adversarial network into traditional depth-image-based rendering framework with no explicit depth information provided. Secondly, it utilized novel perceptual constraints without relying on pristine images, including the blind synthesized image quality metric and no-reference structure similarity. The entire framework is fully convolutional, producing hallucinated results in an end-to-end way. Particularly, the whole framework is independent of supervision information. Experimental results demonstrate that the proposed method produces pleasant virtual views in comparison with supervised methods, thereby can be beneficial to practical applications.

Unintentional investigations: Truth in drama and “all that ‘ordinary’ in the phrase ‘ordinary language philosophy’ means”

This paper explores the notion of truth in relation to literature. It opens with a critical exposition of some dominant tendencies in contemporary aesthetics, in which narrow views of truth and reference guide the aesthetic investigations in harmful ways. One of the problems with such as view is not merely that it becomes difficult to talk about truth in art, but that it also makes the idea that we can learn something from literature problematic. The effort of this paper is thus to open up for a variety of notions of truth, that are not immediately tied to the notion of representation or correspondence. We need a way of talking about truth in art. The effort to explore a notion of truth in art that is not tied to narrow views about reference, and which broadens our sense of “aboutness” goes, in this paper, via a reading of Harold Pinter’s Nobel Lecture from 2005, together with some reflections inspired by some of Stanely Cavell’s reflections about the relevance of reflecting upon ordinary language. It is argued that literature engages in a form of conceptual reflection, by means of making the sense of our concepts clear and by challenging philosophical preconceptions about what our concepts must mean. What we can learn from art is thus not necessarily toed to either representation or authorial intent, but comes into view by means of the literary exercises that often (but certainly not always) require a conceptual sensitivity; that is, by means of careful attention to what words mean and what follows from them in specific contexts of use.

Hybrid dual stream blender for wide baseline view synthesis

Free navigation of a scene requires warping some reference views to some desired target viewpoint and blending them to synthesize a virtual view. Convolutional Neural Networks (ConvNets) based methods can learn both the warping and blending tasks jointly. Such methods are often designed for moderate inter-camera baseline distance and larger kernels are required for warping if the baseline distance increases. Algorithmic methods can in principle deal with large baselines, however the synthesized view suffers from artifacts near disoccluded pixels. We present a hybrid approach where first, reference views are algorithmically warped to the target position and then are blended via a ConvNet. Preliminary view warping allows reducing the size of the convolutional kernels and thus the learnable parameters count. We propose a residual encoder–decoder for image blending with a Siamese encoder to further keep the parameters count low. We also contribute a hole inpainting algorithm to fill the disocclusions in the warped views. Our view synthesis experiments on real multiview sequences show better objective image quality than state-of-the-art methods due to fewer artifacts in the synthesized images.

Seeing‐in and Singling Out: How to Reconcile Pictures with Singular Thought

AbstractAccording to the standard view of pictorial reference, a picture produces singular thought in virtue of both its appearance and its history. Zeimbekis (2010) challenges this view, arguing that the perception of the picture's appearance does not contribute to the production of singular thought. The paper defends the standard view from Zeimbekis' challenge, specifying the roles of appearance and history in pictorial reference. While knowledge about the picture's history allows one to identify the standpoint from which to see the scene depicted, the perception of the picture's appearance exploits that standpoint to single out individuals in that scene.

Virtual view synthesis for the nonuniform illuminated between views in surgical video

For high-quality surgical video virtual view synthesis, a Weighted Autoregressive Interpolation (WAI) algorithm and an Adaptively-enhanced Hole Filling (AHF) are proposed to reduce the artifacts caused by up-sampling and relieve the luma difference. First, high quality up-sampled reference views are acquired by the WAI algorithm. A Piecewise Autoregressive (PAR) model is introduced and the distance weight of pixels is also considered. The precision of the virtual view is improved by the WAI and the texture edges are well preserved. Next, for the AHF, the intermediate view with more structure details is selected as the template. The other intermediate view is calibrated to it. And the luma difference is relieved. Then, a Nearest background Holes Filling algorithm (NHF) is adopted to blend these two intermediate views, in which only background pixels are selected to fill the remaining holes. Combining the WAI with AHF, the visual quality of the surgical virtual video is improved. For the objective quality, the experimental results show that the PSNR of the proposed algorithm is 0.5841 dB higher than the VSRS 1D-Fast algorithm on average. For subjective quality, the proposed method can reduce the artifacts and gain higher subjective quality for the synthesized virtual view of the surgical video.

Role of Deep Learning in Loop Closure Detection for Visual and Lidar SLAM: A Survey.

Loop closure detection is of vital importance in the process of simultaneous localization and mapping (SLAM), as it helps to reduce the cumulative error of the robot’s estimated pose and generate a consistent global map. Many variations of this problem have been considered in the past and the existing methods differ in the acquisition approach of query and reference views, the choice of scene representation, and associated matching strategy. Contributions of this survey are many-fold. It provides a thorough study of existing literature on loop closure detection algorithms for visual and Lidar SLAM and discusses their insight along with their limitations. It presents a taxonomy of state-of-the-art deep learning-based loop detection algorithms with detailed comparison metrics. Also, the major challenges of conventional approaches are identified. Based on those challenges, deep learning-based methods were reviewed where the identified challenges are tackled focusing on the methods providing long-term autonomy in various conditions such as changing weather, light, seasons, viewpoint, and occlusion due to the presence of mobile objects. Furthermore, open challenges and future directions were also discussed.

Automatic orientation cues for intuitive immersive interrogation of 3D echocardiographic images in virtual reality using deep learning

Abstract Funding Acknowledgements Type of funding sources: Other. Main funding source(s): NIHR i4i funded 3D Heart Project Wellcome / EPSRC Centre for Medical Engineering (WT 203148/Z/16/Z) onbehalf 3D Heart Project Background/Introduction: In echocardiography (echo), image orientation is determined by the position and direction of the transducer during examination, unlike cardiovascular imaging modalities such as CT or MRI. As a result, when echo images are first shown their display orientation has no external anatomical landmarks, thus the user has to identify anatomical landmarks in the regions of interest to understand the orientation. Purpose To display an anatomical model of a standard heart, automatically aligned to an acquired patient’s 3D echo image - assisting image interpretation by quickly orienting the viewer. Methods 47 echo datasets from 13 pediatric patients with hypoplastic left heart syndrome (HLHS) were annotated by manually indicating the cardiac axes in both ES and ED volumes. We chose a view akin to the standard four chamber view in healthy hearts as the reference view, showing the AV valves, the right atrium, the left atrium and the hypoplastic ventricle. We then trained a deep convolutional neural network (CNN) to predict the rotation required for re-orientation to the reference view. Three data strategies were explored: 1) using 3D images to estimate orientation, 2) using three orthogonal slices only (2.5D approach) and 3) using the central slice only (2D approach). Three different algorithms were investigated: 1) an orientation classifier, 2) an orientation regressor with mean absolute angle error, and 3) an orientation regressor with geodesic loss. The data was split into training, validation and test sets with a 8:1:1 ratio. The training data was augmented by applying random rotations in the range [−10◦, +10◦] and updating labels accordingly. The model with smallest validation error was applied in tandem with the VR visualisation of the echo volumes. Results Experimental results suggest that a 2.5D CNN classifying discrete integer angles performs best in re-orienting volumetric images to the reference view, with a mean absolute angle error on the test set of 9.0 deg (test set error ranges from 10.8 to 25.9 deg. for other algorithms). An HLHS volumetric data (left) is automatically aligned with the cardiac model (right) using our trained network when loaded in VR as shown in Figure 1. The volume and the model are both cropped at the referencing plane. Conclusion A deep learning network to align 3D echo images to a reference view was successfully trained and then integrated into VR to reorient echo volumes to match a standard anatomical view. This work demonstrates the potential of combining artificial intelligence and VR in medical imaging, although further user study is expected to evaluate its clinical impact. Caption for Abstract Picture The VR user interface informs the user of the 3D echo image orientation, automatically aligning it with an anatomical model, here showing the four chamber apical view. Abstract Figure. Deep learning model integrated into VR

Various density light field image coding based on distortion minimization interpolation

In recent years, the light field (LF) as a new imaging modality has attracted wide interest. The large data volume of LF images poses great challenge to LF image coding, and the LF images captured by different devices show significant differences in angular domain. In this paper we propose a view prediction framework to handle LF image coding with various sampling density. All LF images are represented as view arrays. We first partition the views into reference view (RV) set and intermediate view (IV) set. The RVs are rearranged into a pseudo sequence and directly compressed by a video encoder. Other views are then predicted by the RVs. To exploit the four dimensional signal structure, we propose the linear approximation prior (LAP) to reveal the correlation among LF views and efficiently remove the LF data redundancy. Based on the LAP, a distortion minimization interpolation (DMI) method is used to predict IVs. To robustly handle the LF images with different sampling density, we propose an Iteratively Updating depth image based rendering (IU-DIBR) method to extend our DMI. Some auxiliary views are generated to cover the target region and then the DMI calculates reconstruction coefficients for the IVs. Different view partition patterns are also explored. Extensive experiments on different types LF images also valid the efficiency of the proposed method.

Disocclusion-type aware hole filling method for view synthesis

Depth-image-based rendering is an efficient way to produce content for 3D video and free viewpoint video. However, since the background that is occluded by the foreground objects in the reference view may become visible in the synthesized view, disocclusions are produced. In this paper, a disocclusion-type aware hole filling method is proposed for disocclusion handling. Disocclusions are divided into two types based on the depth value of their boundary pixels: foreground-background (FG-BG) disocclusion and background-background (BG-BG) disocclusion. For FG-BG disocclusion, the depth values of the associated pixels are optimized in the reference image to ensure the removal of ghosts and adaptively divide the disocclusion into some small holes. For BG-BG disocclusion, a foreground removal method is applied to remove the corresponding foreground objects. The removed regions are filled with the surrounding background textures so that the BG-BG disocclusion in the synthesized image can be eliminated. Experimental results indicate that the proposed method outperforms the other methods in the objective and subjective evaluations.

View-invariant action recognition via Unsupervised AttentioN Transfer (UANT)

With wide applications in surveillance and human-robot interaction, view-invariant human action recognition is critical, however, challenging, due to the action occlusion and information loss caused by view change. Current methods mainly seek for a common feature space for different views. However, such solutions become invalid when there exist few common features, e.g. large view change. To tackle the problem, we propose an Unsupervised AttentioN Transfer (UANT) approach for view-invariant action recognition. Other than transferring feature knowledge, UANT transfers attention from one selected reference view to arbitrary views, which correctly emphasizes crucial body joints and their relations for view-invariant representation. In addition, the attention calculation method taking into account both recognition contribution and reliability of skeleton joints generates effective attention. Experiments showed its effectiveness for correctly locating crucial body joints in action sequences. We exhaustively evaluate our approach on the UESTC and the NTU dataset, performing unsupervised view-invariant evaluations, i.e. X-view and Arbitrary-view recognition. Experiment results demonstrate its superiority in view-invariant representation and recognition.

A novel fast search method to find disparity vectors in multiview video coding

In multiview/3D video, the amount of data to be transmitted to the decoder increases proportionally with the number of cameras. One way to efficiently compress such video is to use Multiview Video Coding (MVC) which simultaneously reduces temporal and spatial redundancy within the same view and among multiple views. But, existing disparity estimation methods used to extract the redundancy among views are typically too computationally complex. In this paper, we propose a method for fast disparity estimation in multiview/3D video. First, for each block of the predicted frame, the view-dependent geometry and the depth information are used to find the corresponding block in the reference views. Then, a fast and greedy search method is proposed to search the surrounding areas to find the most similar block. Simulation results show that our proposed method achieves better bitrate and quality with much lower computational complexity compared to state-of-the-art methods.

Light Field Image Coding Using VVC Standard and View Synthesis Based on Dual Discriminator GAN

Light field (LF) technology is considered as a promising way for providing a high-quality virtual reality (VR) content. However, such an imaging technology produces a large amount of data requiring efficient LF image compression solutions. In this paper, we propose a LF image coding method based on a view synthesis and view quality enhancement techniques. Instead of transmitting all the LF views, only a sparse set of reference views are encoded and transmitted, while the remaining views are synthesized at the decoder side. The transmitted views are encoded using the versatile video coding (VVC) standard and are used as reference views to synthesize the dropped views. The selection of non-reference dropped views is performed using a rate-distortion optimization based on the VVC temporal scalability. The dropped views are reconstructed using the LF dual discriminator GAN (LF-D2GAN) model. In addition, to ensure that the quality of the views is consistent, at the decoder, a quality enhancement procedure is performed on the reconstructed views allowing smooth navigation across views. Experimental results show that the proposed method provides high coding performance and overcomes the state-of-the-art LF image compression methods by –36.22% in terms of BD-BR and 1.35 dB in BD-PSNR. The web page of this work is available at https://naderbakir79.github.io/LFD2GAN.html .