Auxiliary Views Research Articles

View Selection of Safety Managers Affects Their Ability to Evaluate Safety Level of Workers

Evaluating the safety level of work behaviors is an important means of safety management through safety managers’ manual visual or video surveillance. This paper aims to grasp the view selection rules of safety managers in correctly observing and analyzing work behavior, which are conducive to safety behavior assessment. Firstly, based on a similarity simulation experiment, the work process of the operators was simulated, and the process was simultaneously monitored from multiple views to obtain surveillance videos. Then, safety managers with different experience levels watched the videos and answered the questionnaire in a questionnaire survey experiment. The authors found the following: (1) Prior experience does not affect the view selection during the observation stage, but it does affect the view selection during the analysis stage. Experienced safety managers perform better in view selection when dealing with complex tasks. (2) The work postures have a significant effect on the view type and their combination order. (3) The view selection of experienced safety managers in the behavior observation and analysis stages is non-constant except for sitting posture operations, while the view selection of amateur managers is always constant. (4) Sitting posture work takes the front view as the main view and the left–right–upper views as auxiliary views; standing posture uses the left and right views as the main views and the front–back–upper views as auxiliary views; and mixed posture takes the left and right views as the main views and the upper–front–back views as auxiliary views. These view selection rules can achieve the highest evaluation performance. These findings can help train or select high-quality safety managers and provide a scientific basis for arranging views during video surveillance.

Multi-view Heterogeneous Graph Neural Networks for Node Classification

Recently, with graph neural networks (GNNs) becoming a powerful technique for graph representation, many excellent GNN-based models have been proposed for processing heterogeneous graphs, which are termed Heterogeneous graph neural networks (HGNNs). However, existing HGNNs tend to aggregate information from either direct neighbors or those connected by short metapaths, thereby neglecting the higher-order information and global feature similarity information in heterogeneous graphs. In this paper, we propose a Multi-View Heterogeneous graph neural network (MV-HGNN) to aggregate these information. Firstly, two auxiliary views, specifically a global feature similarity view and a graph diffusion view, are generated from the original heterogeneous graph. Secondly, MV-HGNN performs two message-passing strategies to get the representation of different views. Subsequently, a transformer-based aggregator is used to get the semantic information. Subsequently, the representations of the three views are fused into a final composite representation. We evaluate our method on the node classification task over three commonly used heterogeneous graph datasets, and the results demonstrate that our proposed MV-HGNN significantly outperforms state-of-the-art baselines.

A 3D Descriptive Geometry Problem-Solving Methodology Using CAD and Orthographic Projection

In solving descriptive geometry (DG) problems, board (i.e., hand drawing) methods are frequently used, despite this discipline is still very important to enhance spatial vision. These methods are very different from CAD tools which are used in the field of design. CAD facilitates the realization of geometric constructions and transformations (i.e., rotation, translation, copying, scaling, alignment, and symmetry, among others) are performed analytically. For this reason, a 3D DG problem-solving methodology using CAD and orthographic projection (CADOP) is introduced. Once the principles of DG, orthographic projection, and CAD fundamentals and tools are described, CADOP is applied to obtain (i) orthogonal views; (ii) principal lines of a plane; (iii) true-size view of a plane; (iv) parallelism, perpendicularity, and distance, and (v) angles. Considering the user coordinate system in CADOP allows one to place the horizontal plane in the suitable position to solve DG problems directly in one step. In the traditional methods, the use of auxiliary views must be carried out in several steps instead. The dynamic management of the 3D view of the scene is facilitated in CADOP, improving its understanding, and achieving the precision inherent in analytical calculations.

Gated Region-Refine pose transformer for human pose estimation

Implementing the transformer for global fusion is a novel and efficient method for pose estimation. Although the computational complexity of modeling dense attention can be significantly reduced by pruning possible human tokens, the accuracy of pose estimation still suffers from the problem of high overlap of candidate regions and severe background noise. Moreover, the undifferentiated fusion of features from different views also leads to a sizeable effective information loss. To address these challenges, we propose a Gated Region-Refine Pose Transformer (GRRPT) for human pose estimation. The proposed GRRPT can obtain the general area of the human body from the coarse-grained tokens and then embed it into the fine-grained ones to extract more details of the joints. Experimental results on COCO demonstrate that performing the Multi-Resolution Attention mechanism learns more refined candidate regions and improves accuracy. Furthermore, we design a Fusion Gate module consisting of two gates to pixel-wise select valid information from the auxiliary views, which significantly alleviates information redundancy. Finally, we evaluate the effectiveness of our method on Human3.6M and our dataset FDU-Motion and achieve state-of-the-art performance.

Learning a Compact Spatial-Angular Representation for Light Field

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of three-dimensional (3-D) deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

QEVIS: Multi-Grained Visualization of Distributed Query Execution.

Distributed query processing systems such as Apache Hive and Spark are widely-used in many organizations for large-scale data analytics. Analyzing and understanding the query execution process of these systems are daily routines for engineers and crucial for identifying performance problems, optimizing system configurations, and rectifying errors. However, existing visualization tools for distributed query execution are insufficient because (i) most of them (if not all) do not provide fine-grained visualization (i.e., the atomic task level), which can be crucial for understanding query performance and reasoning about the underlying execution anomalies, and (ii) they do not support proper linkages between system status and query execution, which makes it difficult to identify the causes of execution problems. To tackle these limitations, we propose QEVIS, which visualizes distributed query execution process with multiple views that focus on different granularities and complement each other. Specifically, we first devise a query logical plan layout algorithm to visualize the overall query execution progress compactly and clearly. We then propose two novel scoring methods to summarize the anomaly degrees of the jobs and machines during query execution, and visualize the anomaly scores intuitively, which allow users to easily identify the components that are worth paying attention to. Moreover, we devise a scatter plot-based task view to show a massive number of atomic tasks, where task distribution patterns are informative for execution problems. We also equip QEVIS with a suite of auxiliary views and interaction methods to support easy and effective cross-view exploration, which makes it convenient to track the causes of execution problems. QEVIS has been used in the production environment of our industry partner, and we present three use cases from real-world applications and user interview to demonstrate its effectiveness. QEVIS is open-source at https://github.com/DBGroup-SUSTech/QEVIS.

Progressive Multi-Scale Fusion Network for Light Field Super-Resolution

Light field (LF) cameras can record multi-view images from a single scene, and these images can provide spatial and angular information to improve the performance of image super-resolution (SR). However, it is a challenge to incorporate distinctive information from different LF views. At the same time, due to the inherent resolution of the image sensor, high spatial and angular resolution are trade-off problems. In this paper, we propose a progressive multi-scale fusion network (PMFN) to improve the LFSR performance. Specifically, a progressive feature fusion block (PFFB) based on an encoder-and-decoder structure is designed to implicitly align disparities and integrate complementary information between complementary views. The core module of the PFFB is a dual-branch multi-scale fusion module (DMFM), which can integrate the information from a reference view and auxiliary views to produce a fusion feature. Each DMFM consists of two parallel branches, which have different receptive fields to fuse hierarchical features from complementary views. Three DMFMs with a dense connection are used in the PFFB, which can fully exploit multi-level features to improve the SR performance. Experimental results on both synthetic and real-world datasets demonstrate that the proposed model achieves state-of-the-art performance among existing methods. Moreover, quantitative results show that our method can also generate faithful details.

Learning to restore light fields under low-light imaging

Light Field (LF) images have the unique advantage of recording scenes from multiple viewpoints, which provides many applications, such as refocusing and depth estimation. However, low-light conditions can severely influence these applications. In this paper, we propose a two-stage deep learning framework for the LF restoration under low-light imaging. First, there is a multi-to-one (MTO) network, which restores each view separately by utilizing multiple auxiliary views. All the views share the same feature extractor, with an efficient spatial-channel attention mechanism to extract more informative features. A channel-attention feature fusion (CAFF) module is designed to selectively fuse more useful complementary information from the auxiliary views, with a learnable global scalar to adjust the importance of the auxiliary features. Then, the outputs of the MTO network are further enhanced by an (all-to-all) ATA network, which uses spatial and angular residual blocks to process all the views synchronously for fully encoding the spatial-angular information. Extensive experiments have been conducted to demonstrate the superior performance and robustness of our method, i.e., it can restore the luminance, spatial details and angular geometries of the LF images under various light levels effectively.

Various density light field image coding based on distortion minimization interpolation

In recent years, the light field (LF) as a new imaging modality has attracted wide interest. The large data volume of LF images poses great challenge to LF image coding, and the LF images captured by different devices show significant differences in angular domain. In this paper we propose a view prediction framework to handle LF image coding with various sampling density. All LF images are represented as view arrays. We first partition the views into reference view (RV) set and intermediate view (IV) set. The RVs are rearranged into a pseudo sequence and directly compressed by a video encoder. Other views are then predicted by the RVs. To exploit the four dimensional signal structure, we propose the linear approximation prior (LAP) to reveal the correlation among LF views and efficiently remove the LF data redundancy. Based on the LAP, a distortion minimization interpolation (DMI) method is used to predict IVs. To robustly handle the LF images with different sampling density, we propose an Iteratively Updating depth image based rendering (IU-DIBR) method to extend our DMI. Some auxiliary views are generated to cover the target region and then the DMI calculates reconstruction coefficients for the IVs. Different view partition patterns are also explored. Extensive experiments on different types LF images also valid the efficiency of the proposed method.

Hunting High and Low: Visualising Shifting Correlations in Financial Markets

AbstractThe analysis of financial assets’ correlations is fundamental to many aspects of finance theory and practice, especially modern portfolio theory and the study of risk. In order to manage investment risk, in‐depth analysis of changing correlations is needed, with both high and low correlations between financial assets (and groups thereof) important to identify. In this paper, we propose a visual analytics framework for the interactive analysis of relations and structures in dynamic, high‐dimensional correlation data. We conduct a series of interviews and review the financial correlation analysis literature to guide our design. Our solution combines concepts from multi‐dimensional scaling, weighted complete graphs and threshold networks to present interactive, animated displays which use proximity as a visual metaphor for correlation and animation stability to encode correlation stability. We devise interaction techniques coupled with context‐sensitive auxiliary views to support the analysis of subsets of correlation networks. As part of our contribution, we also present behaviour profiles to help guide future users of our approach. We evaluate our approach by checking the validity of the layouts produced, presenting a number of analysis stories, and through a user study. We observe that our solutions help unravel complex behaviours and resonate well with study participants in addressing their needs in the context of correlation analysis in finance.

Visual Analysis System for Market Sales Data of Agricultural Products

In order to realize the automation of large data analysis and mining, a series of advanced algorithms, such as data cleaning and clustering, are used to deeply excavate the market sales data of agricultural and sideline products and it provide the necessary data basis for the decision layer of the enterprise. We designed and developed a visual analysis system of market sales data orients to products and business circles in the article. First, we clean and filter the original market sales data, and then establish the model based on clustering analysis in machine learning. Finally, the automatic display of analysis result views are achieved through excellent visualization framework technology. The experimental results show that in the actual sales data onto agricultural products, the characteristics of the business circle that the system automatically aggregates are obvious, and the combined display mode of the main and auxiliary views can show the characteristics of the sales data of a commercial circle from the macro and micro perspectives, respectively. It not only utilizes the ability of computer automatic analysis, but also fully exploits people’s cognitive ability of visual information and it helps people understand the information, knowledge and wisdom behind the big data of the market sales more intuitively and efficiently. Based on web visualization technology, a case of visual analysis of market sales data onto agricultural products and business circles is presented in this paper. And it has practical reference significance of other typical marketing analysis, especially for agricultural products market.

Visibility widgets for unveiling occluded data in 3D terrain visualization

In 3D visualization of terrain, occlusion has detrimental impact on discovering, accessing, and spatially relating embedded data. This can lead to significant loss of information. To address this issue, we introduce visibility widgets: compact auxiliary views that visualize occluded data, their locations, and related properties, and thus support users in revealing hidden information. The widgets comprise different occlusion management strategies for detecting and analyzing invisible data inside as well as outside of the field of view. In addition, each widget is designed complementary and transient to facilitate interactive exploration and navigation while not monopolizing the user’s attention. We demonstrate the application of visibility widgets focusing the visual exploration of weather data in 3D terrain.

Multiple Continuous Virtual Paths Based Cross-View Action Recognition

In this paper, we propose a novel method for cross-view action recognition via multiple continuous virtual paths which connect the source view and the target view. Each point on one virtual path is a virtual view which is obtained by a linear transformation of an action descriptor. All the virtual views are concatenated into an infinite-dimensional feature to characterize continuous changes from the source to the target view. To utilize these infinite-dimensional features directly, we propose a virtual view kernel (VVK) to compute the similarity between two infinite-dimensional features, which can be readily used to construct any kernelized classifiers. In addition, a constraint term is introduced to fully utilize the information contained in the unlabeled samples which are easier to obtain from the target view. The rationality behind the constraint is that any action video belongs to only one class. To further explore complementary visual information, we utilize multiple continuous virtual paths. The original source and target views are projected to different auxiliary source and target views using the random projection technique. Then we fuse all the VVKs generated from all pairs of auxiliary views. Our method is verified on the IXMAS and MuHAVi datasets, and the experimental results demonstrate that our method achieves better performance than the state-of-the-art methods.

Super resolution for multiview mixed resolution images in transform-domain with optimal weight

Mixed resolution framework has been applied in stereoscopic video coding system to reduce data-size and bit rate. Recently, Super Resolution (SR) problem in mixed resolution format has drawn a lot of attention in order to provide the users with high quality 3D visual experience. In this paper we present a novel SR scheme, which not only borrows the high frequency content from neighboring full resolution frame, but also makes use of all high resolution (HR) warped images to enhance low resolution (LR) view image. The correlation between the target LR view and auxiliary HR views determines the weight of these high frequency components, which will be merged into LR frame to construct the output HR image. The DCT is used for frequency decomposition and frequency integration. Experimental results show that the proposed method achieves higher performance in both subjective and objective evaluations.

Multiple Description Coding for Stereoscopic Videos With Stagger Frame Order

Due to the prediction structures employed in video coding, the loss of one packet will affect many following frames. In this paper, a multiple description coding scheme with stagger frame order is proposed for stereoscopic 3-D videos. First, the reference and auxiliary views in stereoscopic sequences will be asymmetrically encoded into one description, whereas the other description will be formed in the same way with one dumb frame delay. Because of the stagger frame order, the coarsely encoded B frames will be inserted into different positions of the two descriptions. If a certain frame encoded with I/P mode is lost, then its corresponding B-frame version will be employed to compensate for the loss. In each description, the quantization steps of B frames are tuned based on a closed-form solution that considers the video contents, network status, frame positions in the group of picture, and the layer of the views. For further improvement, a fusing scheme is provided. The experimental results demonstrate that the proposed scheme outperforms state-of-the-art schemes. Specifically, up to 1.3-dB gain is achieved in the case of packet loss, and 2-dB gain is obtained for the side/central performance.

Fast Encoder Design for Multi-view Video

Multi-view video coding is an international encoding standard that attains good performance by fully utilizing temporal and inter-view correlations. However, it suffers from high computational complexity. This paper presents a fast encoder design to reduce the level of complexity. First, when the temporal correlation of a group of pictures is sufficiently strong, macroblock-based inter-view prediction is not employed for the non-anchor pictures of B-views. Second, when the disparity between two adjacent views is above some threshold, frame-based inter-view prediction is disabled. Third, inter-view prediction is not performed on boundary macroblocks in the auxiliary views, because the references for these blocks may not exist in neighboring views. Fourth, finer partitions of inter-view prediction are cancelled for macroblocks in static image areas. Finally, when estimating the disparity of a macroblock, the search range is adjusted according to the mode size distribution of the neighboring view. Compared with reference software, these techniques produce an average time reduction of 83.65%, while the bit-rate increase and peak signal-to-noise ratio loss are less than 0.54% and 0.05dB, respectively.

An Adaptive Error Concealment Method Based on Fuzzy Reasoning for Multi-View Video Coding

This study proposes an adaptive error concealment selection process in order to improve the quality of reconstructed images which have low computation complexity for multi-view video coding. In this study, the proposed algorithm uses the motion vector variances and texture histogram bins in the blocks neighboring the damaged block in order to define two correlations; the degree of motion and the degree of texture similarity. To increase the accuracy of the suitable EC for the damaged macro-block, these two correlations are used to select a suitable error concealment method for reconstructing the damaged macro-block through fuzzy reasoning. The motion degree indicates the motion complexity for the damaged macro-block, and the texture similarity degree indicates the spatial continuity between the damaged macro-block and its neighboring block. The proposed adaptive error concealment is selected to reduce concealing time on homogenous damaged blocks and to improve the reconstructed image quality on non-homogenous damaged blocks. The experiment results show that the proposed algorithm can reduce the concealing time by at least 1.16 s and can improve the peak signal-to-noise (PSNR) by 0.13 dB–2.44 dB and 0.11 dB–3.52 dB in the main views and auxiliary views, respectively. This proposed algorithm is suitable for the multi-view video transmission on the consumer electronic fields such as 3D mobile devices, 3D television.

MB-based Decoder Buffer Reduction for Multi-view Video Coding

The hierarchical B picture (HBP) coding structure has been developed to improve the coding efficiency of multi-view video coding (MVC). HBP coding structure adopts temporal prediction and inter-view prediction to reduce the residual of the data in motion estimation. Since all the reference frames must be exhaustively searched for inter-view and temporal predictions, the encoder may take high computation time and need a large buffer size for the decoded picture. Hence, this paper proposes an effective reference frames determination method for motion estimation in HBP to reduce the computation time and buffer size. The proposed algorithm uses the relationship between the degree of motion and its predicted direction to select the prediction reference frame in the inter-view or temporal domain. The decoder buffer of the proposed algorithm only stores the reference region MBs instead of the entire reference picture, so that the decoder buffer size is significantly reduced. The experiment results show that the encoding time for the main views and the auxiliary views can be reduced by up to 65.39 % and 66.92 %, respectively. The decoder buffer size and bandwidth can be reduced by up to 70 % and 46.67 %, respectively. The results show that the proposed method not only achieves low computational complexity, but also reduces the decoded buffer size while maintaining high quality.

Constructing a wire-frame from views on arbitrary view planes for objects with conic sections inclined to all view planes

A new and efficient approach to construct a 3D wire-frame of an object from its orthographic projections is described. The input projections can be two or more and can include regular and complete auxiliary views. Each view may contain linear, circular and other conic sections. The output is a 3D wire-frame that is consistent with the input views. The approach can handle auxiliary views containing curved edges. This generality derives from a new technique to construct 3D vertices from the input 2D vertices (as opposed to matching coordinates that is prevalent in current art). 3D vertices are constructed by projecting the 2D vertices in a pair of views on the common line of the two views. The construction of 3D edges also does not require the addition of silhouette and tangential vertices and subsequently splitting edges in the views. The concepts of complete edges and n-tuples are introduced to obviate this need. Entities corresponding to the 3D edge in each view are first identified and the 3D edges are then constructed from the information available with the matching 2D edges. This allows the algorithm to handle conic sections that are not parallel to any of the viewing directions. The localization of effort in constructing 3D edges is the source of efficiency of the construction algorithm as it does not process all potential 3D edges. Working of the algorithm on typical drawings is illustrated.

Evaluation of Asymmetric Stereo Video Coding and Rate Scaling for Adaptive 3D Video Streaming

It is well known that the human visual system can perceive high frequencies in 3D stereo video, even if that information is present in only one of the views. Therefore, the best perceived 3D stereo video quality may be achieved by asymmetric coding where the reference and auxiliary (right and left) views are coded at unequal PSNR. However, the questions of what is the best level of asymmetry in order to maximize the perceived quality and whether asymmetry should be achieved by spatial resolution reduction or PSNR (quality) reduction have been open issues. We conducted extensive subjective tests, which indicate that if the reference view is encoded at sufficiently high quality and the auxiliary view is encoded at a lower quality but above a certain PSNR threshold, then the degradation in 3D video quality is unnoticeable. Since asymmetric coding by PSNR reduction gives finer control over achievable PSNR values over spatial resolution reduction, asymmetry by PSNR reduction allows us to encode at a point more close to this just-noticeable asymmetry PSNR threshold; hence will be preferred over the spatial resolution reduction method. Subjective tests also indicate that below this just-noticeable asymmetry threshold, where subtle artifacts start to appear, symmetric coding performs better than asymmetric coding in terms of perceived 3D video quality. Therefore, we show that the choice between asymmetric vs. symmetric coding depends on PSNR; hence, the available total bitrate. This paper also proposes a novel asymmetric scalable stereo video coding framework to enable adaptive stereoscopic video streaming taking full advantage of these observations and subjective test results.