Computer Vision Perspective Research Articles

A multi-featured expression recognition model incorporating attention mechanism and object detection structure for psychological problem diagnosis

Expression is the main method for judging the emotional state and psychological condition of the human body, and the prediction of changes in facial expressions can effectively determine the mental health of a person, thus avoiding serious psychological or psychiatric disorders due to early negligence. From a computer vision perspective, most researchers have focused on studying facial expression analysis, and in some cases, body posture is also considered. However their performance is more limited under unconstrained natural conditions, which requires more information to be used in human emotion analysis. In this paper, we design an Adaptive Multi-End Fusion Attention Mechanism suitable for extracting human body information based on the deep learning framework, depending on human expressions, postures and the environment they are in and add it to an object detection model to obtain the information we need from different regions of the human body and face and features of different sizes and use fusion networks for feature fusion and classification, and from different test methods to confirm that this fusion approach for expression recognition and prediction is feasible. This model achieves an average accuracy of 34.51 % in the Emotic contextual expression recognition dataset.

Super-resolution reconstruction of 3D digital rocks by deep neural networks

Digital rock technology provides valuable insights into the pore structure and fluid flow properties of geoenergy resources. Artificial intelligence technology in vision and image processing, especially the image super-resolution, has great potential for digital rock reconstruction and resolution enhancement. However, the analyzed core samples are typically sandstones/carbonates in micro-scale resolutions and in two-dimensional (2D) space, whereas the shale rocks in nano-scale resolutions for unconventional resources or three-dimensional (3D) digital cores are rarely investigated. Additionally, previous studies primarily emphasized image quality from a computer vision perspective, with little consideration given to estimating physical properties of digital rocks using super-resolution techniques.This study presents a very deep super-resolution (VDSR) algorithm, specifically designed to generate high-resolution 3D digital rock images, for nano-scale shale matrix and micro-scale hydraulic fractures. We compare both image quality and permeability accuracy between the original high-resolution images and the super-resolution images reconstructed by the proposed method. The results reveal that the reconstructed images using the proposed method closely resemble the actual images, and effectively reduce errors in permeability computations. This study highlights the applicability of the proposed VDSR algorithm in establishing the detailed structures of 3D nano-scale shale matrix and hydraulic fractured rocks, thus advancing super-resolution techniques in digital core analysis for geoenergy resources development.

Learning Structured Relation Embeddings for Fine-Grained Fashion Attribute Recognition

Fashion attribute recognition is a not-new topic, but rather a core task in understanding fashion from the perspective of computer vision. This paper proposes a structured relation-aware network (sRA-Net), which exploits multiple hidden relations in fashion images to enrich and achieve accurate attribute representations to boost the performance of fashion attribute recognition. Specifically, it deconstructs the features of a clothing fashion item into three levels, including low-level attribute-related image region information, mid-level attribute dependency information, and high-level clothing look information. To learn these multi-relational embeddings, we present three relation-aware attention mechanisms. The attribute attention mechanism describes the relationship among different attribute vectors through self-attention and uses the attention map to update the attribute embedding. Then, the spatial attention mechanism associates the attribute with the image features and enhances the attribute embedding by leveraging the attribute-related image region. Finally, the channel attention mechanism selects attribute-related image feature channels to obtain a more fine-grained attribute embedding. Furthermore, we introduce structure-aware embedding to constrain attribute recognition in images from a global perspective by identifying the inner structure of the clothing. Without bells and whistles, sRA-Net outperforms all state-of-the-art attribute recognition methods on two mainstream fashion attribute datasets, namely the DeepFashion-C dataset and iFashion-Attribute dataset, with over 1%-3% improvement.

An intelligent anti-infection ventilation strategy: From occupant-centric control and computer vision perspectives

An intelligent anti-infection ventilation strategy: From occupant-centric control and computer vision perspectives

An Intelligent Method for Predicting Pacific Oyster (Crassostrea gigas) Freshness Using Deep Learning Fused with Malondialdehyde and Total Sulfhydryl Groups Information.

To achieve a non-destructive and rapid detection of oyster freshness, an intelligent method using deep learning fused with malondialdehyde (MDA) and total sulfhydryl groups (SH) information was proposed. In this study, an "MDA-SH-storage days" polynomial fitting model and oyster meat image dataset were first built. AleNet-MDA and AlxNet-SH classification models were then constructed to automatically identify and classify four levels of oyster meat images with overall accuracies of 92.72% and 94.06%, respectively. Next, the outputs of the two models were used as the inputs to "MDA-SH-storage days" model, which ultimately succeeded in predicting the corresponding MDA content, SH content and storage day for an oyster image within 0.03 ms. Furthermore, the interpretability of the two models for oyster meat image were also investigated by feature visualization and strongest activations techniques. Thus, this study brings new thoughts on oyster freshness prediction from the perspective of computer vision and artificial intelligence.

Video Moment Retrieval via Comprehensive Relation-Aware Network

Video moment retrieval aims to retrieve a target moment from an untrimmed video that semantically corresponds to the given language query. Existing methods commonly treat it as a regression task or a ranking task from the perspective of computer vision. Most of these works neglect comprehensive relations between video content and language context at a multi-granularity level and fail to efficiently model temporal relations among different video moments. In this paper, we formulate video moment retrieval into video reading comprehension by treating the input video as a text passage and language query as a question. To tackle the above impediments, we propose a Comprehensive Relation-aware Network (CRNet) to perceive comprehensive relations from extensive aspects. Specifically, we unite visual and textual features simultaneously at both clip-level and moment-level to thoroughly exploit inter-modality information, leading to a coarse-and-fine cross-modal interaction. Moreover, a background suppression module is introduced to restrain irrelevant background clips, meanwhile, a novel IoU attention mechanism and graph attention layer are efficiently devised to focus on the dependencies among highly-correlated video moments for the best choice selection. In-depth experiments on three public datasets TACoS, ActivityNet Captions, and Charades-STA demonstrate the superiority of our solution.

The Right Spin: Learning Object Motion from Rotation-Compensated Flow Fields

A good understanding of geometrical concepts as well as a broad familiarity with objects lead to excellent human perception of moving objects. The human ability to detect and segment moving objects works in the presence of multiple objects, complex background geometry, motion of the observer and even camouflage. How we perceive moving objects so reliably is a longstanding research question in computer vision and borrows findings from related areas such as psychology, cognitive science and physics. One approach to the problem is to teach a deep network to model all of these effects. This is in contrast with the strategy used by human vision, where cognitive processes and body design are tightly coupled and each is responsible for certain aspects of correctly identifying moving objects. Similarly, from the computer vision perspective there is evidence that classical, geometry-based techniques are better suited to the “motion-based” parts of the problem, while deep networks are more suitable for modeling appearance. In this work, we argue that the coupling of camera rotation and camera translation can create complex motion fields that are difficult for a deep network to untangle directly. We present a novel probabilistic model to estimate the camera’s rotation given the motion field. We then rectify the flow field to obtain a rotation-compensated motion field for subsequent segmentation. This strategy of first estimating camera motion, and then allowing a network to learn the remaining parts of the problem, yields improved results on the widely used DAVIS benchmark as well as the more recent motion segmentation data set MoCA (Moving Camouflaged Animals).

Automating seismic‐well tie via self‐supervised learning

AbstractAs an essential process in subsurface interpretation, seismic‐well tie aims at calibrating the well measurements in depth domain with the seismic records in time domain for reliable reservoir mapping and modelling. Such a process usually requires massive manual efforts, primarily extracting source wavelets to synthesize seismograms from well measurements and stretching/squeezing synthetic seismograms to match actual seismic signals, and thus becomes time‐consuming and labour intensive with the amount of wells expanding in a field. This paper formulates the seismic‐well tie problem from the perspective of computer vision and, considering the lack of manually prepared training labels in most of real projects, proposes automating it by a self‐supervised workflow of two key components. The first component is to extract time‐variant wavelets in the target interval using a dual‐task autoencoder, which is optimized by maximizing the spectrum similarity between the extracted wavelet and the actual seismic. The second component is to estimate the corresponding time‐shift between a pair of the synthetic seismogram and the actual seismic using a flow net, which is trained by a pseudo dataset derived from the actual well measurements. More specifically, the data generation consists with defining one‐dimensional time‐shift curves to warp the original less accurate time‐depth relationships and then synthesizing the corresponding seismograms based on the convolutional model at all available wells. When turning to the stage of inference at a target well, the proposed workflow automatically extracts a representative wavelet, generates the corresponding synthetic seismogram from the original time‐depth relationship and estimates the time‐shift curve necessary for revising the original time‐depth relationship that would lead to an optimal tying with the actual seismic at the well. The proposed workflow is tested on two field datasets, and both results demonstrate improved tying over traditional methods such as statistical wavelet extraction and dynamic time warping.

Neural Architecture Search Survey: A Computer Vision Perspective.

In recent years, deep learning (DL) has been widely studied using various methods across the globe, especially with respect to training methods and network structures, proving highly effective in a wide range of tasks and applications, including image, speech, and text recognition. One important aspect of this advancement is involved in the effort of designing and upgrading neural architectures, which has been consistently attempted thus far. However, designing such architectures requires the combined knowledge and know-how of experts from each relevant discipline and a series of trial-and-error steps. In this light, automated neural architecture search (NAS) methods are increasingly at the center of attention; this paper aimed at summarizing the basic concepts of NAS while providing an overview of recent studies on the applications of NAS. It is worth noting that most previous survey studies on NAS have been focused on perspectives of hardware or search strategies. To the best knowledge of the present authors, this study is the first to look at NAS from a computer vision perspective. In the present study, computer vision areas were categorized by task, and recent trends found in each study on NAS were analyzed in detail.

Ethical Considerations in Artificial Intelligence: A Comprehensive Disccusion from the Perspective of Computer Vision

This paper delves deeply into the multifaceted ethical challenges within the realm of computer vision, focusing intently on various ethical dimensions inherent in this cutting-edge field. It emphasizes the pressing need to address ethical concerns related to AI technologies, including algorithmic fairness, informed consent, public engagement, robust privacy protocols, transparency, and the integration of human judgment through human-in-the-loop systems. The study underscores the vital importance of collaboration among diverse stakeholders, including governments, businesses, academia, and society, to promote responsible and equitable AI practices within computer vision.Through meticulous examination, the paper highlights the urgency of balancing technological advancement with ethical considerations. It advocates for the development and implementation of ethical principles, ensuring that AI technologies align with societal values and promote fairness, transparency, and accountability. The collaborative efforts among various sectors are crucial to fostering an ethical framework that guides the responsible deployment of AI in the field of computer vision. By integrating ethical consciousness into the core of technological innovation, this approach aims to create a symbiotic relationship between artificial intelligence and society, ultimately benefiting humanity as a whole.

Visual Memory Neural Network for Artistic Graphic Design

Artistic graphic design is the aesthetic result of the designer’s fusion of various elements, with a high degree of independence. Considering the lack of significant visual design scope and aesthetic indicators of graphic design, our research aims to build an upgraded network model that can categorize different types of artistic graphics with labels and realize the free combination of graphic solutions. We realize the scheme reorganization of artistic graphic design from the perspective of computer vision and propose the artistic graphic design method based on memory neural network. We built a computer vision environment and reconstructed the computer vision network to set up an independent deep camera vision range calculation law. Considering the artistic graphic region segmentation problem, we propose the self-attentive mechanism, which can quantitatively segment different artistic graphic regions according to temporal features, before arranging them in a sequence to obtain the graphic region feature vector. We also add the LSTM structure based on the attention mechanism to match with the self-attention features of the graphical region segmentation module and pass the matched attention feature vector to the LSTM network to extract the labeled text feature information of the graphs. To test the effectiveness of our method, we build a database of artistic graphics and set up an adaptive training process. We also compared deep learning methods of the same type, and the experimental results proved that our method outperformed other deep methods in artistic graphic design by keeping the scheme reorganization accuracy and quantitative evaluation of artistic models above 90%.

Animal pose estimation: A closer look at the state-of-the-art, existing gaps and opportunities

Over the past few years, research on animal pose estimation in computer vision field has grown in many aspects such as 2D and 3D pose estimation, 3D mesh reconstruction, and behavior prediction. Promoted by deep learning, more and more animal pose estimation tools and animal pose datasets have also been made publicly available. However, compared to human pose estimation, which already has high accuracy and high applicability for complex scenes, animal pose estimation is still at a preliminary stage. The huge domain shift between each species, the scarce datasets, and uncooperative research subjects all pose intractable challenges to the development of robust and accurate animal pose estimation algorithms. In this review paper, we summarize the recent (from 2013 to 2021) work in animal pose estimation from computer vision perspective in order to present the state-of-the-art approaches and highlight the challenges they face in this field. We first categorize the various methods of animal pose estimation and present them according to several keywords. Also, we sort and introduce the released annotated image, video, and 3D models of animal poses as well as a promising substitute for real dataset. We also report the performances of the existing algorithms and visualize their results. Finally, we provide an in-depth analysis of the persisting obstacles in this field based on existing work, and offer potential solutions.

A comparison of deep learning methods for seismic impedance inversion

Deep learning is widely used for seismic impedance inversion, but few work provides in-depth research and analysis on designing the architectures of deep neural networks and choosing the network hyperparameters. This paper is dedicated to comprehensively studying on the significant aspects of deep neural networks that affect the inversion results. We experimentally reveal how network hyperparameters and architectures affect the inversion performance, and develop a series of methods which are proven to be effective in reconstructing high-frequency information in the estimated impedance model. Experiments demonstrate that the proposed multi-scale architecture is helpful to reconstruct more high-frequency details than a conventional network. Besides, the reconstruction of high-frequency information can be further promoted by introducing a perceptual loss and a generative adversarial network from the computer vision perspective. More importantly, the experimental results provide valuable references for designing proper network architectures in the seismic inversion problem.

A topography-aware approach to the automatic generation of urban road networks

Existing deep-learning tools for road network generation have limited applications in flat urban areas due to their overreliance on the geometric and spatial configurations of street networks and inadequate considerations of topographic information. This paper proposes a new method of street network generation based on a generative adversarial network by designing a pre-positioned geo-extractor module and a geo-merging bypath. The two improvements employ the complementary use of geometric configurations and topographic features to automate street network generation in both flat and hilly urban areas. Our experiments demonstrate that the improved model yields a more realistic prediction of street configurations than conventional image inpainting techniques. The model’s effectiveness is further enhanced when generating streets in hilly areas. Furthermore, the geo-extractor module provides insights from the computer vision perspective in recognizing when topographic information should be considered and which topographic information should receive more attention.

Measuring the rogue wave pattern triggered from Gaussian perturbations by deep learning.

Weak Gaussian perturbations on a plane wave background could trigger lots of rogue waves (RWs), due to modulational instability. Numerical simulations showed that these RWs seemed to have similar unit structure. However, to the best of our knowledge, there are no relative results to prove that these RWs have the similar patterns for different perturbations, partly due to that it is hard to measure the RW pattern automatically. In this work, we address these problems from the perspective of computer vision via using deep neural networks. We propose a rogue wave detection network (RWD-Net) model to automatically and accurately detect RWs in the images, which directly indicates they have the similar computer vision patterns. For this purpose, we herein meanwhile have designed and release the corresponding dataset, termed as rogue wave dataset-10K (RWD-10K), which has 10191 RW images with bounding box annotations for each RW unit. In our detection experiments, we get 99.29% average precision on the test splits of the proposed dataset. Finally, we derive our metric, termed as the density of RW units, to characterize the evolution of Gaussian perturbations and obtain the statistical results on them.

Application of Emotional Factors of Ink Symbols Evaluated by Network Model in Modern Visual Image Design

Accurate and fast recognition of ink symbols can enhance the perception ability of intelligent ink symbols in the environment, provide information input for the operation of intelligent ink symbols, and improve the perfection of ink painting. This paper proposes an end-to-end deep neural network model based on YOLOv3 for overall and individual recognition of ink symbols from a computer vision perspective. Ink symbol images generated by the simulation software are used for learning the overall and individual ink symbol detection models. Experiments demonstrate that the YOLOv3 detection algorithm has a good detection effect on ink symbolic targets, and the recognition of individual ink symbols has higher accuracy and flexibility, which provides a preliminary solution idea to solve the ink symbol information perception problem.

Extracting Objects’ Spatial–Temporal Information Based on Surveillance Videos and the Digital Surface Model

Surveillance systems focus on the image itself, mainly from the perspective of computer vision, which lacks integration with geographic information. It is difficult to obtain the location, size, and other spatial information of moving objects from surveillance systems, which lack any ability to couple with the geographical environment. To overcome such limitations, we propose a fusion framework of 3D geographic information and moving objects in surveillance video, which provides ideas for related research. We propose a general framework that can extract objects’ spatial–temporal information and visualize object trajectories in a 3D model. The framework does not rely on specific algorithms for determining the camera model, object extraction, or the mapping model. In our experiment, we used the Zhang Zhengyou calibration method and the EPNP method to determine the camera model, YOLOv5 and deep SORT to extract objects from a video, and an imaging ray intersection with the digital surface model to locate objects in the 3D geographical scene. The experimental results show that when the bounding box can thoroughly outline the entire object, the maximum error and root mean square error of the planar position are within 31 cm and 10 cm, respectively, and within 10 cm and 3 cm, respectively, in elevation. The errors of the average width and height of moving objects are within 5 cm and 2 cm, respectively, which is consistent with reality. To our knowledge, we first proposed the general fusion framework. This paper offers a solution to integrate 3D geographic information and surveillance video, which will not only provide a spatial perspective for intelligent video analysis, but also provide a new approach for the multi-dimensional expression of geographic information, object statistics, and object measurement.

Series Arc Fault Detection Based on Wavelet Compression Reconstruction Data Enhancement and Deep Residual Network

Series arc fault is the main cause of electrical fire. Because of the complex load types and the randomness of arc fault in low-voltage distribution system, it is difficult to obtain fault data for data-driven machine learning algorithm to achieve higher accuracy. Therefore, an arc fault detection model based on residual network (ResNet) is proposed from the perspective of computer vision, and an appropriate data enhancement method is given. We first analyze the time-domain current signals of different load arc faults by multilayer discrete wavelet analysis. The obtained five-layer discrete wavelet detail values are fused into a matrix, and the coefficient matrix is then converted into a phase space image (RGB color space) using a colormap index. The phase space feature map is made into a classification dataset according to the load and fault type, which is input into ResNet in the form of three channels (RGB) for convolution and classification recognition. Then, aiming at the over-fitting phenomenon of neural network caused by small sample, a data enhancement method based on wavelet compression reconstruction is proposed. Finally, we set up different datasets to compare our method with typical neural network regularization methods. The results show that our method effectively solves the over-fitting phenomenon of deep network ResNet152 and improves the accuracy of ResNet 50/101/152 to 97.91%, 96.30%, and 97.69%, respectively.

PRNU registration under scale and rotation transform based on convolutional neural networks

Assessing if an image comes from a specific device is fundamental in many application scenarios. The most promising techniques to solve this problem rely on the Photo Response Non Uniformity (PRNU), a unique trace left during image acquisition. A PRNU fingerprint is computed from several images of a given device, then it is compared with the probe residual noise by means of correlation. However, such a comparison requires that PRNUs are synchronized: even small image transformations can spoil this task. Most of the attempts to solve the registration problem rely on time consuming brute-force search, which is prone to missing detections and false positives. In this paper, the problem is addressed from a computer vision perspective, exploiting recent image registration techniques based on deep learning, and focusing on scaling and rotation transformations. Experiments show that the proposed method is both more accurate and faster than state-of-the-art approaches.

Understanding rooftop PV panel semantic segmentation of satellite and aerial images for better using machine learning

l Revealing highly class-imbalance and non-concentrated distribution of PV panel image data. l Finding the notable resolution threshold for effective PV panel semantic-segmentation. l Exploring homogeneous texture feature and heterogenous color feature of PV panel image data. l Providing recommendations on further improvements of PV panel semantic-segmentation. The photovoltaic (PV) industry boom and increased PV applications call for better planning based on accurate and updated data on the installed capacity. Compared with the manual statistical approach, which is often time-consuming and labor-intensive, using satellite/aerial images to estimate the existing PV installed capacity offers a new method with cost-effective and data-consistent features. Previous studies investigated the feasibility of segmenting PV panels from images involving machine learning technologies. However, due to the particular characteristics of PV panel semantic-segmentation, the machine learning tools need to be designed and applied with careful considerations of the issue formulation, data quality, and model explainability. This paper investigated the characteristics of PV panel semantic-segmentation from the perspective of computer vision. The results reveal that the PV panel image data has several specific characteristics: highly class-imbalance and non-concentrated distribution; homogeneous texture and heterogenous color features; and the notable resolution threshold for effective semantic-segmentation. Moreover, this paper provided recommendations for data obtaining and model design, aiming at each observed character from the viewpoints of recent solutions in computer vision, which can be helpful for future improvement of the PV panel semantic-segmentation.