V2 Dataset Research Articles

Multi-Scale Context Enhanced Network for Monocular Depth Estimation

Monocular depth estimation is a classical computer vision task. At present, most CNN methods cannot effectively combine high-level and low-level features, leading to the loss of details and blurring of boundaries. To solve the problem, we propose a Multi-Scale Context Enhanced Network (MCEN) to learn more abundant context and expand its receptive field for high-accuracy estimation. Our method employs CRE-HRNet (Context and Receptive Enhanced High-Resolution Network) with four branches ranging from low-dimension to high-dimension features to obtain richer contextual information and extract multi-scale features. It then uses RM (Refinement Module) adopting the residual dilated convolution to retains detailed information and improve the receptive field. Finally, non-local block enables our network to capture the longdistance context through its special non-local operation. Experiments with the NYU Depth V2 dataset show its outstanding performance.

An efficient supervised framework for music mood recognition using autoencoder‐based optimised support vector regression model

Music is the art of ‘language of emotions’. Recently, music mood recognition is an emerging task. An efficient supervised framework for music mood recognition using autoencoder-based optimised support vector regression (SVR) model is developed for the music emotion recognition. Our main intention is to increase the accuracy of emotion classification of music by considering text-dependent and non-text-dependent features. For the high level feature representation, stacked autoencoder is used with two hidden layers. Modified K-Medoid-based brain storm optimisation-based support vector regression (SVR_KMBSO) model is utilised for the emotion classification. Using the K-Medoid-based brain storm algorithm, the optimal parameters of the SVR are selected. The proposed framework utilises ISMIR2012 dataset and NJU_V1 dataset for English and for Hindi; online songs are also gathered and used for the music mood recognition. All the three datasets include songs based on four emotions like happy, angry, relax and sad. The experimental results are evaluated and compared with the existing classifiers including SVR, deep belief network (DBN) and Recurrent neural network (RNN). The proposed method SVR_KMBSO achieved high accuracy using three different datasets.

Dilated Fully Convolutional Neural Network for Depth Estimation from a Single Image

Depth prediction plays a key role in understanding a 3D scene. Several techniques have been developed throughout the years, among which Convolutional Neural Network has recently achieved state-of-the-art performance on estimating depth from a single image. However, traditional CNNs suffer from the lower resolution and information loss caused by the pooling layers. And oversized parameters generated from fully connected layers often lead to a exploded memory usage problem. In this paper, we present an advanced Dilated Fully Convolutional Neural Network to address the deficiencies. Taking advantages of the exponential expansion of the receptive field in dilated convolutions, our model can minimize the loss of resolution. It also reduces the amount of parameters significantly by replacing the fully connected layers with the fully convolutional layers. We show experimentally on NYU Depth V2 datasets that the depth prediction obtained from our model is considerably closer to ground truth than that from traditional CNNs techniques.

Visual question answering based on local-scene-aware referring expression generation

Visual question answering requires a deep understanding of both images and natural language. However, most methods mainly focus on visual concept; such as the relationships between various objects. The limited use of object categories combined with their relationships or simple question embedding is insufficient for representing complex scenes and explaining decisions. To address this limitation, we propose the use of text expressions generated for images, because such expressions have few structural constraints and can provide richer descriptions of images. The generated expressions can be incorporated with visual features and question embedding to obtain the question-relevant answer. A joint-embedding multi-head attention network is also proposed to model three different information modalities with co-attention. We quantitatively and qualitatively evaluated the proposed method on the VQA v2 dataset and compared it with state-of-the-art methods in terms of answer prediction. The quality of the generated expressions was also evaluated on the RefCOCO, RefCOCO+, and RefCOCOg datasets. Experimental results demonstrate the effectiveness of the proposed method and reveal that it outperformed all of the competing methods in terms of both quantitative and qualitative results.

SOSD-Net: Joint semantic object segmentation and depth estimation from monocular images

Depth estimation and semantic segmentation play essential roles in scene understanding. The state-of-the-art methods employ multi-task learning to simultaneously learn models for these two tasks at the pixel-wise level. They usually focus on sharing the common features or stitching feature maps from the corresponding branches. However, these methods lack in-depth consideration on the correlation of the geometric cues and the scene parsing. In this paper, we first introduce the concept of semantic objectness to exploit the geometric relationship of these two tasks through an analysis of the imaging process, then propose a Semantic Object Segmentation and Depth Estimation Network (SOSD-Net) based on the objectness assumption. To the best of our knowledge, SOSD-Net is the first network that exploits the geometry constraint for simultaneous monocular depth estimation and semantic segmentation. In addition, considering the mutual implicit relationship between these two tasks, we exploit the iterative idea from the expectation–maximization algorithm to train the proposed network more effectively. Extensive experimental results on the Cityscapes and NYU v2 dataset are presented to demonstrate the superior performance of the proposed approach.

Detecting DeepFake, FaceSwap and Face2Face facial forgeries using frequency CNN

The face of a person plays a vital role in any communication or visual content. To enhance this visual content, popular and easy accessible editing tools are used. However, there malicious usage is spreading disharmony in the society, by tampering video evidences, defaming a person’s image etc. Therefore a robust detection method is required to authenticate the visual content. Thus, a novel method is proposed to detect facial forgeries. The proposed method extracts faces from a target video and convert them into frequency domain using two dimensional global discrete Cosine transform (2D- GDCT). Thereafter, a 3 layered frequency convolutional neural network (fCNN) is employed to detect forged facial image. The proposed method is trained and tested on FaceForensics++ dataset and Celeb-DF(v2) dataset. In addition, its robustness is evaluated on standardized benchmark dataset and compared with the state-of-the-art methods to prove its effectiveness.

Boundary-induced and scene-aggregated network for monocular depth prediction

Monocular depth prediction is an important task in scene understanding. It aims to predict the dense depth of a single RGB image. With the development of deep learning, the performance of this task has made great improvements. However, two issues remain unresolved: (1) The deep feature encodes the wrong farthest region in a scene, which leads to a distorted 3D structure of the predicted depth; (2) The low-level features are insufficient utilized, which makes it even harder to estimate the depth near the edge with sudden depth change. To tackle these two issues, we propose the Boundary-induced and Scene-aggregated network (BS-Net). In this network, the Depth Correlation Encoder (DCE) is first designed to obtain the contextual correlations between the regions in an image, and perceive the farthest region by considering the correlations. Meanwhile, the Bottom-Up Boundary Fusion (BUBF) module is designed to extract accurate boundary that indicates depth change. Finally, the Stripe Refinement module (SRM) is designed to refine the dense depth induced by the boundary cue, which improves the boundary accuracy of the predicted depth. Several experimental results on the NYUD v2 dataset and the iBims-1 dataset illustrate the state-of-the-art performance of the proposed approach. And the SUN-RGBD dataset is employed to evaluate the generalization of our method. Code is available at https://github.com/XuefengBUPT/BS-Net.

Abstract PS11-04: Computational drug repositioning for the identification of new agents to sensitize drug-resistant breast tumors across treatment arms and molecular subtypes

Abstract Introduction: One of the principal limiting factors to achieving cures in patients with cancer is drug resistance. Drug repositioning is the application of FDA-approved drug compounds for novel indications beyond the scope of the drug’s original intended use. This approach offers advantages over traditional drug development by reducing development costs and providing shorter paths to approval, as drug safety has already been established during the drug’s original regulatory process. One approach for computational drug repositioning involves generating a disease gene expression signature and then identifying a drug that can reverse this disease signature. In this study, we extracted drug resistance signatures from the I-SPY 2 TRIAL by comparing gene expression profiles of responder and non-responder patients stratified by treatment and molecular subtype. We then applied our drug repositioning pipeline to predict compounds that can reverse the gene expression profiles of these drug resistance signatures. We hypothesize that reversing these drug resistance signatures will resensitize tumors to treatment and improve patient outcome. Methods: We first generated the drug resistance signatures by performing differential expression between responders (RCB 0/I) and non-responders (RCB III) within treatment arms and molecular subtypes. An optimal log fold-change cutoff was selected for each signature by identifying the cutoff that best separates the responder and non-responder samples using k-means clustering. We then applied our drug repositioning pipeline to identify compounds that significantly reverse these signatures using the drug perturbation profiles generated in a breast cancer cell line in the Connectivity Map v2 dataset. Briefly, the pipeline uses a non- parametric, rank-based pattern-matching strategy based on the Kolmogorov-Smirnov (KS) statistic to assess the enrichment of resistance genes in a ranked drug gene expression list. Significance of each prediction is estimated from a null distribution of scores generated from random gene signatures. Results: We found that few individual genes are shared among the resistance signatures across the treatment arms and molecular subtypes, with the most common genes present in only 5/17 of the treatment arm and molecular subtype groups. At the pathway-level, however, we found that immune-related pathways are generally enriched among the responders and estrogen-response pathways are generally enriched among the non-responders. Although most of our drug predictions are unique to treatment arms and molecular subtypes, our drug repositioning pipeline identified the selective estrogen receptor degrader (SERD) fulvestrant as a compound that can potentially reverse resistance across a majority of the treatment arms and molecular subtypes. Conclusion: We applied our drug repositioning pipeline to identify novel agents to sensitize drug-resistant tumors in the I-SPY 2+ clinical trial and identified a SERD, fulvestrant, as a potential candidate for multiple molecular subtypes and treatment arms. Citation Format: Katharine Yu, Amrita Basu, Christina Yau, Denise Wolf, Gillian Hirst, Laura Sit, Nicholas O’Grady, Thelma Brown, I-SPY 2 TRIAL Investigators, Angela DeMichele, Don Berry, Nola Hylton, Doug Yee, Laura Esserman, Laura van 't Veer, Marina Sirota. Computational drug repositioning for the identification of new agents to sensitize drug-resistant breast tumors across treatment arms and molecular subtypes [abstract]. In: Proceedings of the 2020 San Antonio Breast Cancer Virtual Symposium; 2020 Dec 8-11; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2021;81(4 Suppl):Abstract nr PS11-04.

Wavelet Transform and Deep Convolutional Neural Network-Based Smart Healthcare System for Gastrointestinal Disease Detection.

This work presents a smart healthcare system for the detection of various abnormalities present in the gastrointestinal (GI) region with the help of time-frequency analysis and convolutional neural network. In this regard, the KVASIR V2 dataset comprising of eight classes of GI-tract images such as Normal cecum, Normal pylorus, Normal Z-line, Esophagitis, Polyps, Ulcerative Colitis, Dyed and lifted polyp, and Dyed resection margins are used for training and validation. The initial phase of the work involves an image pre-processing step, followed by the extraction of approximate discrete wavelet transform coefficients. Each class of decomposed images is later given as input to a couple of considered convolutional neural network (CNN) models for training and testing in two different classification levels to recognize its predicted value. Afterward, the classification performance is measured through the following measuring indices: accuracy, precision, recall, specificity, and F1 score. The experimental result shows 97.25% and 93.75% of accuracy in the first level and second level of classification, respectively. Lastly, a comparative performance analysis is carried out with several other previously published works on a similar dataset where the proposed approach performs better than its contemporary methods.

Cocktail Glass Network: Fast Depth Estimation Using Channel to Space Unrolling

Depth-estimation from a single input image can be used in applications such as robotics and autonomous driving. Recently, depth-estimation networks with UNet encoder/decoder structures have been widely used. In these decoders, operations are repeated to gradually increase the image resolution, while decreasing the channel size. If the upsampling operation at a high magnification can be processed at once, the amount of computation in the decoder can be dramatically reduced. To achieve this, we propose a new network structure, i.e., a cocktail glass network. In this network, convolution layers in the decoder are reduced, and a novel fast upsampling method is used that is known as channel-to-space unrolling, which converts thick channel data into high-resolution data. The proposed method can be easily implemented using simple reshaping operations; therefore, it is suitable for reducing the depth-estimation network. Considering the experimental results based on the NYU V2 and KITTI datasets, we demonstrate that the proposed method reduces the amount of computation in the decoder by half, while maintaining the same level of accuracy; it can be used in both lightweight and large-model-capacity networks.

Fusion of Handcrafted and Deep Features for Forgery Detection in Digital Images

Content authentication of digital images has captured the attention of forensic experts and security researchers due to a multi-fold increase in the dissemination of multimedia data through the open and vulnerable Internet. Shrewd attackers successfully devise novel ways to challenge state-of-art forensic tools used for forgery detection in digital images. Feature engineering approaches have yielded up to 97% accuracy on benchmarked datasets. Deep learning approaches have shown promising results in various image classification problems but cannot find hidden patterns in digital images, which can reliably detect image forgeries. State-of-art accuracy of deep learning approaches for forgery detection is up to 98% on benchmarked datasets. The objective of the proposed approach is to further escalate the detection accuracy, pushing it near 100%. In this paper, a synergy of handcrafted features based on color characteristics and deep features using the image’s luminance channel is employed to mine patterns responsible for accurate forgery detection. In the first Stream, 648-D Markov-based features are computed from the quaternion discrete cosine transform of the image. In the second Stream, the luminance channel of YCbCr colorspace is used to extract the Local Binary Pattern of the image. Further, local binary feature maps are fed to the pre-trained ResNet-18 model to obtain a 512-D feature vector called ’ResFeats’ from the last layer of the model’s convolutional base portion. The handcrafted features from Stream I and ResFeats from Stream II are combined to form an 1160-D feature vector. Further, classification is performed using a shallow neural network, and the method is tested on CASIA v1 and CASIA v2 datasets. The accuracy of the proposed fusion-based approach is 99.3% on benchmark datasets.

Monocular Depth Estimation Based on Multi-Scale Depth Map Fusion

Monocular depth estimation is a basic task in machine vision. In recent years, the performance of monocular depth estimation has been greatly improved. However, most depth estimation networks are based on a very deep network to extract features that lead to a large amount of information lost. The loss of object information is particularly serious in the encoding and decoding process. This information loss leads to the estimated depth maps lacking object structure detail and have non-clear edges. Especially in a complex indoor environment, which is our research focus in this paper, the consequences of this loss of information are particularly serious. To solve this problem, we propose a Dense feature fusion network that uses a feature pyramid to aggregate various scale features. Furthermore, to improve the fusion effectiveness of decoded object contour information and depth information, we propose an adaptive depth fusion module, which allows the fusion network to fuse various scale depth maps adaptively to increase object information in the predicted depth map. Unlike other work predicting depth maps relying on U-NET architecture, our depth map predicted by fusing multi-scale depth maps. These depth maps have their own characteristics. By fusing them, we can estimate depth maps that not only include accurate depth information but also have rich object contour and structure detail. Experiments indicate that the proposed model can predict depth maps with more object information than other prework, and our model also shows competitive accuracy. Furthermore, compared with other contemporary techniques, our method gets state-of-the-art in edge accuracy on the NYU Depth V2 dataset.

Multi-Channel Remote Photoplethysmography Correspondence Feature for 3D Mask Face Presentation Attack Detection

With the advancement of 3D printing technologies, 3D mask presentation attack becomes a critical challenge in face recognition. To tackle the 3D mask presentation attack detection (PAD), remote Photoplethysmography (rPPG) is employed as an intrinsic detection cue which is independent of the mask material and appearance quality. Although the effectiveness of existing rPPG-based methods has been verified, they may not be robust enough when rPPG signals are contaminated by noise. To identify the heartbeat information from the noisy raw rPPG signals, we propose a new 3D mask PAD feature, multi-channel rPPG correspondence feature (MCCFrPPG) with the global noise-aware template learning and verification framework. To further boost the discriminability, temporal variation of the rPPG signal is considered and extracted through the multi-channel time-frequency analysis scheme. This paper also extends HKBU-MARs V2 dataset with more customized high-quality masks and increases the number of videos by two times. Comprehensive experiments were performed on existing 3D mask datasets and the extended HKBU-MARs V2+, which totally covers 3 types of masks, 12 different light settings and 6 cameras. The results not only justify the effectiveness and robustness of the proposed MCCFrPPG on 3D mask attacks but also indicate its potential on handling the replay attack with camera motion and dim light.

Encoder-Decoder Structure With the Feature Pyramid for Depth Estimation From a Single Image

We address the problem of depth estimation from a single monocular image in the paper. Depth estimation from a single image is an ill-posed and inherently ambiguous problem. In the paper, we propose an encoder-decoder structure with the feature pyramid to predict the depth map from a single RGB image. More specifically, the feature pyramid is used to detect objects of different scales in the image. The encoder structure aims to extract the most representative information from the original image through a series of convolution operations and to reduce the resolution of the input image. We adopt Res2-50 as the encoder to extract important features. The decoder section uses a novel upsampling structure to improve the output resolution. Then, we also propose a novel loss function that adds gradient loss and surface normal loss to the depth loss, which can predict not only the global depth but also the depth of fuzzy edges and small objects. Additionally, we use Adam as our optimization function to optimize our network and speed up convergence. Our extensive experimental evaluation proves the efficiency and effectiveness of the method, which is competitive with previous methods on the Make3D dataset and outperforms state-of-the-art methods on the NYU Depth v2 dataset.

Mixed-Scale Unet Based on Dense Atrous Pyramid for Monocular Depth Estimation

Monocular depth estimation is an undirected problem, so constructing a network to predict better image depth information is an important research topic. This paper proposes a mixed-scale Unet network (MAPUnet) with a dense atrous pyramid based on the coder-decoder structure widely used in computer vision. We innovatively introduce the Unet++ structure of the image segmentation network for depth estimation. We reset the number of convolutional layers of the network under the framework of the Unet++ network and innovatively connect the decoders densely. Moreover, by choosing the appropriate size of the atrous radius, we form a dense atrous pyramid based on different feature layers to better connect the features in the deep and shallow layers of the network. To verify the effectiveness of the proposed algorithm, we test the network on the KITTI dataset and the NYU Depth V2 dataset. We compare the network with the current state-of-the-art methods. The proposed method has higher accuracy and has steadily improved relative to the threshold of accuracy and root-mean-square error. We also conduct ablation studies, studies targeting the effectiveness of the network framework, and discussions on the convergence time and parameter complexity of the network. We will open-source the code at https://github.com/yang-yi-fan/MAPUnet .

Vision-Based Fall Detection Using ST-GCN

Falls are a growing issue in society and has become a hot topic in the healthcare domain. Falls are more likely to occur to due to age or health problems such as cardiovascular issues and muscle weakness. In this work we focus on fall detection. The aftereffects of falls often lead to the use of prescription pain medications. We are motivated to help prevent suicide attempts by overdose in the Canadian correctional services. Most previous studies were based on hand-crafted features which limit the robustness and generality of the system. We therefore propose a general vision-based system, using Spatial Temporal Graph Convolutional Networks (ST-GCN). This system has proven its efficiency and robustness in the action recognition domain. Contrary to previous works, this model can be applied directly to new data without the need to retrain the model while offering good accuracy. Additionally, with the help of transfer learning we can solve the insufficient data problem. By using three public datasets: the NTU RGB-D dataset, the TST Fall detection dataset v2 and the Fallfree dataset to validate our method, we achieved a 100% accuracy, surpassing the state-of-the-art.

Efficient Framework for Genetic Algorithm-Based Correlation Power Analysis

Various Artificial Intelligence (AI) techniques are combined with classic side-channel methods to improve the efficiency of attacks. Among them, Genetic-Algorithms-based Correlation Power Analysis (GA-CPA) is proposed to launch attacks on hardware cryptosystems to extract the secret key efficiently. However, the convergence efficiency of GA-CPA is unsatisfactory due to two problems: the randomly generated initial population generally have low fitness, and the mutation operation in each iteration hardly produces high-quality individuals because of the confusion and diffusion characteristics of S-boxes. In this paper, we propose an analysis framework of GA-CPA which focuses on solving these two problems. First, we explore the list of candidate key bytes which is the result of Correlation Power Analysis (CPA) on a limited number of power traces, so that the population can be initialized with high quality candidates. Second, we improve the mutation operation by guiding the candidate key to mutate in a higher-fitness direction instead of randomly. Third, we make full use of the fitness calculation method and combine it with key enumeration algorithms to further improve the efficiency of key recovery. Simulation experimental results show that our method reduces the number of traces by 33.3% and 43.9% compared to CPA with key enumeration and GA-CPA respectively when the success rate is fixed to 90%. Real experiments performed on SAKURA-G confirm that the number of traces required in our method is much less than the numbers of traces required in CPA and GA-CPA. Besides, we adjust our method to deal with DPA contest v1 dataset, and achieve a better result of 40.76 traces than the winning proposal of 42.42 traces. The computation cost of our proposal is nearly 16.7% of the winner.

On deep learning techniques to boost monocular depth estimation for autonomous navigation

Inferring the depth of images is a fundamental inverse problem within the field of Computer Vision since depth information is obtained through 2D images, which can be generated from infinite possibilities of observed real scenes. Benefiting from the progress of Convolutional Neural Networks (CNNs) to explore structural features and spatial image information, Single Image Depth Estimation (SIDE) is often highlighted in scopes of scientific and technological innovation, as this concept provides advantages related to its low implementation cost and robustness to environmental conditions. In the context of autonomous vehicles, state-of-the-art CNNs optimize the SIDE task by producing high-quality depth maps, which are essential during the autonomous navigation process in different locations. However, such networks are usually supervised by sparse and noisy depth data, from Light Detection and Ranging (LiDAR) laser scans, and are carried out at high computational cost, requiring high-performance Graphic Processing Units (GPUs). Therefore, we propose a new lightweight and fast supervised CNN architecture combined with novel feature extraction models which are designed for real-world autonomous navigation. We also introduce an efficient surface normals module, jointly with a simple geometric 2.5D loss function, to solve SIDE problems. We also innovate by incorporating multiple Deep Learning techniques, such as the use of densification algorithms and additional semantic, surface normals and depth information to train our framework. The method introduced in this work focuses on robotic applications in indoor and outdoor environments and its results are evaluated on the competitive and publicly available NYU Depth V2 and KITTI Depth datasets.

Multi-Attention Network for Unsupervised Video Object Segmentation

In recently years, some useful unsupervised video object segmentation methods that emphasize the common information in videos have been proposed. Despite the effectiveness of these methods, they ignore the information from the shallow layers of the network and thus fail to segment the details of the objects. To address this problem, we propose a multi-attention network for unsupervised video object segmentation (MANet). Recent studies show that the deep layers of networks are sensitive to high-level semantic information but messy details, while it is opposite for shallow layers. From this insight, a multi-attention module is designed by taking into account the information from the shallow layers in addition to that from the deep layers. This module can distinguish the primary object and segment the details of the object effectively by enhancing the common information between video frames while combing the features from the shallow layers and the deep layers. Experimental results on the DAVIS-2016 and SegTrack v2 datasets show that our network outperforms the state-of-the-art methods.

The study of skeleton description reduction in the human fall-detection task

Accurate and reliable real-time fall detection is a key aspect of any intelligent elderly people care system. A lot of modern RGB-D cameras can provide a skeleton description of a human figure as a compact pose presentation. This makes it possible to use this description for further analysis without access to real video and, thus, to increase the privacy of the whole system. The skeleton description reduction based on the anthropometrical characteristics of a human body is proposed. The experimental study on the TST Fall Detection dataset v2 by the Leave-One-Person-Out method shows that the proposed skeleton description reduction technique provides better recognition quality and increases the overall performance of a Fall-Detection System.