NYU Dataset Research Articles

MQGA: A quantitative analysis of brain network hubs using multi-graph theoretical indices

Recent advancements in large-scale network studies have shown that connector hubs and provincial hubs are vital for coordinating complex cognitive tasks by facilitating information transfer between and within specialized modules. However, current methods for identifying these hubs often lack standardized measurement criteria, hindering quantitative analysis. This study proposes a novel computational method utilizing multi-graph theoretical index calculations to quantitatively analyze hub attributes in brain networks. Using benchmark network, random simulation network (N = 100), resting fMRI data from the ADHD-200 NYU dataset (HC = 110, ADHD = 146), and the Peking dataset (HC = 120, ADHD = 83), we introduce the Multi-criteria Quantitative Graph Analysis (MQGA) method, which employs betweenness centrality, degree centrality, and participation coefficient to determine the connector (con) hub index and provincial (pro) hub index. The method's accuracy, reliability, and stability were validated through correlation analysis of hub indices and labels, vulnerability tests, and consistency analysis across subjects. Results indicate that as network sparsity increases, the con hub index increases while the pro hub index decreases, with the optimal hub node index at 4 % sparsity. Vulnerability tests revealed that removing con nodes had a greater impact on network integrity than removing pro nodes. Both con and pro exhibited stability in consistency analyses, but con was more stable. The stability of hub scores in disease groups was significantly lower than in the healthy control group. High con values were found in the precuneus, postcentral gyrus, and precentral gyrus, whereas high pro values were identified in the precentral gyrus, postcentral gyrus, superior parietal lobule, precuneus, and superior temporal gyrus. This approach enhances the accuracy and sensitivity of hub node identification, facilitating precise comparisons and producing consistent, replicable results, advancing our understanding of brain network hub nodes, their roles in cognitive processes, and their implications for brain disease research.

Enhanced ADHD classification through deep learning and dynamic resting state fMRI analysis.

Attention Deficit Hyperactivity Disorder (ADHD) is characterized by deficits in attention, hyperactivity, and/or impulsivity. Resting-state functional connectivity analysis has emerged as a promising approach for ADHD classification using resting-state functional magnetic resonance imaging (rs-fMRI), although with limited accuracy. Recent studies have highlighted dynamic changes in functional connectivity patterns among ADHD children. In this study, we introduce Skip-Vote-Net, a novel deep learning-based network designed for classifying ADHD from typically developing children (TDC) by leveraging dynamic connectivity analysis on rs-fMRI data collected from 222 participants included in the NYU dataset within the ADHD-200 database. Initially, for each subject, functional connectivity matrices were constructed from overlapping segments using Pearson's correlation between mean time series of 116 regions of interest defined by the Automated Anatomical Labeling (AAL) 116 atlas. Skip-Vote-Net was then developed, employing a majority voting mechanism to classify ADHD/TDC children, as well as distinguishing between the two main subtypes: the inattentive subtype (ADHDI) and the predominantly combined subtype (ADHDC). The proposed method was evaluated across four classification scenarios: (1) two-class classification of ADHD from TD children using balanced data, (2) two-class classification between ADHD and TD children using unbalanced data, (3) two-class classification between ADHDI and ADHDC, and (4) three-class classification among ADHDI, ADHDC, and TD children. Using Skip-Vote-Net, we achieved mean classification accuracies of 97% ± 1.87 and 97.7% ± 2.2 for the balanced and unbalanced classification cases, respectively. Furthermore, the mean classification accuracy for discriminating between ADHDI and ADHDC reached 99.4% ± 1.21. Finally, the proposed method demonstrated an average accuracy of 98.86% ± 1.03 in classifying ADHDI, ADHDC, and TD children collectively. Our findings highlight the superior performance of Skip-Vote-Net over existing methods in the classification of ADHD, showcasing its potential as an effective diagnostic tool for identifying ADHD subtypes and distinguishing ADHD from typically developing children.

ASD-GANNet: A Generative Adversarial Network-Inspired Deep Learning Approach for the Classification of Autism Brain Disorder.

The classification of a pre-processed fMRI dataset using functional connectivity (FC)-based features is considered a challenging task because of the set of high-dimensional FC features and the small dataset size. To tackle this specific set of FC high-dimensional features and a small-sized dataset, we propose here a conditional Generative Adversarial Network (cGAN)-based dataset augmenter to first train the cGAN on computed connectivity features of NYU dataset and use the trained cGAN to generate synthetic connectivity features per category. After obtaining a sufficient number of connectivity features per category, a Multi-Head attention mechanism is used as a head for the classification. We name our proposed approach "ASD-GANNet", which is end-to-end and does not require hand-crafted features, as the Multi-Head attention mechanism focuses on the features that are more relevant. Moreover, we compare our results with the six available state-of-the-art techniques from the literature. Our proposed approach results using the "NYU" site as a training set for generating a cGAN-based synthetic dataset are promising. We achieve an overall 10-fold cross-validation-based accuracy of 82%, sensitivity of 82%, and specificity of 81%, outperforming available state-of-the art approaches. A sitewise comparison of our proposed approach also outperforms the available state-of-the-art, as out of the 17 sites, our proposed approach has better results in the 10 sites.

Enforcing high frequency enhancement in deep networks for simultaneous depth estimation and dehazing

Single image dehazing and monocular depth estimation algorithms are crucial for environmental perception for autonomous driving. However, few algorithms can simultaneously perform single-image dehazing and depth estimation to solve the problems of perceiving surrounding information for unmanned vehicles in haze. The current algorithm based on the Transformer and Convolutional Neural Networks cannot effectively extract high frequency information and low frequency information in the image, resulting in poor image dehazing and depth estimation ability. Therefore, we design the DADENet network to solve these problems. We enhanced sensitivity to high-frequency information by incorporating Sobel operators and Gaussian blur into the residual unit structure, resulting in the ResNet-Edge-Encoder block. Furthermore, we improved the capability to extract both high-frequency and low-frequency information from feature maps by effectively combining depthwise convolutions with attention structures in the Transformer-Edge-Decoder block. Additionally, we designed the Depth-Transmission loss function to leverage the relationship between depth maps and transmission maps, thereby enhancing the accuracy of both single-image dehazing and depth estimation tasks. Extensive experiments on the NYU, KITTI and Citycapes datasets demonstrate that our DADENet algorithm effectively performs single-image dehazing and depth estimation, surpassing classical and state-of-the-art algorithms.

BaSICNet: Lightweight 3-D Hand Pose Estimation Network Based on Biomechanical Structure Information for Dexterous Manipulator Teleoperation

The use of hand pose for dexterous manipulator teleoperation is an attractive method to the control of the multi-fingered manipulators. Furthermore, the advancement of the deep learning and depth sensors has encouraged the development of the 3D hand pose estimation. However, developing a 3D hand pose estimation method with an accurate and real-time performance is still a difficult task in computer vision. In this paper, a lightweight depth-based network named biomechanical structure information cascade network (BaSICNet) is proposed by considering the global and local structure of human hands to improve the performance through a cascade network and a bone-constraint loss function. In addition, the BaSICNet is applied to a five-fingered dexterous manipulator platform to achieve visual hand-based teleoperation. Extensive evaluations on two public datasets show that the BaSICNet can produce accurate and fast 3D hand poses (9.15mm and 7.59mm mean errors on NYU and MSRA datasets with 114.7 fps), and can achieve superior 3D hand pose estimation balance of accuracy and speed when compared with state-of-the-art (SOAT) methods. Experiment results on the dexterous manipulator platform also show that the BaSICNet can be applied well for teleoperation.

BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation.

Monocular depth estimation (MDE) is a fundamental task in computer vision and has drawn increasing attention. Recently, some methods reformulate it as a classification-regression task to boost the model performance, where continuous depth is estimated via a linear combination of predicted probability distributions and discrete bins. In this paper, we present a novel framework called BinsFormer, tailored for the classification-regression-based depth estimation. It mainly focuses on two crucial components in the specific task: 1) proper generation of adaptive bins and 2) sufficient interaction between probability distribution and bins predictions. To specify, we employ a Transformer decoder to generate bins, novelly viewing it as a direct set-to-set prediction problem. We further integrate a multi-scale decoder structure to achieve a comprehensive understanding of spatial geometry information and estimate depth maps in a coarse-to-fine manner. Moreover, an extra scene understanding query is proposed to improve the estimation accuracy, which turns out that models can implicitly learn useful information from the auxiliary environment classification task. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that BinsFormer surpasses state-of-the-art MDE methods with prominent margins. Code and pretrained models are made publicly available at https://github.com/zhyever/Monocular-Depth-Estimation-Toolbox/tree/ main/configs/binsformer.

Multi-modal fusion architecture search for camera-based semantic scene completion

Camera-based Semantic scene completion (SSC) aims to infer the 3D volumetric occupancy and semantic categories of a scene simultaneously from a single RGB image. The main challenge of camera-based SSC is the lack of geometry information compared with RGB-D SSC. Although the estimated depth from RGB image will help SSC to some extent, the depth prediction quality is far from the demand of SSC. To solve this problem, we propose a NAS-based multi-modal fusion method to incorporate the semantic and geometry information from other intermediate representations (predicted depth and predicted 2D segmentation) to form a more robust 2D feature representation. A key idea of this design is that explicit 2D semantic information could alleviate the misleading information of 3D distortions introduced by estimated depth. Specifically, we propose the Confidence-Block to automatically learn an optimal architecture for routing and obtaining the depth prediction confidence. We propose the two-level fusion search space by decomposing the fusion search space into fusion stage search space and fusion operation search space. Moreover, we propose a confidence-aware 2D–3D projection module to alleviate the 3D projection error. Extensive experiments show that our method outperforms the state-of-the-art method by a large margin using a single RGB image on NYU and NYUCAD datasets.

CATNet: Convolutional attention and transformer for monocular depth estimation

Monocular depth estimation has received more and more attention due to its wide range of application scenarios. In this paper, we propose a novel simple framework, called CATNet, which treats monocular depth estimation as an ordinal regression problem. At present, in order to obtain higher performance, the research on monocular depth estimation is achieved by increasing the amount of calculation and parameters of the model. Based on this, we propose a novel simple encoder–decoder architecture that aims to reduce the SOTA model parameters and complexity while keeping the depth estimation accuracy as high as possible rather than aiming for extremely lightweight. Meanwhile, in order to further refine the multi-scale information extracted by the encoder, we propose a Multi-dimensional Convolutional Attention (MCA) module. To enhance the extraction of global information for accurate pixel classification, we propose a Dual Attention Transformer (DAT) module to extract global features of images. Furthermore, experimental results on the KITTI and NYU datasets demonstrate that the advantage of our proposed framework is that it achieves almost equivalent depth estimation performance to the current SOTA with fewer parameters and lower complexity. To the best of our knowledge, CATNet is the first work that achieves nearly the same depth estimation accuracy as Transformer-based large model encoders with so few parameters.

Lightweight 3D hand pose estimation by cascading CNNs with reinforcement learning

This paper proposes a novel strategy for lightweight 3D hand pose estimation. The strategy decomposes the estimation process into feature extraction and feature exploitation, where feature extraction performs dimension reduction on the original input and outputs feature vectors. Feature exploitation is further analyzed and considered as a path optimization problem, and reinforcement learning (RL) is proved to be capable of tackling the problem accurately. A framework cascading convolutional neural networks (CNNs) and RL is next introduced to validate the effectiveness of the proposed strategy, where two different backbones are used to extract features, and RL is extended into continuous space to enhance accuracy. Ablation studies and experiments are carried out on NYU and ICVL datasets using the proposed strategy with continuous RL. The results show that the accuracy of continuous RL exceeds discrete RL, and the rapidity and accuracy leads the backbones. Comparative studies show the strategy achieves leading rapidity and accuracy in single-view depth-based methods.

DepthFormer: Exploiting Long-range Correlation and Local Information for Accurate Monocular Depth Estimation

This paper aims to address the problem of supervised monocular depth estimation. We start with a meticulous pilot study to demonstrate that the long-range correlation is essential for accurate depth estimation. Moreover, the Transformer and convolution are good at long-range and close-range depth estimation, respectively. Therefore, we propose to adopt a parallel encoder architecture consisting of a Transformer branch and a convolution branch. The former can model global context with the effective attention mechanism and the latter aims to preserve the local information as the Transformer lacks the spatial inductive bias in modeling such contents. However, independent branches lead to a shortage of connections between features. To bridge this gap, we design a hierarchical aggregation and heterogeneous interaction module to enhance the Transformer features and model the affinity between the heterogeneous features in a set-to-set translation manner. Due to the unbearable memory cost introduced by the global attention on high-resolution feature maps, we adopt the deformable scheme to reduce the complexity. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins. The effectiveness of each proposed module is elaborately evaluated through meticulous and intensive ablation studies.

Seed correlation analysis based on brain region activation for ADHD diagnosis in a large-scale resting state data set.

Attention-deficit/hyperactivity disorder (ADHD) is a neurodevelopmental disorder of multifactorial pathogenesis, which is often accompanied by dysfunction in several brain functional connectivity. Resting-state functional MRI have been used in ADHD, and they have been proposed as a possible biomarker of diagnosis information. This study's primary aim was to offer an effective seed-correlation analysis procedure to investigate the possible biomarker within resting state brain networks as diagnosis information. Resting-state functional magnetic resonance imaging (rs-fMRI) data of 149 childhood ADHD were analyzed. In this study, we proposed a two-step hierarchical analysis method to extract functional connectivity features and evaluation by linear classifiers and random sampling validation. The data-driven method-ReHo provides four brain regions (mPFC, temporal pole, motor area, and putamen) with regional homogeneity differences as second-level seeds for analyzing functional connectivity differences between distant brain regions. The procedure reduces the difficulty of seed selection (location, shape, and size) in estimations of brain interconnections, improving the search for an effective seed; The features proposed in our study achieved a success rate of 83.24% in identifying ADHD patients through random sampling (saving 25% as the test set, while the remaining data was the training set) validation (using a simple linear classifier), surpassing the use of traditional seeds. This preliminary study examines the feasibility of diagnosing ADHD by analyzing the resting-state fMRI data from the ADHD-200 NYU dataset. The data-driven model provides a precise way to find reliable seeds. Data-driven models offer precise methods for finding reliable seeds and are feasible across different datasets. Moreover, this phenomenon may reveal that using a data-driven approach to build a model specific to a single data set may be better than combining several data and creating a general model.

MuTr: Multi-Stage Transformer for Hand Pose Estimation from Full-Scene Depth Image.

This work presents a novel transformer-based method for hand pose estimation-DePOTR. We test the DePOTR method on four benchmark datasets, where DePOTR outperforms other transformer-based methods while achieving results on par with other state-of-the-art methods. To further demonstrate the strength of DePOTR, we propose a novel multi-stage approach from full-scene depth image-MuTr. MuTr removes the necessity of having two different models in the hand pose estimation pipeline-one for hand localization and one for pose estimation-while maintaining promising results. To the best of our knowledge, this is the first successful attempt to use the same model architecture in standard and simultaneously in full-scene image setup while achieving competitive results in both of them. On the NYU dataset, DePOTR and MuTr reach precision equal to 7.85 mm and 8.71 mm, respectively.

TriHorn-Net: A model for accurate depth-based 3D hand pose estimation

3D hand pose estimation methods have made significant progress recently. However, estimation accuracy is often far from sufficient for specific real-world applications, and thus there is significant room for improvement. This paper proposes TriHorn-Net, a novel model that uses specific innovations to improve hand pose estimation accuracy on depth images. The first innovation is decomposition of the 3D hand pose estimation into the estimation of 2D joint locations in the depth image space (UV), and the estimation of their corresponding depths aided by two complementary attention maps. This decomposition prevents depth estimation, which is a more difficult task, from interfering with the UV estimations at both the prediction and feature levels. The second innovation is PixDropout, which is, to the best of our knowledge, the first appearance-based data augmentation method for hand depth images. Experimental results demonstrate that the proposed model outperforms the state-of-the-art methods on three public benchmark datasets, achieving 5.73 mm, 7.13 mm, and 7.68 mm on the ICVL, MSRA, and NYU datasets, respectively. The proposed model achieves this performance with a relatively fast average inference speed of 9.25 ms per frame on an NVIDIA 1080Ti GPU, which, as discussed in the paper, places the proposed model among the fastest-performing 3D hand pose estimation models. Our implementation is available at https://github.com/mrezaei92/TriHorn-Net.

Nested DWT–Based CNN Architecture for Monocular Depth Estimation

Applications such as medical diagnosis, navigation, robotics, etc., require 3D images. Recently, deep learning networks have been extensively applied to estimate depth. Depth prediction from 2D images poses a problem that is both ill-posed and non-linear. Such networks are computationally and time-wise expensive as they have dense configurations. Further, the network performance depends on the trained model configuration, the loss functions used, and the dataset applied for training. We propose a moderately dense encoder-decoder network based on discrete wavelet decomposition and trainable coefficients (LL, LH, HL, HH). Our Nested Wavelet-Net (NDWTN) preserves the high-frequency information that is otherwise lost during the downsampling process in the encoder. Furthermore, we study the effect of activation functions, batch normalization, convolution layers, skip, etc., in our models. The network is trained with NYU datasets. Our network trains faster with good results.

From Front to Rear: 3D Semantic Scene Completion Through Planar Convolution and Attention-Based Network

Semantic Scene Completion (SSC) aims to reconstruct complete 3D scenes with precise voxel-wise semantics from the single-view incomplete input data, a crucial but highly challenging problem for scene understanding. Although SSC has seen significant progress due to the introduction of 2D semantic priors in recent years, the occluded parts, especially the rear-view of the scenes, are still poorly completed and segmented. To ameliorate this issue, we propose a novel deep learning framework for 3D SSC, named Planar Convolution and Attention-based Network (PCANet), to effectively extend high-precision predictions of the front-view surface to the rear-view occluded areas. Specifically, we decompose the traditional convolutional layer into three successive planar convolutions to form a Planar Convolution Residual (PCR) block, which maintains the planar features of the 3D scene. Afterward, the Planar Attention Module (PAM) is proposed to capture three different planar attentions and harvest the global context from the front surface to the rear occluded areas to improve the overall accuracy. Extensive experiments on the real NYU and NYUCAD datasets and the synthetic SUNCG-RGBD dataset demonstrate that our proposed framework can generate high-quality SSC results in both front and rear views and outperforms the state-of-the-art approaches trained in an end-to-end manner without additional data.

Occlusion Handler Density Networks for 3D Multimodal Joint Location of Hand Pose Hypothesis

Predicting the pose parameters during the hand pose estimation (HPE) process is an ill-posed challenge. This is due to severe self-occluded joints of the hand. The existing approaches for predicting pose parameters of the hand, utilize a single-value mapping of an input image to generate final pose output. This way makes it difficult to handle occlusion especially when it comes from the multimodal pose hypothesis. This paper introduces an effective method of handling multimodal joint occlusion using the negative log-likelihood of a multimodal mixture-of-Gaussians through a hybrid hierarchical mixture density network (HHMDN). The proposed approach generates multiple feasible hypotheses of 3D poses with visibility, unimodal and multimodal distribution units to locate joint visibility. The visible features are extracted and fed into the Convolutional Neural Networks (CNN) layer of the HHMDN for feature learning. Finally, the effectiveness of the proposed method is proved on ICVL, NYU, and BigHand public hand pose datasets. The imperative results show that the proposed method in this paper is effective as it achieves a visibility error of 30.3mm, which is less error compared to many state-of-the-art approaches that use different distributions of visible and occluded joints.

A Survey on GAN-Based Data Augmentation for Hand Pose Estimation Problem

Deep learning solutions for hand pose estimation are now very reliant on comprehensive datasets covering diverse camera perspectives, lighting conditions, shapes, and pose variations. While acquiring such datasets is a challenging task, several studies circumvent this problem by exploiting synthetic data, but this does not guarantee that they will work well in real situations mainly due to the gap between the distribution of synthetic and real data. One recent popular solution to the domain shift problem is learning the mapping function between different domains through generative adversarial networks. In this study, we present a comprehensive study on effective hand pose estimation approaches, which are comprised of the leveraged generative adversarial network (GAN), providing a comprehensive training dataset with different modalities. Benefiting from GAN, these algorithms can augment data to a variety of hand shapes and poses where data manipulation is intuitively controlled and greatly realistic. Next, we present related hand pose datasets and performance comparison of some of these methods for the hand pose estimation problem. The quantitative and qualitative results indicate that the state-of-the-art hand pose estimators can be greatly improved with the aid of the training data generated by these GAN-based data augmentation methods. These methods are able to beat the baseline approaches with better visual quality and higher values in most of the metrics (PCK and ME) on both the STB and NYU datasets. Finally, in conclusion, the limitation of the current methods and future directions are discussed.

End‐to‐end global to local convolutional neural network learning for hand pose recovery in depth data

Despite recent advances in 3-D pose estimation of human hands, thanks to the advent of convolutional neural networks (CNNs) and depth cameras, this task is still far from being solved in uncontrolled setups. This is mainly due to the highly non-linear dynamics of fingers and self-occlusions, which make hand model training a challenging task. In this study, a novel hierarchical tree-like structured CNN is exploited, in which branches are trained to become specialised in predefined subsets of hand joints called local poses. Further, local pose features, extracted from hierarchical CNN branches, are fused to learn higher order dependencies among joints in the final pose by end-to-end training. Lastly, the loss function used is also defined to incorporate appearance and physical constraints about doable hand motions and deformations. Finally, a non-rigid data augmentation approach is introduced to increase the amount of training depth data. Experimental results suggest that feeding a tree-shaped CNN, specialised in local poses, into a fusion network for modelling joints' correlations and dependencies, helps to increase the precision of final estimations, showing competitive results on NYU, MSRA, Hands17 and SyntheticHand datasets.

DarkASDNet: Classification of ASD on Functional MRI Using Deep Neural Network

Non-invasive whole-brain scans aid the diagnosis of neuropsychiatric disorder diseases such as autism, dementia, and brain cancer. The assessable analysis for autism spectrum disorders (ASD) is rationally challenging due to the limitations of publicly available datasets. For diagnostic or prognostic tools, functional Magnetic Resonance Imaging (fMRI) exposed affirmation to the biomarkers in neuroimaging research because of fMRI pickup inherent connectivity between the brain and regions. There are profound studies in ASD with introducing machine learning or deep learning methods that have manifested advanced steps for ASD predictions based on fMRI data. However, utmost antecedent models have an inadequacy in their capacity to manipulate performance metrics such as accuracy, precision, recall, and F1-score. To overcome these problems, we proposed an avant-garde DarkASDNet, which has the competence to extract features from a lower level to a higher level and bring out promising results. In this work, we considered 3D fMRI data to predict binary classification between ASD and typical control (TC). Firstly, we pre-processed the 3D fMRI data by adopting proper slice time correction and normalization. Then, we introduced a novel DarkASDNet which surpassed the benchmark accuracy for the classification of ASD. Our model's outcomes unveil that our proposed method established state-of-the-art accuracy of 94.70% to classify ASD vs. TC in ABIDE-I, NYU dataset. Finally, we contemplated our model by performing evaluation metrics including precision, recall, F1-score, ROC curve, and AUC score, and legitimize by distinguishing with recent literature descriptions to vindicate our outcomes. The proposed DarkASDNet architecture provides a novel benchmark approach for ASD classification using fMRI processed data.

Single Image Depth Estimation Using Edge Extraction Network and Dark Channel Prior

The key to the depth estimation from a single image lies in inferring the distance of various objects without copying texture while maintaining clear object boundaries. In this paper, we propose depth estimation from a single image using edge extraction network and dark channel prior (DCP). We build an edge extraction network based on generative adversarial networks (GANs) to select valid depth edges from a number of edges in an image. We use DCP to generate a transmission map that is able to represent distance from the camera. Transmission map is generated by conducting minimum value filtering on DCP. First, we concatenate the transmission map with the original RGB image to form a tensor, i.e. RGB + T. Second, we generate an initial depth image from the tensor through the generator by inferring depth from stacked residual blocks. Third, we compare the edge map of the initial depth image with that of the input RGB image to select valid depth edges. Both edge maps are generated by the edge extraction network. Finally, we distinguish real and fake on the generated depth image using a discriminator, and enhances the performance of the generator. Various experiments on NYU, Make3D and MPI Sintel datasets demonstrate that the proposed network generates clear edges in depth images as well as outperforms state-of-the-art methods in terms of visual quality and quantitative measurements.