Long-term Temporal Information Research Articles

Markov Progressive Framework, a Universal Paradigm for Modeling Long Videos.

Compared to images, video, as an increasingly mainstream visual media, contains more semantic information. For this reason, the computational complexity of video models is an order of magnitude larger than their image-level counterparts, which increases linearly with the square number of frames. Constrained by computational resources, training video models to learn long-term temporal semantics end-to-end is quite a challenge. Currently, the main-stream method is to split a raw video into clips, leading to incomplete fragmentary temporal information flow and failure of modeling long-term semantics. To solve this problem, in this paper, we design the Markov Progressive framework (MaPro), a theoretical framework consisting of the progressive modeling method and a paradigm model tailored for it. Inspired by natural language processing techniques dealing with long sentences, the core idea of MaPro is to find a paradigm model consisting of proposed Markov operators which can be trained in multiple sequential steps and ensure that the multi-step progressive modeling is equivalent to the conventional end-to-end modeling. By training the paradigm model under the progressive method, we are able to model long videos end-to-end with limited resources and ensure the effective transmission of long-term temporal information. We provide detailed implementations of this theoretical system on the mainstream CNN- and Transformer-based models, where they are modified to conform to the Markov paradigm. The theoretical paradigm as a basic model is the lower bound of the model efficiency. With it, we further explore more sophisticated designs for CNN and Transformer-based methods specifically. As a general and robust training method, we experimentally demonstrate that it yields significant performance improvements on different backbones and datasets. As an illustrative example, the proposed method improves the SlowOnly network by 4.1 mAP on Charades and 2.5 top-1 accuracy on Kinetics. And for TimeSformer, MaPro improves its performance on Kinetics by 2.0 top-1 accuracy. Importantly, all these improvements are achieved with a little parameter and computation overhead. We hope the MaPro method can provide the community with new insight into modeling long videos.

Adaptive temporal transformer method for short-term wind power forecasting considering shift in time series distribution

In wind power prediction, the input probability distributions in the different sub-periods are shifted owing to the strong randomness of the input features, such as wind speed and direction. This may violate the assumption for machine learning that the training and test data meet the condition of being independent and identically distributed, resulting in an insufficient generalization ability of the prediction model that is trained with the training data and applied to unknown test data. To address this problem, this study proposes an adaptive temporal transformer method for short-term wind power forecasting. First, a temporal transformer model with a gate recurrent unit and multi-head attention layers was used to extract the short- and long-term temporal information of the multiple input variables. Then, an adaptive learning strategy consisting of two stages—temporal distribution characterization and temporal distribution matching—was developed to explore the common knowledge hidden in each sub-period. The case results for an actual wind farm in northwest China showed that the proposed method could effectively weaken the adverse effects of the shifts in time series distribution on forecasting and improve the accuracy and generalization of short-term wind power prediction.

LSTPNet: Long short-term perception network for dynamic facial expression recognition in the wild

In-the-wild dynamic facial expression recognition (DFER) is a very challenging task, and previous methods based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformers emphasize the extraction of either short-term temporal information or long-term temporal information from facial video sequences. Different from existing methods, this paper proposes a long short-term perceptimon network (LSTPNet) for dynamic facial expression recognition, taking into account the joint perception of the above two temporal cues to benefit the DFER task. Specifically, we propose a long short-term temporal Transformer (LSTformer) which can perceive both long-term and short-term temporal information effectively. In addition, we introduce a temporal channel excitation (TCE) module extended from the previous notable efficient channel attention (ECA) module, in order to establish temporal attention for intermediate features within the backbone network, and obtain more temporally representative features. Experimental results on three benchmark datasets demonstrate the state-of-the-art performance of the proposed LSTPNet. The code will be available at https://github.com/LLFabiann/LSTPNet/.

Learning Complementary Spatial-Temporal Transformer for Video Salient Object Detection.

Besides combining appearance and motion information, another crucial factor for video salient object detection (VSOD) is to mine spatial-temporal (ST) knowledge, including complementary long-short temporal cues and global-local spatial context from neighboring frames. However, the existing methods only explored part of them and ignored their complementarity. In this article, we propose a novel complementary ST transformer (CoSTFormer) for VSOD, which has a short-global branch and a long-local branch to aggregate complementary ST contexts. The former integrates the global context from the neighboring two frames using dense pairwise attention, while the latter is designed to fuse long-term temporal information from more consecutive frames with local attention windows. In this way, we decompose the ST context into a short-global part and a long-local part and leverage the powerful transformer to model the context relationship and learn their complementarity. To solve the contradiction between local window attention and object motion, we propose a novel flow-guided window attention (FGWA) mechanism to align the attention windows with object and camera movements. Furthermore, we deploy CoSTFormer on fused appearance and motion features, thus enabling the effective combination of all three VSOD factors. Besides, we present a pseudo video generation method to synthesize sufficient video clips from static images for training ST saliency models. Extensive experiments have verified the effectiveness of our method and illustrated that we achieve new state-of-the-art results on several benchmark datasets.

Visual-semantic network: a visual and semantic enhanced model for gesture recognition

Gesture recognition has attracted considerable attention and made encouraging progress in recent years due to its great potential in applications. However, the spatial and temporal modeling in gesture recognition is still a problem to be solved. Specifically, existing works lack efficient temporal modeling and effective spatial attention capacity. To efficiently model temporal information, we first propose a long- and short-term temporal shift module (LS-TSM) that models the long-term and short-term temporal information simultaneously. Then, we propose a spatial attention module (SAM) that focuses on where the change primarily occurs to obtain effective spatial attention capacity. In addition, the semantic relationship among gestures is helpful in gesture recognition. However, this is usually neglected by previous works. Therefore, we propose a label relation module (LRM) that takes full advantage of the relationship among classes based on their labels’ semantic information. To explore the best form of LRM, we design four different semantic reconstruction methods to incorporate the semantic relationship information into the class label’s semantic space. We perform extensive ablation studies to analyze the best settings of each module. The best form of LRM is utilized to build our visual-semantic network (VS Network), which achieves the state-of-the-art performance on two gesture datasets, i.e., EgoGesture and NVGesture.

Financial sentiment classification with fresh and hot public opinions

Financial sentiment analysis aims to extract public opinion about an institution to help financial researchers make better decisions. To predict sentiment more accurately, it is necessary for models to improve their capability to capture long-term temporal information and support multi-user interaction. However, existing methods only analyze sentiment based on one comment from a user, which fails to fully exploit the latent emotions of the public, and they lack effective temporal modeling and interaction capabilities. In this paper, we analyze a company from two perspectives to alleviate the above issues: (1) the fresh opinions can reflect timely public attitudes towards a company, while (2) the hot opinions provide the most influential views. A comprehensive exploration of fresh and hot financial sentiment can help researchers make more accurate determinations. To this end, we propose a novel financial sentiment classification framework (FSCN), that can capture temporal information and interact with the opinions of users to make a more comprehensive decision. Our approach takes into account the inherent temporal dependencies in public opinions and combines both views of information to achieve an accurate classification of financial sentiment. Specifically, the FSCN contains (1) a multi-opinion extractor to filter and extract features from massively fresh and hot opinions, respectively. (2) a fresh-hot bilinear pooling (FHBP) module to effectively fuse fresh and hot features. Additionally, to verify the effectiveness of the proposed method, we crawl data from the Internet and create a real-world public opinion dataset that consists of 79,350 comments from 837 companies. Extensive experiments demonstrate that our framework achieves state-of-the-art results on this real-world dataset and is capable of providing reliable service in the financial system. Codes will be released at https://github.com/zjfgh2015/FSCN.

MLA-TCN: Multioutput Prediction of Dam Displacement Based on Temporal Convolutional Network with Attention Mechanism

The displacement of concrete dams effectively reflects their structural integrity and operational status. Therefore, establishing a model for predicting the displacement of concrete dams and studying the evolution mechanism of dam displacement is essential for monitoring the structural safety of dams. Current data-driven models utilize artificial data that cannot reflect the actual status of dams for network training. They also have difficulty extracting the temporal patterns from long-term dependencies and obtaining the interactions between the targets and variables. To address such problems, we propose a novel model for predicting the displacement of dams based on the temporal convolutional network (TCN) with the attention mechanism and multioutput regression branches, named MLA-TCN (where MLA is multioutput model with attention mechanism). The attention mechanism implements information screening and weight distribution based on the importance of the input variables. The TCN extracts long-term temporal information using the dilated causal convolutional network and residual connection, and the multioutput regression branch achieves simultaneous multitarget prediction by establishing multiple regression tasks. Finally, the applicability of the proposed model is demonstrated using data on a concrete gravity dam within 14 years, and its accuracy is validated by comparing it with seven state-of-the-art benchmarks. The results show that the MLA-TCN model, with a mean absolute error (MAE) of 0.05 mm, a root-mean-square error (RMSE) of 0.07 mm, and a coefficient of determination (R2) of 0.99, has a comparably high predictive capability and outperforms the benchmarks, providing an accurate and effective method to estimate the displacement of dams.

Sparse self-attention guided generative adversarial networks for time-series generation

AbstractRemarkable progress has been achieved in generative modeling for time-series data, where the dominating models are generally generative adversarial networks (GANs) based on deep recurrent or convolutional neural networks. Most existing GANs for time-series generation focus on preserving correlations across time. Although these models may help in capturing long-term dependencies, their capacity to pay varying degrees of attention over different time steps is inadequate. In this paper, we propose SparseGAN, a novel sparse self-attention-based GANs that allows for attention-driven, long-memory modeling for regular and irregular time-series generation through learned embedding space. This way, it can yield a more informative representation for time-series generation while using original data for supervision. We evaluate the effectiveness of the proposed model using synthetic and real-world datasets. The experimental findings indicate that forecasting models trained on SparseGAN-generated data perform comparably to forecasting models trained on real data for both regularly and irregularly sampled time series. Moreover, the results demonstrate that our proposed generative model is superior to the current state-of-the-art models for data augmentation in the low-resource regime and introduces a novel method for generating realistic synthetic time-series data by leveraging long-term structural and temporal information.

A Spatio-Temporal Graph Convolutional Network for Air Quality Prediction

Air pollution is a pressing issue that poses significant threats to human health and the ecological environment. The accurate prediction of air quality is crucial to enable management authorities and vulnerable populations to take measures to minimize their exposure to hazardous pollutants. Although many methods have been developed to predict air quality data, the spatio-temporal correlation of air quality data is complex and nonstationary, which makes air quality prediction still challenging. To address this, we propose a novel spatio-temporal neural network, GCNInformer, that combines the graph convolution network with Informer to predict air quality data. GCNInformer incorporates information about the spatial correlations among different monitoring sites through GCN layers and acquires both short-term and long-term temporal information in air quality data through Informer layers. Moreover, GCNInformer uses MLP layers to learn low-dimensional representations from meteorological and air quality data. These designs give GCNInformer the ability to capture the complex and nonstationary relationships between air pollutants and their surrounding environment, allowing for more accurate predictions. The experimental results demonstrate that GCNInformer outperforms other methods in predicting both short-term and long-term air quality data. Thus, the use of GCNInformer can provide useful information for air pollutant prevention and management, which can greatly improve public health by alerting individuals and communities to potential air quality hazards.

DMNet: A Personalized Risk Assessment Framework for Elderly People With Type 2 Diabetes.

Type 2 diabetes is the most common chronic disease for the elderly people. This disease is difficult to be cured and causes continued medical expenses. The early and personalized risk assessment of type 2 diabetes is necessary. So far, various type 2 diabetes risk prediction methods have been proposed. However, these methods have three major issues: 1) not fully considering the importance of personal information and rating information of healthcare system, 2) not adopting the long-term temporal information, and 3) not comprehensively capturing the correlation between the diabetes risk factor categories. To address these issues, the personalized risk assessment framework for elderly people with type 2 diabetes is needed. However, it is very challenging due to two reasons, namely imbalanced label distribution and high-dimensional features. In this paper, we propose diabetes mellitus network framework (DMNet) for type 2 diabetes risk assessment of elderly people. Specifically, we propose tandem long short-term memory to extract the long-term temporal information of different diabetes risk categories. In addition, the tandem mechanism is used to capture the correlation between the diabetes risk factor categories. To balance the label distribution, we adopt the method of synthetic minority over-sampling technique with Tomek links. To form the better feature representations, we utilize entity embedding to solve the problem of high-dimensional features. To evaluate the performance of our proposed method, we conduct the experiments on a real-world dataset called Research on Early Life and Aging Trends and Effects. The experiment results show that DMNet outperforms the baseline methods in terms of six evaluation metrics (i.e., accuracy of 0.94, balanced accuracy of 0.94, precision of 0.95, F1-score of 0.95, recall of 0.95 and AUC of 0.94).

A Two-Stream Neural Network for Pose-Based Hand Gesture Recognition

Pose-based hand gesture recognition has been widely studied in the recent years. Compared with full body action recognition, hand gesture involves joints that are more spatially closely distributed with stronger collaboration. This nature requires a different approach from action recognition to capturing the complex spatial features. Many gesture categories, such as “Grab” and “Pinch,” have very similar motion or temporal patterns posing a challenge on temporal processing. To address these challenges, this article proposes a two-stream neural network with one stream being a self-attention-based graph convolutional network (SAGCN) extracting the short-term temporal information and hierarchical spatial information, and the other being a residual-connection-enhanced bidirectional independently recurrent neural network (IndRNN) for extracting long-term temporal information. The SAGCN has a dynamic self-attention mechanism to adaptively exploit the relationships of all hand joints in addition to the fixed topology and local feature extraction in the GCNs. The proposed method effectively takes advantage of the GCN and IndRNN to capture the temporal-spatial information. The widely used Dynamic Hand Gesture dataset (two evaluation protocols) and First-Person Hand Action dataset are used to validate its effectiveness and our method achieves state-of-the-art performance with 96.31%, 94.05%, and 90.26%, respectively, in terms of recognition accuracy.

A Global 250-m Downscaled NDVI Product from 1982 to 2018

Satellite-based normalized difference vegetation index (NDVI) time series data are useful for monitoring the changes in vegetation ecosystems in the context of global climate change. However, most of the current NDVI products cannot effectively reconcile high spatial resolution and continuous observations in time. Here, to produce a global-scale, long-term, and high-resolution NDVI database, we developed a simple and new data downscaling approach. The downscaling algorithm considers the pixel-wise ratios of the coefficient of variation (CV) between the coarse- and fine-resolution NDVI data and relative changes in the NDVI against a baseline period. The algorithm successfully created a worldwide monthly NDVI database with 250 m resolution from 1982 to 2018 by translating the fine spatial information from MODIS (Moderate-resolution Imaging Spectroradiometer) data and the long-term temporal information from AVHRR (Advanced Very High Resolution Radiometer) data. We employed the evaluation indices of root mean square error (RMSE), mean absolute error (MAE), and Pearson’s correlation coefficient (Pearson’s R) to assess the accuracy of the downscaled data against the MODIS NDVI. Both the RMSE and MAE values at the regional and global scales are typically between 0 and 0.2, whereas the Pearson’s R values are mostly above 0.7, which implies that the downscaled NDVI product is similar to the MODIS NDVI product. We then used the downscaled data to monitor the NDVI changes in different plant types and places with significant vegetation heterogeneity, as well as to investigate global vegetation trends over the last four decades. The Google Earth Engine platform was used for all the data downscaling processes, and here we provide a code for users to easily acquire data corresponding to any part of the world. The downscaled global-scale NDVI time series has high potential for the monitoring of the long-term temporal and spatial dynamics of terrestrial ecosystems under changing environments.

SleepContextNet: A temporal context network for automatic sleep staging based single-channel EEG

Background and objective: Single-channel EEG is the most popular choice of sensing modality in sleep staging studies, because it widely conforms to the sleep staging guidelines. The current deep learning method using single-channel EEG signals for sleep staging mainly extracts the features of its surrounding epochs to obtain the short-term temporal context information of EEG epochs, and ignore the influence of the long-term temporal context information on sleep staging. However, the long-term context information includes sleep stage transition rules in a sleep cycle, which can further improve the performance of sleep staging. The aim of this research is to develop a temporal context network to capture the long-term context between EEG sleep stages.Methods: In this paper, we design a sleep staging network named SleepContextNet for sleep stage sequence. SleepContextNet can extract and utilize the long-term temporal context between consecutive EEG epochs, and combine it with the short-term context. we utilize Convolutional Neural Network(CNN) layers for learning representative features from each sleep stage and the representation features sequence learned are fed into a Recurrent Neural Network(RNN) layer for learning long-term and short-term context information among sleep stage in chronological order. In addition, we design a data augmentation algorithm for EEG to retain the long-term context information without changing the number of samples.Results: We evaluate the performance of our proposed network using four public datasets, the 2013 version of Sleep-EDF (SEDF), the 2018 version of Sleep-EDF Expanded (SEDFX), Sleep Heart Health Study (SHHS) and the CAP Sleep Database. The experimental results demonstrate that SleepContextNet outperforms state-of-the-art techniques in terms of different evaluation metrics by capturing long-term and short-term temporal context information. On average, accuracy of 84.8% in SEDF, 82.7% in SEDFX, 86.4% in SHHS and 78.8% in CAP are obtained under subject-independent cross validation.Conclusions: The network extracts the long-term and short-term temporal context information of sleep stages from the sequence features, which utilizes the temporal dependencies among the EEG epochs effectively and improves the accuracy of sleep stages. The sleep staging method based on forward temporal context information is suitable for real-time family sleep monitoring system.

Attention-based spatial–temporal neural network for accurate phase recognition in minimally invasive surgery: feasibility and efficiency verification

Abstract Laparoscopic surgery, as a representative minimally invasive surgery (MIS), is an active research area of clinical practice. Automatic surgical phase recognition of laparoscopic videos is a vital task with the potential to improve surgeons’ efficiency and has gradually become an integral part of computer-assisted intervention systems in MIS. However, the performance of most methods currently employed for surgical phase recognition is deteriorated by optimization difficulties and inefficient computation, which hinders their large-scale practical implementation. This study proposes an efficient and novel surgical phase recognition method using an attention-based spatial–temporal neural network consisting of a spatial model and a temporal model for accurate recognition by end-to-end training. The former subtly incorporates the attention mechanism to enhance the model’s ability to focus on the key regions in video frames and efficiently capture more informative visual features. In the temporal model, we employ independently recurrent long short-term memory (IndyLSTM) and non-local block to extract long-term temporal information of video frames. We evaluated the performance of our method on the publicly available Cholec80 dataset. Our attention-based spatial–temporal neural network purely produces the phase predictions without any post-processing strategies, achieving excellent recognition performance and outperforming other state-of-the-art phase recognition methods.

Efficient Two-Stream Network for Online Video Action Segmentation

Temporal action segmentation is a task of predicting frame-level classes of untrimmed long-termed videos. It can be widely used in various applications including customer analysis, data collection, video indexing, and surveillance. Learning good representations from videos such as motion and spatial representation is a critical factor for model performance. Furthermore, the used model should make reliable decisions with low latency for practical use in a resource-constrained environment. However, many works in action segmentation consider just an accuracy using the pre-calculated representations from heavy 3-dimensional(3D) networks. In this paper, we propose a two-stream action segmentation pipeline that can learn motion and spatial information efficiently and operate online. While the temporal stream combines frame-grouping and TSM for capturing short-term dynamics and long-term temporal information at the same time, the spatial stream captures information on color and appearance complementary to representations from the temporal stream. In addition, the results of both the streams are combined by a cross-attention module to provide a desired classification result for the task. Since it can be operated without heavy 3D convolutional neural networks (CNNs), it takes much less memory and computation than conventional 3D-CNN-based methods. Our proposed network using a non-overlapping sliding window achieved segmentation performance on two action segmentation datasets comparable to many recent works that require full temporal resolution and pre-calculated features of 3D CNNs.

Looking Beyond Two Frames: End-to-End Multi-Object Tracking Using Spatial and Temporal Transformers.

Tracking a time-varying indefinite number of objects in a video sequence over time remains a challenge despite recent advances in the field. Most existing approaches are not able to properly handle multi-object tracking challenges such as occlusion, in part because they ignore long-term temporal information. To address these shortcomings, we present MO3TR: a truly end-to-end Transformer-based online multi-object tracking (MOT) framework that learns to handle occlusions, track initiation and termination without the need for an explicit data association module or any heuristics. MO3TR encodes object interactions into long-term temporal embeddings using a combination of spatial and temporal Transformers, and recursively uses the information jointly with the input data to estimate the states of all tracked objects over time. The spatial attention mechanism enables our framework to learn implicit representations between all the objects and the objects to the measurements, while the temporal attention mechanism focuses on specific parts of past information, allowing our approach to resolve occlusions over multiple frames. Our experiments demonstrate the potential of this new approach, achieving results on par with or better than the current state-of-the-art on multiple MOT metrics for several popular multi-object tracking benchmarks.

MTT: Multi-Scale Temporal Transformer for Skeleton-Based Action Recognition

In the task of skeleton-based action recognition, long-term temporal dependencies are significant cues for sequential skeleton data. State-of-the-art methods rarely have access to long-term temporal information, due to the limitations of their receptive fields. Meanwhile, most of the recent multiple branches methods only consider different input modalities but ignore the information in various temporal scales. To address the above issues, we propose a multi-scale temporal transformer (MTT) in this letter, for skeleton-based action recognition. Firstly, the raw skeleton data are embedded by graph convolutional network (GCN) blocks and multi-scale temporal embedding modules (MT-EMs), which are designed as multiple branches to extract features in various temporal scales. Secondly, we introduce transformer encoders (TE) to integrate embeddings and model the long-term temporal pattern. Moreover, we propose a task-oriented lateral connection (LaC) aiming to align semantical hierarchies. LaC distributes input embeddings to the downstream transformer encoders (TE), according to semantical levels. The classification headers aggregate results from TE and predict the action categories at last. The proposed method is shown efficiency and universality during experiments and achieves the state-of-the-art on three large datasets, NTU-RGBD 60, NTU-RGBD 120 and Kinetics-Skeleton 400.

A Data Augmentation Method for Skeleton-Based Action Recognition with Relative Features

In recent years, skeleton-based human action recognition (HAR) approaches using convolutional neural network (CNN) models have made tremendous progress in computer vision applications. However, using relative features to depict human actions, in addition to preventing overfitting when the CNN model is trained on a few samples, is still a challenge. In this paper, a new motion image is introduced to transform spatial-temporal motion information into image-based representations. For each skeleton sequence, three relative features are extracted to describe human actions. The three relative features are consisted of relative coordinates, immediate displacement, and immediate motion orientation. In particular, the relative coordinates introduced in our paper not only depict the spatial relations of human skeleton joints but also provide long-term temporal information. To address the problem of small sample sizes, a data augmentation strategy consisting of three simple but effective data augmentation methods is proposed to expand the training samples. Because the generated color images are small in size, a shallow CNN model is suitable to extract the deep features of the generated motion images. Two small-scale but challenging skeleton datasets were used to evaluate the method, scoring 96.59% and 97.48% on the Florence 3D Actions dataset and UTkinect-Action 3D dataset, respectively. The results show that the proposed method achieved a competitive performance compared with the state-of-the-art methods. Furthermore, the augmentation strategy proposed in this paper effectively solves the overfitting problem and can be widely adopted in skeleton-based action recognition.

Foraging efficiency in temporally predictable environments: is a long-term temporal memory really advantageous?

Cognitive abilities enabling animals that feed on ephemeral but yearly renewable resources to infer when resources are available may have been favoured by natural selection, but the magnitude of the benefits brought by these abilities remains poorly known. Using computer simulations, we compared the efficiencies of three main types of foragers with different abilities to process temporal information, in spatially and/or temporally homogeneous or heterogeneous environments. One was endowed with a sampling memory, which stores recent experience about the availability of the different food types. The other two were endowed with a chronological or associative memory, which stores long-term temporal information about absolute times of these availabilities or delays between them, respectively. To determine the range of possible efficiencies, we also simulated a forager without temporal cognition but which simply targeted the closest and possibly empty food sources, and a perfectly prescient forager, able to know at any time which food source was effectively providing food. The sampling, associative and chronological foragers were far more efficient than the forager without temporal cognition in temporally predictable environments, and interestingly, their efficiencies increased with the level of temporal heterogeneity. The use of a long-term temporal memory results in a foraging efficiency up to 1.16 times better (chronological memory) or 1.14 times worse (associative memory) than the use of a simple sampling memory. Our results thus show that, for everyday foraging, a long-term temporal memory did not provide a clear benefit over a simple short-term memory that keeps track of the current resource availability. Long-term temporal memories may therefore have emerged in contexts where short-term temporal cognition is useless, i.e. when the anticipation of future environmental changes is strongly needed.

Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level.

Depression is a severe psychological condition that affects millions of people worldwide. As depression has received more attention in recent years, it has become imperative to develop automatic methods for detecting depression. Although numerous machine learning methods have been proposed for estimating the levels of depression via audio, visual, and audiovisual emotion sensing, several challenges still exist. For example, it is difficult to extract long-term temporal context information from long sequences of audio and visual data, and it is also difficult to select and fuse useful multi-modal information or features effectively. In addition, how to include other information or tasks to enhance the estimation accuracy is also one of the challenges. In this study, we propose a multi-modal adaptive fusion transformer network for estimating the levels of depression. Transformer-based models have achieved state-of-the-art performance in language understanding and sequence modeling. Thus, the proposed transformer-based network is utilized to extract long-term temporal context information from uni-modal audio and visual data in our work. This is the first transformer-based approach for depression detection. We also propose an adaptive fusion method for adaptively fusing useful multi-modal features. Furthermore, inspired by current multi-task learning work, we also incorporate an auxiliary task (depression classification) to enhance the main task of depression level regression (estimation). The effectiveness of the proposed method has been validated on a public dataset (AVEC 2019 Detecting Depression with AI Sub-challenge) in terms of the PHQ-8 scores. Experimental results indicate that the proposed method achieves better performance compared with currently state-of-the-art methods. Our proposed method achieves a concordance correlation coefficient (CCC) of 0.733 on AVEC 2019 which is 6.2% higher than the accuracy (CCC = 0.696) of the state-of-the-art method.