Monocular Depth Estimation Methods Research Articles

Monocular Absolute Depth Estimation from Motion for Small Unmanned Aerial Vehicles by Geometry-Based Scale Recovery.

In recent years, there has been extensive research and application of unsupervised monocular depth estimation methods for intelligent vehicles. However, a major limitation of most existing approaches is their inability to predict absolute depth values in physical units, as they generally suffer from the scale problem. Furthermore, most research efforts have focused on ground vehicles, neglecting the potential application of these methods to unmanned aerial vehicles (UAVs). To address these gaps, this paper proposes a novel absolute depth estimation method specifically designed for flight scenes using a monocular vision sensor, in which a geometry-based scale recovery algorithm serves as a post-processing stage of relative depth estimation results with scale consistency. By exploiting the feature correspondence between successive images and using the pose data provided by equipped navigation sensors, the scale factor between relative and absolute scales is calculated according to a multi-view geometry model, and then absolute depth maps are generated by pixel-wise multiplication of relative depth maps with the scale factor. As a result, the unsupervised monocular depth estimation technology is extended from relative depth estimation in semi-structured scenes to absolute depth estimation in unstructured scenes. Experiments on the publicly available Mid-Air dataset and customized data demonstrate the effectiveness of our method in different cases and settings, as well as its robustness to navigation sensor noise. The proposed method only requires UAVs to be equipped with monocular camera and common navigation sensors, and the obtained absolute depth information can be directly used for downstream tasks, which is significant for this kind of vehicle that has rarely been explored in previous depth estimation studies.

RCFNet: Related Cross-level Feature Network with Cascaded Self-distillation for Monocular Depth Estimation

Monocular depth estimation (MDE) is a challenging yet crucial computer vision task, which aims to generate accurate depth maps from a single image. Existing MDE approaches mainly rely on extracting and fusing diverse information from multi-level features to improve prediction accuracy. However, these methods often apply a traditional feature pyramid structure, neglecting a comprehensive exploration of feature fusion paths across multiple levels. Moreover, a single-feature fusion strategy has limited ability to optimize the network. We propose a novel related cross-level feature network (RCFNet) with cascaded self-distillation for monocular depth estimation, including a cross-level feature enhancement (CLFE) module, a hierarchical feature cross refinement (HFCR) module, and a cascaded self-distillation (CSD) module. The CLFE module integrates cross-level features to further exploit the highest-level features, where a channel attention mechanism with hybrid weight operations is deployed to enhance the initial features. The HFCR module adaptively captures strongly correlated complementary information through a window-based multi-head cross-attention mechanism to generate refined features. Meanwhile, the CSD module with a hierarchical feature transformation loss is proposed, which can be viewed as a virtual teacher to progressively extract discriminative features within the network for better gradient flow improvement. Extensive experiments on NYUv2 and KITTI datasets demonstrate that our method outperforms existing SOTA MDE methods in terms of accuracy capacity and robustness.

SIM-MultiDepth: Self-Supervised Indoor Monocular Multi-Frame Depth Estimation Based on Texture-Aware Masking

Self-supervised monocular depth estimation methods have become the focus of research since ground truth data are not required. Current single-image-based works only leverage appearance-based features, thus achieving a limited performance. Deep learning based multiview stereo works facilitate the research on multi-frame depth estimation methods. Some multi-frame methods build cost volumes and take multiple frames as inputs at the time of test to fully utilize geometric cues between adjacent frames. Nevertheless, low-textured regions, which are dominant in indoor scenes, tend to cause unreliable depth hypotheses in the cost volume. Few self-supervised multi-frame methods have been used to conduct research on the issue of low-texture areas in indoor scenes. To handle this issue, we propose SIM-MultiDepth, a self-supervised indoor monocular multi-frame depth estimation framework. A self-supervised single-frame depth estimation network is introduced to learn the relative poses and supervise the multi-frame depth learning. A texture-aware depth consistency loss is designed considering the calculation of the patch-based photometric loss. Only the areas where multi-frame depth prediction is considered unreliable in low-texture regions are supervised by the single-frame network. This approach helps improve the depth estimation accuracy. The experimental results on the NYU Depth V2 dataset validate the effectiveness of SIM-MultiDepth. The zero-shot generalization studies on the 7-Scenes and Campus Indoor datasets aid in the analysis of the application characteristics of SIM-MultiDepth.

Complete contextual information extraction for self-supervised monocular depth estimation

Self-supervised learning methods are increasingly important for monocular depth estimation since they do not require ground-truth data during training. Although existing methods have achieved great success for better monocular depth estimation based on Convolutional Neural Networks (CNNs), the limited receptive field of CNNs usually is insufficient to effectively model the global information, e.g., relationship between foreground and background or relationship among objects, which are crucial for accurately capturing scene structure. Recently, some studies based on Transformers have attracted significant interest in computer vision. However, duo to the lack of spatial locality bias, they may fail to model the local information, e.g., fine-grained details with an image. To tackle these issues, we propose a novel self-supervised learning framework by incorporating the advantages of both the CNNs and Transformers so as to model the complete contextual information for high-quality monocular depth estimation. Specifically, the proposed method mainly includes two branches, where the Transformer branch is considered to capture the global information while the Convolution branch is exploited to preserve the local information. We also design a rectangle convolution module with pyramid structure to perceive the semi-global information, e.g. thin objects, along the horizontal and vertical directions within an image. Moreover, we propose a shape refinement module by learning the affinity matrix between pixel and its neighborhood to obtain accurate geometrical structure of scenes. Extensive experiments evaluated on KITTI, Cityscapes and Make3D dataset demonstrate that the proposed method achieves the competitive result compared with the state-of-the-art self-supervised monocular depth estimation methods and shows good cross-dataset generalization ability.

Edge-Enhanced Dual-Stream Perception Network for Monocular Depth Estimation

Estimating depth from a single RGB image has a wide range of applications, such as in robot navigation and autonomous driving. Currently, Convolutional Neural Networks based on encoder–decoder architecture are the most popular methods to estimate depth maps. However, convolutional operators have limitations in modeling large-scale dependence, often leading to inaccurate depth predictions at object edges. To address these issues, a new edge-enhanced dual-stream monocular depth estimation method is introduced in this paper. ResNet and Swin Transformer are combined to better extract global and local features, which benefits the estimation of the depth map. To better integrate the information from the two branches of the encoder and the shallow branch of the decoder, we designed a lightweight decoder based on the multi-head Cross-Attention Module. Furthermore, in order to improve the boundary clarity of objects in the depth map, a loss function with an additional penalty for depth estimation error on the edges of objects is presented. The results on three datasets, NYU Depth V2, KITTI, and SUN RGB-D, show that the method presented in this paper achieves better performance for monocular depth estimation. Additionally, it has good generalization capabilities for various scenarios and real-world images.

F[formula omitted]Depth: Self-supervised indoor monocular depth estimation via optical flow consistency and feature map synthesis

Self-supervised monocular depth estimation methods have been increasingly given much attention due to the benefit of not requiring large, labelled datasets. Such self-supervised methods require high-quality salient features and consequently suffer from severe performance drop for indoor scenes, where low-textured regions dominant in the scenes are almost indiscriminative. To address the issue, we propose a self-supervised indoor monocular depth estimation framework called F2Depth. A self-supervised optical flow estimation network is introduced to supervise depth learning. To improve optical flow estimation performance in low-textured areas, only some patches of points with more discriminative features are adopted for finetuning based on our well-designed patch-based photometric loss. The finetuned optical flow estimation network generates high-accuracy optical flow as a supervisory signal for depth estimation. Correspondingly, an optical flow consistency loss is designed. Multi-scale feature maps produced by finetuned optical flow estimation network perform warping to compute feature map synthesis loss as another supervisory signal for depth learning. Experimental results on the NYU Depth V2 dataset demonstrate the effectiveness of the framework and our proposed losses. To evaluate the generalization ability of our F2Depth, we collect a Campus Indoor depth dataset composed of approximately 1500 points selected from 99 images in 18 scenes. Zero-shot generalization experiments on 7-Scenes dataset and Campus Indoor achieve δ1 accuracy of 75.8% and 76.0% respectively. The accuracy results show that our model can generalize well to monocular images captured in unknown indoor scenes.

Deep learning-assisted 3D laser steering using an optofluidic laser scanner.

Laser ablation is an effective treatment modality. However, current laser scanners suffer from laser defocusing when scanning targets at different depths in a 3D surgical scene. This study proposes a deep learning-assisted 3D laser steering strategy for minimally invasive surgery that eliminates laser defocusing, increases working distance, and extends scanning range. An optofluidic laser scanner is developed to conduct 3D laser steering. The optofluidic laser scanner has no mechanical moving components, enabling miniature size, lightweight, and low driving voltage. A deep learning-based monocular depth estimation method provides real-time target depth estimation so that the focal length of the laser scanner can be adjusted for laser focusing. Simulations and experiments indicate that the proposed method can significantly increase the working distance and maintain laser focusing while performing 2D laser steering, demonstrating the potential for application in minimally invasive surgery.

Attention-Based Monocular Depth Estimation Considering Global and Local Information in Remote Sensing Images

Monocular depth estimation using a single remote sensing image has emerged as a focal point in both remote sensing and computer vision research, proving crucial in tasks such as 3D reconstruction and target instance segmentation. Monocular depth estimation does not require multiple views as references, leading to significant improvements in both time and efficiency. Due to the complexity, occlusion, and uneven depth distribution of remote sensing images, there are currently few monocular depth estimation methods for remote sensing images. This paper proposes an approach to remote sensing monocular depth estimation that integrates an attention mechanism while considering global and local feature information. Leveraging a single remote sensing image as input, the method outputs end-to-end depth estimation for the corresponding area. In the encoder, the proposed method employs a dense neural network (DenseNet) feature extraction module with efficient channel attention (ECA), enhancing the capture of local information and details in remote sensing images. In the decoder stage, this paper proposes a dense atrous spatial pyramid pooling (DenseASPP) module with channel and spatial attention modules, effectively mitigating information loss and strengthening the relationship between the target’s position and the background in the image. Additionally, weighted global guidance plane modules are introduced to fuse comprehensive features from different scales and receptive fields, finally predicting monocular depth for remote sensing images. Extensive experiments on the publicly available WHU-OMVS dataset demonstrate that our method yields better depth results in both qualitative and quantitative metrics.

DPDFormer: A Coarse-to-Fine Model for Monocular Depth Estimation

Monocular depth estimation attracts great attention from computer vision researchers for its convenience in acquiring environment depth information. Recently classification-based MDE methods show its promising performance and begin to act as an essential role in many multi-view applications such as reconstruction and 3D object detection. However, existed classification-based MDE models usually apply fixed depth range discretization strategy across a whole scene. This fixed depth range discretization leads to the imbalance of discretization scale among different depth ranges, resulting in the inexact depth range localization. In this article, to alleviate the imbalanced depth range discretization problem in classification-based monocular depth estimation (MDE) method we follow the coarse-to-fine principle and propose a novel depth range discretization method called depth post-discretization (DPD). Based on a coarse depth anchor roughly indicating the depth range, the DPD generates the depth range discretization adaptively for every position. The depth range discretization with DPD is more fine-grained around the actual depth, which is beneficial for locating the depth range more precisely for each scene position. Besides, to better manage the prediction of the coarse depth anchor and depth probability distribution for calculating the final depth, we design a dual-decoder transformer-based network, i.e., DPDFormer, which is more compatible with our proposed DPD method. We evaluate DPDFormer on popular depth datasets NYU Depth V2 and KITTI. The experimental results prove the superior performance of our proposed method.

BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation.

Monocular depth estimation (MDE) is a fundamental task in computer vision and has drawn increasing attention. Recently, some methods reformulate it as a classification-regression task to boost the model performance, where continuous depth is estimated via a linear combination of predicted probability distributions and discrete bins. In this paper, we present a novel framework called BinsFormer, tailored for the classification-regression-based depth estimation. It mainly focuses on two crucial components in the specific task: 1) proper generation of adaptive bins and 2) sufficient interaction between probability distribution and bins predictions. To specify, we employ a Transformer decoder to generate bins, novelly viewing it as a direct set-to-set prediction problem. We further integrate a multi-scale decoder structure to achieve a comprehensive understanding of spatial geometry information and estimate depth maps in a coarse-to-fine manner. Moreover, an extra scene understanding query is proposed to improve the estimation accuracy, which turns out that models can implicitly learn useful information from the auxiliary environment classification task. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that BinsFormer surpasses state-of-the-art MDE methods with prominent margins. Code and pretrained models are made publicly available at https://github.com/zhyever/Monocular-Depth-Estimation-Toolbox/tree/ main/configs/binsformer.

IterDepth: Iterative Residual Refinement for Outdoor Self-Supervised Multi-Frame Monocular Depth Estimation

Self-supervised monocular depth estimation has been a challenging task in computer vision for a long time, and it relies on only monocular or stereo video for its supervision. To address the challenge, we propose a novel multi-frame monocular depth estimation method called IterDepth, which is based on an iterative residual refinement network. IterDepth extracts depth features from consecutive frames and computes a 3D cost volume measuring the difference between current and previous features transformed by PoseCNN (pose estimation convolutional neural network). We reformulate depth prediction as a residual learning problem, revamping the dominating depth regression to enable high-accuracy multi-frame monocular depth estimation. Specifically, we design a gated recurrent depth fusion unit that seamlessly blends depth features from the cost volume, image features, and the depth prediction. The unit updates the hidden states and refines the depth map through iterative refinement, achieving more accurate predictions than existing methods. Our experiments on the KITTI dataset demonstrate that IterDepth is 7× faster in terms of FPS (frames per second) than the recent state-of-the-art DepthFormer model with competitive performance. We also test IterDepth on the Cityscapes dataset to showcase its generalization capability in other real-world environments. Moreover, IterDepth can balance accuracy and computational efficiency by adjusting the number of refinement iterations and performs competitively with other CNN-based monocular depth estimation approaches. Code will be made publicly available.

Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields.

Text-driven 3D scene generation is widely applicable to video gaming, film industry, and metaverse applications that have a large demand for 3D scenes. However, existing text-to-3D generation methods are limited to producing 3D objects with simple geometries and dreamlike styles that lack realism. In this work, we present Text2NeRF, which is able to generate a wide range of 3D scenes with complicated geometric structures and high-fidelity textures purely from a text prompt. To this end, we adopt NeRF as the 3D representation and leverage a pre-trained text-to-image diffusion model to constrain the 3D reconstruction of the NeRF to reflect the scene description. Specifically, we employ the diffusion model to infer the text-related image as the content prior and use a monocular depth estimation method to offer the geometric prior. Both content and geometric priors are utilized to update the NeRF model. To guarantee textured and geometric consistency between different views, we introduce a progressive scene inpainting and updating strategy for novel view synthesis of the scene. Our method requires no additional training data but only a natural language description of the scene as the input. Extensive experiments demonstrate that our Text2NeRF outperforms existing methods in producing photo-realistic, multi-view consistent, and diverse 3D scenes from a variety of natural language prompts. Our code and model will be available upon acceptance.

Joint Learning of Image Deblurring and Depth Estimation Through Adversarial Multi-Task Network

Self-supervised monocular depth estimation methods have achieved remarkable results on natural clear images. However, it is still a serious challenge to directly recover depth information from blurred images caused by long-time exposure while camera fast moving. To address this issue, we propose a unified framework for simultaneous deblurring and depth estimation (SDDE), which has higher coupling performance and flexibility compared with the simple concatenation strategy of deblurring model and depth estimation model. This framework mainly benefits from three features: (1) a novel Task-aware Fusion Module (TFM) to adaptively select the most relevant intermediate shared features for the dual decoder network by aggregating multi-scale features, (2) a unique Spatial Interaction Module (SIM) to learn higher-order representation in the encoder stage to better describe complex boundaries of different classes in high-dimensional space, and focuses on the task-related region by modeling the pairwise spatial correlation of the holistic tensor, (3) a Priors-Based Composite Regularization term to jointly optimize the shared encoder-dual decoder network. This work was evaluated on multiple datasets, including: Stereo blur [1], KITTI [2],NYUv2 [3], REDS [4] and our own large-scale stereo blur dataset, resulting in state-of-the-art results for depth estimation and image deblurring, respectively.

Monocular Depth Estimation for Glass Walls with Context: A New Dataset and Method.

Traditional monocular depth estimation assumes that all objects are reliably visible in the RGB color domain. However, this is not always the case as more and more buildings are decorated with transparent glass walls. This problem has not been explored due to the difficulties in annotating the depth levels of glass walls, as commercial depth sensors cannot provide correct feedbacks on transparent objects. Furthermore, estimating depths from transparent glass walls requires the aids of surrounding context, which has not been considered in prior works. To cope with this problem, we introduce the first Glass Walls Depth Dataset (GW-Depth dataset). We annotate the depth levels of transparent glass walls by propagating the context depth values within neighboring flat areas, and the glass segmentation mask and instance level line segments of glass edges are also provided. On the other hand, a tailored monocular depth estimation method is proposed to fully activate the glass wall contextual understanding. First, we propose to exploit the glass structure context by incorporating the structural prior knowledge embedded in glass boundary line segment detections. Furthermore, to make our method adaptive to scenes without structure context where the glass boundary is either absent in the image or too narrow to be recognized, we propose to derive a reflection context by utilizing the depth reliable points sampled according to the variance between two depth estimations from different resolutions. High-resolution depth is thus estimated by the weighted summation of depths by those reliable points. Extensive experiments are conducted to evaluate the effectiveness of the proposed dual context design. Superior performances of our method is also demonstrated by comparing with state-of-the-art methods. We present the first feasible solution for monocular depth estimation in the presence of glass walls, which can be widely adopted in autonomous navigation.

End-to-end learning for joint depth and image reconstruction from diffracted rotation

Monocular depth estimation is an open challenge due to the ill-posed nature of the problem at hand. Deep learning techniques proved capable of producing acceptable depth estimation accuracy but the lack of robust depth cues within RGB images severally limits their performance. Coded aperture-based methods using phase and amplitude masks encode strong depth cues within 2D images by means of depth-dependent Point Spread Functions (PSFs) at the price of a reduced image quality. In this paper, we propose a novel end-to-end learning approach for depth from diffracted rotation. A phase mask that produces a Rotating Point Spread Function (RPSF) as a function of defocus is jointly optimized with the weights of a depth estimation neural network. To this aim, we introduce a differentiable physical model of the aperture mask and exploit an accurate simulation of the camera imaging pipeline. Our approach requires a significantly less complex model and less training data, yet it outperforms existing methods for monocular depth estimation on indoor benchmarks. In addition, we address the image degradation problem by incorporating a non-blind and nonuniform image deblurring module to recover the sharp all-in-focus image from its blurred counterpart.

Accurate detection and depth estimation of table grapes and peduncles for robot harvesting, combining monocular depth estimation and CNN methods

Precision agriculture is a growing field in the agricultural industry and it holds great potential in fruit and vegetable harvesting. In this work, we present a robust accurate method for the detection and localization of the peduncle of table grapes, with direct implementation in automatic grape harvesting with robots. The bunch and peduncle detection methods presented in this work rely on a combination of instance segmentation and monocular depth estimation using Convolutional Neural Networks (CNN). Regarding depth estimation, we propose a combination of different depth techniques that allow precise localization of the peduncle using traditional stereo cameras, even with the particular complexity of grape peduncles. The methods proposed in this work have been tested on the WGISD (Embrapa Wine Grape Instance Segmentation) dataset, improving the results of state-of-the-art techniques. Furthermore, within the context of the EU project CANOPIES, the methods have also been tested on a dataset of 1,326 RGB-D images of table grapes, recorded at the Corsira Agricultural Cooperative Society (Aprilia, Italy), using a Realsense D435i camera located at the arm of a CANOPIES two-manipulator robot developed in the project. The detection results on the WGISD dataset show that the use of RGB-D information (mAP=0.949) leads to superior performance compared to the use of RGB data alone (mAP=0.891). This trend is also evident in the CANOPIES Grape Bunch and Peduncle dataset, where the mAP for RGB-D images (mAP=0.767) outperforms that of RGB data (mAP=0.725). Regarding depth estimation, our method achieves a mean squared error of 2.66 cm within a distance of 1 m in the CANOPIES dataset.

EVALUATING MONOCULAR DEPTH ESTIMATION METHODS

Abstract. Depth estimation from monocular images has become a prominent focus in photogrammetry and computer vision research. Monocular Depth Estimation (MDE), which involves determining depth from a single RGB image, offers numerous advantages, including applications in simultaneous localization and mapping (SLAM), scene comprehension, 3D modeling, robotics, and autonomous driving. Depth information retrieval becomes especially crucial in situations where other sources like stereo images, optical flow, or point clouds are not available. In contrast to traditional stereo or multi-view methods, MDE techniques require fewer computational resources and smaller datasets. This research work presents a comprehensive analysis and evaluation of some state-of-the-art MDE methods, considering their ability to infer depth information in terrestrial images. The evaluation includes quantitative assessments using ground truth data, including 3D analyses and inference time.

Scene-aware refinement network for unsupervised monocular depth estimation in ultra-low altitude oblique photography of UAV

Using depth estimation joint target detection networks to locate targets in the UAV field of view is a novel application in the depth estimation research field. The presence of more depth variations and low-texture regions in the ultra-low altitude oblique photographic images make them trickier to train for an excellent depth estimation network compared to autonomous driving scenarios. This presents a challenge in achieving optimal training. This study investigates the problem of unsupervised monocular depth estimation for ultra-low altitude oblique photography images. It aims to make subsequent advanced vision tasks better benefit from excellent depth estimation results in terms of overcoming complex scenes. The lack of effective back-projection directionality in training using adjacent frames is attributed to the extensive low-textured areas contained in the training data for complex ultra-low altitude oblique photography. We propose a self-supervised scene-aware refinement learning architecture from the perspective of enhancing feature perception to deal with such problems. The architecture consists of a multi-resolution feature fusion depth network and a perceptual refinement network (PRNet), together with a pose network to enhance regional differences in complex environments from a refined feature context perspective to obtain higher quality depth maps. We rethink the problem of depth information recovery and design the edge information aggregation (EIA) module, which is configured in the decoder section to refine the local region depth detail representation. We design several loss terms to constrain the training of the network in order to improve the quality of depth estimation. Our method is compared with six state-of-the-art self-supervised monocular depth estimation methods on three datasets (UAVid 2020, WildUAV, UAV ula). The experimental results demonstrate that our model achieves the best performance in most scenarios. The code and the private dataset (UAV ula) can be publicly available at https://github.com/takisu0916/MRFEDepth.

DepthFormer: Exploiting Long-range Correlation and Local Information for Accurate Monocular Depth Estimation

This paper aims to address the problem of supervised monocular depth estimation. We start with a meticulous pilot study to demonstrate that the long-range correlation is essential for accurate depth estimation. Moreover, the Transformer and convolution are good at long-range and close-range depth estimation, respectively. Therefore, we propose to adopt a parallel encoder architecture consisting of a Transformer branch and a convolution branch. The former can model global context with the effective attention mechanism and the latter aims to preserve the local information as the Transformer lacks the spatial inductive bias in modeling such contents. However, independent branches lead to a shortage of connections between features. To bridge this gap, we design a hierarchical aggregation and heterogeneous interaction module to enhance the Transformer features and model the affinity between the heterogeneous features in a set-to-set translation manner. Due to the unbearable memory cost introduced by the global attention on high-resolution feature maps, we adopt the deformable scheme to reduce the complexity. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins. The effectiveness of each proposed module is elaborately evaluated through meticulous and intensive ablation studies.

Attention Mechanism Used in Monocular Depth Estimation: An Overview

Monocular depth estimation (MDE), as one of the fundamental tasks of computer vision, plays important roles in downstream applications such as virtual reality, 3D reconstruction, and robotic navigation. Convolutional neural networks (CNN)-based methods gained remarkable progress compared with traditional methods using visual cues. However, recent researches reveal that the performance of MDE using CNN could be degraded due to the local receptive field of CNN. To bridge the gap, various attention mechanisms were proposed to model the long-range dependency. Although reviews of MDE algorithms based on CNN were reported, a comprehensive outline of how attention boosts MDE performance is not explored yet. In this paper, we firstly categorize recent attention-related works into CNN-based, Transformer-based, and hybrid (CNN–Transformer-based) approaches in the light of how the attention mechanism impacts the extraction of global features. Secondly, we discuss the details and contributions of attention-based MDE methods published from 2020 to 2022. Then, we compare the performance of the typical attention-based methods. Finally, the challenges and trends of the attention mechanism used in MDE are discussed.