Particle tracking velocimetry (PTV) is a powerful technique for global and nonintrusive flow field measurement, which shows a great potential to improve the spatial resolution compared to other flow visualization technologies (e.g., correlation-based particle image velocimetry). However, performing PTV under flow conditions with high flow speed and high particle density is still a challenge. In addition, more and more research sacrifices the computational efficiency for high accuracy using complex iterative algorithms. To address these problems, we propose a deep particle tracking network, called deep neural network for PTV (DeepPTV), for learning the complex fluid flow motion from two consecutive particle sets efficiently and accurately. First, the local spatial geometry information from neighboring particles is aggregated for each particle along with robust features (i.e., the relative distance to the neighbors), which help to preserve the properties of fluids. Second, to cope with the problem caused by nonuniform seeding density of particles, the multiscale features are combined together in each hierarchy of the neural network. Furthermore, motivated by the convection flow phenomena, the proposed DeepPTV model adopts a novel network architecture named convection architecture to estimate the flow field in a hierarchical framework, namely, from large-scale motion to small-scale motion. Experimental evaluations on both artificial and laboratory particle images demonstrate that the proposed framework can provide satisfactory accuracy that rivals the state-of-the-art methods. Moreover, the presented high efficiency makes it a promising algorithm for real-time estimation and real-time flow control problems.