Mesh adaptation is crucial in numerical simulation, providing optimal resource allocation for accurately capturing physical phenomena. However, when applied to Computational Fluid Dynamics (CFD) problems with complex multi-scale properties, existing adaptation methods face huge challenges due to the high computational cost of solving auxiliary partial differential equations (PDEs) and the difficulty in aligning the flow features with mesh geometric features. In this work, an end-to-end data-driven mesh adaptation framework, Flow2Mesh, is proposed to address these challenges by adopting a hybrid modeling strategy to construct the mapping from pixelated flow-fields to graph-based meshes. It achieves a rapid and accurate one-step mesh adaptation via a perceptual feature network (PFN) and a mesh movement network (MMN). PFN extracts the global perceptual features from flow-fields to enhance flow feature representation and mesh resolution independence. In MMN, these features are utilized to deform the initial mesh to a topology-invariant adaptive mesh by a proposed physically driven mesh convolutional network. It considers the inherent mesh geometric information for efficient node feature aggregation and alignment of mesh density with a flow-field structure. To generate high-quality adaptive meshes, various mesh-related losses are designed to regularize the mesh movement and alleviate the mesh tangling. Experiments in CFD scenarios demonstrate the generalization of our model to different design parameters and mesh configurations. It takes three orders of magnitude less time to generate similar meshes than the PDE-based method. The results exhibit the potential of Flow2Mesh to be a flexible and reliable tool for rapid mesh adaptation in scientific and industrial fields.