Rapidly and accurately predicting on-site peak ground velocity (PGV) is important for earthquake hazard mitigation. Traditional methods used to predict PGV involve a single physics-based parameter, like the peak displacement (Pd) or squared velocity integral (IV2) techniques; deep-learning methods involve a single neural network model, like the convolutional neural network (CNN) or recurrent neural network (RNN) models, to extract feature for estimating the PGV. Here, based on the training dataset from earthquake events occurred in Japan, we construct hybrid deep-learning network (HybridNet) for predicting on-site PGV, which is consist of CNN and RNN feature extraction blocks. We use physics-based feature time series, waveforms and a site feature from a single station as the input of HybridNet model; additionally, we concatenate the features from the CNN block, RNN block and site feature to predict the on-site PGV. We show that concerning the standard deviation of error, the mean absolute error and the coefficient of determination for PGV prediction, HybridNet model exhibits better performance on the test dataset than the baseline models. Additionally, potential damage zone (PDZ) can be predicted by interpolating the predicted PGVs at the stations. Based on the predicted PGV of the HybridNet model, we investigate the feasibility of PDZ estimation on five earthquakes (M≥6.5). And we find that within a few seconds after the arrival of P-wave, the predicted PDZ is consistent well with the PGV ShakeMap obtained from the U.S. Geological Survey.