Fueled in part by recent applications in neuroscience, the multivariate Hawkes process has become a popular tool for modeling the network of interactions among high-dimensional point process data. While evaluating the uncertainty of the network estimates is critical in scientific applications, existing methodological and theoretical work has primarily addressed estimation. To bridge this gap, we develop a new statistical inference procedure for high-dimensional Hawkes processes. The key ingredient for the inference procedure is a new concentration inequality on the first- and second-order statistics for integrated stochastic processes, which summarize the entire history of the process. Combining recent martingale central limit theorem with the new concentration inequality, we then characterize the convergence rate of the test statistics in a continuous time domain. Finally, to account for potential non-stationarity of the process in practice, we extend our statistical inference procedure to a flexible class of Hawkes processes with time-varying background intensities and unknown transition functions. The finite sample validity of the inferential tools is illustrated via extensive simulations and further applied to a neuron spike train dataset. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.