Time-delay embedding is an increasingly popular starting point for data-driven reduced-order modeling efforts. In particular, the singular value decomposition (SVD) of a block Hankel matrix formed from successive delay embeddings of the state of a dynamical system lies at the heart of several popular reduced-order modeling methods. In this paper, we show that the left singular vectors of this Hankel matrix are a discrete approximation of space-time proper orthogonal decomposition (POD) modes, and the singular values are square roots of the POD energies. Analogous to the connection between the SVD of a data matrix of snapshots and space-only POD, this connection establishes a clear interpretation of the Hankel modes grounded in classical theory, and we gain insights into the Hankel modes by instead analyzing the equivalent discrete space-time POD modes in terms of the correlation matrix formed by multiplying the Hankel matrix by its conjugate transpose. These insights include the distinct meaning of rows and columns, the implied norm in which the modes are optimal, the impact of the time step between snapshots on the modes, and an interpretation of the embedding dimension/height of the Hankel matrix in terms of the time window on which the modes are optimal. Moreover, the connections we establish offer opportunities to improve the convergence and computation time in certain practical cases, and to improve the accuracy of the modes with the same data. Finally, popular variants of POD, namely the standard space-only POD and spectral POD, are recovered in the limits that snapshots used to form each column of the Hankel matrix represent flow evolution over short and long times, respectively.