The importance of bond markets in the financial industry stems from its dimension, its direct relevance for other asset classes and for the overall economy. In this paper, we conduct the first study of bond yield forecasting using deep learning long short-term memory (LSTM) networks, validating the potential of LSTMs networks for that purpose, and identifying the LSTM's memory advantage over standard feedforward neural networks, in particular, the multilayer perceptron (MLP). Specifically, we model the 10-year Euro government bond yield using univariate LSTMs with different input sequences (6, 21 and 61 time steps), considering five forecasting horizons, from next day to 20 days ahead. We compare those LSTM models with MLPs, both univariate as well as using the most relevant features for each forecasting horizon. The results show that the univariate LSTM model with additional memory is capable of achieving similar results as the multivariate MLP using information from markets and the economy. Moreover, the direct comparison of models in identical conditions, i.e. small input sequence of 5 time steps, leads to results with LSTMs that are similar or better with lower standard deviations. Furthermore, with the LSTMs, shorter forecasting horizons require smaller input sequences and vice-versa. In summary, the results are encouraging for its use incorporating macroeconomic and market information, adjusting the input sequence length to the forecasting horizon considered.