The problem of adaptive modulation with outdated channel state information (CSI) is considered. Best existing approach to tackle this problem relies on using a (non-)linear auto-regressive moving average (ARMA) model to predict current CSI from outdated values. This approach is valid only if the wireless channel variations over time behave in a linear or smooth enough nonlinear fashion, which is not necessarily the case. We propose a deep reinforcement learning based adaptive modulation (DRL-AM) approach that can handle this limitation. While DRL-AM is more complex than (non-)linear AR(MA), it performs significantly better as corroborated via numerical results on real channel measurements. Furthermore, compared to capacity-achieving codes, complexity is moved from receiver to transmitter making this approach suitable for receiving nodes with limited resources such as internet of things (IoT) devices.