The widely used Bayesian filtering has a solid theoretical foundation and an efficient computational architecture, but it suffers from first-order Markovianity and the inability to learn from offline data. While recent research has utilized deep learning to overcome these limitations, issues remain with respect to theoretical rigor and scalability, resulting in the non-guaranteed optimality of filtering. In this work, we approach these problems from a biomimetic perspective. Firstly, a memory-based state evolution model is constructed inspired by the human cognitive mechanism, which enables recursive filtering while incorporating long-term state information. Secondly, a memory-biomimetic deep Bayesian filtering algorithm with good interpretability is designed, which improves the filtering process at multiple levels by naturally integrating recurrent neural networks and Bayesian filtering architecture. Thirdly, the Gaussian approximation implementation of the proposed filtering method is derived, which brings an efficient computational process with good analytical properties. Extensive experiments for airborne target tracking are performed to demonstrate the excellent estimation performance of the proposed method.