Track quality significantly impacts passenger comfort and safety. The maintenance of track quality involves routine monitoring of track geometry and track component defects. In a drive-by monitoring framework, vibration data are collected from operational trains providing an effective and low-cost track monitoring system. This framework enables each train to inspect multiple lines without the need to install and maintain sensors on each line. In conventional data-driven frameworks that utilize machine learning or statistical models, geometrical defects diagnosis model is often constructed using supervised learning. Supervised learning requires response data and corresponding damage states (i.e., labels) for each railway line to learn the line-specific damage diagnosis model. However, labelled data are frequently unavailable for many lines in practical situations, making it challenging to construct a damage diagnosis model. Furthermore, using data directly from source line to construct a damage diagnosis model for the target line can lead to inaccurate results. To this end, in this paper, a novel drive-by monitoring framework is proposed based on Transfer Learning (TL) concept. This approach transfers the geometrical defects diagnosis model learned from one line to a new line. Firstly, condition-sensitive time domain features are extracted from the raw data. Then, a deep learning model based on the long short-term memory (LSTM) network is trained on a dataset from one line (source domain) and fine-tuned for a new dataset using a small number of samples from another line (target domain). The effectiveness of this framework is compared through the classification performance using real-world monitoring datasets collected from two different lines of the France railways network. The results demonstrate up to a 22% increase in defect detection accuracy compared to the basic method.