Due to technical barriers and economic costs, malicious circuits, known as hardware Trojans, are easily implanted in the complicated integrated circuit design and manufacturing process, which can lead to many disastrous consequences, such as denial of service, information leakage, performance degradation, etc. Research on how to detecting hardware Trojans has grown into a significantly open issue over the past decade. While, for very large scale integrated circuits, numerous new challenges deserve our full attention, including golden-free chip reference, automatic feature engineering, hardware Trojan localization, and scalable framework. In response to the above challenges, a fine-grained gate-level hardware Trojan detection approach is proposed in this paper, named GateDet, from improving earlier circuit graph modeling to developing a detection framework based on Bidirectional Graph Convolution Networks with a timely information fusion strategy. GateDet achieves automatic feature circuit extraction and further overcomes the original neighborhood limitation of Bidirectional Graph Convolution Network. Moreover, for large-scale training, it comprehensively considers the problems of sample imbalance and boundary network, and develops a circuit directed graph sampling method based on GraphSAINT, which improves the training performance of the directed graph framework. From experiments, GateDet shows high scalability on 24 benchmarks of TrustHub. It could be used to learn about adaptive structural feature extraction for different Trojans simultaneously. Compared to the existing gate-level detections, the fine-grained results of GateDet are more accurate and can be used to track suspicious structures, reducing manual review.