Infrared small target detection for aerial remote sensing is crucial in both civil and military fields. For infrared targets with small sizes, low signal-to-noise ratio, and little detailed texture information, we propose a Res-SwinTransformer with a Local Contrast Attention Network (RSLCANet). Specifically, we first design a SwinTransformer-based backbone to improve the interaction capability of global information. On this basis, we introduce a residual structure to fully retain the shallow detail information of small infrared targets. Furthermore, we design a plug-and-play attention module named LCA Block (local contrast attention block) to enhance the target and suppress the background, which is based on local contrast calculation. In addition, we develop an air-to-ground multi-scene infrared vehicle dataset based on an unmanned aerial vehicle (UAV) platform, which can provide a database for infrared vehicle target detection algorithm testing and infrared target characterization studies. Experiments demonstrate that our method can achieve a low-miss detection rate, high detection accuracy, and high detection speed. In particular, on the DroneVehicle dataset, our designed RSLCANet increases by 4.3% in terms of mAP@0.5 compared to the base network You Only Look Once (YOLOX). In addition, our network has fewer parameters than the two-stage network and the Transformer-based network model, which helps the practical deployment and can be applied in fields such as car navigation, crop monitoring, and infrared warning.