Ignition location prediction is crucial for wildfire incident investigation and events reconstruction. However, existing models mainly focus on simulating the wildfire forward and rarely trace the ignition backward. In this paper, a novel transformer-based neural network named ILNet was proposed to predict the ignition location backward from the final wildfire perimeter. The ILNet first concatenated all wildfire-driven data as a composite image and divided it into several regular patches. Then, the self-attention mechanism was adopted to extract global spatial features with a variable scale among these patches. These features were further decoded to output semantic masks of growth phase and ignition phase. The geometric center of ignition phase was defined as the ignition location. Finally, a real wildfire was chosen as the study case. The results show the competitive performance of ILNet model (MIoU: 88.45%, IDE_N: 1.99%, computation time: 0.57s), enabling to improve the traditional field work for government agencies.