Nowadays, the aluminum alloy building structure is at risk of the accidental impact loadings, which seriously threatens the personal security and the safety of the property. However, the existing research on the local lateral impact performance of the aluminum foam-filled tube applied to building structures is limited. In this study, impact tests on 9 aluminum foam-filled tubes with fixed boundaries were conducted by using a drop hammer impact test device to investigate the dynamic performance of the foam-filled tube under local lateral low-velocity impact loadings. The failure modes, impact force histories, displacement histories, and permanent deformation of the foam-filled tubes were obtained and compared with those of the empty tubes under the same impact loading. The finite element (FE) model was also established via the software ANSYS/LS-dyna and the accuracy of the FE method was verified against impact test results. Based on the experimental and numerical results, the aluminum foam-filled aluminum tube shows better lateral impact resistant performance than the empty tube due to the support effect of the aluminum foam on the tube wall by good energy absorption and deformation characteristics. Especially, filling the aluminum foam prevents the generation of cracks for the composite tube near the impact position when the impact energy is high. The influence law of the aluminum foam filler and its density, shape of hammer head on the lateral impact resistant performance of the aluminum foam-filled composite tube was then investigated based on the experimental and numerical results by consideration of the damage level, energy absorption, impact force–lateral displacement relationship, and permanent deformation. By filling the aluminum foam into the empty tube, the energy absorption is improved by up to 67.2% and 44.9%, respectively, when the hammer displacement reaches 20 mm and 40 mm within the impact load condition conducted in this study.