Most existing weakly-supervised object localization (WSOL) methods have improved training procedures for better localization performance. However, the inference procedure has been overlooked. We observe that the useful information for localization is missed by the current inference practice of WSOL. To address this limitation, we propose a new test-time ingredient for WSOL: binarizing the penultimate feature map and their corresponding weights of the last linear layer. With this simple remedy, the proposed method consistently improves the localization performance of the existing training methods for WSOL. Extensive evaluation including with three different backbone networks on three different WSOL benchmarks validates its effectiveness. In addition, we demonstrate our method is also able to improve weakly-supervised semantic segmentation performances on PASCAL VOC dataset. Lastly, since our method is only applied during the testing phase, our performance gain comes with negligible computational overheads.