In fact, several categories in a large dataset are not difficult for recent advanced deep neural networks to recognize. Eliminating them for a challenging smaller subset will assist the early network proposals in taking a quick trial of verification. To this end, we propose an efficient rescaling method based on the validation outcomes of a pre-trained model. Firstly, we will take out the sensitive images of the lowest-accuracy classes of the validation outcomes. Each of such images is then considered to identify which label it was confused with. Gathering the lowest-accuracy classes along with the most confused ones can produce a smaller subset with a higher challenge for quick validation of an early network draft. Finally, a rescaling application is introduced to rescale two popular large datasets (ImageNet and Places365) for different tiny subsets (i.e., ReINΩ and RePLΩ respectively). Experiments for image classification have proved that neural networks obtaining good performance on the original datasets also achieve good results on their rescaled subsets. For instance, MobileNetV1 and MobileNetV2 with 70.6% and 72% on ImageNet respectively obtained 46.53% and 47.47% on its small subset ReIN30, which only contains about 39000 images. It can be observed that the better performance of MobileNetV2 on ImageNet correspondingly leads to the better rate on its rescaled subset. Appropriately, utilizing these rescaled sets would help researchers save time and computational costs in the way of designing deep neural architectures. All codes related to the rescaling proposal and the resultant subsets are available at http://github.com/nttbdrk25/ImageNetPlaces365.