Most convolutional neural network (CNN) designs are still bottlenecked by costly computational load, which impedes their utilization in some industrial applications. To address this issue, a novel design framework is proposed in this paper, which fuses three design strategies that can be applied to the majority of the state-of-the-art popular block-based CNN models to lighten their computing budget, while maintaining comparable accuracies. The novel concept of sparse skip connections is introduced as a block-level design strategy in the models, where each module output is used at both an input to the next module and an input to a three-step-away module to efficiently exploit feature reuse across the blocks. In module-level design, a proportional channel split operation strengthened by concatenation is employed to achieve accuracy and model size trade-off. As part of the layer-level strategy, an equal number of input and output channels in layers is selected so that the degree of parallelization is increased for a faster inference time. To demonstrate the framework’s efficacy, comprehensive experiments are conducted performing highly competitive visual recognition tasks, on the ImageNet datasets for image classification and on the MS COCO dataset for object detection. The evaluation results, which are based on DenseNet, ResNet, ShiftNet, ShuffleNet, and ShuffleNet-v2, verify that the proposed SSP framework is notably capable of reducing the parameter number, FLOPs, and inference time of existing CNN models with quite alike accuracies.