Image modeling towards sport scenes plays an important role in sport image classification and analysis. Traditional algorithms for sport image modeling required carefully hand-crafted features, which cannot be popularized in practical application, especially with the emergence of massive-scale data. Weakly-supervised learning algorithms have shown effectiveness in modeling data with image-level labels. Thus, in this paper, we propose a weakly-supervised learning based method for sport image modeling without utilizing bounding box annotations, which can be used for various sport image applications. More specifically, we first collect large-scale sport images from existing datasets and Internet, and we annotate them at image-level labels. Subsequently, we leverage region proposal generation algorithm to select discriminative regions that can effectively represent the category of images. Each region is fed into a pre-trained CNN architecture to extract deep representation. Afterwards, we design an improved multiple discriminant analysis (MDA) algorithm to project these datapoints to a subspace that can more easily to distinguish different sport categories. Comprehensive experiments have shown the effectiveness and robustness of our proposed method.