Lithology identification is a necessary task for activities of reservoir evaluation and underground engineering construction. Recently, the intelligent lithology identification method based on deep learning of image has attracted more and more attention with its excellent performance. However, the existing models are not developed specifically for lithology identification, and there is still room for improving accuracy and a large number of redundant calculations during inference. Therefore, it is necessary to develop a lightweight model specifically for lithologic identification. In order to realize accurate, rapid and effective lithology identification, we propose a novel integrated strategy of easy pruning, parameter searching, and reparameterization for lightweight intelligent lithology identification. Firstly, a model dedicated to lithology classification and easy to pruning is constructed by using the multi-branch structure; secondly, a pruning parameter search method based on differential evolution of expressivity is proposed; and finally, the model is further compressed using reparameterization. We validate the entire method using 160 lithologies. P3DNet has an accuracy of 97.69% on 160 lithologies. The pruning method based on differential evolution of expressivity can realize the optimal pruning under the condition of predetermined accuracy. The P3DNet with 134,380 parameters can achieve an accuracy of 97.50% on 160 lithologies, and it takes only 3.84 ms for the NVIDIA GeForce GTX 2080 Ti GPU to infer an image with a resolution of 224×224×3. The pruned P3DNet is 39.54% faster than MobileNet v3 and 50.65% faster than ShuffleNet V2. Our method can significantly reduce the requirements of the image-based identification method on device memory, computing power, etc., and greatly expand the application scenarios of the intelligent identification method. Relevant research can provide strong support for rapid and intelligent identification of lithology in site survey and construction.