In image retrieval applications for multi-users, users can retrieve similar images in a database. The server can collude with the user and exploit the search patterns to infer the information of other users, which may cause search pattern leakage. However, current forward privacy solutions can only partially solve the problem. In this paper, a multi-user image retrieval scheme, called multi-user image retrieval with suppression of search pattern (MIRSP), is presented to suppress the search pattern leakage. This scheme uses a blockchain structure to establish an optimized tree index. Meanwhile, compressive sensing is used to convert high-dimensional feature vectors into similarity-preserving binary codes, which reduces the similarity calculation cost and guarantees the security of nodes. Then, a path shuffling algorithm is proposed to hide the identifier and position of the search nodes. Finally, a fully trusted certificate authority center is added to ensure the security management and distribution of keys and avoid unnecessary communication overhead. Our experimental evaluation shows that the proposed MIRSP is 3 × faster than the state-of-the-art schemes in search computation times, which demonstrates that MIRSP contributes significantly in providing security and privacy protection in multi-user scenarios.