Public Administrative Procedures (APs) are processes, implementation methods, documents, and requirements or conditions prescribed by government agencies or authorized individuals to address specific tasks related to individuals or organizations. However, organizations and citizens still face challenges in easily and conveniently accessing information and public administrative services. This paper investigates advanced techniques to address the challenges of document retrieval for public administrative services. We implement a hybrid retrieval process, combining traditional retrieval models such as TF-IDF and BM25 with modern models such as SBERT and fine-tuning models. Results demonstrate that combining these models significantly enhances retrieval performance. The ensemble model of BM25 and finetuned SBERT achieved the highest F2 score, indicating superior effectiveness in information retrieval.