Nowadays, the extensively used lead sulfide (PbS) quantum dot (QD) hole transport layer (HTL) relies on layer-by-layer method to replace long chain oleic acid (OA) ligands with short 1,2-ethanedithiol (EDT) ligands for preparation. However, the inevitable significant volume shrinkage caused by this traditional method will result in undesired cracks and disordered QD arrangement in the film, along with adverse increased defect density and inhomogeneous energy landscape. To solve the problem, a novel method for EDT passivated PbS QD (PbS-EDT) HTL preparation using small-sized benzoic acid (BA) as intermediate ligands is proposed in this work. BA is substituted for OA ligands in solution followed by ligand exchange with EDT layer by layer. With the new method, smoother PbS-EDT films with more ordered and closer QD packing are gained. It is demonstrated stronger coupling between QDs and reduced defects in the QD HTL owing to the intermediate BA ligand exchange. As a result, the suppressed nonradiative recombination and enhanced carrier mobility are achieved, contributing to ≈20% growth in short circuit current density (Jsc) and a 23.4% higher power conversion efficiency (PCE) of 13.2%. This work provides a general framework for layer-by-layer QD film manufacturing optimization.