Drought stress is an important factor affecting soybean yield. Improving drought tolerance of soybean varieties can increase yield and yield stability when the stress occurs. Identifying QTL related to drought tolerance using molecular marker-assisted selection is able to facilitate the development of drought-tolerant soybean varieties. In this study, we used a high-yielding and drought-sensitive cultivar ‘Zhonghuang 35’ and a drought-tolerant cultivar ‘Jindou 21’ to establish F6:9 recombinant inbred lines. We constructed a high-density genetic map using specific locus amplified fragment sequencing (SLAF-Seq) technology. The genetic map contained 8078 SLAF markers distributing across 20 soybean chromosomes with a total genetic distance of 3780.98 cM and an average genetic distance of 0.59 cM between adjacent markers. Two treatments (irrigation and drought) were used in the field tests, the Additive-Inclusive Composite Interval Mapping (ICIM-ADD) was used to call QTL, and plant height and seed weight per plant were used as the indicators of drought tolerance. We identified a total of 23 QTL related to drought tolerance. Among them, seven QTL (qPH2, qPH6, qPH7, qPH17, qPH19-1, qPH19-2, and qPH19-3) on chromosomes 2, 6, 7, 17, and 19 were related to plant height, and five QTL (qSWPP2, qSWPP6, qSWPP13, qSWPP17, and qSWPP19) on chromosomes 2, 6, 13, 17, and 19 were related to seed weight and could be considered as the major QTL. In addition, three common QTL (qPH6/qSWPP6, qPH17/qSWPP17, and qPH19-3/qSWPP19) for both plant height and seed weight per plant were located in the same genomic regions on the same chromosomes. Three (qPH2, qPH17, and qPH19-2) and four novel QTL (qSWPP2, qSWPP13, qSWPP17, and qSWPP19) were identified for plant height and seed weight per plant, respectively. Two pairs of QTL (qPH2/qSWPP2 and qPH17/qSWPP17) were also common for both plant height and seed weight per plant. These QTL and closely linked SLAF markers could be used to accelerate breeding for drought tolerant cultivars via MAS.