This study aimed to introduce sediment properties (total organic carbon (TOC), acid-volatile sulfides (AVS), particle size distribution) into sediment quality guideline-based risk quotients to assess the potential toxicity of metals (Ni, Cu, Zn, Cr, Cd, and Pb) released from sediments. Sediment was collected at three times points in 20 sampling sites in Kaohsiung Harbor. The Microtox® toxicity test was used to assess the sediment toxicity and the relationship between sediment toxicity and risk quotient estimated based on the metal concentration was constructed. To improve the toxicity prediction and modify the risk quotient according to the sediment properties, stepwise multiple linear regression (MLR) models that have been tested over wide ranges of TOC, AVS, and particle size distribution to determine the key sediment properties. Common multimetal indices, including the pollution load index, modified degree of contamination index, Nemerow pollution index, potential ecological risk index, and total toxic risk index, were compared with sediment toxicity to evaluate the degrees of correlation. By modifying the relationship between metal toxicity and the risk quotient by including TOC and AVS, the prediction showed that sediments in Kaohsiung Harbor were generally of slight acute toxicity to acute toxicity to organisms, with sampling sites near an industrial zone showing a higher probability of high acute toxicity. In particular, the acute risk of adverse effects on aquatic organisms from sediments in the Salt River estuary was significantly higher than that at other sites, which was consistent with the results of assessment based on the multimetal indices. This study suggests that the MLR-based approach may facilitate the adoption of updated site-specific metals standards that more accurately account for the parameters affecting metal bioavailability than metal concentration standard alone.