This study aims at demonstrating label-free drug-response-patterns assessment of different tumor spheroids and drug types by dynamic optical coherence tomography (D-OCT). The study involved human breast cancer (MCF-7) and colon cancer (HT-29) spheroids. The MCF-7 and HT-29 spheroids were treated with paclitaxel (Taxol; PTX) and the active metabolite of irinotecan SN-38, respectively. The drugs were applied with 0 (control), 0.1, 1, and 10 μM concentrations and the treatment durations were 1, 3, and 6 days. A swept-source OCT microscope equipped with a repeated raster scanning protocol was used to scan the spheroids. Logarithmic intensity variance (LIV) and late OCT correlation decay speed (OCDS_l) algorithms were used to visualize the tumor spheroid dynamics. LIV and OCDS_l images visualized different response patterns of the two types of spheroids. In addition, spheroid morphology, LIV, and OCDS_l quantification showed different time-courses among the spheroid and drug types. These results may indicate different action mechanisms of the drugs. The results showed the feasibility of D-OCT for the evaluation of drug response patterns of different cell spheroids and drug types and suggest that D-OCT can perform anti-cancer drug testing.