We study the $D$-wave charmed baryons of $SU(3)$ flavor $\mathbf{\bar3}_F$ using the method of QCD sum rule in the framework of heavy quark effective theory (HQET). We find that the $\Lambda_c(2880)$, $\Xi_c(3055)$ and $\Xi_c(3080)$ can be well described by the $D$-wave $SU(3)$ $\mathbf{\bar 3}_F$ charmed baryon multiplets of $J^P=3/2^+$ and $5/2^+$, which contain two $\lambda$-mode orbital excitations, i.e., the $\Lambda_c(2880)$ has $J^P=5/2^+$, and the $\Xi_c(3055)$ and $\Xi_c(3080)$ have $J^P = 3/2^+$ and $5/2^+$, respectively. Our results also suggest that the $\Lambda_c(2880)$ has a partner state, the $\Lambda_c(3/2^+)$ of $J^P=3/2^+$. Its mass is around $2.81 ^{+0.33}_{-0.18}$ GeV, and the mass difference between it and the $\Lambda_c(2880)$ is $28^{+45}_{-24}$ MeV. We also evaluate the masses of their bottom partners.