In this work, we study, in a systematic way, dust-acoustic modulated envelope structures such as rogue waves (unstable waves) and dark soliton (stable waves) collisions in a complex plasma with nonthermal ions and Boltzmann electrons. In the present plasma system, we can have both negative and positive potential structures associated with the nonlinear dust-acoustic structures. Therefore, we derived the modified Korteweg–de Vries (mKdV) equation, by using the reductive perturbation technique, to describe the nonlinear structures at critical plasma parameters. For studying the properties of the modulated envelope structures, the mKdV equation transformed to a nonlinear Schrödinger equation. Depending on the modulational instability analysis, the stability and instability regions for the propagating nonlinear modulated waves have been determined precisely. After that, the properties of the dust-acoustic rogue waves are examined within the instability regions. Moreover, the effects of physical parameters, such as the ion-to-electron temperature ratio and the ion nonthermal parameter on the profile of dust-acoustic rogue waves are examined. Furthermore, our investigations extended to study the head-on collisions of two-dark solitons in the stability regions. Using the extended Poincaré–Lighthill–Kuo perturbation method, the dark solitons in the present plasma system develop according to two quasi-Korteweg–de Vries equations. After that, the phase shifts induced by the face-to-face collisions between two-dark solitons are obtained analytically. Also, the effects of the above physical parameters on the phase shifts are reported. The results may have relevance in space and laboratory dusty plasmas.