In the transition to full deployment of connected vehicles (CVs), the CV penetration rate plays a key role in bridging the gap between partial and complete traffic information. Several innovative methods have been proposed to estimate the CV penetration rate using only CV data. However, these methods, as point estimators, may lead to biased estimations or suboptimal solutions when applied directly in modeling or system optimization. To avoid these problems, the uncertainty and variability in the CV penetration rate must be considered. Recently, a probabilistic penetration rate (PPR) model was developed for estimating such uncertainties. The key model input is a constrained queue length distribution composed exclusively of queues formed by red signals in undersaturation conditions with no residual vehicles. However, in real-world scenarios, due to random arrivals, residual vehicles are commonly carried over from one cycle to another in temporary overflow cycles in undersaturation conditions, which seriously restricts the applicability of the PPR model. To address this limitation, this paper proposes a Markov-constrained queue length (MCQL) model that can model the complex effects of residual vehicles on the CV penetration rate uncertainty. A constrained queue with residual vehicles is decomposed into four vehicle groups: observable constrained residual vehicles, unobservable constrained residual vehicles, unconstrained residual vehicles, and new arrivals. Although the first vehicle group is observable in the former cycle, the focus of this work is to model the residual vehicles from the second and third vehicle groups in combination with the new arrivals. The MCQL model includes four sub-models, namely, the residual-vehicle model, convolutional constrained queue model, constrained residual queue model, and observable residual queue model, to isolate and derive the distribution of the constrained vehicle set formed by the three latter vehicle groups. This distribution is then substituted into the PPR model to estimate the uncertainty. Comprehensive VISSIM simulations and applications to real-world datasets demonstrate that the proposed MCQL model can accurately model the residual-vehicle effect and estimate the uncertainty. Thus, the applicability of the PPR model is truly extended to real-world settings, regardless of the presence of residual vehicles. A simple stochastic CV-based adaptive signal control example illustrates the potential of the proposed model in real-world applications.