Software is used in diverse settings and depends on development and testing environments. Software development should improve the reliability, quality, cost, and stability of software, making the software testing period crucial. We proposed a software reliability model (SRM) that considers the uncertainty of software environments and the fault detection rate function as a Rayleigh distribution, with an explicit mean value function solution in the model. The goodness-of-fit of the proposed model relative to several existing nonhomogeneous Poisson process (NHPP) SRMs is presented based on three software application failure datasets. Further, a cost model is also presented that addresses the error-removal risk level and required time. The optimal testing release policy for minimizing the expected total cost (ETC) is also determined for NHPP SRMs. The impact of the software environment is studied by varying it, and the optimal release times and minimum ETCs are compared. The goodness-of-fit comparison confirmed that the proposed model has more accurate prediction values than other models. Further, whereas the existing models applied to the cost model do not change after a certain operation period, the proposed model yields changes in release time even for long operating periods.