In this paper we consider an intrusion tolerant system with two detection modes; automatic detection mode and manual detection mode for intrusions, which is called SITAR (Scalable Intrusion Tolerant Architecture), and describe the dynamic transition behavior by a continuous-time semi-Markov chain (CTSMC). Based on the embedded Markov chain (EMC) approach, we derive the steady-state probability of the CTSMC, the steady-state system availability and the mean time to security failure (MTTSF). Especially, we show necessary and sufficient conditions to exist the optimal switching time from an automatic detection mode to a manual detection mode, which maximizes the steady-state system availability. Next, we develop an adaptive mode control scheme to estimate the optimal switching time without specifying the associated probability distribution function, whose idea behind is based on a statistically non-parametric algorithm by means of the total time on test concept. Numerical examples through a comprehensive simulation study are presented for illustrating the optimal switching of detection mode, and investigating the asymptotic property of the resulting adaptive mode control scheme.