Engineering quantum devices requires reliable characterization of the quantum system, including qubits, quantum operations (also known as instruments) and the quantum noise. Recently, quantum gate set tomography (GST) has emerged as a powerful technique for self-consistently describing quantum states, gates, and measurements. However, non-Markovian correlations between the quantum system and environment impact the reliability of GST. To address this, we propose a self-consistent operational framework called instrument set tomography (IST) for non-Markovian GST. Based on the stochastic quantum process, the instrument set describes instruments and system-environment (SE) correlations. We introduce a linear inversion IST (LIST) to describe instruments and SE correlations without physical constraints. The disharmony of linear relationships between instruments is detected. Furthermore, we propose a physically constrained statistical method based on the maximum likelihood estimation for IST (MLE-IST) with adjustable dimensions. MLE-IST shows significant flexibility in adapting to different types of devices, such as noisy intermediate-scale quantum (NISQ) devices, by adjusting the model and constraints. Experimental results demonstrate the effectiveness and necessity of simultaneously describing instruments and SE correlations. Specifically, the LIST and MLE-IST obtains significant improvement on average square error reduction in the imperfect implemented simulations by orders of −23.77 and −6.21, respectively, compared to their comparative methods. Remarkably, real-chip experiments indicate that a polynomial number of parameters with respect to the Markovian order are sufficient to characterize non-Markovian quantum noise in current NISQ devices. Consequently, IST provides an essential and self-consistent framework for characterizing, benchmarking, and developing quantum devices in terms of the instrument set.