In this paper a shell formulation is proposed for analyzing the torsional instability of carbon nano-peapods (CNPs), i.e., the hybrid structures composed of C60 fullerenes encapsulated inside carbon nanotubes (CNTs), based on the nonlocal elasticity theory. The nonlocal elasticity theory, as a well-known non-classical continuum theory, is capable to capture small scale effects which appear due to the discontinuities in nano-structures. Based on the derived formulation, the critical torsional moments for a pristine (10,10) CNT and C60@(10,10) CNP are investigated as case studies. The results for the (10,10) CNT are compared with those of the available molecular dynamics simulations in the literature, and accordingly the appropriate value of the small scale coefficient appearing in the constitutive equations of the nonlocal theory is estimated for CNTs. Then, the critical torsional moment for the C60@(10,10) CNP is predicted. It is observed that the presence of the encapsulated C60 fullerenes inside the (10,10) CNT causes an increase in the torsional instability resistance of the CNT more than 100%.