Amorphous zinc-niobium-tin-oxide (a-ZNTO), a new amorphous oxide semiconductor, was proposed for fabricating thin-film transistors (TFTs). a-ZNTO thin films were grown by magnetron sputtering and used as channel layers of TFTs. The effects of niobium content on film properties and device behaviors were investigated in detail. Nb could serve as a carrier suppressor by reducing oxygen vacancies in a-ZNTO films and lower the interfacial trap states in a-ZNTO TFTs. The enhanced performances of a-ZNTO TFTs were achieved at an optimized Nb content of $x=0.2$ for a Nb:Zn:Sn atomic ratio of $x$ :4:7. The reduced channel sizes are very effective to improve TFT performances, with an ON/OFF current ratio of $\sim 10^{8}$ and a field-effect mobility of 13.2 cm2/Vs. The indium-free a-ZNTO TFTs may have the potential for applications in next-generation displays.