Speech masking is a technique used for protecting speech confidentiality, which is realised by adding maskers to cover target speech. Numerous studies have demonstrated the native and non-native differences in perceiving masked speech, but few have compared the effect of masker types from the perspective of effectiveness of speech privacy and their annoyance levels. This paper reports on the result of an English word perception task by native and non-native listeners where six types of maskers, both speech-like and non-speech-like, are implemented, as well as the annoyance levels of each masker. Natives, non-natives residing in an English-speaking country, and non-natives residing in a non-English-speaking country were tasked to write down sentences when masking sound was present, and also to rate the annoyance levels of the maskers. Results showed that language background significantly affected the performance of unmasked speech, the difference between the two non-native listeners became minimal in the presence of maskers, and maskers annoyed the listener groups differently, calling for a discussion in the selection of a universally effective masker to increase speech privacy.