A new clustering technique based on most allied directional neighbors is proposed to suppress low and high-density impulse noise from digital images. Most allied neighbors exhibit a vital role in estimation as well restoration of appropriate gray level value of corrupted pixels. In first phase, most allied directional neighbors, i.e., pixels directly attached to central pixel and the directional pixels (horizontal, vertical and two diagonal directions) next to attached pixels in the processing window, are partitioned into two equal size clusters based on gradient values. Cluster with a minimum sum of gradient values (most similar neighbors) and the one with relatively large gradient values are passed to fuzzy inference system to infer the current pixel to be noisy-free, edge or a noisy. In second phase, a switching technique opts one of the three options depending upon fuzzy membership degrees and local information to restore the corrupted pixel value. A nonparametric approach based on local information for dynamic threshold setting using fuzzy logic makes the proposed filter computationally effective and adaptive to process a large number of images without user-defined parameters. Simulation results based on well-known quantitative measures indicate the supremacy of the proposed filter for random-valued impulse noise as well as salt and peppers noise.