The design procedure of a novel compact ultra-wideband (UWB) filter with high selectivity and low insertion loss is presented in this paper. The proposed filter consists of a simple planar combline filter at the top of a substrate and a non-intuitive shape of the defected ground structure (DGS) on the other side. The UWB filter is obtained by optimizing the shape of DGS and the combline parameters with a mixed integer genetic algorithm. The mixed integer genetic algorithm optimizes both continuous and binary values simultaneously. The continuous values define the combline parameters while the binary values define the shape of DGS. The detail of the design procedure is presented and validated by fabricating a prototype of the proposed filter. The measured insertion loss of the proposed filter in the mid-band frequency is less than 0.5 dB. The measurement results also show that the proposed UWB filter with a very compact structure has five transmission zeroes at the upper and lower stop bands.