We present a search for pair production of a fourth generation t' quark and its antiparticle, followed by their decays to a W boson and a jet, based on an integrated luminosity of 5.3/fb of proton-antiproton collisions at sqrt{s}=1.96 TeV collected by the D0 Collaboration at the Fermilab Tevatron Collider. We set upper limits on the t't'bar production cross section that exclude at the 95% C.L. a t' quark that decays exclusively to W+jet with a mass below 285 GeV. We observe a small excess in the muon+jets channel which reduces the mass range excluded compared to the expected limit of 320 GeV in the absence of a signal.