The big bottleneck in scaling PDE-based codes to petaflop computing is scalability of effective preconditioners. We have developed and implemented an effective and scalable low energy basis preconditioner (LEBP) for elliptic solvers, leading to computational savings of an order of magnitude with respect to other preconditioners. The efficiency of LEBP relies on the implementation of parallel matrix-vector multiplication required by coarse solver to handle the h-scaling. We provide details on optimization, parallel performance and implementation of the coarse grain solver and show scalability of LEBP on the IBM Blue Gene and the Cray XT3.