The natural scenarios show that the images which are taken under low light environments experiences weak luminosity and low-contrast issues frequently. One of the vital units of image contrast enhancement accelerator is a Histogram computation. In this paper, a new VLSI structure is developed for contrast enhancement using optimized memory-based histogram analysis. Initially, a comparator-less parallel rank ordering filter is proposed to remove the noise in the pre-processing stage. Then, the pipelining and parallel process of the memory-based histogram analysis is enhanced by introducing an enhanced transient search optimization (ETSO) approach that selects suitable data to perform the data comparison process in the processing elements (PEs). In addition, a flexible digital comparators (FDC), pipelined structure of bilinear interpolation and reciprocal units are introduced in the proposed contrast enhancement accelerator to save area and power usage. Also, the histogram memory unit is designed by considering dual data flow with self-configurable characteristics for the selection of appropriate data flow in the contrast enhancement application. At last, the stated VLSI structures for both histogram computation and enhancement unit are implemented in FPGA for meeting their speed and resource constraints. The key factors of the proposed design are lowest delay of 5.256 ns and lower power consumption of 0.289 W against existing accelerators.