In recent years, researchers are paying much attention towards the video compression techniques due to the increasing need of storing huge video information in less storage space. H.264 is a popular algorithm for video compression, but it requires more attention towards its performance to improve the visual quality and the compression rate. In literature, there are several algorithms for video compression by providing a good motion computation scheme which is a crucial part of H.264. But these techniques affect the visual quality and the compression rate of the video. By considering this, two important contributions are presented in this paper. The first contribution is a new motion compensation technique through adaptive assisted block search algorithm. The second contribution is devising of new objective function for selecting the best blocks handling rate-distortion tradeoffs. Accordingly, Adaptive Shape Assisted Block Search Algorithm and fuzzy holoentropy-enabled cost function are developed for motion vector computation. The performance analysis is done using two metrics such as Peak Signal Noise Ratio and Compression Ratio to prove the visual quality and compression. Five different video are used for the experimentation and the result shows that the proposed algorithm attained the highest PSNR of 41 dB when compared with elastic model-based compression and H.264.