The high energy GeV emission of gamma-ray bursts (GRBs), detected by \emph{Fermi}/LAT, has a significantly different morphology compared to the lower energy MeV emission, detected by \emph{Fermi}/GBM. Though the late time GeV emission is believed to be synchrotron radiation produced via an external shock, this emission as early as the prompt phase is puzzling. Meaningful connection between these two emissions can be drawn only by an accurate description of the prompt MeV spectrum. We perform a time-resolved spectroscopy of the GBM data of long GRBs having significant GeV emission, using a model consisting of 2 blackbodies and a power-law. We examine in detail the evolution of the spectral components and found that GRBs having high GeV emission (GRB 090902B and GRB 090926A) have a delayed onset of the power-law component, in the GBM spectrum, which lingers at the later part of the prompt emission. This behaviour mimics the flux evolution in LAT. In contrast, bright GBM GRBs with an order of magnitude lower GeV emission (GRB 100724B and GRB 091003) show a coupled variability of the total and the power-law flux. Further, by analyzing the data for a set of 17 GRBs, we find a strong correlation between the power-law fluence in the MeV and the LAT fluence (Pearson correlation: r=0.88 and Spearman correlation: $\rho=0.81$). We demonstrate that this correlation is not influenced by the correlation between the total and the power-law fluences at a confidence level of 2.3$\sigma$. We speculate the possible radiation mechanisms responsible for the correlation.