SummaryThe knowledge of the effects of instability and heterogeneity on displacements, primarily enhanced oil recovery, and carbon dioxide storage are well known, although they remain difficult to predict. The usual recourse to modeling these effects is through numerical simulation. Simulation remains the gold standard for prediction; however, its results lack generality, being case-specific. There are also several analytic models for displacements that are usually more informative than simulation results. However, these methods apply to steady-state, incompressible flow.Carbon dioxide injection for storage uses compressible fluids and, in the absence of producers, will not approach steady-state flow (Wu et al. 2017). Consequently, it is unlikely that storage will be in reservoirs of open boundaries (steady-state flow). Flow of compressible fluid necessitates the use of closed or partially sealed boundaries, a factor that is consistent with compressible flow.This work deals with the conditions that cause the onset of incipient viscous fingering or Saffman-Taylor (ST) instability. The actual growth and propagation of fingers, a subject of much recent literature, is not discussed here. The original ST formalism of M > 1 for gravity-free flow is highly restrictive: it is for linear flow of nonmixing incompressible fluids in steady-state flow. In this work, we relax the incompressible flow restriction and thereby broaden the ST criterion to media that have sealing and/or partially sealing outer boundaries. We use the nonlinear partial differential equation for linear flow and developed analytic solutions for a tracer flow analog and also for a two-fluid compressible flow. The analysis is restricted to stabilized flow and to constant compressibility fluids, but we are not restricted to small compressibility fluids.There is no transition (mixing) zone between displacing and displaced fluids; the displacement is piston-like. The absence of a transition zone means that the results apply to both miscible and immiscible displacements, absent dispersion, or local capillary pressure. The assumption of a sharp interface is to focus on the combined effect of mobility ratio and compressibility. We use the product of the fluid compressibility and pressure drop (cfΔP) to differentiate the compressibility groups (Dake 1978; Dranchuk and Quon 1967), where ΔP is defined as the pressure drop within the specific fluid region.The results will be based on proposed analytical solutions compared to numerical simulation. The proposed formulation is less restrictive than the original ST formalism of M > 1 and allows evaluation of viscous fingering initiation or ST stability criterion in the presence of different boundary conditions (open vs. closed boundaries) with compressible fluids under the stated assumptions, which is the scope of this work.The key contribution here is the effect of external boundaries, which consequently makes necessary the use of compressible fluids. Absent compressibility, the necessary condition for the growth of a viscous finger is simply the mobility ratio, M > 1. It is the objective of this work to study how the ST criterion is affected by the presence of sealing and partially sealing outer boundaries with the consequent inclusion of compressible flows as in carbon dioxide storage and enhanced oil recovery by gas injection.The results show that adding compressibility always makes displacements more unstable for stabilized background flow, even for a favorable mobility ratio (M < 1) at extremely large compressibility (e.g., cf > 5×10−3 1/psi). For a sealed external boundary (no production or leakage), displacements will become more stable as a front approaches an external boundary for all mobility ratios (M) investigated.