When a thin binary layer grows epitaxially on an elemental substrate, the composition often modulates in the plane of the layer. The layer may even demix when the bulk of the same composition is miscible. Sometimes the layer separates into two phases, forming periodic stripes or other regular patterns. The size of the separated phases may be in the range 1–100 nm, and stable against coarsening on annealing. This paper develops a thermodynamic framework to study these remarkable phenomena. For an epilayer less than a few monolayers thick, the excess energy cannot be attributed to individual sources of superficial misfit. Instead, we lump the epilayer and adjacent monolayers of the substrate into a single superficial object, and specify the excess surface energy for the object. The variation of the surface energy density with the variation of the strain defines surface stress. When the composition modulates in the epilayer, the surface stress is nonuniform, deforms the substrate, and reduces the total energy. Consequently, the composition-dependent surface stress tends to refine phases, resisting coarsening. In a stability analysis, we perturb a uniform concentration field into a sinusoidal field with a small amplitude and an arbitrary wavelength. The entropy of mixing stabilizes the uniform layer for long wavelengths. To stabilize the uniform layer for short wavelengths, we assume that the surface energy density also depends on concentration-gradient. A stability condition shows the combined elastic, entropic, and gradient effects. We also outline a dynamical system, which can be used to study pattern emergence and evolution during annealing.