Organically modified layered double hydroxide (O-LDH) was successfully synthesized by coprecipitation and characterized through Fourier transform infrared spectroscopy (FTIR) and X-ray diffraction (XRD). O-LDH/poly(butyl acrylate–vinyl acetate) (P(BA-VAc) emulsion was prepared via in situ polymerization, then ammonium polyphosphate (APP) was introduced to obtain the O-LDH/APP/P(BA-VAc) flame-retardant latex. The flame retardancy and thermal behavior of the latexes after evaporation of water were investigated by the limiting oxygen index (LOI) test, vertical burning test (UL-94), and thermal gravimetric analysis (TGA). Compared with the APP/P(BA-VAc) composite, the LOI value of O-LDH/APP/P(BA-VAc) containing 0.5 wt.% O-LDH at the same total additive loading increased to over 30.0%, and its UL-94 increased from no rating to V-0. TG data show that the amount of residues increases obviously when O-LDH is added. The LOI values increased with increasing amounts of char residues. The morphology and microstructure of residues generated during the LOI test were investigated by scanning electron microscopy (SEM). FTIR and X-ray photoelectron spectroscopy (XPS) were used to determine the composition of the residues formed after combustion at 450 °C for 5 min to support a fundamental analysis of the mechanism of char formation.