Previous experiments have shown that spherical colloidal particles relax to equilibrium slowly after they adsorb to a liquid-liquid interface, despite the large interfacial energy gradient driving the adsorption. The slow relaxation has been explained in terms of transient pinning and depinning of the contact line on the surface of the particles. However, the nature of the pinning sites has not been investigated in detail. We use digital holographic microscopy to track a variety of colloidal spheres-inorganic and organic, charge-stabilized and sterically stabilized, aqueous and non-aqueous-as they breach liquid interfaces. We find that nearly all of these particles relax logarithmically in time over timescales much larger than those expected from viscous dissipation alone. By comparing our results to theoretical models of the pinning dynamics, we infer the area per defect to be on the order of a few square nanometers for each of the colloids we examine, whereas the energy per defect can vary from a few kT for non-aqueous and inorganic spheres to tens of kT for aqueous polymer particles. The results suggest that the likely pinning sites are topographical features inherent to colloidal particles-surface roughness in the case of silica particles and grafted polymer "hairs" in the case of polymer particles. We conclude that the slow relaxation must be taken into account in experiments and applications, such as Pickering emulsions, that involve colloids attaching to interfaces. The effect is particularly important for aqueous polymer particles, which pin the contact line strongly.