The fundamental possibility of implementing light-controlled atom-by-atom assembly of silver clusters beginning with dimers and trimers via photostimulated diffusion of individual atoms initially adsorbed on the surface of AgCl and ZnS single crystals by mass-spectrometric ion deposition is shown. Possible mechanisms of this process are discussed, including the charge exchange in adatoms via alternating capture of holes and electrons accompanied by the transition to neighboring, energetically more favorable adsorption sites; hopping displacements of adatoms over the surface by gaining kinetic energy upon nonradiative recombination of photocarriers from their levels; and low-barrier diffusion of short-lived negatively charged adatoms with a captured electron.