The simulated annealing algorithm is applied to the design of binary holographic optical elements for the connection of single-mode structures. The system energy used with the simulated annealing algorithm is the absolute square of the overlap between the transmitted wave front and the modal characteristics of the receiving device; the system variables are the binary states of the hologram pixels. The plausibility of the obtained binary patterns is discussed for examples involving plane and spherical modal characteristics. The ability of the annealing process to avoid local minimum trapping is demonstrated.