In this paper, we present and design a tensor holographic metasurface (THMS) capable of radiating multiple beams with different polarizations in both forward and backward directions. The proposed THMS manipulates the fundamental TE mode surface waves (SWs) to produce bidirectional leaky-wave radiation using bilayer anisotropic metal patches. To achieve the independent control of the direction and polarization of bidirectional beams, the nature of the capacitive impedance element supporting TE mode SWs is exploited. A dual-feed scheme exciting orthogonal TE SWs is proposed for THMS generating circularly polarized beams with good performance. As a proof of concept, a bidirectional THMS integrated with two WR-28 waveguide sources based on the orthogonal-feeds scheme, which simultaneously radiate linear- and circular-polarized four beams, is designed, fabricated, and measured at 35 GHz. The experiment results are in good agreement with the simulated ones, achieving a peak gain of 19.2 (19.6) dBi, a -3 dB gain bandwidth of 9.14% (4.3%), for circularly-polarized (linearly-polarized) beams and a 3 dB axial ratio bandwidth of 13% for circularly-polarized beams. The sidelobe levels and cross polarization levels of all beams are below -15 dB and -22 dB, respectively.