Asymmetric Y-junctions, compared with mode coupling-based devices, possess considerably smaller wavelength dependence and thus are more promising for ultra-broadband mode (de)multiplexing in integrated optics. However, these devices also feature relatively high mode crosstalk and insertion loss. Here, we show that the mode crosstalk and loss of an asymmetric Y-junction can be significantly reduced by optimizing the waveguide shape of the Y-junction using an adjoint-based inverse design. Based on such inverse-designed asymmetric Y-junctions, we realize ultra-compact, broadband, and low crosstalk silicon photonic TE00 & TE1 and TE0 & TE2 mode (de)multiplexers with sizes of only 4.5 × 1.2 µm2 and 6 × 1.4 µm2, respectively. From simulations it is shown that the TE0 & TE1 and TE0 & TE2 mode (de)multiplexers contain wide bandwidths of 160 nm (1460-1620 nm) and 140 nm (1460-1600 nm), respectively, over which the mode crosstalks are below about -20 dB, and the losses are <0.41 dB and <0.88 dB, respectively. The experimental results show that in the corresponding TE0 & TE1 and TE0 & TE2 mode division multiplexing systems, the crosstalks are less than -15.5 dB and -15 dB over the spectral ranges of 1453-1580 nm and 1460-1566 nm, respectively, and the losses are <1.7 dB at 1520 nm and <8.24 dB over the entire measured wavelength range.