Two-loop multi-leg form factors in off-shell kinematics require knowledge of planar and nonplanar double box Feynman diagrams with massless internal propagators. These are complicated functions of Mandelstam variables and external particle virtualities. The latter serve as regulators of infrared divergences, thus making these observables finite in four space-time dimensions. In this paper, we use the method of canonical differential equations for the calculation of (non)planar double box integrals in the near mass-shell kinematical regime, i.e., where virtualities of external particles are much smaller than the Mandelstam variables involved. We deduce a basis of master integrals with uniform transcendental weight based on the analysis of leading singularities employing the Baikov representation as well as an array of complementary techniques. We dub the former asymptotically canonical since it is valid in the near mass-shell limit of interest. We iteratively solve resulting differential equations up to weight four in terms of multiple polylogarithms.