Model reduction is a common task within the simulation, control, and optimization of complex dynamical systems. For instance, in control and optimization problems for instationary partial differential equations (PDEs), the associated large-scale linear systems resulting from a (semi-) discretization have to be solved repeatedly. Model reduction is a more and more accepted technique to attack these problems in reasonable time. Here, we investigate model reduction for parabolic PDEs based on balanced truncation (BT). The system theoretic properties of BT and the related method of singular perturbation approximation (SPA) provide desirable features for the reduced-order system. The major computational task in BT is the solution of large-scale Lyapunov or Stein equations, depending on whether the semidiscretized or fully discretized system is considered. Therefore, the method is often considered to be of limited use for really large-scale applications, i.e., problems that require $\gg1$ GByte of storage, so that memory access becomes an issue. Here, we develop effective implementations of BT and SPA by exploiting the structure of the underlying PDE problem. This is achieved by a data-sparse approximation of the (semi-)discretized differential operator using the hierarchical matrix format. Furthermore, we review how to integrate the corresponding formatted arithmetic in the sign function method for computing approximate solution factors of the Lyapunov equations and suggest an iterative scheme for solving Stein equations employing formatted arithmetic. These approaches are well suited for a class of practically relevant problems and allow the application of BT and SPA to systems coming from two-dimensional (2D) and three-dimensional (3D) finite element method (FEM) and boundary element method (BEM) discretizations. The resulting methods are most beneficial in the context of BEM discretizations for 3D problems where large-scale dense matrices define the (semi-)discretized control system. We also analyze the additional approximation error in the reduced-order model introduced by the hierarchical matrix approximation. The efficiency of the methods is demonstrated for linear, scalar convection-diffusion-reaction equations. The numerical results confirm that the model reduction/BT error and the hierarchical matrix approximation error can be balanced by choosing the tolerances and parameters involved appropriately.