The technique of $$\mathcal {H}$$ -matrices was introduced to deal with (large-scale) dense matrices in an elegant way. It provides a data-sparse format and allows an approximate matrix algebra of nearly optimal complexity. The primary step in the construction of an $$\mathcal {H}$$ -matrix is how to approximate the subblocks of a given dense matrix by matrices that have low numerical rank. Randomized algorithms have recently been introduced as a highly efficient tool for computing approximate factorization of low-rank matrices. In this paper, we consider various randomized algorithms to construct an $$\mathcal {H}$$ -matrix and perform the corresponding $$\mathcal {H}$$ -matrix algebra. In particular, by taking advantages of randomization, we present a simple but fast algorithm of complexity $$\mathcal {O}(4k^2n+k^3 )$$ for truncating an $$n\times n$$ low-rank matrix of (fixed) rank k. The corresponding efficient deterministic algorithm has the complexity $$\mathcal {O}(12k^2n+23k^3 )$$ . We provide numerical examples applying to a BEM model associated with Laplace equation in one and two dimensions to show the efficiency of the proposed randomized methods versus the previously known efficient deterministic methods. The gain of efficiency about 10–25% is achievable when a moderate oversampling parameter $$p=5$$ is used in the computations. In this case, the experimental results show that the proposed algorithm not only has a lower cost but also is more accurate than the conventional methods.