Due to the high polarizability of finite-length carbon nanotubes (CNTs) in the quasi-static regime, they can be considered as building blocks for the fabrication of high dielectric constant material. Our theoretical estimations, based on an effective medium approach and solutions of a boundary value problem for individual CNT, predict that composite materials comprising short-length CNTs can have very high dielectric constants (up to 300) and low dielectric loss tangents (below 0.03) in the terahertz range. In order to prove this, 500–1000 nm thick films comprising single- and multi-walled CNTs of both long (0.5–2 μm) and short (0.1–0.4 μm) lengths have been fabricated. The analysis, based on the time-domain terahertz spectroscopy in the range 0.2–1.0 THz, demonstrated a decrease in the dielectric loss tangents of the CNT-based materials with a reduction in CNT length. In the terahertz range, the films comprising short-length CNTs had a relative effective permittivity with a large real part (25–136) and dielectric loss tangent (0.35–0.60).