We study the solution of the linear least-squares problem $\min_x \| b - A x \|^2_2$ where the matrix $A \in {\Bbb R}^{m \times n}$ ($ m \ge n$) has rank $n$ and is large and sparse. We assume that $A$ is available as a matrix, not an operator. The preconditioning of this problem is difficult because the matrix $A$ does not have the properties of differential problems that make standard preconditioners effective. Incomplete Cholesky techniques applied to the normal equations do not produce a well-conditioned problem. We attempt to bypass the ill-conditioning by finding an $n \times n$ nonsingular submatrix $B$ of $A$ that reduces the Euclidean norm of $AB^{-1}$. We use $B$ to precondition a symmetric quasi-definite linear system whose condition number is then independent of the condition number of $A$ and has the same solution as the original least-squares problem. We illustrate the performance of our approach on some standard test problems and show it is competitive with other approaches.