Topological measurement-based quantum computation (MBQC) enables one to carry out universal fault-tolerant quantum computation via single-qubit Pauli measurements with a family of large entangled states called cluster states as resources. Raussendorf's three-dimensional cluster states (RTCSs) based on the surface codes are mainly considered for topological MBQC. In such schemes, however, the fault-tolerant implementation of the logical Hadamard, phase ($Z^{1/2}$), and $T$ ($Z^{1/4}$) gates which are essential for building up arbitrary logical gates has not been achieved to date without using state distillation, while the controlled-NOT (CNOT) gate does not require it, to best of our knowledge. State distillation generally consumes many ancillary logical qubits, thus it is a severe obstacle against practical quantum computing. To solve this problem, we suggest an MBQC scheme via a family of cluster states called color-code-based cluster states (CCCSs) based on the two-dimensional color codes instead of the surface codes. We define logical qubits, construct elementary logical gates, and describe error correction schemes. We show that all the logical Clifford gates including the CNOT, Hadamard, and phase gates can be implemented fault-tolerantly without state distillation, although the fault-tolerant $T$ gate still requires it. We further prove that the minimal number of physical qubits per logical qubit in a CCCS is at most approximately 1.8 times smaller than the case of an RTCS. We lastly show that the error threshold of MBQC via CCCSs for logical-$Z$ errors is 2.7-2.8%, which is comparable to the value for RTCSs, assuming a simple error model where physical qubits have $X$-measurement or $Z$ errors independently with the same probability.