Rocks and minerals buried in the earth’s surface usually undergo weathering processes and change color in the burying environment. A kind of yellow Chinese stamp stone named “Lumu stone”, which is buried in a yellowish weathering crust (yellowish soil), was selected to investigate its color changes in the weathering processes. In this study, the appearance features, mineral components, micromorphology, spectroscopy characteristics, and color causation of the “Lumu stone” were studied by using X-ray powder diffraction (XRD), scanning electron microscopy (SEM), an electron probe microanalyzer (EPMA), a laser ablation inductively coupled plasma mass spectrometer (LA-ICP-MS), and a UV-Visible (UV-Vis) spectrum. The “Lumu stone” usually exhibits a darker yellow outer layer and a lighter yellow core, suggesting that yellow color permeated into the stone from the surface to the core gradually and the color is secondary forming. The results from XRD and SEM show the studied samples are mainly composed of dickite and illite. The individual particles of the dickite and illite are about 2–5 μm, randomly distributing in the three-dimensional space and constituting voids among the particles. The acid pickling experiments using HCl coupled with KSCN confirmed that the mineral phases that caused the yellow color of the matrix are iron oxide and hydroxide. On the other hand, goethite and hematite were observed gathering in the yellow and brown-red cracks on the “Lumu stone” by SEM study. However, iron oxide and hydroxide in the matrix were difficult to observe and detect among the dickite and illite aggregates by SEM and XRD methods. It indicates that they may be nanoscale in size and very low in content. According to the calculation of the second derivative of Kubelka-Munk (K-M) transformed diffuse reflection spectroscopy (DRS) curves obtained from UV-Vis, the characteristic peaks of goethite and hematite were found in the yellow matrix, and their contributions to the color were confirmed. The concentrations of goethite and hematite were calculated to be 0.32 to 1.87 g/kg and 0.22 to 0.93 g/kg in the studied samples, respectively. In this study, a series of methods were employed to detect very low levels of goethite and hematite in the samples undergoing weathering processes. Additionally, nanoscale goethite and hematite were considered newly formed minerals when buried in the weathering processes and may gradually move into the voids among phyllosilicate particles. Therefore, they turned the “Lumu stone” yellow.