Compaction processes within mudstone samples across varying depths (723.5–3213.5 m) in the Vienna Basin were investigated, focusing on porosity changes and the resulting micromechanical response in the clay-rich, fine-grained fraction (“clay matrix”). A novel approach combining nanoindentation mapping with machine learning data analysis was developed to efficiently extract representative micromechanical parameters of the clay matrix. Emphasis was put on capturing the properties of the fine-grained, clay-dominated composite rather than individual mineral phases, enabling the analysis of compaction-induced strength changes at microscale. A significant enhancement in the mechanical strength of the clay matrix with increasing burial depth was observed. Reduced elastic modulus (Er) and hardness (H) increased from 6.8 ± 3.4 to 22.6 ± 7.5 GPa and from 0.2 ± 0.2 to 0.9 ± 0.2 GPa, respectively, over the depth interval of 2490 m. Moreover, a strong correlation between depth and porosity and consequently micromechanical properties of the clay matrix exists. This highlights the substantial influence of intra-clay porosity on mechanical properties, which follow a general compaction trend with depth. Broad ion beam-scanning electron microscopy (BIB-SEM) analysis was used for textural investigations at micro-to nanoscale and indicated that the reduction in porosity predominantly resulted from mechanical compaction rather than mineral diagenesis, which would have been noticeable by signs of significant dissolution or cementation. The correlation coefficient matrix between multiscale porosity measurements and nanoindentation affirmed that the porosity loss was closely linked to the enhanced mechanical properties of the fine-grained composite clay matrix, while other compositional variables like total clay mineral content showed weak correlations. Empirical mathematical equations were derived to describe the mechanical properties as generalized functions of depth and porosity. These may be used as geomechanical input for future basin analysis and geoenergy applications (e.g., seal rock studies in a geological storage context). This study introduces a new approach to unravel compaction processes in fine-grained rocks at microscale, emphasizing the interplay between intra-clay porosity and micromechanical changes during proceeding burial.