In this work, the matrix characterization (mineralogy, total and local chemical composition, and total organic (TOC) and inorganic carbon (TIC) contents) of different types of sediments from mining- and metallurgy-influenced areas and the assessment of the impact of the matrix on the association of potentially hazardous metals with the mineral phases of these samples, which affect their mobility in the environment, are presented. For these purposes, sediment samples with different origins and from different locations in the environment were analyzed. Anthropogenic sediments from metal-rich post-flotation tailings (Lintich, Slovakia) represent waste from ore processing, natural river sediments from the Hornád River (Košice, Slovakia) represent areas influenced predominantly by the metallurgical industry, and lake sediments from a water reservoir Ružín (inflow from the Hornád and Hnilec Rivers, Slovakia) represent the impact of the metallurgical and/or mining industries. The total metal contents were determined by X-ray fluorescence (XRF) analysis, the local chemical and morphological microanalysis by scanning electron microscopy with energy-dispersive spectroscopy (SEM-EDS), and the TOC and TIC contents by infrared (IR) spectrometry. The mobility/bioavailability of Cu, Pb, and Zn in/from sediments at the studied areas was assessed by ethylenediaminetetraacetic acid (EDTA) and acetic acid (AA) extraction and is discussed in the context of the matrix composition. The contents of selected potentially hazardous elements in the extracts were determined by the high-resolution continuum source flame atomic absorption spectrometry (HR-CS FAAS).