Despite ongoing concerns about trace metal and metalloid (trace metals) exposure risks from indoor dust, there has been limited research examining their sources and relationship to outdoor soils. Here we determine the concentrations and sources for potentially toxic trace metals arsenic (As), chromium (Cr), copper (Cu), manganese (Mn), lead (Pb) and zinc (Zn) and their pathways into homes in Sydney, Australia, using home-matched indoor dust (n = 166), garden soil (n = 166), and road dust samples (n = 51). All trace metals were more elevated indoors versus their matched garden soil counterparts. Indoor Cu and Zn dust concentrations were significantly more enriched than outdoor dusts and soils, indicating indoor sources were more relevant for these elements. By contrast, even though Pb was elevated in indoor dust, garden soil concentrations were correspondingly high, indicating that it remains an important source and pathway for indoor contamination. Elevated concentrations of As, Pb and Zn in garden soil and indoor dust were associated with home age (>50 years), construction materials, recent renovations and deteriorating interior paint. Significant correlations (p < 0.05) between road dust and garden soil Cu concentrations, and those of As and Zn in soil and indoor dust, and Pb across all three media suggest common sources. Scanning electron microscopy (SEM) analysis of indoor dust samples (n = 6) showed that 57% of particles were derived from outdoor sources. Lead isotopic compositions of soil (n = 21) and indoor dust (n = 21) were moderately correlated, confirming the relevance of outdoor contaminants to indoor environments. This study illustrates the source, relationship and fate of trace metals between outdoor and indoor environments. The findings provide insight into understanding and responding to potentially toxic trace metal exposures in the home environment.