This paper presents the results of a stepwise investigation of the quantity and quality of food losses along the Swiss potato supply chain. Quantitative data were collected from field trials, from structured interviews with wholesalers, processors and retailers, and from consumer surveys in combination with a 30-day diary study. The “Swiss trade customs for potatoes” pose the basis for the qualitative evaluation of losses. The influences of technological, institutional (business and economy; legislation and policy), and social drivers on the generation of fresh potato and processed potato products losses were assessed. Losses due to quality standards driven by food safety and consumer preferences for certain aesthetic standards have been evaluated too. Across the entire potato value chain, approximately 53–55% of the initial fresh potato production and 41–46% of the initial processing potato production are finally lost. Losses between organic and non-organic supply chains differ from 2% to 5%. From the total initial fresh potato production, 15–24% gets lost during agricultural production, a further 12–24% at wholesalers, 1–3% at retailers, and 15% at private households. In comparison, 5–11% of the initial production gets lost at wholesalers, a further 14–15% during processing, 0% at retailers, and 2% at private households. Losses during agricultural production do not vary much (13–25%) between fresh and processing potatoes. Approximately half of total potato losses occur because potatoes do not meet quality standards. 25–34% of these quality-driven losses are caused by food safety reasons, and the remainder are caused by consumer preferences or suitability for storage. In total, social drivers (e.g., consumer preferences, behavior, or socio-demographical factors) are responsible for two-thirds to three-fourths of all fresh potato losses and 40–45% of all processing potato losses. Technological drivers cause circa one-third of the total processing potato losses. The majority of the rejected potatoes are used as animal feed (67–90%) in Switzerland. Approximately 30% of all fresh potato losses are disposed while just 4–5% of all processing potato losses are thrown away. The interviewed experts stated that lower quality specifications might cause lower loss rates at the first stages of the supply chain but higher ones at the later stages due to worse storage or processing abilities of potatoes with defects, which also might affect proper tubers.