Coffee is Uganda’s biggest export commodity, produced mainly by an estimated one million smallholder farmers (<2.5ha). Arabica (Coffea arabica L.) and Robusta (Coffea canephora Pierre ex Froehn.) are the two coffee species grown. Robusta is dominantly cultivated at lower elevations (<1400m) such as in Central and Northern Uganda and Arabica is dominant at higher elevations (>1400m) such as Eastern, Southwest, and Northwest Uganda. Actual yields are far below (<30%) potential due to various biotic, abiotic, and management constraints, yet there is no quantitative information on site-specific production constraints and the yield gaps attributed to those constraints. In this study, yields and diverse production factors were monitored in 254 plots of five major coffee growing regions (i.e., Central, North, East, Southwest, and Northwest). Boundary line analysis was applied to evaluate the relative importance of the individual production factors in limiting coffee production and to quantify the associated yield gaps at regional level. The impacts of rainfall variation on coffee yield were evaluated separately by regression analysis. The results of boundary line analysis indicated that biotic constraints (coffee twig borer) and poor management practices (unproductive coffee trees and low coffee plant density) restricted Robusta production in the Central region; poor soil nutrient status (especially potassium) and lack of mulching were the causes of yield loss of Robusta grown in the Northern region. For Arabica, unfavorable soil properties (high soil pH and phosphorus concentration) and excessive number of shade trees were the most important constraints in the East; high soil magnesium concentration and poor mulching limited coffee yield in the Southwest; poor soil nutrient status (especially phosphorus and potassium) and low coffee plant density were the important yield limitations in the Northwest. Average explained yield gaps of individual coffee plot due to the most important production constraints were 45%, 52%, 57%, 49%, and 50% of attainable yield, respectively, in the Central, Northern, Eastern, Southwest, and Northwest regions. Considerably less annual precipitation was received in 2009/2010 coffee growing season compared with that in the previous three years (2006–2008). Seasonal rainfall shortage occurred in the Southwest was a significant limitation to Arabica production, while excessive rainfall across the whole growing season was associated with yield reduction in the Eastern and Northwest regions. We conclude that there was a large yield gap for both Robusta and Arabica coffee grown in Uganda. Boundary line analysis allows the evaluation of relative importance of individual production constraint directly in the plot. The important production constraints varied strongly depending on the regions, which calls for site-specific management implementations. Soil fertility can be improved by implementing integrated soil fertility management (ISFM) that makes use of nutrients from the soil, recycled crop residues, mulch and chemical fertilizers. Attentions should also be given to other management practices such as coffee plant density, unproductive coffee trees and shade trees etc.