Grapheme-color synesthesia is a neurological phenomenon in which viewing a grapheme elicits an additional, automatic, and consistent sensation of color. Color-to-letter associations in synesthesia are interesting in their own right, but also offer an opportunity to examine relationships between visual, acoustic, and semantic aspects of language. Research using large populations of synesthetes has indeed found that grapheme-color pairings can be influenced by numerous properties of graphemes, but the contributions made by each of these explanatory factors are often confounded in a monolingual dataset (i.e., only English-speaking synesthetes). Here, we report the first demonstration of how a multilingual dataset can reveal potentially-universal influences on synesthetic associations, and disentangle previously-confounded hypotheses about the relationship between properties of synesthetic color and properties of the grapheme that induces it. Numerous studies have reported that for English-speaking synesthetes, “A” tends to be colored red more often than predicted by chance, and several explanatory factors have been proposed that could explain this association. Using a five-language dataset (native English, Dutch, Spanish, Japanese, and Korean speakers), we compare the predictions made by each explanatory factor, and show that only an ordinal explanation makes consistent predictions across all five languages, suggesting that the English “A” is red because the first grapheme of a synesthete's alphabet or syllabary tends to be associated with red. We propose that the relationship between the first grapheme and the color red is an association between an unusually-distinct ordinal position (“first”) and an unusually-distinct color (red). We test the predictions made by this theory, and demonstrate that the first grapheme is unusually distinct (has a color that is distant in color space from the other letters' colors). Our results demonstrate the importance of considering cross-linguistic similarities and differences in synesthesia, and suggest that some influences on grapheme-color associations in synesthesia might be universal.