The large abundance of small open reading frames (smORFs) in prokaryotic and eukaryotic genomes and the plethora of smORF-encoded small proteins became only apparent with the constant advancements in bioinformatic, genomic, proteomic, and biochemical tools. Small proteins are typically defined as proteins of < 50 amino acids in prokaryotes and of less than 100 amino acids in eukaryotes, and their importance for cell physiology and cellular adaptation is only beginning to emerge. In contrast to antimicrobial peptides, which are secreted by prokaryotic and eukaryotic cells for combatting pathogens and competitors, small proteins act within the producing cell mainly by stabilizing protein assemblies and by modifying the activity of larger proteins. Production of small proteins is frequently linked to stress conditions or environmental changes, and therefore, cells seem to use small proteins as intracellular modifiers for adjusting cell metabolism to different intra- and extracellular cues. However, the size of small proteins imposes a major challenge for the cellular machinery required for protein folding and intracellular trafficking and recent data indicate that small proteins can engage distinct trafficking pathways. In the current review, we describe the diversity of small proteins in prokaryotes and eukaryotes, highlight distinct and common features, and illustrate how they are handled by the protein trafficking machineries in prokaryotic and eukaryotic cells. Finally, we also discuss future topics of research on this fascinating but largely unexplored group of proteins.