Swarms of autonomous agents are useful in many applications due to their ability to accomplish tasks in a decentralized manner, making them more robust to failures. Due to the difficulty in running experiments with large numbers of hardware agents, researchers often make simplifying assumptions and remove constraints that might be present in a real swarm deployment. While simplifying away some constraints is tolerable, we feel that two in particular have been overlooked: one, that agents in a swarm take up physical space, and two, that agents might be damaged in collisions. Many existing works assume agents have negligible size or pass through each other with no added penalty. It seems possible to ignore these constraints using collision avoidance, but we show using an illustrative example that this is easier said than done. In particular, we show that collision avoidance can interfere with the intended swarming behavior and significant parameter tuning is necessary to ensure the behavior emerges as best as possible while collisions are avoided. We compare four different collision avoidance algorithms, two of which we consider to be the best decentralized collision avoidance algorithms available. Despite putting significant effort into tuning each algorithm to perform at its best, we believe our results show that further research is necessary to develop swarming behaviors that can achieve their goal while avoiding collisions with agents of non-negligible volume.