Radio frequency identification (RFID) is an emerging wireless communication technology which allows objects to be identified automatically. An RFID system consists of a set of readers and several objects, equipped with small and inexpensive computer chips, called tags. In a dense RFID system, where several readers are placed together to improve the read rate and correctness, readers and tags can frequently experience packet collision. High probability of collision impairs the benefit of multiple reader deployment and results in misreading. A common approach to avoid collision is to use a distinct frequency channel for interrogation for each reader. Various multi-channel anti-collision protocols have been proposed for RFID readers. However, due to their heuristic nature, most algorithms may not achieve optimal system performance. In this paper, we systematically design two optimization-based distributed channel selection and randomized interrogation algorithms for dense RFID systems. For this purpose, we develop elaborate models for the reader-to-tag and reader-to-reader collision problems. The first algorithm is fully distributed and is guaranteed to find a local optimum of a max-min fair resource allocation problem for RFID systems. The second algorithm is semi-distributed and achieves the global optimal system performance. Max-min fair optimality balances the performance and the processing load among readers. Simulation results show that our algorithms have significantly better performance than the previous heuristic algorithms.