We present a differentiable formulation of rigid-body contact dynamics for objects and robots represented as compositions of convex primitives. Classical physics engines rely on non-differentiable collision detection modules. More recent optimization-based approaches simulating contact between convex primitives rely on a bilevel formulation that separates collision detection and contact simulation. These latest approaches are unreliable in realistic contact simulation scenarios because isolating the collision detection problem introduces contact location non-uniqueness. Our approach combines contact simulation and collision detection into a unified single-level optimization problem. This disambiguates the collision detection problem in a physics-informed manner. Our formulation features improved simulation robustness and a reduction in computational complexity when compared to a similar <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">differentiable</i> simulation baseline. We illustrate the contact and collision differentiability on a robotic manipulation task requiring optimization-through-contact. We provide a numerically efficient implementation of our formulation called <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/simon-lc/Silico.jl</uri> Silico.jl.