Miniaturized multi-directional stereo vision sensor with a wide field of view (FOV) and high measurement precision is highly desirable for internal inspection in confined spaces. However, the current analysis of the design and optimization strategies for the sensor layout are not quite comprehensive. To make optimal use of the sensor and achieve high measurement accuracy, a thorough understanding of this strategy is required. In this work, we present a quad-directional stereo vision sensor based on a single camera and a dual mirror pyramid with no constraint on the placement of the devices. Geometrical analysis of the sensor is conducted by ray-tracing to examine effects of the parameters on the measurement precision, such as model accuracy, depth of field, and optical path difference. On this basis, we have investigated the method to optimize the structural parameters to maximize the overall axial and radial FOV and the utilization efficiency of the sensor. The design and optimization strategies are summarized to optimally set up the structural parameters and a miniaturized prototype of the sensor is developed and verified by both simulations and experiments.