Developing a general framework with a novel stochastic offset strategy for the design of optimized RF pulses and time-varying spatially non-linear ΔB0 shim array fields for restricted slice excitation and refocusing with refined magnetization profiles within the intervals of the fixed voxels. Our framework uses the decomposition property of the Bloch equations to enable joint design of RF-pulses and shim array fields for restricted slice excitation and refocusing with auto-differentiation optimization. Bloch simulations are performed independently on orthogonal basis vectors, Mx, My, and Mz, which enables designs for arbitrary initial magnetizations. Requirements for refocusing pulse designs are derived from the extended phase graph formalism obviating time-consuming sub-voxel isochromatic simulations to model the effects of crusher gradients. To refine resultant slice-profiles because of voxelwise optimization functions, we propose an algorithm that stochastically offsets spatial points at which loss is computed during optimization. We first applied our proposed design framework to standard slice-selective excitation and refocusing pulses in the absence of non-linear ΔB0 shim array fields and compared them against pulses designed with Shinnar-Le Roux algorithm. Next, we demonstrated our technique in a simulated setup of fetal brain imaging in pregnancy for restricted-slice excitation and refocusing of the fetal brain. Our proposed framework for optimizing RF pulse and time-varying spatially non-linear ΔB0 shim array fields achieve high fidelity restricted-slice excitation and refocusing for fetal MRI, which could enable zoomed fast-spin-echo-MRI and other applications.