Pursuing a bottom-up approach to explore which flavor symmetry could serve as an explanation of the observed fermion masses and mixings, we discuss an extension of the standard model (SM) where the flavor structure for both quarks and leptons is determined by a spontaneously broken S4 and the requirement that its particle content is embeddable simultaneously into the conventional SO(10) grand unified theory (GUT) and a continuous flavor symmetry G_f like SO(3)_f or SU(3)_f. We explicitly provide the Yukawa and the Higgs sector of the model and show its viability in two numerical examples which arise as small deviations from rank one matrices. In the first case, the corresponding mass matrix is democratic and in the second one only its 2-3 block is non-vanishing. We demonstrate that the Higgs potential allows for the appropriate vacuum expectation value (VEV) configurations in both cases, if CP is conserved. For the first case, the chosen Yukawa couplings can be made natural by invoking an auxiliary Z2 symmetry. The numerical study we perform shows that the best-fit values for the lepton mixing angles theta_12 and theta_23 can be accommodated for normal neutrino mass hierarchy. The results for the quark mixing angles turn out to be too small. Furthermore the CP-violating phase delta can only be reproduced correctly in one of the examples. The small mixing angle values are likely to be brought into the experimentally allowed ranges by including radiative corrections. Interestingly, due to the S4 symmetry the mass matrix of the right-handed neutrinos is proportional to the unit matrix.