In the field of soft matter research, the characteristic behavior of both nematic-isotropic (N-I) and smectic-A nematic(Sm-A N) phase transitions has gained considerable attention due to their several attractive features. In this work, a high-resolution measurement of optical birefringence (Δn) has been performed to probe the critical behavior at the N-I and Sm-A N phase transitions in a binary system comprising the rodlike octylcyanobiphenyl and a laterally methyl substituted hockey-stick-shaped mesogen, 4-(3-n-decyloxy-2-methyl-phenyliminomethyl)phenyl 4-n-dodecyloxycinnamate. For the investigated mixtures, the critical exponent β related to the limiting behavior of the nematic order parameter close to the N-I phase transition has come out to be in good conformity with the tricritical hypothesis. Moreover, the yielded effective critical exponents (α', β', γ') characterizing the critical fluctuation near the Sm-A N phase transition have appeared to be nonuniversal in nature. With increasing hockey-stick-shaped dopant concentration, the Sm-A N phase transition demonstrates a strong tendency to be driven towards a first-order nature. Such a behavior has been accounted for by considering a modification of the effective intermolecular interactions and hence the related coupling between the nematic and smectic order parameters, caused by the introduction of the angular mesogenic molecules.