In this study, we investigate the profound impacts of the sex determination pattern in the invasive sea lamprey on ecological dynamics and focuses on how the pattern influences the population based upon the interplay of system stability. In the initial segment, we employ the Lotka-Volterra model and system dynamics to study lamprey sex ratios correlation with ecosystem stability. Focusing on food impact on lamprey sex ratio, it can be delved that its sex determination maintains the prey population at a consistently low level, thereby affecting population stability. The following segment explores lamprey sex ratios evaluations utilizing system dynamics model based on Analytic Hierarchy Process (AHP). Cellular Automata (CA) is employed for cross-validations, revealing nuanced insights into the adaptive advantages and vulnerabilities of lampreys reproductive strategy, highlighting the resilience of lamprey populations under natural pressures. Our modeling, with visualizations and simulations, supports findings and highlights avenues for future research. This study contributes to the evaluations of bio sex ratio switching, contributing for the possible ecosystem conservation strategies.