In the present study, we propose and analyze an epidemic mathematical model for malaria dynamics, considering multiple recurrent phenomena: relapse, reinfection, and recrudescence. A limitation in hospital bed capacity, which can affect the treatment rate, is modeled using a saturated treatment function. The qualitative behavior of the model, covering the existence and stability criteria of the endemic equilibrium, is investigated rigorously. The concept of the basic reproduction number of the proposed model is obtained using the concept of the next-generation matrix. We find that the malaria-free equilibrium point is locally asymptotically stable if the basic reproduction number is less than one and unstable if it is larger than one. Our observation on the malaria-endemic equilibrium of the proposed model shows possible multiple endemic equilibria when the basic reproduction number is larger or smaller than one. Hence, we conclude that a condition of a basic reproduction number less than one is not sufficient to guarantee the extinction of malaria from the population. To test our model in a real-life situation, we fit our model parameters using the monthly incidence data from districts in Central Sumba, Indonesia called Wee Luri, which were collected from the Wee Luri Health Center. Using the first twenty months’ data from Wee Luri district, we show that our model can fit the data with a confidence interval of 95%. Both analytical and numerical experiments show that a limitation in hospital bed capacity and reinfection can trigger a more substantial possibility of the appearance of backward bifurcation. On the other hand, we find that an increase in relapse can reduce the chance of the appearance of backward bifurcation. A non-trivial result appears in that a higher probability of recrudescence (treatment failure) does not always result in the appearance of backward bifurcation. From the global sensitivity analysis using a combination of Latin hypercube sampling and partial rank correlation coefficient, we found that the initial infection rate in humans and the mosquito infection rate are the most influential parameters in determining the increase in total new human infections. We expand our model as an optimal control problem by including three types of malaria interventions, namely the use of bed net, hospitalization, and fumigation as a time-dependent variable. Using the Pontryagin maximum principle, we characterize our optimal control problem. Results from our cost-effectiveness analysis suggest that hospitalization only is the most cost-effective strategy required to control malaria disease.