Purpose The purpose of this paper is to introduce an artificial bee colony-based Kalman filter algorithm along with an extended objective function to ensure the optimality of the estimator of the quadrotor in the presence of unknown measurement noise statistics. Design/methodology/approach Six degree-of-freedom mathematical model of the quadrotor is derived. Position controller for the quadrotor is designed. Kalman filter-based estimation algorithm is implemented in the sensor feedback loop. Artificial bee colony-based hybrid algorithm is used as an optimization method to handle the unknown noise statistics. Existing objective function is extended with a penalty term. Mathematical proof of the extended objective function is derived. Results of the proposed algorithm is compared with de facto genetic algorithm-based Kalman filter. Findings Artificial bee colony algorithm-based Kalman filter and extended objective function duo are able to optimize the measurement noise covariance matrix with an absolute error as low as 0.001 [m2]. Proposed method and function is capable of reducing the noise from 2 to 0.09 [m] for x-axis, 3.4 to 0.14 [m] for y-axis and 3.7 to 0.2 [m] for z-axis, respectively. Originality/value The motivation behind this paper is to bring a novel optimization-based solution for the estimation problem of the quadrotor when the measurement noise statistics are unknown along with an extended objective function to prevent the infeasible solutions with mathematical convergence analysis.