Elliptic curve cryptography (ECC), as one of the public key cryptography systems, has been widely applied to many security applications. It is challenging to implement a scalar multiplication (SM) operation which has the highest computational complexity in ECC. In this study, we propose a hardware processor which achieves high speed and high security for ECC. We first present a three-clock cycle, divide-and-conquer multiplication algorithm which greatly reduces the number of execution cycles of multiplication. We then propose a dedicated multiplication hardware structure which reuses the multiplier and optimizes data path delay. To keep multiplication running in non-idle status and executing in parallel with other modular operations, the operation scheduling of point addition and point doubling has been re-designed and optimized based on an effective segmentation and pipeline strategy. Finally, under the premise of similar computing and hardware overhead, we propose an improved high-security SM algorithm which involves random points to resist side-channel attacks. On a 55 nm complementary metal oxide semiconductor application specific integrated circuit platform, the processor costs 463k gates and requires 0.028 ms for one SM. Our results indicate that the ECC processor is superior to other state-of-the-art designs reported in the literature in terms of speed and area-time product metrics.