In a virtualized software defined network (VSDN), PACKET_IN messages of switches must pass through the hypervisor in order to reach the corresponding controller. Hence, the latency experienced by a network element is the sum of latency from network element to hypervisor and the latency from the hypervisor to the controller corresponding to the network element. Therefore, the locations of both the hypervisors and controllers determine the latency of network elements in a virtualized environment. In this paper, we propose a strategy for determining the placement of controllers in a VSDN while fixing the hypervisor(s) in the physical network. We also propose an approach for jointly optimizing the placement of hypervisors and controllers in a VSDN. The objective is to minimize the worst case latency between the network element and its corresponding controller. Furthermore, we propose a generalized model which can be used not only to optimize the worst case latency, but also to optimize other objectives such as the average latency, the maximum average latency, and the average maximum latency. The proposed problems are formulated as integer linear programs. We evaluated the performance of our proposed strategies using the AT&T network of Internet Topology Zoo and the Internet 2 OS3E topology, and compared with the hypervisor placement problem. Evaluations demonstrate that the proposed methods outperform the existing hypervisor placement approach with respect to the various performance metrics.