Optimizing the virtual machine (VM) migration is an important issue of server consolidation in the cloud data center. By leveraging the content similarity among the memory of VMs, the time and the amount of transferred data in VM migration, as well as the pressure of network traffic, can be reduced. There are two problems in server consolidation: (1) determining which VMs should be migrated from the overloaded hosts (VM selection problem) and (2) how to place these VMs to the destination hosts (VM placement problem). By exploiting the content similarity, we redefine the above two problems into one problem to minimize the transferred memory data in VM migration. Given a fixed host overloaded threshold, an approximation algorithm is proposed to solve the problem with one overloaded host and one destination host. For the case of multiple overloaded hosts and destination hosts, two heuristic algorithms are presented with fixed and dynamic overloaded threshold respectively. We conduct a real workload trace based simulation to evaluate the performance of our algorithms. The result shows that our algorithms can produce fewer transferred VM memory data and consume less energy than existing policies.