Knowledge graph (KG) is a kind of structured human knowledge of modeling the relationships between real-world entities. High quality KG is of crucial importance for many knowledge-based applications, e.g., question answering, recommender systems, etc. This paper studies the problem of entity alignment in KGs to promote knowledge fusion. Existing methods model the semantic representation of entities by using graph structural information or attribute information of the KG and then align the entities across different domains by calculating the distances between entities’ embeddings. However, these methods only consider the node-to-node similarity in the alignment procedure while the edge-to-edge similarity is ignored. Our research hypothesis is that the graph edge alignment information is critical in entity alignment. We reformulate the knowledge entity alignment as a quadratic assignment problem (QAP) by adding relation alignment under the one-to-one mapping constraint. To solve the notorious QAP in a large-scale heterogeneous graph like KG, we propose a model, dual neighborhood consensus network (DNCN), which approximately decomposes the QAP into two small-scale linear assignment problems, i.e., entity alignment and relation alignment. After that, an edge-coloring propagation method is proposed to refine the coarse entity alignment result using the relation correspondence. Theoretical proof shows that this method can guarantee the isomorphism between local sub-graphs. The performance of DNCN is evaluated using the DBP15K and DWY100K benchmarks. Experimental results show that DNCN achieves the best performance on the DBP15K benchmark, and is computationally efficient. Ablation studies verify the importance of graph edge alignment information.