Deep learning has recently become a crucial tool to solve many complex problems and has the potential to revolutionize industries. With the widespread adoption of the Internet of Things, there are now many devices with limited computational resources that are capable of running deep learning models. It has opened up new opportunities to implement deep learning in the edge environment so that decisions can be made locally without sending data to a cloud server for processing. However, because of limited resources, the model’s performance and communication overhead are challenges when deploying learning models to edge devices. The focus of this paper is to investigate and propose a federated recognition architecture for object classification in a distributed edge intelligence environment. Specifically, we build an edge server that includes a voting module and a feedback module to improve the overall accuracy of object classification. The voting module aggregates predictions of multiple edge devices whereas the feedback module sends the voting results to edge devices to adjust the local deep learning model. We build edge devices based on the EdgeX platform which makes it easy to manage data and optimize communication overheads. Because the edge server and edge nodes only exchange prediction results, our proposed architecture ensures security with sensitive data as well as deep learning model architecture. By testing on the image dataset, we evaluate the proposed architecture’s performance and show that it outperforms individual local models in terms of accuracy. Furthermore, our experiments demonstrate that with the feedback mechanism, the deep learning model is constantly updated with new data to maintain accuracy and avoid being outdated. Besides, we prove the real-time processing speed by collecting the delay time of the proposed model. The results show that our proposed architecture has the potential to be deployed in practical applications such as smart cities, and surveillance systems.