• We introduce a simple aspect-representation for 3D objects for 6DoF pose estimation. • This representation is used to cluster appearances of an object as aspects. • Proposed model predicts both the aspect of a given object and the semantic-key points. • Multi-tasking to high confidence key-point vectors hence more accurate results. • Proposed method is significantly faster due to small backbone and limited PnP Space. We present a novel approach to the problem of estimating a given object's 6DoF pose from a single RGB image. Recent works focus on a multi-stage approach, which first detects key-points followed by perspective -n-point pose estimation algorithm and a pose refinement procedure. We show that adding a classifier block estimating the predefined aspects of the objects improves the multi-stage process. This is due to the fact that the additional classifier acts as a constraint simplifying the required neural network and at the same time yielding better key-point selection. We reduce the search space for the key-point selection and exclude false-positives by mapping the appearance of an object to an aspect. The simplified neural network allows faster inference and a smaller footprint. Our experiments show that our hypothesis performs similar to the state-of-the-art on three different datasets. We also show that an off-the-shelf refinement process can further improve our results to surpass state-of-the-art on several objects. Another advantage is, the proposed pipeline can run efficiently on real-time due to the smaller neural network backbone used. The code to replicate this research will be publicly available at https://github.com/greymad/6DoFPoseAspects