PurposeThis paper aims to apply the augmented reality (AR) technology to assembly design in the early design stage. A proof‐of‐concept system with AR interface is developed.Design/methodology/approachThrough AR interface, designers can design the assembly on the real assembly platform. The system helps users to design the assembly features to provide proper part‐part constraints in the early design stage. The virtual assembly features are rendered on the real assembly platform using AR registration techniques. The new evaluated assembly parts can be generated in the AR interface and assembled to assembly platform through assembly features. The model‐based collision detection technique is implemented for assembly constraint evaluation.FindingsWith AR interface, it would be possible to combine some of the benefits of both physical and virtual prototyping (VP). The AR environment can save a lot of computation resource compared to a totally virtual environment. Working on real assembly platform, designers have more realistic feel and the ability to design an assembly in a more intuitive way.Research limitations/implicationsMore interaction tools need to be developed to support the complex assembly design efficiently.Practical implicationsThe presented system encourages designers to consider the assembly issues in the early design stage. The primitive 3D models of assembly parts with proper part‐part constraints are generated using the system before doing detailed geometry design.Originality/valueA new markerless registration approach for AR system is presented. This generic approach can be also used for other AR applications.