The Virtual Work Optimization Method (VWOM) is used to optimize tall buildings with semi rigid connections. The VWOM is an automated method that minimizes the mass of a structure with a given geometry, deflection criteria and load cases, while adhering to building code requirements. Members are selected from a user defined database to meet strength and stiffness criteria. The rotational stiffness of connections can be specified and varied, from rigid to pinned. The method is explained using an example of a two storey one bay steel frame with six members. Four case studies are presented: (i) a six storey two bay frame, (ii) a ten storey one bay steel building, (iii) a fifteen storey three bay steel frame and (iv) a twenty four storey three bay building. The examples are compared to results found in the literature. Structures were optimized for a range of connection rotational stiffnesses. The VWOM obtained results up to 44.6% lighter than published work. Optimized results showed that in most cases the semi rigid connection produced the same results as the rigid connections. This is due to the rapid plateauing of the optimized mass as the connection stiffness increases. Thus, the typical definition of semi rigid and rigid rotational stiffnesses (for example Eurocode 3) are both found on this plateau. If some members are lightly loaded, broad semi rigid and rigid bands can be produced. The effect in these cases is that structures with rigid connections are lighter than those with semi rigid connections as no compensation in the member stiffness is required to meet deflection criteria.