Concrete-encased steel composite columns are preferred for their exceptional ductility and strength, particularly in high-rise buildings. This research aims to enhance both the strength and ductility of these composite columns by increasing the height of the steel profile. Typically, hexagonal or circular openings, referred to as castellated elements, are incorporated into the steel profile to achieve this height increase. This study employed a topology optimization method to identify the ideal opening shape for the steel profile in concrete-encased steel composite columns. The analysis revealed a sinusoidal-like opening shape, which was then refined for manufacturing. The optimal opening shape was used to increase the height of the existing steel profile, and nonlinear analyses were conducted to evaluate the effectiveness of this new optimized steel profile in concrete-encased steel composite columns. Two concrete-encased steel composite columns were designed: one with the optimal steel profile and the other with a standard steel I profile. ANSYS APDL 19.0 software was used to simulate an experiment based on an existing concrete-encased steel column to validate the nonlinear analysis. The verification analysis demonstrated a remarkable similarity between the experimental and numerical load–displacement graphs, indicating that the numerical analysis was reliable. In the analysis of the composite columns, both axial and lateral forces were applied in the nonlinear analyses. The axial force was applied at 15% of the column’s capacity, while the lateral force was applied until the composite column reached a state of failure. The results of the nonlinear analyses allowed for a comparison of load–displacement curves and the performance of the composite columns. In comparison to the standard steel I profile, the steel profile with the optimal opening shape increased load-carrying capacity by approximately 19% and energy absorption capacity by approximately 24%.