Double-cladding Yb3+-doped fiber with octagonal core is designed and fabricated using solution doping techniques. According to the simulated results, the octagonal-core fiber with core/cladding diameter of 30/250 μm has decreased the number of mode supported from 6 to 4, and the fiber with core/cladding diameter of 10/130 μm exhibits single-mode performance. Additionally, to confirm that the octagonal core does not influence the laser power and efficiency, CW and AO Q-switched pulsed fiber lasers using the 10/130 μm fiber are presented. The CW fiber laser has a single-mode output of 7.25 W with a slope efficiency of 80 %. For a pulsed fiber laser, an average power of 1.5 W, pulse width of 245 ns, is amplified to 18 W.