We report a passively Q-switched and mode-locked erbium-doped fiber laser (EDFL) based on PtSe2, a new two-dimensional material, as a saturable absorber (SA). Self-started Q-switching at 1560 nm in the EDFL was achieved at a threshold pump power of 65 mW, and at the maximum pump power of 450 mW, the maximum single Q-switched pulse energy is 143.2 nJ. Due to the polarization-dependent characteristics of the PtSe2-based SA, the laser can be switched from the Q-switched state to the mode-locked state by adjusting the polarization state. A mode-locked pulse train with a repetition rate of 23.3 MHz and a pulse width of 1.02 ps can be generated when the pump power increases to about 80 mW, and the stable mode-locked state is maintained until the pump power reaches its maximum 450 mW. The maximum single mode-locked pulse energy is 0.53 nJ. This is the first time to our knowledge that successful generation of stable Q-switched and mode-locked pulses in an Er-doped fiber laser has been obtained by using PtSe2 as a saturable absorber.