With the development of serverless computing, developers can implement and deploy business applications as a combination of stateless functions. Although originally proposed for cloud computing, serverless computing is gradually applied to cloud-edge systems for service deployment to provide users with high-quality, low-latency services. However, optimized service deployment in 6G networks is a very challenging issue because of the vast number of deployable devices in the network, and its permutations are highly exponential. In this paper, we propose optimized service deployment schemes for online and offline, respectively, to minimize the overall latency at a lower cost. (1) First, a SD-SRF algorithm based on the greedy algorithm is proposed to optimize the service deployment for a multi-layer edge network, which consists of two phases: SR and SF. (a) Services are deployed in the nearest ancestor devices in the routing tree of all such service requests with the least cost of deployment. (b) When the deployment cost of moving some service replicas to devices in the lower layer is less than the benefit, the service will fall. (2) However, the offline algorithm relies on the availability of prior information heavily, such as request arrival pattern and number, which is difficult to obtain. Therefore, this paper proposes a PPO-MSD algorithm for optimal deployment online, where a Markov decision process (MDP) is modeled. Extensive simulation results show that PPO-MSD outperforms existing algorithms in terms of overall delay and utility, and its performance is close to the optimal ones obtained by SD-SRF, with the SD-SRF and PPO-MSD algorithms reducing the delay on average by 32.40% and 9.91%.