Connected and automated vehicle (CAV) technology makes it possible to track and control the movement of vehicles, thus providing enormous potential to improve intersection operations. In this paper, we study the traffic signal control problem at an isolated intersection in a CAV environment, considering mixed traffic including various types of vehicles and pedestrians. Both the vehicle delay and the pedestrian delay are incorporated into the model formulation. This introduces some additional complexity, as any benefits to pedestrians will come at the expense of higher delays for the vehicles. Thus, some valid questions we answer in this paper are as follows: Under which circumstances could we provide priority to pedestrians without over penalizing the vehicles at the intersection? How important are the connectivity and autonomy associated with CAV technology in this context? What type of signal control algorithm could be used to minimize person delay accounting for both vehicles and pedestrians? How could it be solved efficiently? To address these questions, we present a model that optimizes signal control (i.e., vehicle departure sequence), automated vehicle trajectories, and the treatment of pedestrian crossing. In each decision step, the weighted sum of the vehicle delay and the pedestrian delay (e.g., the total person delay) is minimized by the joint optimization on the basis of the predicted departure sequences of vehicles and pedestrians. Moreover, a near-optimal solution of the integrated problem is obtained with an ant colony system algorithm, which is computationally very efficient. Simulations are conducted for different demand scenarios and different CAV penetration rates. The performance of the proposed algorithm in terms of the average person delay is investigated. The simulation results show that the proposed algorithm has potential to reduce the delay compared to an actuated signal control method. Moreover, in comparison to a CAV-based signal control that does not account for the pedestrian delay, the joint optimization proposed here can achieve improvement in the low- and moderate-vehicle-demand scenarios.