In this paper, the time-varying group formation control for linear multi-agent systems under directed communication topology is investigated from an observer viewpoint. Different from the existing works on the time-varying group formation, the groups herein could have a cyclic partition, which is more common in real applications than the topology with the acyclic groups. The leaderless time-varying group formation problem is studied first. An observer-based distributed protocol is presented for each agent, where the observer is used to estimate the unmeasurable state utilizing the output information. Then, to broaden the scope of applications, we further study the leader-following case, in which there exists a leader with nonzero and bounded input for each subgroup. To tackle this problem, we take the input of the leader as a disturbance, and develop the new forms of control protocols with nonlinear functions. Furthermore, for both cases, it is shown that under the formation feasible conditions, the desired time-varying group formation can be achieved if the strong enough intra-group coupling is selected and the corresponding digraph of each subgraph contains a directed spanning tree. Finally, two simulation examples are given.