For non-ellipsoidal extended targets and group targets tracking, using an ellipsoid to approximate the target extension may not be accurate enough because of lacking shape and orientation information. In consideration of this, we model a non-ellipsoidal extended target or target group as a combination of multiple ellipsoidal sub-objects, each represented by a random matrix. Besides, in practical tracking scenes, the knowledge of detection probability is of critical importance, which, however, is prior unknown due to the fluctuation of target parameters. Thus based on the aforementioned random sub-matrices model, an improved generalized labeled multi-Bernoulli (GLMB) filter is presented for the tracking of non-ellipsoidal extended targets or group targets in the case of unknown detection probability. In addition, a cheaper Gibbs sampling based joint prediction and updating implementation of the proposed GLMB filter is given, within which the extended targets or group targets are modeled using Beta gamma Gaussian inverse Wishart (BGGIW) distributions, achieving the estimation to detection probability, measurement rate, and extension state of each sub-object and the kinematic state with the positions of all sub-objects and unified kinematics for each extended target or target group. The simulation results demonstrate the effectiveness of the proposed filter.