Purpose: A method for 4D (3D+time) segmentation of bone and cartilage surfaces of the knee joint imaged by magnetic resonance (MR) is reported. Accurate segmentation is necessary to understand osteoarthritis (OA) disease processes and quantify the efficacy of DMOAD (disease-modifying OA drug) drug trials. Manual segmentation is tedious, time-consuming (several hours per joint) and irreproducible. We have previously reported an automated 3D approach based on layered optimal graph segmentation of multiple objects and surfaces (LOGISMOS). This study extends 3D LOGISMOS to 3D+time (4D) LOGISMOS to simultaneously segment four cartilage and bone surfaces (femur, tibia) in longitudinal sequences of follow-up MRI's. The key benefit of using our automated 4D LOGISMOS method is that information from all time points of the temporal sequence contributes to the single optimal solution that utilizes temporal and spatial context between adjacent time points. Methods: Our 4D LOGISMOS approach consists of two major steps: a) Constraining topological relationships between identical pre-segmentation meshes between time-points using rigid registration, and b) enforcing physiologically permissible maximum changes between time-points by introducing inter-time point graph arcs that longitudinally link the temporally corresponding locations of the respective bones and cartilages of the femur and tibia. Robust hierarchical random-forest classifiers were used to derive cartilage cost functions trained using segmentation examples. Once segmented, automated sub-plate analysis was used to identify segmentation accuracy in medial/lateral regions of the femur (cMF/cLF) and tibia (cMT/cLT). To assess segmentation accuracy, 54 patients at baseline and 12-month follow-up visits from the OAI (Osteoarthritis Initiative) progressive cohort were analyzed (108 3D MR DESS datasets) and compared with the manually-traced independent standard (IS). To assess the temporal consistency of the segmentations, a large sub-cohort of 399 patients with five yearly follow-ups (baseline, 12, 24, 36, 48 month; 1,995 DESS MRI’s) were analyzed to show the benefits of 4D over 3D analysis. Since no independent standard was available for this cohort, consistency assessment was based on a hypothesis that cartilage thickness losses over the five-year period would be monotonically decreasing and that this hypothesis can be tested using longitudinal cartilage thickness correlation coefficient R. Higher occurrence of analyses satisfying this hypothesis indicates a larger success rate – histograms of R-values were analyzed to determine whether 4D LOGISMOS outperformed the 3D approach. Results: Tables show signed and unsigned cartilage border positioning errors using the 4D and 3D methods at baseline and 12 months with green boxes representing significant improvement while red boxes indicate a worsened performance. Analyzing histograms of R-values for different sub-plates showed a significantly (p<0.05) better performance of the 4D LOGISMOS when compared to that of 3D.Table 1Baseline signed and unsigned cartilage surface positioning errors between 4D and 3D LOGISMOS on the major femoral and tibial sub-plates.Table 212-month signed and unsigned cartilage surface positioning errors comparison between 4D and 3D LOGISMOS on the major femoral and tibial sub-plates.Table 212-month signed and unsigned cartilage surface positioning errors comparison between 4D and 3D LOGISMOS on the major femoral and tibial sub-plates. Conclusions: The proposed 4D LOGISMOS segmentation algorithm outperformed the 3D approach and demonstrated the benefits of using temporal contextual information in studies with multiple follow-up MRIs. Temporal-context aware image segmentation techniques offer a significant improvement of segmentation performance in hard-to-analyze longitudinal studies of OA progression.