AbstractPurpose This work introduces ongoing research on computer‐aided retinal surgery. A content‐based video retrieval system is presented: given a video stream captured by a digital camera monitoring the current surgery, the system retrieves similar videos in video archives. These informations could guide the surgery steps or generate surgical alerts if the current surgery shares complications with archived videos.Methods We propose to use data compression to extract video features. 1: motion vectors are derived from MPEG‐4 stream. 2: image sequence segmentation is performed by a k‐means clustering. 3: we used Kalman filter to track region displacements between consecutive frames and therefore characterize region trajectories. Finally, we combined this motion information with residual consisting of the difference between original input images and predicted images. To compare videos, we adopted an extension of fast dynamic time warping.Results The system was applied to a small dataset of 24 video‐recorded retinal surgeries (621s +‐ 299s). Images have a definition of 720x576 pixels. An ophthalmic surgeon has divided each video into three new videos, each corresponding to one step of the membrane peeling procedure: Injection, Coat, Vitrectomy. The effectiveness of the proposed method, measured by ROC curve, is interesting (Az ≅ 0.73).Conclusion A novel CBVR system, allowing retrieval of medical video, has been presented. Experiments on the dataset of retinal surgery steps validate the semantic relevance of retrieved results in ophthalmic applications.