Purpose: In this paper, we describe a method for recovering the tissue properties directly from medical images and study the correlation of tissue (i.e., prostate) elasticity with the aggressiveness of prostate cancer using medical image analysis. Methods: We present a novel method that uses geometric and physical constraints to deduce the relative tissue elasticity parameters. Although elasticity reconstruction, or elastograph, can be used to estimate tissue elasticity, it is less suited for in-vivo measurements or deeply seated organs like prostate. We develop a method to estimate tissue elasticity values based on pairs of images, using a finite-element-based biomechanical model derived from an initial set of images, local displacements, and an optimization-based framework. Results: We demonstrate the feasibility of a statistically based classifier that automatically provides a clinical T-stage and Gleason score based on the elasticity values reconstructed from computed tomography images. Conclusion: We study the relative elasticity parameters by performing cancer grading/staging prediction and achieve up to 85% accuracy for cancer staging prediction and up to 77% accuracy for cancer grading prediction using a feature set, which includes recovered relative elasticity parameters and patient age information.