Medical cannabis with cancer as a qualifying condition has become legalized in more states, but currently there are no standardized measures of perceived benefits and harms of cannabis use in cancer. This study surveyed a population-based sample of cancer survivors (n = 1539) with various types of cancer including breast (25%), prostate (17%), and gastrointestinal (11%) cancers. Item response theory analyses were used to evaluate the items for measuring perceived benefits and harms. Item response theory evaluates survey items by estimating the accuracy (analogous to reliability) and severity reflected by each item. Item response theory analyses showed all the items were accurate (reliable) measures of perceived benefits or harms. The perceived benefits items assessed beliefs well from low to high levels of perceived benefits. The perceived harms items assessed beliefs from moderate to high levels of perceived harms. The items can be used in future studies to standardize measurement while allowing some customization.