Editor's Spotlight/Take 5: How Large a Study Is Needed to Detect TKA Revision Rate Reductions Attributable to Robotic or Navigated Technologies? A Simulation-based Power Analysis.

Abstract

Here’s a personality test. What is your reaction to the following claim: “Navigated, robot-assisted [your favorite procedure here] results in more-precise implant alignment than does manual surgery.” It seems to me that a prudent reaction would be to ask more questions: How much more precise? What advantages might a patient notice, if any, from this increase in precision? Are there potential harms or added costs associated with the new approach? Our specialty’s reaction has been otherwise. Despite the fact that no well-designed study about a new implant-alignment tool, navigation system, or surgical robot has been shown to improve patient-reported outcomes or implant durability, and despite the fact that the best studies on those innovations—including systematic reviews, network meta-analyses, registry reports, and long-term follow-up studies of randomized trials—have found no differences that a patient might perceive [1, 2, 8, 9, 13], the use of expensive, unproven technology in our specialty is growing fast. The global use of robotic-assisted knee arthroplasty, for example, will reach approximately 700,000 procedures per year by 2030, sustaining a compound annual growth rate of 8% from 2021 to 2030 [5]. I can only speculate as to why this might be. Certainly, as a tech-driven specialty, I think many of us enjoy thinking about ways that new tools may make us more effective; why not try them out? I can attest that robotic and navigated systems are enjoyable to operate, and they feel—indeed are—very precise when used correctly. These sensations certainly provide positive feedback to the surgeon, and may contribute to a surgeon’s decision to continue to use them. It also seems likely that gut-level intuition plays a role here. Accelerometers make everything from our cellphones to the airplanes we fly in more capable; why wouldn’t they be useful to guide our hands—or a robot’s hands—through a complicated operation in three dimensions and six degrees of freedom? Finally, it’s possible that some of the buzz is driven by intellectual property interests or industry consulting relationships, though I’d hate to believe this is a large part of it. Regardless, there are important reasons not to use new approaches until they’re proven superior in ways patients might perceive: costs, surgical time, and perhaps most importantly, the risks inherent to uncertainty and novelty. Unintended consequences can be severe, surprising, and anything but hypothetical. Real patients—folks’ moms and dads—get hurt when we use something we think works well rather than making sure we know it works well [10, 12]. I’ve come to believe that surrogate endpoints, like the percentage of implant-alignment outliers, need to be viewed with great skepticism when we’re dealing with generally well-behaved operations like primary joint arthroplasty. We need to ask for more than just improved precision before deciding to use these tools in our practices. Improved implant survivorship would be nice. For this reason, I was so excited to read this month’s Editor’s Spotlight article, “How Large a Study Is Needed to Detect TKA Revision Rate Reductions Attributable to Robotic or Navigated Technologies? A Simulation-based Power Analysis” [6], which was performed by a high-powered clinical-methodological combo team from the University of British Columbia. Using robust mathematical simulation approaches and source data from a systematic search process, they projected that technology-assisted TKA (robotic and/or navigated) may yield slight advantages in terms of implant survivorship over the very long run. How slight and how long? The authors’ simulations suggested a survivorship edge associated with new tech over conventional TKA of 1.4% to 2% at 15 years. At 10 years, they calculated a number needed to treat of 100 patients. This means one would have to perform—and pay for—100 robotic TKAs in place of that many conventional TKAs in order to avert one revision procedure. Back to the personality screen: Does this mean that we should perform a randomized trial enrolling more than 5000 patients and running it out into the second decade in order to be adequately powered to verify that such a small difference actually materializes? Or does it make more sense to say that such a small—and, at this point, hypothetical—difference is unlikely to be worth the cost and time even to do such a study? (To say nothing of the cost, time, and risk of buying, deploying, and learning how to use all those robots or navigation systems). This study is important even if you do not replace knees, and regardless of whether you want your next operation performed by human hands or those of a robot. I say so because the methods the authors used help us to think about how, when, and with what kind of evidentiary support we might decide to try something new in the operating room. Another angle in this paper, which I think will have wide appeal, is that while we’re used to seeing studies end with the suggestion “More research is needed …”, the fact is this may not always be true. If we need 5000 patients and 15 years to find a difference in favor of an expensive robot or navigation system, that difference must be very small, indeed, and almost certainly not worth searching for with a big, resource-intensive trial. We should focus our attention elsewhere. These are provocative topics. Join me as I discuss them with Matthew Hickey BEng and PhD candidate, and Bassam Masri MD, authors of “How Large a Study Is Needed to Detect TKA Revision Rate Reductions Attributable to Robotic or Navigated Technologies? A Simulation-based Power Analysis,” in the Take 5 interview that follows. Take 5 Interview with Matthew Hickey BEng and PhD candidate, and Bassam Masri MD, authors of “How Large a Study Is Needed to Detect TKA Revision Rate Reductions Attributable to Robotic or Navigated Technologies? A Simulation-based Power Analysis” Seth S. Leopold MD:Congratulations on this thought-provoking, and I think in some ways game-changing, study. Before getting into the knee stuff, let’s take a broad look at why your approach is so important: Studies like this can inform how surgeons and healthcare systems decide whether to adopt any of a variety of new treatments. Consider the big picture for a moment: How do you see this kind of research influencing physicians’ and policymakers’ choices in that regard? Matthew D. Hickey BEng: The decision to adopt new medical technologies can be a difficult one that should typically be driven either by perceived improvements in patient outcomes or reductions in overall healthcare costs. While our study focuses specifically on the challenges of detecting changes in revision rates attributable to technology-assisted intervention in TKA, I think there is a broader lesson here. Since patient characteristics and clinical capabilities can vary in important ways across practice settings, and since both of those things affect a patient’s risk of revision, it would rarely make sense to use a single treatment approach in all settings. Rather than zeroing in on changes in patient outcomes resulting from using technology assistance in routine TKA, we believe that more focus should be placed on looking for ways to further personalize or customize individual treatments within particular practice settings to achieve the most cost-effective improvements in overall patient outcomes. This is where we believe the use of simulation tools, like the ones we used in our paper, can be of real benefit to physicians and policymakers. Simulation-based approaches can help us estimate how much of a benefit proposed treatments may have on particular patients in any number of different practice settings, and aid physicians and policymakers in making more informed decisions concerning when and where to use more advanced (and presumably more costly) techniques before implementing them. Dr. Leopold:We often talk about costs of care, but costs of research loom large, as well. As I suggested in the commentary, more research is not always needed; sometimes, we know enough to say that a line of inquiry is not sufficiently promising to justify further investment. At the same time, it seems clear that there will be a large role for technology-assisted approaches in surgery as we move ahead. What is the most responsible way to decide what to study and when to quit studying it, and how can studies like yours help guide those conversations? Bassam Masri MD: Well, Dr. Leopold, a knee is not a knee is not a knee when it comes to knee replacement outcomes. Most knees are straightforward, and the outcome is reasonably predictable. However, others are more difficult; for example, knees in patients with unusual anatomy, prior trauma, or obesity, and perhaps even knees in younger patients (such that the device will be at risk for loosening for many more years), all may need to be considered differently. Most of the research on technology-assisted surgery has shown a signal that there may be a potential for better outcomes, but, as our study showed, it will be next to impossible to prove that this improvement can deliver greater value at the level of a given practice, particularly in smaller practices. I think future research in this area should seek to identify who can benefit most from this technology to see whether we can have a substantial difference in outcome, and so that we can design feasible studies that either demonstrate the value of the technology, or show that it lacks value. The beauty of simulation is that we can look at many of these factors using computational models before subjecting patients to the unknowns of new technology. If these mathematical models show that a clinical study is feasible, then we can proceed with such research. This is something that a clinician alone cannot do, and a scientist or an engineer alone cannot do. Team-based work between engineers and clinicians can help bridge these gaps and help us come up with better answers. Dr. Leopold:It feels easier to take a chance on expensive new technology that may have a learning curve associated with costs or even risks if the standard treatment is unreliable or inconsistent. If someone had a robot that could consistently help patients resolve infected pelvic discontinuities after revision THA, or stabilize a skeletal metastasis while curing the underlying malignancy, we’d be all-in. But primary lower extremity arthroplasty is generally reliable, and most of its problems are not caused by 1° or 2° malalignments. Why do you think so much effort has been spent developing expensive, complicated systems that seem like they might offer at most—as you found—incremental differences in well-behaved procedures? Dr. Masri: You hit the nail on the head here. Lower extremity arthroplasty is so good and so mature that developing newer, fancier implants will not alter outcomes and will not change the marketplace. However, considering the high volume of these procedures, any technology that promises better outcomes, faster recovery, or easier surgery will likely be profitable. The additional precision offered by these technologies cannot be argued with, and it is much easier to accept this technology because it makes intuitive sense. The benefit to industry is the high margin for these devices and consumables, without having to take the risk of a new implant or technology that may cause more harm. I am not against robotics, but until they can be reliably shown not to add substantial costs (that is, by offsetting their fixed costs with reductions in operative times or by producing demonstrable ancillary benefits), I agree that it is difficult to argue that there’s much of a case for shifting our practices. But I do think that, given the large numbers of arthroplasties performed annually, even a moderate improvement in outcome, if it could be achieved at low cost, would have a large overall impact. Dr. Leopold:Your study suggests only a small reduction in the risk of revision if technology-assisted TKA is applied generally, but in the paper and in a previous question, you pointed to some potential benefits to selected patients whose risk of revision is greater. I’m excited to see how that turns out. But now, let’s look a bit more broadly: Might there be other kinds of benefits to navigation and perhaps even robotic tools—perhaps in surgical education, perhaps even beyond TKA—and if so, what would it take for our specialty to make the most of those benefits, and how might the costs be borne? Mr. Hickey: While a projected reduction in risk of revision of 2% at 15 years may seem relatively small (and, as we’ve shown, difficult to discern against a backdrop of a roughly 93% survival at that same time point), it does represent almost a 30% relative reduction in overall revisions (7.5% down to as low as 5.5% in our simulation), which I consider to be noteworthy, even if this reduction cannot, on its own, justify using technology assistance. If it could be shown (more convincingly than has been done to date) that technology assistance provides other benefits such as reduced episode-of-care costs [3, 4, 11] or improved patient functional scores [14], then such considerations could also influence decisions about its use. Another intriguing potential benefit, as you point out, is in surgical education, both for residents and experienced surgeons. Repeated anecdotal evidence that the process of using technology-assisted TKA improves surgeons’ conventional surgery was borne out in a study that showed that an experienced TKA surgeon had better coronal alignment results in a series of conventional manual procedures performed after a series of navigated surgeries than in a similar series of conventional procedures performed prior to using the navigation system [7]. This improvement could be due to a better conceptual understanding of the procedure or of the key decisions, or a better intuitive sense of the alignment targets. Unfortunately, since “educational value” would likely not be considered reimbursable, any additional costs would probably have to be funded either through medical schools’ training budgets or directly by the companies selling the implants. Either way, there will be continued pressure to bring the capital and per-case costs down to justify wider adoption. Dr. Leopold:Let’s throw in one specific question here for the knee people: You suggested that more-promising avenues of study for technology-assisted TKA might involve patients with higher BMI or younger age, presumably because they may be at higher risk for revision. It takes a certain number of repetitions, and ongoing practice, to use tech-assisted systems effectively; keeping them on the bench for everyday use and only sending them into the game for the tough moments seems a recipe for trouble. I also wonder whether lower-volume surgeons—who also care for patients with higher BMIs, as well as younger patients—would be able to use these systems at all if they try to limit their use to the hardest procedures. How would you balance out these tradeoffs? Dr. Masri: You are absolutely right. It makes little sense to have a tool sitting on the bench to be used only once in a while, because the learning curve will almost certainly kill any potential benefit. This is why, if we can prove that these devices can improve outcomes in more difficult cases, robotics should only be available at centralized referral centers. This way, lower volume centers can legitimately, and without guilt or criticism, cherry-pick the simpler procedures and refer the harder ones to centers that are better equipped to handle them. The current problem is that compensation models do not reward centers for performing more-difficult procedures. Bundled payments are an example of such a model. No one wants to do only difficult procedures and allow his or her colleagues to cherry-pick the easier ones. If we decide to collaborate and do what’s best for patients by referring the most difficult procedures to centers that specialize in them, providing them with the right tools, and modifying compensation schemes that value the burdens associated with case complexity, then tertiary referral centers will not be penalized for being the good Samaritans. This will take a culture shift in how we all work, I think.