It’s not often that a few doctors ensconced in the ivy towers of academia can get a lumbering dinosaur like the Food & Drug Administration to turn on a dime, let alone during the week between Christmas and the new year.

But that’s exactly what University of California at San Francisco researchers Sanket S. Dhruva, Lisa Bero and Rita Redberg did after the Journal of the American Medical Assn. published their paper “Strength of Study Evidence Examined by the FDA in Premarket Approval of Cardiovascular Devices.”

The study, which looked at 78 pre-market approvals for cardiovascular devices from 2000 to 2008, touched such a nerve at the Rockville, Md., headquarters of the FDA that acting Center for Devices and Radiological Health head honcho Jeffrey Shuren had the watchdog agency’s own internal study of the device approval process released early. Shuren, during a spin control session with the New York Times, conceded the researchers’ main point: The agency needs to do a better job regulating the evidence used to clear medical devices.

MassDevice caught up with Redberg and Dhruva, who was a medical student under Redberg during the study, to discuss their findings and get their reactions to the foofaraw the study kicked up.

MassDevice: Were you surprised by the reaction of the FDA to your report? After all, it’s not often the acting head of the Centers For Devices and Radiological Health goes on the offensive over a journal article.

Rita Redberg: I’d say accusing us of inaccuracies is on the offensive. Although we certainly stand by all of our numbers and would be happy to make them all public. They were all in the article anyway, but everything we based them on was all public as well. But I was surprised by the ferocity of it. We certainly just tried to report the data; the data certainly wasn’t what we were expecting and it certainly wasn’t what the FDA was expecting. I didn’t excpect the FDA would be thrilled, but I didn’t think they would attack the messenger and claim inaccuracies in our data.

Sanket Dhruva: We’re very glad that the FDA is starting to move. The data behind medical device approvals was under the radar and now it’s not under the radar, so I’m excited that the FDA is starting to think about it and is starting to make changes.

There’s been a lot of discussion about pharmaceuticals and randomized, double-blind trials after the Vioxx controversy and anytime anything happens with a drug. But our perspective is that, at least with cardiovascular devices, these are the sickest patients with heart problems and we’re going to put a device in them, which is going to be implanted most of the time, that’s something where we feel the strength of the data should be even stronger than it is in pharmaceuticals. Because a pharmaceutical can definitely have an impact, but eventually the drug goes out of the body when you stop taking it.

So that’s why we feel very strongly that devices are no longer under the radar and it’s very important to think about what’s going into in terms of the data.

MassDevice: What was the inspiration behind the study?

RR: My co-authors Dr. Dhruva, Lisa Bero and I have all been interested in technology assessment in FDA device issues. I’m particularly interested in the quality of evidence we use to approve new devices and then to use them, which comes fairly quickly after approval. That was my interest. Dr. Bero has done some studies on the drug approval process, but I was particularly interested because I’m a cardiologist and Dr. Dhruva and I have worked before, looking at the the quality of evidence that Medicare uses for coverage decisions.

No one had looked the FDA approval process for cardiovascular devices — or any devices for that matter — very closely. That was about a year-and-a-half ago, in the summer of 2008, and I shared my plans with the FDA. I told them that was what I was doing because I requested confidential data, which I am authorized to access because I am a member of the FDA expert panel. The FDA told me I shouldn’t use the confidential data, so everything we used was publicly available on the agency’s website, which is a summary of the safety and effectiveness of the device, which I believe is written by the companies.

SD: I was a medical student at UCSF last year and had worked with Dr. Redberg earlier. I want to be a cardiologist and I’m very interested in cardiovascular devices, because they’re exciting; they have such an impact on human lives. But we hadn’t heard much about what quality of evidence was behind a lot of these devices. It was more just sensing that we don’t know what’s going on in terms of data. When someone’s critically ill, or having a heart attack, putting in a stent is going to save their life. But it’s also going to stay in for the rest of their lives.

Another inspiration was preemption and how that means the FDA’s approval of these PMAs essentially prevents consumers from suing and how that’s changed for pharmaceuticals with the Wyeth v. Levine case. So it’s even more imporant, because if the FDA approves a device and someone gets injured, we want to make sure the device is approved on the best possible evidence.

MassDevice: What pre-conceived notions, if any, did you have that were either proven or dispelled by the findings?

RR: We looked at pre-market approvals, which is the highest risk category, so we expected to find a lot of high-quality randomized controlled trials. We were surprised; when we first started the study we thought we’d try to see how many of these studies had actually been published in the peer-reviewed literature. But once we started reading the PMAs, we realized it would be important to first categorize the kind of data that we were looking at.

SD: I would have expected to have seen stronger evidence, as well as more thorough follow-up, more clearly delineated clinical endpoints, but at the same time there’s an amazing variety of devices out there. There is a lot that goes into the device approval process, including all the testing that’s not just clinical trials. Although I would argue that clinical trials are the most important, obviously there’s a lot of effort that goes into that and knowing these are the devices that make an impact on the health of our patients. But at the same time we came back expecting a little bit stronger evidence.

MassDevice: One interesting statistic is that 65 percent of PMA clearances were based on a single study. What does this say about the thoroughness of the research?

RR: Well, I think it’s more thorough if you have more than one study. There were a lot of things like that; I think only about 25 percent were randomized, only 14 percent were blinded, the longest follow up period we found was one year. There were studies we found that didn’t list how many people were in the study, in which they didn’t tell us anything about the population in the study. They overwhelmingly used surrogate endpoints that, in some cases, didn’t seem to have a lot of clinical significance. There were a lot of issues, it wasn’t just that they based a lot of them on one study.

SD: Absolutely, we would have expected to have more studies, especially knowing that a lot of these are going to be implanted in patients, and as we know in several different clinical incidences the findings of one study have often been modified by the findings of a second study. I would have preferred to see multiple studies and would have expected to see stronger evidence. And that’s one of the reasons we did per-PMA analysis, because we wanted to see how many PMAs had at least one randomized study, one blinded study and at least one center in the United States.

MassDevice: Can you walk us through the mechanics of the study?

SD: We went to the FDA website and we looked for anything that was cardiovascular and anything that was an original PMA filing. Only the ones that are approved are made publicly available, so we did that search on October 31, 2008. At that time we looked at everything that was publicly available, 78 PDFs for the 78 PMAs that we looked for. Those files range from about 15 pages to about 70 pages. We got most our data from the clinical testing section and the safety section and we started putting it into a pretty massive spreadsheet and then tabulated all the data afterwards, which took several months. The data was also confirmed by Dr. Bero and Dr. Redberg. We wanted to make sure our data was as accurate as possible and we stand by it.

MassDevice: Explain your hypothesis that these may be prone to bias. Do you mean the examiner may have bias in favor of the company?

RR: A fair number did not have any control group, I believe there was about half that didn’t have a control group. And of the ones that did, 30 percent of them were historical control groups, which I had actually never heard of prior to looking the pre-market applications. I thought you only enrolled your control group at the same time. But historical controls meant that they used a control group from a different study. So you can pick and chose them, so we thought that allowed for bias.

The whole strength of doing a randomized study is that you take a population that shows all the same traits, you haven’t chosen them in any way, and you randomly separate them into two groups, one that gets your intervention and one that doesn’t. Then you know that everything you’re seeing is from the intervention, because you’re controlling everything else. If you don’t do that, then the population is possibly biased because there are other characteristics that could have been separating what you’re studying and the control population.

MassDevice:Has this changed your opinion about the safety of cardiovascular devices that are already on the market?

RR: We really weren’t looking at safety and effectiveness of devices on the market. We were looking at the quality of evidence that’s used on which to base the device approval.

MassDevice: But if you’re saying that the evidence used to base an approval on is not adequate, doesn’t that lead to questions about the safety and effectiveness of approved devices?

RR: I think it leads to more uncertainty.

SD: I think that we can do a better job of getting stronger evidence for our high-risk cardiovascular devices.

MassDevice:What improvements would you suggest?

RR: We’d like to see high-quality data of safety and effectiveness in the trials that the FDA uses to base approvals for cardiovascular devices; more randomized studies, more blinded studies, more clinically significant endpoints instead of surrogate endpoints. And I believe the FDA has already announced that they are making changes, so we’re very pleased about that.

MassDevice: Has the FDA contacted you directly, or has it all been through the press?

RR:The FDA has not contacted me directly.