Inside Medidata: Reproducing Trial Results
We live and breathe clinical development at Medidata, and we spend a lot of time talking about clinical transformation (check out one of our happy hours sometime).
So, to give you a taste of our culture, we thought it would be fun to share a conversation that evolved over our company intranet between our president, Glen de Vries, and Dhruv Vasishtha, a senior business analyst on the product management team. After Glen and Dhruv kicked off the discussion, a number of other people chimed in as well.
Keep the conversation going and share your thoughts below!
Dhruv: Glen, I’d love to hear your thoughts on the recent Quartz article “Nearly All of Our Medical Research is Wrong.” It discusses the lack of reproducibility in medical research and some of the foundations of the problem. A couple of takeaways from it include:
- If you test enough different subgroups, you’re bound to find an association by pure chance and that association will be spurious and irreproducible
- Since negative data is not reported, study failures are rarely seen and outsized emphasis is placed on successful results
Glen: It’s an interesting article. This is a legitimate problem in lab research and clinical trials. I'm not sure if I buy the fact that successes are inappropriately amplified because of unseen, outsized negative data sets. However I think that more data availability and sharing can help (on the positive and negative side) better clinical trial designs lead to better outcomes. I think it can be done with broader new study designs and through data sharing across completed studies
I also think human biology is incredibly complicated. It’s more often than not a pretty bad assumption to expect that patients in study A are identical to patients in Study B. But that's exactly why I think there's such a compelling scientific opportunity for us to help our clients look at data in better ways across studies!
Dhruv: Interesting that you identified trial design as the area driving outcomes. I completely agree with the fallacy of patient A's biochemistry being similar enough to patient B’s, therefore allowing for reproducible results. That being said, my sense is that in the future, genomic testing will be so cheap and thorough that trials will be run for smaller segments of sub-populations in order to find tailored therapies.
I know someone currently working on a company, MedGenome, that is sequencing genomes of heterogenous communities in India. I wonder if we'll get to a point where you can run trials on a sufficiently scientifically similar patient group where results are reproducible.
Glen: I agree that in the future we'll be able to demonstrate that two patient populations are similar enough to expect a similar response from both (and therefore there is something unexpected, and perhaps scientifically interestingly, if you don't get it). And I also agree that cheap genetic testing and sequencing will be hugely helpful in that regard.
One caution/caveat on genetic testing and sequencing: I think that phenotype is much more complex and much more specific than genotype. The effect of current and past environment on gene expression has a huge effect on physiology and is much more difficult to accurately measure than the underlying genetic potential encoded in a cell.
Dhruv: The phenotype point is really interesting and I remember you talking about it in the past. Taking that into account, I see how something like trial design or cross-study analysis becomes the lowest common denominator in driving efficiencies and improvements in research outcomes.
Glen: Exactly. Having the end-to-end structured info on design and subjects (phenotype and genotype), and how the analyses are done gives us an amazing "denominator."
David Metz, Product Management, New York: Reader beware: The article isn't particularly clear about this, but the findings of the "53 landmark cancer studies" are preclinical in nature. Before a therapy is released to market, it would undergo Phase I-III trials where any spurious preclinical findings would be discovered and prevent the product from hitting the market. It's not like randomized clinical trials are suddenly a farce because of these findings.
Jim McPherson, Product Management, New York: On a related note, last year at DIA there was a general theme where companies wished that a large portion of the "control" group could just be data-driven from existing history. For example, after identifying the IP group breakdown (ethnicity, gender, socio-economic status, etc) they could punch in those parameters and get a randomized "control group" of health history, but without recruiting a large subset of new subjects who wouldn't be experiencing the drug. That way they could compare the drug's efficacy and adverse events with the history.
Glen: This is great timing. For those who didn't hear it, we were asked about data sharing on the earnings call. I went "on record" saying it was a good thing, but that it's not trivial to make it work. ... which is where we come in.
Lara Cornelius, Data Operations, Conshohocken, PA: If the pharma industry can be encouraged to create consistency between companies in the way clinical trials are structured, there will be much greater success in reproducibility. Data driven control groups are actively used in preclinical studies, however these data stem from internally run historical control studies for which the protocol and data collection methods are perfectly aligned with the subsequent "test article" studies. Throw in genetically identical rodents plus identical "nurturing" (housing, HVAC, feed) and you've created a historical baseline that only need be repeated if a new parameter needs to be measured. Obviously this is much more complicated on the human trial side, but if we can collect and aggregate the "nurturing" info and pair it with the medical data, there would be no argument against using historical data as a control.
Glen’s talk at the HITLAB Seminar Series is just around the corner! Join us on February 25 at Columbia University for, “Behavioral and Cognitive Phenotypes: How Instrumented People Will Change Every Part of Healthcare and Epidemiology.” You can sign up here.