Hard Drugs

Jacob Trefethen:
Today, we're gonna talk about Moore's law meets its nemesis, Eroom.

Saloni Dattani:
We should definitely keep that in.

[podcast jingle]

Saloni Dattani:
Moore's law is a trend that describes how the number of transistors in an integrated circuit, like a computer chip, has doubled roughly every two years since the 1970s, with minimal changes in cost. That means computer chips have become more and more powerful over time.

Jacob Trefethen:
In drug development, the opposite seems to have happened. Since at least the 1950s, it's become more and more expensive to develop new drugs. We're getting fewer new drugs per billion dollars spent on research and development, and as a science funder, I took that personally. So, researchers have called this trend Eroom's law, Moore's law, backwards.

Saloni Dattani:
This trend seems pretty strange. Given how much our technology has advanced, why has it become more expensive to develop new drugs rather than less?

Jacob Trefethen:
Even more strangely, the decline seems to have levelled off in the last decade. What's going on?

Saloni Dattani:
In this episode of Hard Drugs, we'll explore Eroom's law and what's behind it. Is there any hope to reverse this? Can AI solve this?

Jacob Trefethen:
AI meets its match.

Saloni Dattani:
Hahahaha. All right. How excited are you for this episode?

Jacob Trefethen:
I'm excited. I came across Eroom's law a few years ago and thought it was, it's one of those articulations that really sticks with you and gets stuck in your craw, as they say. That means we're going to have to tackle it head on if we want to develop more treatments, cures, and prevention tools for people with diseases.

Saloni Dattani:
I agree. I think it's also pretty scary. Most of what we are interested in is reforms and things that can bring new medicines to more people. But if it's getting exponentially more expensive to develop new drugs, that probably means this overarching trend is going to be stronger than any of these reforms that people are making. It seems like all we can do is slow down its impact, but that's pretty depressing. I think the reason for this episode for me is that we should try to figure out what's actually going on and try to reverse it, if that's even possible.

Jacob Trefethen:
Let's get into it.

Saloni Dattani:
Let's get into it. Let's start with some trivia. Here's your first question. According to Eroom's law, how many years does it take for the cost of development per new drug to double?

Jacob Trefethen:
Okay, Moore's law.

Saloni Dattani:
Is it doubling every two years?

Jacob Trefethen:
Sometimes people say 18 months, but call it two years. Do I think Eroom's law is two years? No, it's not that bad. Things aren't doubling their expensiveness every two years. So it's got to be longer than that.

Saloni Dattani:
Phew!

Jacob Trefethen:
With exponentials, I always recall that if you're growing at 10% a year, you double in seven years. So I'll go with seven years.

Saloni Dattani:
Interesting idea. That's pretty close. It's nine years. That's the historical trend that's described in the original paper. This trend was described by Jack Scannell, a researcher, along with some of his colleagues in 2012. He coined the phrase, and it describes the trend between 1950 and 2010. In the 1950s, the number of drugs you used to get per billion dollars of R&D spending was around 50 new drugs. But now you get about one drug per billion dollars in R&D. These are inflation-adjusted figures.

Jacob Trefethen:
We've declined 50x. They're a really bad start to the episode.

Saloni Dattani:
Well, let's get to an even more depressing question then. Some diseases have attracted a lot of R&D spending, without any big successes so far. For one particular disease, it's estimated that private companies spent around $42.5 billion trying to develop new drugs between 1995 and 2021 to tackle this disease. This excludes preclinical research, which is hard to measure. But what's the disease?

Jacob Trefethen:
I have my guess. I think it's Alzheimer's.

Saloni Dattani:
It is Alzheimer's! Ding, ding, ding!

Jacob Trefethen:
We're going to get haters in the chat because of course, so I guess up until 2021, there was really not any success. And then we had a few drugs come through. They're not ideal, I would say, even those though. But yeah, Alzheimer's has been just so tough during that period.

Saloni Dattani:
Yeah, I would say they're not big successes. Like, they do maybe work, but even the people who claim that they work would probably say that they're kind of modest effects.

Jacob Trefethen:
And, you know, only tens of thousands of Americans take them, not millions. So, yeah, we haven't reached a real big change yet.

Saloni Dattani:
That's quite depressing. What would you say are some of the reasons that it's so much harder in Alzheimer's than in other diseases?

Jacob Trefethen:
Oh gosh, how long do you have? I mean, there's some of the drivers. We need to do an episode on Alzheimer's at some point.

Saloni Dattani:
Top three reasons?

Jacob Trefethen:
Okay, top three reasons. We don't understand the drivers of Alzheimer's very well. It develops over decades. So it's hard to run clinical trials or to really understand progression. And it's hard to measure things going on in the brain because there's a skull in the way.

Saloni Dattani:
That's so annoying that there's that skull.

Jacob Trefethen:
Yeah.

Saloni Dattani:
Such a shame.

Jacob Trefethen:
If only we could design around that.

Saloni Dattani:
If only we had x-ray vision.

Jacob Trefethen:
Speak for yourself.

Saloni Dattani:
Okay, a lot of what we're talking about is spending in R&D. I think that spending on R&D and how expensive it is to develop new drugs also leads to further consequences on how expensive it is to sell those drugs or to purchase those drugs later on. So here's a question about the most expensive drug launched. I'm going to give you the name. That's a clue. It's called Lenmeldy.

Jacob Trefethen:
Lenmeldy.

Saloni Dattani:
It's a gene therapy for children with a rare inherited disease called metachromatic leukodystrophy. This became the most expensive medicine when it launched in 2024. What was its list price for treating one patient in America? This time we're focusing on the list price, not the cost of treatment or the R&D spending.

Jacob Trefethen:
I probably saw headlines at the time, but I can't recall the specifics. It's definitely going to be over a million dollars. I'm going to say two million dollars.

Saloni Dattani:
$2 million... The answer is $4.25 million for this one-off treatment.

Jacob Trefethen:
Oh my gosh. So someone's insurance paid that?

Saloni Dattani:
I guess so. I mean, it's pretty rare, so I guess it would only be a small number of people taking it. This is a gene therapy that involves editing blood stem cells to add a functional version of the gene that people with this disease are missing. But it's really expensive, and each treatment has to be manufactured individually. So you have to collect the child's blood stem cells, genetically modify them in a lab, check the quality, and then replace the cells into the same child. A complicated process.

Jacob Trefethen:
Yes, I'm not sure. I've got a spare 4 million lying around, actually. But anyway, keep going.

Saloni Dattani:
Keep going. Okay, now let's go into the reverse. Now for some good news about a cheaper medicine.

Jacob Trefethen:
Okay.

Saloni Dattani:
This is kind of interesting because I've been writing an article about smallpox eradication, and what I find really interesting about smallpox eradication is how cheap it was. It cost about $3 billion in today's money to eradicate smallpox worldwide in the 1960s and 70s.

Jacob Trefethen:
I mean, that is completely wild. We spend way more on controlling individual infectious diseases per year. How frustrating. Okay.

Saloni Dattani:
One of the reasons that it was so cheap was that smallpox vaccines themselves are very cheap to produce. They were first developed in the late 18th century when Edward Jenner developed the first smallpox vaccine, and there have been loads of improvements since then. So smallpox vaccines became really cheap by the 1960s. But just how cheap? How much did each dose of smallpox vaccine cost in the 1960s?

Jacob Trefethen:
Wow, I wonder even how to come up with a guess there. I mean, I know what vaccines cost to produce with current cost structures, mostly by Indian companies today. They can get really cheap. You know, we're talking under a dollar, depending on how you make it. Yeah, maybe back then it was cheap. Let's go with... 25 cents a dose.

Saloni Dattani:
Wow. Here's what it was. In 1960s money, the vaccine costs about two cents per dose. And if you count the cost of vaccination, including like the people vaccinating and the transport, that's about 10 cents per dose, which is about 74 cents in today's dollars.

Jacob Trefethen:
So the actual product itself was less than 25 cents, I said.

Saloni Dattani:
Right. It was really cheap.

Jacob Trefethen:
That's a bit different.

Saloni Dattani:
That was very different from the previous question, which was for $4.25 million. If only everything was as cheap as smallpox vaccines.

Jacob Trefethen:
And at the end of this episode, we'll show you how.

Saloni Dattani:
Okay.

Jacob Trefethen:
Any other questions?

Saloni Dattani:
Oh, yeah, here's your final fun question.

Jacob Trefethen:
Okay, I better get this right.

Saloni Dattani:
This question is about GLP-1s, which have been on the news for a while now. Ozempic, for example, was developed in Denmark. Denmark is home to Ozempic and Wegovy, which are different brand names for semaglutide. What share of Denmark's GDP growth since 2019 came from pharmaceuticals?

Jacob Trefethen:
Oh, from pharmaceuticals. Okay, so that will almost all be GLP-1s, probably. Well, GDP growth. I mean, I've read the headlines that the revenue from GLP-1s is some shocking proportion of total GDP, let alone GDP growth. And it's all presumably new revenue because the product didn't exist before. So if it were 2% of total GDP or something wild and they're growing, well, that sounds high, doesn't it? Let's say they're growing at, I don't even know. Yeah, let's say it's 2% and they're growing at 4% and that means it's a half of all GDP growth.

Saloni Dattani:
That was also very good. It was very close. It's actually 40% according to the OECD.

Jacob Trefethen:
Oh, that was very close.

Saloni Dattani:
Okay, so maybe let's get back to Eroom's law. The trend describes how the cost of bringing a new drug to market roughly doubles every nine years. These are inflation-adjusted. It's generally measured in how many billions are spent in R&D versus how many new drugs you get approved for that spending.

Jacob Trefethen:
It's like, how much does it cost on the R&D side before you get an approval of a drug? And I think that the thing you're doing the division by is an FDA-approved drug? What's the specific thing that we're measuring?

Saloni Dattani:
It's new molecular entities approved in the U.S. So this only counts new drugs that are not just reformulations of existing drugs. If you just look at new drugs that have been approved by the FDA that are available in the U.S. and the number of those per billion dollars of R&D spending.

Jacob Trefethen:
So that's pretty crisp. Okay, it's new FDA-approved, actual new molecules or drugs of some form per R&D spend. You know, despite being crisp, you could poke at whether that's the right metric to hang your hat on in that, you know, some drugs matter a lot more than others. But okay, we don't want to get too fancy here.

Saloni Dattani:
Right. Yeah, so that is the trend. One thing that I found surprising was I read this 2012 paper by Jack Scannell and his colleagues, and I thought, well, if this trend has been going on for 70 years, how come the term was only just coined now? Like, didn't people notice it before? And it turned out that people actually did notice it before. The overall trend has been described for many decades. I read this 1978 paper that was also describing the trend. That paper said in its introduction that the trend had already been widely discussed. So I guess people have known about it for a really long time. But the brilliance of this paper was that it coined it a great name and gave some explanations for it.

Jacob Trefethen:
The most important thing. Can I ask a clarification about Eroom's law?

Saloni Dattani:
Sure. Yeah.

Jacob Trefethen:
When they do the counting up of R&D spend, do they add up all things that pharmaceutical companies book as R&D expenses and that is it? And they just assume that's all pointed towards new chemical entities? Do they add in the NIH? How do they count that?

Saloni Dattani:
That is a great question, and it's something that I'm interested in chatting about because it's actually really hard to measure. Because as you've kind of hinted towards, it's really hard to decide what counts as R&D spending for a drug. You have these figures that come from surveys and filings from companies on how much they're spending on R&D overall, but how much are they spending per drug? That's quite hard to measure. Because of that, there are lots of really different estimates of how much people are spending on R&D in drug development world per drug. The numbers range a lot. So I'm curious about like, if I asked how much is the average R&D spending per drug, what you would guess?

Jacob Trefethen:
The classic thing that people will say that I've heard is, but I have my own opinions, is, well, it used to cost a billion dollars and now it's two. What I mean by that is taking into account failures along the way. So that's not that one drug costs that much. It's that of all the drugs you tried to get to the finish line with one cost $2 billion.

Saloni Dattani:
Yeah, that's what I've heard as well. What I found kind of interesting was that even with that definition, the estimates range really widely. The range is roughly $300 million per drug to $2.8 billion per drug.

Jacob Trefethen:
You mean for a given category, or how is that defined?

Saloni Dattani:
No, for all new molecular drugs, on average.

Jacob Trefethen:
Oh, sorry, you just mean like no one knows if the true answer is 300 million or 2 billion?

Saloni Dattani:
Exactly. Yep.

Jacob Trefethen:
Interesting. That's pretty messed up.

Saloni Dattani:
It's pretty messy. There are a lot of different estimates here. If you think about how this statistic is measured, you're trying to measure the number of drugs that you get per R&D spending. You can easily get the number of new drugs from FDA approval records. That's pretty easy. You can also find the number of failed drugs from clinical trial registries and just asking companies. Then you can also easily get the total R&D spending that firms are making from surveying them or from industry reports or from SEC filings.

But the hard part is estimating how much is spent per drug. How do you attribute the total R&D spending to the projects instead of all of the drugs that are in their portfolio? That comes from asking firms about how much they're spending per projects or per clinical trial phase. You might just ask experts or do surveys. There have been a few big surveys where they've just asked private companies about some of these assumptions. Just how much is spent on R&D before clinical trials begin versus after and then how they're dividing it up. Or you kind of make some assumptions on how much it costs per patient, how much clinical trials cost, how long they spend. But because it's kind of voluntary information that they're providing and that it's private and that different firms might differ a lot, you end up with really different answers.

Then there's also different decisions that you can make on which research costs you count. Like does preclinical research count? Does like NIH spending count? Or is it just what companies are doing? Should you include tax credits? How do you adjust for inflation? Then I think the most tricky part is actually what time period to use in the analysis when you're thinking about R&D spending. Like some drugs end up getting approved after decades of research and other drugs maybe just a few years. Depending on how much time period you put in the denominator, you could get very different answers as well.

Jacob Trefethen:
Can I add more complexity still?

Saloni Dattani:
Yes, confuse everyone even more.

Jacob Trefethen:
Perfect. In the U.S., at least in some other countries, it's tax preferential to have expenses be booked as R&D versus other expenses. So firms have an incentive to overcount their R&D. So I wonder whether that's changed over time and has some effect on this graph or whether it's been roughly constant. So it's maybe less of an issue for the time relative claims we're making. There's another issue in recent years, which is that there are many drugs getting approved in China, being developed by Chinese companies with different R&D cost structures getting approved in China. And that will probably grow. So the FDA, the US FDA approval list might start becoming less representative of frontier drugs over time. So it's, yeah, it's, this one's tough.

Saloni Dattani:
That's very tricky. I think it makes me wonder how much this affects the trend line overall. I'm sort of like, I think that because I used to work at Our World in Data, and I sometimes used to just dig into like, the appendices of these papers, like, where does this data point come from? Occasionally, the answer is this is not very good data.

Jacob Trefethen:
I always feel so bad. I quote R&D statistics all the time, and they're always from like, McKinsey 2023 said 200 billion. And I'm like, oh God, that's probably completely made up.

Saloni Dattani:
I think the problem with those as well, sorry to McKinsey listeners out there, is that often industry reports don't have a detailed public methodology that you can actually probe and try to understand. The same is true with these figures where they're coming from industry surveys and things like that. I'm just like, where is this from? I don't know.

Jacob Trefethen:
I had a sort of horrifying experience in my last job where we would fund R&D and then we'd get inquiries from people who were trying to track R&D spending. We were part of the surveys, you know, and they would always say, okay, it looks like you spent X million on R&D this year. And I'd consult the internal figures and I was like, that's off by like 3X. Like, why would they think that?

Saloni Dattani:
But instead of asking you, they just guessed and said, is that true?

Jacob Trefethen:
They guess based on public records. Despite the fact that we were very public funder and had lots of records, what in fact ends up being counted under different definitions can really swing things by 3x.

Saloni Dattani:
Oh.

Jacob Trefethen:
It's like, is the spending happening in this year or was it committed in this year? Is it how much of it is? So yeah, I've been on the other side of these surveys and you don't end up feeling... confident about what is in the public domain after you get surveyed.

Saloni Dattani:
Let's take this trend now and kind of probe what's behind it. So I think our conclusion is that it's pretty hard to measure R&D spending, but the decline is so large, and you can also kind of find the same thing if you look at other metrics of R&D productivity that is probably real, but what's behind it? There are actually a lot of different hypotheses of the causes of Eroom's law. Do you have a favorite or one that comes to mind?

Jacob Trefethen:
I do think there are multiple contributors, so I don't want to come off like there's one explanation only. However, I would say the explanation that has often reared its head in my professional life as a science funder is that the, I don't know how to say this pithily, but I got taught this by a mentor once, and once you see this statistical fact, you see it everywhere.

Let's say you have a placebo-controlled trial where there's no treatment for something. You know, say stroke. Let's say you're back in the day where there was no treatment for stroke. So when you're trying your new drug, you're going against a high mortality situation. Maybe there's a 20% chance this patient dies and you're giving a treatment that reduces that to 10%, say, made up numbers. You do not need many people in a control arm and a treatment arm to detect that effect because it's pretty large. You know, you're saving a lot of lives pretty quickly. Now your drug gets approved. Wonderful. That's amazing.

Now you're trying to develop another stroke drug, which is the situation I'm in right now, actually, with ischemic stroke. I mean, there's several drugs, but there's, you know, a main one. And it is unethical to withhold the existing treatment from people. That would be monstrous. So to prove that yours is more effective is statistically harder. And it's not, let's say you want to go from 10% to 5%. That is not twice as hard, it's four times as hard. So it's that for a given level of statistical power, say 80% power, and for a given level of significance, your p-value of 5%, the sample size that you need to detect an effect scales with the inverse square root.

Saloni Dattani:
Right. You're comparing now this new drug or this new treatment to what's already there. And what's already there is much better than it used to be.

Jacob Trefethen:
Exactly, which is wondrous. Basically the mathematical thing here is it's something fundamental about detecting knowledge in populations. This isn't just medicine, but the gist is, if you are trying to detect the effect from half the original mortality rate, you need four times as many participants in the trial. If you're trying to detect the effect from a third of the original mortality rate, you need nine times as many participants in the trial. It scales terribly. And cost of clinical trials is you have some fixed costs, but then the majority of costs scales with the number of participants.

So, you know, this is all a way of saying part of the driver of Eroom's law is incredible. We've already invented some amazing medicines. So, you know, that's how I'm coming out of this screed with a positive ending. It's great that there's already some stroke treatments. But yeah, it just means that you are statistically in a square rootly worse situation.

Saloni Dattani:
I think that is my favorite explanation of Eroom's law.

Jacob Trefethen:
Really?

Saloni Dattani:
That's the one that I find most convincing.

Jacob Trefethen:
Woo! Okay. Listeners, we didn't coordinate on that. It just happens we both can see the truth.

Saloni Dattani:
We just happen to have the same opinions sometimes. Do you know the other name for this explanation?

Jacob Trefethen:
Tell me.

Saloni Dattani:
The better than the Beatles problem.

Jacob Trefethen:
Oh, I have heard that. Yes, yes, I have. The better than the Beatles problem, where the gist is, okay, well, if you're releasing new music, your listeners could just listen to the Beatles. So, good luck.

Saloni Dattani:
Yeah. Yeah, you're basically the bar rises over time because you've made so many successful drugs already that now the new drugs that you make have to be better and better. I think that's both because of the efficacy reasons that you've developed some effective drugs, but also that the older drugs have gone off patent and become generics in some cases, so they can out-compete the new drugs on price as well. So it becomes statistically more challenging to detect the benefits of new drugs and requires larger or longer clinical trials. I think that's pretty convincing as an explanation.

Jacob Trefethen:
Well, since we both agree, I'm going to have to pick the other side.

Saloni Dattani:
What's the other side?

Jacob Trefethen:
Okay, let me pick the other side. There are plenty of problems in medicine where you can still do placebo-controlled trials. It's perfectly ethical, and so why don't we see more progress in those areas? But let's say you are trying to make weight loss drugs, GLP-1s. Those are all placebo-controlled, and the FDA is an approval body. All it is saying is you've got to make something that works, and if you're going to claim it works and is safe, you have to show that. It's not saying you have to compare against all your competitors. And, you know, maybe you'll lose out in doctors' offices or in the marketplace if you don't beat your competitors. But those trials were, you know, how come we didn't get those earlier? Those were placebo controlled.

Saloni Dattani:
I think there's a clarification that I would add to that, which is that even if within the trial you're comparing your new drug to a placebo, that doesn't sort of prevent people from taking other treatments outside of the trial. And they often do. And you're not kind of withholding the standard care from them. So in practice, you are sometimes comparing it with what's already out there. But you're right that in some diseases, what's already out there is not very effective.

Jacob Trefethen:
Right. And, you know, there's some absolutely unsolved diseases. Like Alzheimer's, this is not the main problem. We still have Alzheimer's. How come we're 50x less productive than we used to be? So it can't just be the Beatles problem. It can't just be Paul McCartney withholding those drugs. He can't have these Alzheimer's drugs. He's Irish now.

Saloni Dattani:
I love how every time you try out an accent, it ends up being Irish. I have a question for you related to this, which is, what is something that you think peaked long ago, culturally speaking? It could be a genre or anything, like how it's hard to beat the Beatles. What else from the past do you think it's hard to beat?

Jacob Trefethen:
I mean, look at what I'm wearing right now. I'm wearing like a classic black jumper where I look, well, I'll leave it to listeners' judgment.

Saloni Dattani:
Mm-hmm. I'm wearing a white t-shirt, so we are in contrast.

Jacob Trefethen:
You're wearing a white t-shirt. I mean, those are both some pretty old school winners. That's hard to beat.

Saloni Dattani:
My answer is film plots and dialogue. Controversial. But I just think they were better in the past. I think that today, a lot of the most popular films are sequels and franchises, and I just find that pretty boring. Also I think, so I used to watch a lot of old Hollywood movies from like the 1940s to 60s. One thing that I think they do better is just the dialogue. It's just funnier. It's, I guess, yeah, I think it's just much cleverer.

Jacob Trefethen:
Really? No, I watched some of these old films and I'm like, this was the bar? This is so bad. Like the acting in the world's most famous films is stunningly bad in my opinion.

Saloni Dattani:
Hmm. I think the part that I agree with is what I find weird about old films is the pacing. Everything happens so slowly.

Jacob Trefethen:
Yeah.

Saloni Dattani:
Sometimes you have these shots where you're just looking at a door for five seconds or something. You're like, oh my God.

Jacob Trefethen:
Totally.

Saloni Dattani:
But the dialogue, I think, is actually really good.

Jacob Trefethen:
Okay, I have to go back and watch. Give me a suggestion and I'll watch.

Saloni Dattani:
You should watch Casablanca. It has amazing dialogue.

Jacob Trefethen:
I will do that. Of all the gin bars in all the town, I gotta find you in this one.

Saloni Dattani:
Also, one of the best examples of what you're saying, I think, is like sci-fi and fantasy films from before were so bad compared to now. I watched the old King Kong movie from the 1930s at the British Film Institute here. And it was so boring, I actually fell asleep. One of the reasons it was so bad, in my opinion, was that their special effects were not very developed at the time.

So you might think like, how did they develop King Kong in the 1930s? The answer is they just had a stuffed animal. And it was just clever angles that made it seem much bigger than it really was. And this whole film I was just watching thinking, oh my God, these people are trying to fight against a stuffed animal.

Jacob Trefethen:
It didn't scare you?

Saloni Dattani:
No.

Jacob Trefethen:
Okay, funnily enough, I want to watch that more than the Casablanca one, but okay.

Saloni Dattani:
Okay, so back to Eroom's law. The explanation that I guess we both find pretty convincing is the better-than-the-Beatles problem. Because we have more and more existing treatments, it becomes harder and harder to test the effects of new drugs and to find benefits that are better than them. What are some other explanations of Eroom's law that you can think of?

Jacob Trefethen:
I want to be clear, I don't think that explains it all.

Saloni Dattani:
Oh, sure.

Jacob Trefethen:
I think it's a big deal, but I don't think it explains it all. I think there's more mystery left. I mean, I think to get at some of the other explanations, I almost want to describe my pop history understanding of how penicillin was first tried in humans.

Saloni Dattani:
Okay.

Jacob Trefethen:
My understanding is that we're in 1941, and someone comes in, you know, I think a middle-aged man with a bacterial infection. Wait, you don't believe I was there?

Saloni Dattani:
I just like how it was one middle-aged man.

Jacob Trefethen:
So, someone comes in, they have a really bad bacterial infection. The scientists are like, okay, we've purified some penicillin, let's pop it in, see what happens. They try it out within 24 hours. He's feeling amazing. Like, okay, that probably worked. Not necessarily. Maybe it would have resolved. But then they ran out of penicillin, and then he got bad again and died. Okay, so that's N of one. It's actually a fantastic amount of, you know, knowledge gained from that horrifying experience.

Could that have happened today? Absolutely not. You know, so you would today go through a phase one trial to test that penicillin was safe and maybe in healthy volunteers first. Then you would do a phase two to test different doses. Then you do a phase three. In the phase three, you'd be comparing against existing antibiotics we now have, and you'd have the better than the Beatles problem we just discussed. So, you know, you don't get these shocking, incredible discoveries anymore. I think that story contains other aspects of why. So maybe I'll just flip it back around on you. Do you agree that that story contains your other favorite explanations or do you not think so?

Saloni Dattani:
I feel like there's a clarification there because I think that they didn't immediately just go and start using penicillin en masse after that one experience. For one reason that you mentioned was that they ran out of penicillin. They were sort of like, we need to find more productive sources of this fungi. But I think it was basically an early clue that this would work. My understanding is that the USDA collaborated with researchers in Oxford who developed this injectable – was it injectable? maybe it was medically useful penicillin – to run clinical trials and to share data between different manufacturers, and they tried to look for better sources of penicillin.

But I think they did run clinical trials in the end that were bigger than the original one. It's just that the standards were different then, so they wouldn't have done phase one, phase two, phase three. One other explanation people give for Eroom's law, and of course there can be multiple things going on, is the change in regulatory standards for new drugs. The big things that people point to are, in 1962, there was this amendment called the Kefauver-Harris amendment.

In the U.S., before 1962, drug companies were required to do safety testing before they could sell new drugs. But in 1962, there was a requirement to test a new drug's efficacy as well, and that meant you had to do various trials. Randomized control trials kind of developed in the 1930s and 40s, but it was in 1962 when this became an actual requirement.

That increased the standards of evidence and it meant that people did these different phases of trials, which took a lot longer than before.

So when I hear that, my first thought would be, can we see 1962 having a step change on the graph? If you look at the graph, which we'll show in the episode if you're watching it, there isn't an obvious step change in 1962. But I think this isn't necessarily evidence against it because of how long it takes to develop new drugs. Since it takes many years to do research to develop a new drug and get it approved, you'd probably see a gradual change even if these new regulations had an effect.

I think the other- the maybe flip side of this is that a lot of people would say, this was good. This was good to have this amendment to have more efficacy testing beforehand. And maybe it gets more expensive, but that's kind of the point. We want to get rid of the drugs that wouldn't pass that threshold. And we want to know whether drugs are effective before we give them to people.

Jacob Trefethen:
I would not be surprised if that's part of it. I mean, basically, another way to look at that is how much of total R&D expense ends up happening in phase threes that are essentially about confirming an effect before approval? The answer is quite a lot. I mean, I don't have an exact number. I've often heard the number a quarter of R&D expense. If that does happen before an approval in almost every case, then sure enough, you would expect that the dollar goes up per approval. Yeah, so I think that's probably right.

Then there's another question, though, of, you know, there's sort of a deeper epistemological question about like, How do you know whether something works? And would a drug get very far? Even in the absence of as stringent an FDA, doctors still want to know something works before they recommend it to patients. So you still have to somehow generate that knowledge.

So let's say the FDA only required safety data that was robust and glimmers of efficacy rather than confirmed efficacy. Would you still see Eroom's law? I would guess yes. I think there's a demand for knowledge and the benefits aren't obvious enough unless you... Prove them out when the cases are marginal. In some cases, the benefits are obvious. The drug works so well and someone would die otherwise, like in the penicillin case. But most drugs aren't like that.

Saloni Dattani:
Yeah, I think that's right. I think the other question I have when I think of things like that is like, has anyone actually studied how much of an effect these amendments have had? Maybe you can do a comparison between the US and other countries and see how much did R&D spending change over time in the US compared to other countries where that regulation wasn't in place, or maybe it was introduced later on. This is what some people have done.

One example is this paper comparing the UK and the US. Both of them had drug regulation, but they were introduced at different times. In the UK, for example, you had safety research that was doing before a drug was approved. But the efficacy testing requirement only came into place in 1971 instead of 1962, which was the case in the US. So maybe you can compare the trend lines in the US and the UK and see how different they were. So this is essentially a differences-in-differences, I guess, approach to seeing whether there was more of a slowdown in the US than in the UK. This one paper that I was looking at that looked at this, basically did find this. The trend line declined much faster in the U.S. versus in the U.K. Drugs per R&D dollar declined about sixfold in the U.S. versus threefold in the U.K. during that time.

I think it's complicated as well because I think the U.S. regulation also affects the U.K. market of drugs. There's a lot of American pharmaceutical firms that operate in the U.K. and could get their drugs approved there. Then also the U.K. firms might be facing a tougher environment when they're marketing in the U.S. Then also, it's just two countries. Like, what if other things happened in both countries around the same time?

How about this explanation that I often hear from people, which is that ideas are getting harder to find, but the low-hanging fruit has already been picked. This is kind of like the Better Than the Beatles problem, but it's not exactly. It's more that we're depleting the universe of drugs that we can discover.

Jacob Trefethen:
You know, I have mixed feelings about this one. I think in some sense it's obviously true. Once you discover knowledge, it's forever-ish until the library gets burned down.

Saloni Dattani:
Well, as long as you do store it. Maybe some people just keep forgetting and they're not writing things down publicly.

Jacob Trefethen:
Indeed, indeed. But, you know, we have patents, and those patents are public. It's beautiful. We have a store of knowledge. There was just so much more knowledge in that database than there was 100 years ago. You know, let's just talk about knowledge generally rather than drugs, given we're talking about the low-hanging fruit thing. Once you discover that DNA has a double helix structure and that it contains information, you're not going to discover that a second time. And it is an awesome discovery.

So in some sense, it's obviously true that you have low-hanging fruit that has been plucked. Now, in another sense, I think the upside for medicine from where we currently are is still so large that I just don't think we're that far into medicine in some sense. It's like, you know, I had a stomach bug yesterday. We had to stop this recording and delay it a day because of some dodgy food I ate at a book launch. Okay, that's sort of embarrassing. We're still in that situation.

Then more seriously, we are nowhere on some of these big diseases. We're not very far on Alzheimer's. We are not very far on endometriosis. So there's still a lot left to go. And do I think we have some unlocks coming up ahead of us? Definitely. I don't emotionally feel limited in medicine by low-hanging fruit being plucked. Whereas emotionally, I do feel limited in physics by low-hanging fruit being plucked, if I'm being honest.

Saloni Dattani:
I think I agree with you. I have the same feeling where I sort of think, yeah, there's like a trivial way that that's true, but there's just such a huge number of potential diseases that you could be treating. We don't have drugs for a lot of them. Again, like one example of this is rare diseases. There are thousands of rare diseases. Most of them don't have any drugs approved. If it was just about the number of discoveries that you could make, then there are so many things that are left to discover.

I think the other thing that I find confusing about this explanation is that I always sort of think, aren't we getting better at making new discoveries? A lot of our technologies have been improving over time, and some of these new technologies help us to discover new drugs. So I'm sort of thinking, if we're still pretty early on in this understanding of biology or drug development, and our tools are improving a lot, then there should be a lot left to discover, and we should be getting better at it.

Jacob Trefethen:
So why does Eroom's law exist?!

Saloni Dattani:
This is why I prefer the better than the Beatles explanation, but I agree that there are probably multiple things going on. There are a few more explanations that people have, and I think we can just briefly go through them. One of them is that there's more of a trend towards caution, not just from a regulatory perspective, but pharmaceutical companies and doctors are more interested in making sure that drugs are safe. So they're more interested in putting an investment into doing larger clinical trials so that they can prove the safety of new drugs, and so that they don't lose out on money or goodwill if those drugs later turn out to be unsafe.

Jacob Trefethen:
I mean, that's definitely true. Yeah, I totally, that's absolutely true. I mean, people, we've done some episodes on medical history. You have doctors like injecting stuff into their, well, I wouldn't say where, to prove against some other scientists, I knew this caused gonorrhea. You know, you have people who are out there self-administering all sorts of nonsense and generating quite useful evidence from that.

And then the darker side is that they're administering things without informed consent from kids. So there's a dark history of medicine. I'm glad we're past. So some of the caution is rather nice. For example, people have to consent. So that's pretty good. But it's definitely descriptively true. Yeah, I agree.

Saloni Dattani:
There are some examples that I'm thinking of... lobotomies, for example. I think we wouldn't do that today. Did you know that in the past, I think maybe in the 1940s, there was a trend of using, in order to treat psychosis, psychotic episodes, some doctors used insulin coma therapy. They would basically inject patients with insulin so that they fell into a coma. The idea was that this would rid them of the excitement in their brain, which...

Jacob Trefethen:
Oh my god, that's horrible.

Saloni Dattani:
It's kind of crazy and dangerous. Then we also talked about, at one point, how people used malaria infections to treat syphilis. Again, we wouldn't do that today. Thankfully, we have much better drugs available.

Jacob Trefethen:
Didn't we talk about the cholera guy who swallowed some cholera to prove something?

Saloni Dattani:
Was that Barry Marshall? He drunk a goop broth of this bacterium to test whether it caused stomach ulcers.

Jacob Trefethen:
He was like, I think this is going to cause stomach ulcers. Everyone's like, I don't believe you. He's like, I'll prove it by giving myself a stomach ulcer.

Saloni Dattani:
You know, I looked into this story, as I often do, and I found out that it was very misleading.

Jacob Trefethen:
Darn it.

Saloni Dattani:
It's very annoying because it's a great story. But what I found was that he didn't actually develop stomach ulcers as a result of this. He just developed indigestion. Also, that did not convince people.

Jacob Trefethen:
He's like, I'm very indigested. And they're like, okay, you did drink some goopy broth.

Saloni Dattani:
Why did you do that? But thankfully there was lots of other research going on by him and by other researchers that did prove decisively, I think, that this bacteria caused stomach ulcers and in some cases stomach cancer as well.

Jacob Trefethen:
So what I'm thinking of about cholera is Max von Pettenkofer.

Saloni Dattani:
Who is that? Why does this German guy have an Irish accent?

Jacob Trefethen:
Max von Pettenkofer, in 1892, drank a culture of cholera bacteria to prove that the bacteria alone couldn't cause cholera.

Saloni Dattani:
Oh!

Jacob Trefethen:
And he survived and thought that he'd proved his point. But he really did have a live culture Vibrio cholerae, which, spoiler, does cause cholera. And he did develop severe, or he developed diarrhea, but did not develop severe cholera. But basically, that was a punch in the face to Koch, who had thought that bacterium was sufficient to cause cholera.

Saloni Dattani:
Yeah. What do you think the explanation was there? Do you think he was, I don't know.

Jacob Trefethen:
He had guts of steel. He could not get severe cholera because his guts were too good.

Saloni Dattani:
I hope that was true. But it just goes to show these experiments where people just give themselves a disease, I'm sort of like, well, you know, there could be other things going on. You could have guts of steel.

Jacob Trefethen:
Yes, or you could have been infected with cholera before and be immune. I mean, there's many things that could have happened.

Saloni Dattani:
Right.

Jacob Trefethen:
That said, I do think these kind of like N of one studies can be surprisingly revealing. And my personal ethics is that people should be allowed to do more to themselves than others. So, you know, there was that guy who got bitten by snakes to try and create antivenoms.

Saloni Dattani:
Oh yeah.

Jacob Trefethen:
That's, in my personal ethics, kind of awesome.

Saloni Dattani:
Yeah, that was pretty epic.

Jacob Trefethen:
That was pretty epic. So we have some epic people who are still alive in the 21st century.

Saloni Dattani:
There was also that guy who took 300 COVID vaccinations.

Jacob Trefethen:
Yes, no, I think it was 217 is my memory. The German patient, yeah. And he didn't experience side effects. I was like, okay, that seems impossible.

Saloni Dattani:
I want to be that guy because I feel like there were at some point, weren't there places where you could get a free Krispy Kreme or something?

Jacob Trefethen:
Oh, that explains it. So he had 217 Krispy Kremes.

Saloni Dattani:
Why not?

Jacob Trefethen:
They're like, this guy's back again. Krispy Kreme headquarters is like, we've got a problem. We've got a problem. We have a big cost center in Lundberg in Germany.

One thing about low-hanging fruit, if you look at it at the disease level, it's kind of interesting. There's something going on there where... You know, in some sense, it was easier to make antibiotics against bacteria than it is to cure Alzheimer's. It's easier to learn about what kills bacteria, famously in a dish, whereas you can't learn what stops Alzheimer's as easily because it's harder to observe, all sorts of other reasons. So I think there is some aspect of, I'm not surprised that Fleming didn't cure Alzheimer's. Do you know what I mean? There was more knowledge to be gained in cell culture in anti-bacterials.

Saloni Dattani:
I think that's right. There's a point that relates to our better than the Beatles problem, where I think that in the past, a lot of discoveries were also easier to see and to detect. Like antibiotics, for example, or the fact that you can just get rid of an infection by applying a new antibiotic. Or like if you think of the smallpox vaccines or many vaccines that were made in the 19th century, for example, where you don't need to run randomized control trials necessarily to know that they're effective because their effect is so large that they essentially just wipe out the disease in some place.

Over time, those large effects, we're seeing like the reason that we need larger clinical trials is because many of the new drugs have smaller effects that you need to actually do randomization and have a large sample size to detect them.

Jacob Trefethen:
This is the way out of Eroom's law. Do you want me to save that until the end of the episode or reveal it now?

Saloni Dattani:
Oh, reveal it now. What are you going to reveal?

Jacob Trefethen:
Only go after large effects. I'm completely serious.

Saloni Dattani:
Simple.

Jacob Trefethen:
I'm completely serious. People will think I'm joking. But if you have a drug that really works, it is not that hard to detect. The thing that's hard to detect is these 10 percenters. So, you know, that is the way out. If you think AI might change medicine, that is how. It's going to be drugs that are way better, way, way better. Easy, easy, easy to detect and easy to detect quickly. That is the way out.

Saloni Dattani:
I agree, but I also feel like, I don't know, that the modest effects are sometimes important, and sometimes you give them as combination treatments, and as a combination, they're pretty effective. Like with cancer drugs, for example, a lot of individual cancer drugs don't increase survival by that much. But if you give a bunch of them as a regimen, they do. The same is true for when we did our episode on HIV and antiretroviral treatment. Each of the individual drugs didn't lengthen survival by that much. But if you have multiple of them, then boom, you can massively crush the disease.

Jacob Trefethen:
True. Let me keep taking my side though, which is, okay, on cancer drugs, some of them are so good. You know, Gleevec, it got approved before a phase three. You can look at that effect on a graph with no statistics and you're like, oh yeah, that definitely worked. Okay, that obviously worked. So, you know, the reason I should be mocked for this comment, of course, is that, well, yeah, well, it's not actually easy. It's actually quite odd. So that's one of the issues. But I do think it's the way out. It is the way out. You just need, it's not, people get caught up in my, just be better, genuinely.

Saloni Dattani:
Just be better, guys. So, we've talked about Eroom's law, which is an exponential declining trend line. My question for you is, what is your favorite exponential trend over time, aside from Moore's law or Eroom's law?

Jacob Trefethen:
You first.

Saloni Dattani:
My favorite one is the decline in genome sequencing costs. This is not actually an answer because it's not an exponential decline. It's more than exponential that the costs of sequencing a genome have declined so fast that they've been faster than Moore's law.

Jacob Trefethen:
Yeah, Moore's law is kind of pathetic, isn't it?

Saloni Dattani:
Yeah, I mean, meet genetics.

Jacob Trefethen:
Meet genetics. That's a great one.

Saloni Dattani:
In 2001, when the first human genome was sequenced, it cost about $100 million to sequence that one genome. And a few years ago, the cost was about $500. So that is a massive, massive decline. Well done to everyone involved in that.

Jacob Trefethen:
Yeah, that's awesome. Props to Illumina, props to everyone else who...

Saloni Dattani:
You fucking did it.

Jacob Trefethen:
I don't think I've ever heard you swear. Wow. Okay, if we reverse Eroom's law, we get another Saloni swear.

Saloni Dattani:
Okay, so what's your answer? What's your favorite exponential trend?

Jacob Trefethen:
Real wages. Real wages.

Saloni Dattani:
Real wages.

Jacob Trefethen:
I'm going to get absolutely hated on by all the economists who follow the trend lines more specifically, but real wages, roughly speaking, you could map to an exponential that have been going up over time. I think that means that people get to, you know, do more of what they want to do with their lives, you know, pretty cool.

Saloni Dattani:
Is this similar to just the GDP per capita chart over time?

Jacob Trefethen:
That's true, but GDP per capita involves accrual of income to capital, which, you know, that's wonderful too. I don't mean to be rude to capital. Yes, capital is very good. But I think at the end of the day, I'm a little bit more of a people guy. Okay, I mean, at the tail end of explanations for Eroom's law, there's one I've heard from my elitist friends. But they say, actually, there's only a few good people, Maurice Hillemans, you know, a legend. This one guy was head of an R&D lab that developed 40 vaccines. Okay, so you guys better get back to work.

Saloni Dattani:
I mean that's a pretty astonishing record.

Jacob Trefethen:
Yeah, you have to admit. So the elitist theory would say, because there's only a few people who really contribute to drug development, therefore, as you add more costs at the margin, you're adding extra people working in drug development who aren't actually helping that much. What do you think?

Saloni Dattani:
Right, they're just losers.

Jacob Trefethen:
They're just losers and freaks.

Saloni Dattani:
So is the idea that we've just been throwing money at stuff that doesn't work and we haven't increased the number of extremely exceptional individuals working in this field?

Jacob Trefethen:
That's what people would say, yeah.

Saloni Dattani:
And that's why it hasn't... Yeah, okay. Hmm. I don't know. I don't know what I think of that.

Jacob Trefethen:
I think it's false.

Saloni Dattani:
You think it's false?

Jacob Trefethen:
I do think it's false.

Saloni Dattani:
Why do you think it's false?

Jacob Trefethen:
I think that, so there may be some effect that I, you know, I'm probably biased towards things that, explanations that make me feel good, and elitism doesn't make me feel good. That said, it's hard to explain in that worldview why there aren't more people who are really good who've been discovered, you know? So, like, what's happening with those people? If you have some understanding that there's a Maurice Hilleman every million people. Well, we got a lot more people than we used to have, and they got educated, and they've had more nutrition in their early years. So I'm like, there should be a ton of Maurice Hillemans right now.

So I don't really think it quite gets at the problem, even in its own terms. Then in addition, I'm like, no, actually, getting a drug over the finish line is a team sport that takes a ton of people, and it's a little bit. Okay, yeah, it's actually not all about one person. You have to have, for example, participants in research. So even outside of the people who are doing the inventing, but that's a whole nother kettle of fish.

Saloni Dattani:
With Maurice Hilleman specifically, so I've been reading a biography about him, and what is really fascinating about his story is that he almost died three times before he became a scientist. And it's sort of like...

Jacob Trefethen:
What the f- What?

Saloni Dattani:
Yeah, yeah! As a child, he got into near-fatal accidents several times, which is pretty scary, but it's sort of this thing that I'm thinking, which is, we have a lot more people who have much lower mortality risks than they used to. We should be able to find them and educate them, and they probably have more opportunities now to work in drug development than they did before. Who are the other Maurice Hillemans that, you know, did survive?

Jacob Trefethen:
Yes. I mean, the NIH is so big now. It's $50 billion a year. It's supporting so many PhD students. It's supporting so many people. Of course, there are tons of people still overlooked by the system. But you would expect there to be even more overlooked people in history than there are now. So I don't know. It doesn't quite add up to me.

Saloni Dattani:
So, let's talk about the last decade or so, because in the last decade or so, this trend line seems to have slowed down, and the overall long-run decline seems to have plateaued, and the average spending per drug approved has not been getting more expensive since about 2010.

This is kind of interesting because, well, one, it tells us that it's not a law, it's not inevitable for things to get harder and harder, but that might just be because of how we're measuring it, or maybe things are happening differently than in the past than the average, or the types of drugs that are being studied are different than before. I wonder what you think about this. When you've heard about this, do you think it's true? Also, do you have any ideas about why it might be happening?

Jacob Trefethen:
I have not looked deeply into this, whereas I suspect you have looked more deeply into this because I've seen graphs that... Did you make the graph that I've seen on this?

Saloni Dattani:
Probably.

Jacob Trefethen:
Okay, so I'll give my made-up one to begin with, which is I'm always nervous around there's statistical artefacts for this kind of thing. With something that's a really big trend, then it's always fun to be like, oh my gosh, it's reversing. So you've got to be careful before you really call it. And you get a lot of noise year to year. That said, I could come up with stories of why I wouldn't be as surprised. I could give one or you could just tell me the actual answers. Yeah.

Saloni Dattani:
I think there's something to that in that I would be cautious about over-interpreting this yet because there are kind of points in the past where you might have seen what looks like a slowdown as well. But I think if you do a moving average for something like that, you will still see the slowdown. So I sort of, I think it's true. Given that, what would you guess are the explanations?

Jacob Trefethen:
My best guess of that is that it's to do with rare diseases.

Saloni Dattani:
I basically agree with you. I think what's difficult about this as well is that there hasn't been that much research on this question. I think that the research that has been done so far does make me think that's probably the reason. If you look at the number of drug approvals per year, that has increased since around 2010, kind of steadily. A lot of those new drug approvals come from orphan drugs, which are for rare diseases. That actually makes up like roughly half of the drugs approved per year now. So that's quite big.

I think it's generally cheaper to run clinical trials and to develop drugs for rare diseases because you need smaller clinical trials to show their effect. The approval standards are a bit different. And that there are a lot of rare diseases that are caused by a single gene mutation. And you sort of have a much clearer understanding of the biological cause and how to go after it than you do with a lot of other diseases.

So our ability to tackle rare diseases has improved over time, but also there are these incentives and various reasons that pharmaceutical companies are going after them and investing in rare disease drugs.

The other thing that I would say is that if you kind of break down why there's been this slowdown or why the overall trendline has changed, like is it because there are more drugs approved or is it because spending per drug has changed? I think the answer is that drug success rates have increased in clinical trials.

Jacob Trefethen:
I would imagine spending has increased still as well.

Saloni Dattani:
Right. So I think if you put all of that together, I think that the rare disease explanation is probably right. But also I think maybe we should wait a few more years to be really confident about that.

Jacob Trefethen:
That's exciting, though.

Saloni Dattani:
It is exciting, but it doesn't really mean that the overall trend has been reversed, right? We're sort of adding this new category of diseases that people are developing drugs for, rather than making it cheaper to develop new drugs in general. It's good news for rare diseases, but it doesn't mean that we've actually reversed the phenomenon itself. Given the overall trend of Eroom's law, do you think that AI is going to slow down or reverse it?

Jacob Trefethen:
Here is how AI could reverse Eroom's law. At the time that we are recording this episode, there has been demonstrated progress or demonstrated cases of frontier AI systems producing new knowledge, in particular in mathematics. We're recording this at the end of October, 2026. There has been less demonstrated cases of frontier AI systems producing new knowledge in other disciplines. I would say that my expectations are that there will be continued progress in disciplines where verification is easy. One of the things about mathematics is that if you provide a proof or a disproof, you can in some sense verify whether it's a real one that works quickly.

The trouble with drug development is that if you write down the chemical formula for a molecule and the patient population it can most benefit, then to verify that that works, if you are taking this lens, will take 10 years of clinical trials. So the way that you could visualize AI reversing Eroom's law, in my opinion, is it's coming up with the help of scientists with molecules and other treatments that have very large effect sizes, where you can verify in a in vitro model, in a matter of months or weeks, that they are quite likely to be real effects. So in those cases, I think the game is on. The game is on.

If you can have a frontier AI system in discussion with leading scientists in an area that produces a method or protocol written in English that any, you know, of hundreds of labs could carry out in a period that takes less than a few months, which gives you good verification signal on whether that molecule will do what you're hoping. That's wonderful. Now, we can already point to the ways that's going to be tough. There are not always in vitro assays that people are confident will give you that signal.

Saloni Dattani:
Right. We often don't know the biology of the disease well enough to know how to test it in the lab.

Jacob Trefethen:
What do you think of that general picture, though? Does that general picture give you any optimism for reversing Eroom's law?

Saloni Dattani:
I think it does, but I would also say that it reduces the timeline that it takes to test things in clinical trials. If we do have drugs that have large effects, you'll hopefully be able to test them and see their effects much sooner. You might get these phase two trials that are just so big that we just get drugs approved based on that. Or the trials themselves take much shorter time because the effect is so visible early on. So I think that gives me some hope.

The other thing that gives me some hope is that maybe we will find better biomarkers and surrogate endpoints that are faster to test for, and that we won't necessarily have to wait for the entire disease to develop in clinical trials. So we'll be able to get the results faster.

But the flip side to this, I think, is that even to figure out which biomarkers are causal or which surrogate endpoints are causal, you still have to do the long clinical trials that test how those surrogate endpoints are related to the disease. So you still do need to run the long trials at some point. It's just that once you've validated those biomarkers or endpoints, you can then get shorter trials after that.

I would like to plug something that I developed to explore this, which is a game which I called the Clinical Trial Bottlenecks Game.

Jacob Trefethen:
Wait, you were the person who made the clinical trials bottlenecks game? My favorite PC game?

Saloni Dattani:
Yes! That was your favorite game?

Jacob Trefethen:
It changed my life. Well, actually, Toy Story 2 on PlayStation was probably my favorite.

Saloni Dattani:
There was a Toy Story game? I didn't even know that.

Jacob Trefethen:
Wow, you really missed out. You got to play as Buzz Lightyear.

Saloni Dattani:
Well, in this game that I developed, you get to play a little boat in a champagne bottle.

Jacob Trefethen:
What?!

Saloni Dattani:
Can you beat that?

Jacob Trefethen:
Take that, Buzz Lightyear.

Saloni Dattani:
So the reason that I'm plugging this game is because it was essentially on this question, which is how fast can you speed up the clinical trial process by making changes to the design or by improving the effects of new drugs or vaccines? I basically show in various steps how you might actually expect that to change the system overall. So I would recommend playing this game.

Jacob Trefethen:
How do I find it?

Saloni Dattani:
You can just Google search the clinical trial bottlenecks game and you'll find it. So we talked about Eroom's law, and then we talked about Eroom's law slowing down and maybe plateauing in the last 15 years or so. And that that slowdown is probably the result of more of a focus on rare disease drugs, which are generally cheaper to develop. I think we probably should still wait a little bit to be confident in that and to see whether this trend really has stagnated and whether it might reverse. Who knows? I think our conclusion on whether AI will reverse or slow down the trend even more is maybe. Do you have any reasons to think no? Or should we just go with maybe?

Jacob Trefethen:
Did that help?

Saloni Dattani:
That is exactly the answer that I was looking for.

Jacob Trefethen:
I think that my closest words of caution would be that listeners whose mental models have been more formed in the presence of Moore's law in technology may have the wrong intuitions for some of drug development, and I would advocate that they jostle up their neurons a little. That said, does that mean that the trend that we have been on is definitely going to be disastrous for all rest of time? No. And that I do think that you can well be an optimist on some of these things. I think that we have a sense of how to reverse these things, but what is needed is quite difficult.

Saloni Dattani:
I would add that I do think that clinical trials are still a bottleneck to medical progress. And that despite how much progress you can make on developing new drugs, it still is very hard to set up a clinical trial and get it running and operate it and test the effects of a new drug or vaccine. It still takes usually multiple years, and that is still getting harder over time because of the existing drugs that we've developed. I think that overall trend is going to be pretty hard to reverse. I think it's going to be more difficult than people assume, and that we should still try to make more reforms to that process as well as to drug discovery.

Jacob Trefethen:
Sign me up. Sign me up. That's what's going to choke us. I agree. For listeners who want more details on bottlenecks to medical progress in general, you can go back through our archive and search our five-hour episode last year on what does AI progress mean for medical progress, where we go into some of these in more detail. And I'm sure each of us, our thinking is evolving every month on these topics, but that's a good start. Okay, that brings us to the end of the episode. Saloni, what are some of your takeaways about Eroom's law?

Saloni Dattani:
My biggest takeaway is that people have known about this trend of drug development and productivity reducing over time. They've known about this trend for a really long time, since at least the 1960s and 70s. And it's still continued after that for many more decades. So I find that a little bit depressing to think about all researchers who discovered or noticed this trend, they tried to figure out why it was happening, they tried to reverse it, and they mostly failed to do so for decades. It still continued after that.

I'm going to start with that pessimistic note. That makes me wonder, will we fail too? If we try to figure out the causes of Eroom's law or try to reverse it, will we continue to fail just like they did? Who knows? Hopefully not. Maybe the rare disease drug trend reversal is real, and maybe we can reverse it further. But I don't know. There's a bit of caution in me because of that.

Jacob Trefethen:
That makes sense. And we should pinky swear on air right now. You, me, and the listeners, we're going to reverse Eroom's law.

Saloni Dattani:
How do we...?

Jacob Trefethen:
Oh, we can't think of it, so actually we're not bound by that. Okay, next. I learned from you that it's hard to know even basic statistics about what's going on in the world. So there's lots of uncertainty about how much is actually spent on R&D per drug to even come up with this statistic. You know, I hadn't appreciated how wide the uncertainty on the spending side was. I'd had some sense of it. Quite astounding, even though it's measured in dollars, which people care about a lot.

Saloni Dattani:
That, I think, is true. I think what does still give me confidence in the overall trendline is just the fact that it's exponential. If that wasn't the case, maybe my reaction would be, is this real?

Jacob Trefethen:
Yeah, and that it's so large. I mean, my goodness, 50x.

Saloni Dattani:
Yep.

Jacob Trefethen:
Okay, another thing I came in with but just still can't beat the drum on enough is inverse square root, inverse square root. This is a deep fact about the generation of knowledge detecting half the treatment effect size, you need 2x the sample. No, 4x the sample, I said it wrong. You need to half the treatment size, you need 4x the sample. A third, you need 9x the sample. A quarter, you need 16x the sample. And that is your biggest driver of cost of generating knowledge, whether that is in clinical trials or other forms of, you know, randomized control trials and other forms of knowledge generation.

Saloni Dattani:
Just to go over that again, the reason that this inverse square root matters is that it basically helps us to understand why it's becoming harder and harder to test the effects of new drugs. Once you have existing drugs that have reduced the risks of severe disease or death, for example, then seeing an effect of a new drug beyond that becomes exponentially harder. I actually really like this explanation because even though I thought this general idea was pretty convincing as an explanation for Eroom's law, it didn't actually occur to me that it was exponential until you actually spelled that out. I don't know. It kind of clicks in my head now.

Jacob Trefethen:
I had one more lesson from you, which was that Maurice Hilleman almost died multiple times before making 40 vaccines.

Saloni Dattani:
That was very disturbing to learn about. I read it in a biography about his life. Basically, he had multiple near fatal accidents. And I think one of them was an infection that almost killed him before he was an adult. That is very scary to imagine if that happened. I'm very glad it didn't. It makes me think the fact that we have reduced mortality so much over time, especially childhood mortality, means there are a lot more potential Maurice Hillemans out there right now who could be improving the world. And some of them might be listening to this podcast.

Jacob Trefethen:
Oh, I hope so. Okay, and what about reversing Eroom's law? Well, I think that, you know, I'll put my cards on the table that my day job is very related to that, as is yours. I think when you think about, can AI reverse the trend? I think that the punchline is somewhat simple, which is you need shockingly large effect sizes that are easy to detect and that pesky statistics can't get in the way of. And that is going to mean that you just need to be inventing way better drugs, which has all sorts of other implications about how you get there. That's the only way around it. There's no magic trick.

Saloni Dattani:
Yeah, pretty simple. Just go ahead and do better than before. But I think that makes me think, what about the better than the Beatles problem? If you do develop some really effective new drugs, it's going to be even harder to beat them in the future. So will you reverse the trend? Maybe only if you keep going after new diseases each time or something like that.

Jacob Trefethen:
I think that it's a great problem to have if the only driver left of Eroom's law is better than the Beatles because it means we already have great drugs. And there are still many disease categories that don't have great drugs at all. You know, Alzheimer's is one of them and there are plenty others. There's different cancers, there's different, you name your favorites. There's a lot still left to do to benefit people's health. Let's go.

Saloni Dattani:
I think the other thing that the AI progress makes me think of is I do agree that AI could have a large effect. If we do have much better drugs and maybe better biomarkers and surrogate endpoints and things like that, that could speed up clinical trials. But there's still going to be a lot more bottlenecks that remain. And that once you solve these problems, the problems will shift to those other obstacles instead. Things like how easy it is to recruit people into clinical trials and test new diseases, how hard it is to operate clinical trials, how many drugs or vaccines you can test at the same time. And we're still going to have a lot of problems that need solving. So if you're working on something like this, maybe you should think about working on clinical trials as well.

Jacob Trefethen:
I couldn't agree more. I think that it's actually sort of exciting. It feels like there's a bubbling wave of interest in clinical trial reform. If that is something that appeals to you as a listener, I think now is a great time to... Hitch yourself to that wagon or push that wagon forward or whatever the metaphor is because I'm sort of cautiously optimistic that in the coming few years, if people really go for this, we can harness some of the intellectual energy to make policy change too.

Saloni Dattani:
Yeah, just to elaborate on that, the idea behind clinical trial reform that I and other people are working on is that it might be possible to generate the evidence from these experiments much faster and much more cheaply without compromising on rigor or safety. We can make improvements to the process of running clinical trials or to the incentives to run them that could speed up the process.

Jacob Trefethen:
Which sounds like a great place to end. And as with all episodes, it's time to end with our howl.

Both:
Erooooooooooooooommmmmmmmm.