• Dedicating IP to the public domain doesn’t necessarily yield more public benefit than patenting
• Clinical trials for depression are hard to run

Treating depression with magnetic fields sounds, to most people first hearing about it, about as legitimate as treating depression with healing crystals.

But in fact this technique, called Transcranial Magnetic Stimulation (TMS), is well-established, notoriously safe, and fully FDA-approved.

The saga of how this came to be is worth studying for anyone interested in neurotechnology.

The Basics

TMS uses electromagnetic induction to create an electrical current in brain tissue. It does this from outside the skull. No surgery required - in fact no physical objects need touch the body at all.

(Source)

Different TMS coil shapes produce different electric fields in the brain:

(Source)

These days most clinics and researchers use a figure-eight coil, which has a focus in the center of the 8.

TMS induces electrical currents in the brain in short pulses. Single pulses can be used to fire neurons on-demand for neuroscience experiments or diagnostic purposes. But when used therapeutically, patients are typically stimulated with dozens of TMS pulses per second for 30+ minutes a day, 5+ days a week, for 5+ weeks. It’s notoriously noisy.

This is called repetitive-TMS therapy (rTMS, though often just referred to as TMS), and it’s FDA-approved to treat depression, OCD, smoking addiction, migraines, and depression-associated anxiety.

There are other magnetic neurostimulation techniques that use lower-intensity magnetic fields than TMS. But TMS is the most widely used magnetic stimulation method and the only one approved by the FDA.

Early History

People have been putting their heads in magnetic coils for a long time, though not nearly as long as they’ve been running electrical currents through them.

Jacques-Arsène d’Arsonval is apparently the first person to have tried putting people’s heads in a big magnetic field. In 1896 he reported “phosphenes and vertigo, and in some persons, syncope” from the adventure. This effect was independently discovered in Germany and the UK soon after, because apparently reinvention is easier than learning to read French. (Source)

And then in 1914, research on TMS-avant-la-lettre stopped for 30 years, for reasons I don’t know. The timing would seem to implicate WWI, though the last paper on TMS in 1914 was published by researchers at the far-afield University of Washington in Seattle, and all the early TMS authors lived until the 1930s except Thompson, the British rediscoverer, who died in 1916 unrelated to the war.

Though it made a few reappearances after 1946, TMS might have remained undeveloped were it not (reportedly) reinvented again by Prof. Anthony Barker. Barker’s big result (presented in a tiny a 500-word paper)¹ came in 1985, when he figured out how make fingers twitch with painless magnetic stimulation of the head.

This was a big deal. Scientists at the time could make fingers twitch by stimulating the fingers directly, and they could make fingers twitch with painful electric current run across the skull, but painless finger twitches from the head were a breakthrough. Modern TMS was born.

rTMS Takes Off

Even though there were preliminary results of TMS improving mood in conjunction with electrical stimulation in 1987, for the next decade after Barker’s result single-pulse TMS was explored as a research tool and diagnostic tool, not as a therapeutic tool.

But by 1991 Pascual-Leone et al. had shown that rTMS could interfere with visual perception and speech. And in 1994 they also found that the excitatory effects of rTMS in cortex could last for minutes after the stimulation stopped. This reminded researchers of the (presumed) effects of electroconvulsive therapy, an effective treatment for depression among other things. Concurrently, researchers had anecdotally noticed that subjects in vision or speech experiments who received TMS stimulation in their frontal cortex sometimes felt their mood change.

From 1994 to 1996 at least 6 clinical studies (not all controlled) reported TMS altering mood and depression. A lot of these studies’ results were probably flukes or the placebo effect given what we now know to be effective stimulation parameters for TMS, but that’s rarely stopped a field (and maybe shouldn’t).

In 1996 a big meeting of scientists was convened to lay out standards for rTMS use. Such standard-setting might in hindsight seem overcautious given that the only adverse effects of TMS that had been observed were scalp pain and very rarely seizures. But the nascent TMS field was probably keen to avoid the cultural immolation that electroconvulsive therapy had undergone in the 1960s. Prof. Mark George, a pioneer in the field, didn’t even patent the treatment due to fear it would damage its reputation. (Barker before him hadn’t patented his coil design either.) George now thinks this lack of IP protection decreased overall public benefit from TMS by reducing industry investment in the technology.

Despite tepid industry interest and the NIH not funding research into it, academic interest in rTMS grew quickly. Here you can see the numbers of published papers mentioning rTMS after the 1996 conference:

(Source)

Most of these papers aren’t clinical trials, but the interest is the point. In fact this plot underestimates how much work was being done on rTMS, since within a few years of the 1996 meeting, clinics and doctors were offering off-label TMS therapy. Supposedly some countries including Australia, Israel, and Canada approved the treatment around the same time as well, though I haven’t found official evidence of this. Also around this time, commercial vendors started selling rTMS devices to researchers, whereas before researchers had been modifying single-pulse TMS systems or building their own.

All of which is a bit depressing.

Why? Because even with the explosion of research after 1996, it would still take another 12 years for the field to develop rTMS into a legitimate therapy for anything. In fact, as discussed below, the field was lucky the FDA approved rTMS in 2008 at all. Not to mention that the approval was very narrow: only for depression in adults who hadn’t responded to one, but no more than one, antidepressant medication. The field would have to wait another decade for TMS to be approved for OCD and smoking cessation.

I can accept the decades of work up to 1996 as the price of serendipity. But as someone who cares about the therapeutic potential of future neurotechnologies, the decades required to develop reliable TMS therapies demand analysis.

What took so long?

One contributing factor to TMS’s slow path to approval was probably the field’s overwhelming focus on depression.

Depression is of course an important and worthy focus of therapeutic development. But developing depression treatments is particularly slow and hard compared to something like migraine, for which TMS would later be approved. Clinical measures of depression - surveys like the MADRS and HAMD or qualitative patient self-reports - are fickle and noisy. Placebo response rates in depression trials are large, from 30% to 50% — that’s a lot for a treatment to overcome. And it takes a long time to see whether a depression treatment is having a durable effect.

The focus on depression seems to have come from the hope that TMS could supersede the then-as-now-controversial ECT, and maybe because early TMS studies reported mood-altering effects. Some in the TMS field pointed out at the time that disorders associated with more focal neural activity like epilepsy or focal dystonia would be easier to treat than disorders with less-well-understood circuits like depression. But TMS still hasn’t been approved for such focal disorders, so no idea whether that’s true.

Despite my disillusion at a 12+-year clinical translation timeline, one could argue that rTMS for depression was actually developed pretty quickly given how long clinical trials typically take to run. It took about 3-4 years to perform and publish a clinical trial on rTMS (not including time to secure funding), which amounts to only about 4 “cycles” of experimentation between 1996 and 2008.

Clinical trials don’t have to be this slow. rTMS treatments of the kind patients were receiving in this era lasted around 4-6 weeks, with follow-up visits after maybe a few months. The rest of the 3-4 year timelines went into patient recruitment, IRB approval, writing and data analysis, and the publishing process (peer review plus time in press). And in fact publishing, which accounts for probably 6-12 months, doesn’t really count, since researchers share unpublished work with each other all the time. The first two are the big slowdowns, and I don’t think they’ve improved over time. They’ve probably worsened. This is infuriating. Anyway, I’m just venting. Back to the story.

Another reason for the slow translation of TMS, though by no means unique to TMS, is the large parameter space the field had to explore. At minimum the following can be varied:

• Coil shape (see picture at the top)
• Coil position on the head
• Pulse intensity (roughly the amount of current change during a pulse)
• Pulse pattern (continual vs. delivered in bursts, and frequency of pulses)
• Session duration
• Time between sessions
• Number of sessions
• Types of patients recruited
• Combination with other medications or therapy
• What the patient is doing or thinking about during the treatment

Plus things can get more complicated with current waveform and direction.

One thing that’s clear in reviews and meta-analyses from the time is that the easier a parameter was to vary, the more it was explored. For instance: almost every study used commercially available figure-8 coils. This isn’t surprising since changing the coil shape is really hard: you have to re-engineer the high-voltage electronics every time you do since the coil impedance will have changed. At the other extreme, pulse intensity and pattern are the easiest to change and were the most varied between studies. Nevertheless, to state the obvious: there’s no necessary correlation between importance of a parameter to therapeutic efficacy and how easy it is to vary.

Even among the easy-to-vary parameters, though, the field narrowed its exploration early on in two key ways.

One was the division of pulse patterns into “high” (>1 Hz) and “low” (<=1 Hz) frequency. Early observations from 1996 suggested that high-frequency rTMS was excitatory and low-frequency rTMS was inhibitory, and that excitation and inhibition could last for minutes after stimulation. At least this is what was observed in motor cortex, where excitation and inhibition could be measured via the impact on finger twitching. The field stuck to this dichotomy pretty strictly, and also frequently suggested that the excitatory and inhibitory effects were associated with long-term potentiation and depression. This conclusion is almost certainly oversimplified, but it seems to have been good enough.

The other early assumption that the field made was that the left dorsolateral prefrontal cortex (DLPFC) should be excited and the right DLPFC should be inhibited to treat depression. This idea goes all the way back to 1995 and was inspired by lesion and functional imaging work. The left DLPFC remains the main target of TMS for depression. It’s what the FDA approved in 2008.

In hindsight these two assumptions seem to have been spot on. The earliest TMS trials for depression from 1995 and 1996 used nearly exactly the same stimulation setup as would eventually be approved by the FDA in 2008. The main innovation in those 12 years was turning up the pulse intensity by ~40% and doing 4-6 weeks of treatment rather than 1. There were also big innovations in the quality of sham treatments during this time. (“Sham” is the medical device world term for placebo.)

It’s hard to know what to make of this, other than maybe wishing trials with sufficient power had been run sooner. Though Prof. George thinks running trials sooner would have been premature, pointing to what happened with DBS rushing from open-label trials to big trials that flopped.

Two big things the field did underrate at the time were the importance of coil positioning and what the patient thinks about during the treatment.

It’s harder than you might think to position a TMS coil over the right part of the brain, orient it correctly, and keep it in place for a half hour.² The way they used to find a patient’s DLPFC was by measuring 5 cm anterior to wherever TMS made the patient’s thumb twitch. But this isn’t where the PFC is in 1/3 of patients, not to mention the DLPFC. They then started using the 10–20 system, but we know from newer results that this is also pretty inaccurate. Even with modern computers and neuronavigation systems targeting is tricky, and for large trials one has to train technicians/clinicians to target correctly and carefully across lots of patients.

We also know now that what a patient does during stimulation can have a big effect on a TMS treatment. The approved TMS treatment for OCD involves a “Personalized Symptom Provocation” - patients are made to think about something that triggers their symptoms while their brain is stimulated. Between 1996 and 2008, as far as I can tell, patients could do whatever they wanted during TMS treatments: ruminate, read, sleep, etc.

What’s the opposite of a slam dunk?

In 2007, a company called Neuronetics was the first to apply to the FDA for approval of a TMS system for treatment of disease - specifically for approval of their NeuroStar system to treat treatment-resistant depression.

The double-blind, multi-site clinical trial they ran to support their application was the largest that had been run at that time. It took almost 2 years, starting in January 2004, to recruit all 325 patients. The IRBs from all 23 (!) sites involved in the study had to approve the trial.

But the massive investment was worth it, because when the trial finally concluded it…uh…still failed its primary endpoint and arguably all its secondary endpoints too.

And then the FDA approved NeuroStar anyway.

The FDA’s Advisory Committee did justifiably slam Neuronetics for statistical shenanigans when they presented their results. You can read the juicy details in the transcript of FDA deliberations on NeuroStar, Jan 2007. But I think the FDA’s decision was still for the best from a patient risk-benefit perspective. Though the evidence for benefits in the trial were weak, rTMS had (and still has) such a fantastic safety profile that approving NeuroStar and letting post-market studies sort out effectiveness seems preferable to me. And indeed post-market studies have borne this out: TMS works about as well as most depression treatments, which is not that well, but better than no treatment at all.

One thing about the Neuronetics trial that most confuses me is why they didn’t survey patients about what treatment they thought they got in order to test the blind. I have no idea why a trial wouldn’t do this, and doing so might have let them avoid critiques like this:

“36% of the active group compared to only 4% of the sham group complained of [head] pain… When [you adjust for patients feeling pain], p-values for the treatment effect became less significant. In particular, the p-value for the primary MADRS endpoint went from borderline significant to [really bad].” — FDA

To be fair, the FDA never sounds excited about anything. Even in their deliberations on the Pfizer COVID vaccine EUA in December 2020 — the most slam-dunk clinical trial in recent memory — their average tone never exceeded “fairly into it”, and only 17 out of the 22 committee members voted to approve.

But the bottom line is that the NeuroStar approval was pretty far from a slam dunk. Plenty of folks let the FDA know it after the fact, too. And another multi-center trial using very similar stimulation parameters showing TMS had no antidepressant effect was published right after the NeuroStar trial. (Though it was for patients who were also taking antidepressant drugs, whereas the NeuroStar trial was for patients not taking drugs.)

Nevertheless, by 2010, the American Psychiatric Association (APA) had included TMS in its Practice Guidelines. Insurance seems to have taken a little while to come around, but it was mostly covered by 2015, though even today it can apparently be hard to get insurance to cover it.

Recent Developments

This infographic tells the story after 2008 better than I can:

(Source)

TMS has gone on to be approved by the FDA for migraines, OCD, smoking cessation, and depression-associated anxiety. And TMS is regularly used off-label for a lot of things.

But TMS still has lots of room for growth. The more we can learn from neuroimaging about the cortical circuits involved in neuropsychiatric diseases, the more TMS protocols can be developed to target them. And personalized targeting is showing impressive results that might hugely reduce the treatment time and increase efficacy. If the treatment time can be brought down to just a few days, it’d be hard to argue that TMS shouldn’t become a first-line therapy for depression. It has far fewer known adverse effects than any antidepressant drug — the cost and time are what make it infeasible now.

Let’s just hope we can start measuring this growth in years, not decades.

Have feedback? Find a mistake? Please let me know!

• Similar technologies with different names can elicit wildly different cultural and regulatory reactions
• The number of papers published on a technology, or the age of its field of study, or their number of citations therein, tells you almost nothing about its value or effectiveness
• The US used to have a much stronger DIY-health/treat-yourself-at-home ethic
• The FDA cannot be predicted
• For many indications, if the user is asking whether it’s working, it’s not working

People have been putting electricity through their heads for a very long time. The umbrella term for this is transcranial electrical stimulation (tES).

Every discussion of tES must, by tradition, start with the anecdote that people in antiquity put electric fish on their heads to treat headaches. In the modern era, the most famous types of tES include electroconvulsive therapy (ECT), transcranial direct current stimulation (tDCS), and recently temporal interference (TI). But there are, and have been, dozens of varieties of tES differing in electrode placement,¹ waveform, intensity, and duration.

Early Days, ~1900

All due respect to electric fish, tES really got going in the late 19th century with the spread of battery technology. In the “golden age of electrotherapy” in the US around 1900, tES devices nearly identical to modern tDCS devices were “one of the most ubiquitous therapeutic items.” They were sold for anesthesia, nausea, rheumatism, “Female Weakness” (depression?), “Premature Decline in Man” (ED?), and many other ailments, fictional and not. More interesting was that they were sold as much to consumers as to doctors. Self-treatment was considered the norm in medicine until around WWI, and there were strong DIY and direct-to-consumer (D2C) markets for tES. (Source)

The existence of this “golden age” is a bit surprising given that cocaine and opiates were legal at the time. I’d have thought over-the-counter heroin and similarly potent drugs would obviate consumer demand for tES. Equally surprising is the golden age ending after 1910, since that was when the US government began cracking down on drugs.² tES remained unregulated and could have filled that void, but didn’t. Explanations include the rise of psychology, DIY healthcare giving way to the professionalization of medicine, and X-Rays stealing the spotlight.

ECT, 1930s+

tES stayed unpopular from 1910 until the 1930s when Electroconvulsive Therapy (ECT) was developed. ECT, also known as electroshock therapy, involves passing large currents through the skull in order to induce seizures. In the spirit of More Dakka, we’re talking 800 mA compared to the <10 mA in nearly all other forms of tES past and present.

In the early days of ECT the treatment was given to conscious, unanesthetized patients, which was obviously quite unpleasant. By the 1950s the field had gotten around to putting patients under general anesthesia during ECT, resulting in a calm, painless procedure. But the cultural damage was already done. Ken Kesey’s depiction of unanesthetized ECT in One Flew Over the Cuckoo’s Nest, and later the film depiction of the same, is often blamed for the huge decline in the use of ECT after the 1960s. (This is what ECT looks like these days.)

The PR failure of ECT is sad, because ECT is among the most effective treatments for depression, though not necessarily long-lasting. The most common side effect is transient memory loss. While it has slowly crept back into popularity over the years, one wonders why it wasn’t able to rebrand itself, especially given how many other tES techniques managed to do so.

Weak t, 1960s+

As ECT receded in the 1960s, lower-current methods began to advance again.

Electrosleep was the first. Electrosleep was the name of a tES method for insomnia where pulses of low currents (<25 mA, usually <10, and often <1) are passed across the scalp. Invented in 1902, it would have died with the other golden-age tES methods after 1910 had it not been kept alive by research in Russia and Europe.

After reemerging in the US in the 1960s, scientists concluded it wasn’t actually doing anything in the brain directly to cause sleep. It was just a relaxing, electric scalp massage. Due to this it was renamed Cranial Electrostimulation Therapy (CET). And then renamed Transcerebral Electrotherapy (TCET). And then renamed Cranial Electrotherapy Stimulation (CES). When I said I was confused why ECT couldn’t rebrand itself, this is why. (source)

There’s a whole saga of Electrosleep/CET/TCET/CES/scalp-massage-with-a-battery trying to get better treatment from the FDA, but it’s not worth going into unless you really want to get in the weeds of FDA device classification.

Also in the 1960s people started playing around (again) with non-pulsed, DC stimulation, but it fizzled out. And in the spirit of scalp-massage-with-a-battery, more work began to be done targeting cranial nerves for pain and headache relief. And by “work” I mean “scientists with the bravery of a bygone era sticking needles into their own skulls to test their ideas”. This line of research continued to the present day with devices like Cefaly, which is FDA cleared for migraine.

But really not much happened until 2000.

A Thousand Flowers Bloom, and the Return of DIY, 2000+

The modern resurgence of tES began with what were almost certainly two false conclusions.

In 1998 and 2000, two groups reported modulating cerebral excitability with the DC tES method what would eventually become known as tDCS. Based on what we know now about how little current gets through the skull and how tiny the currents they used were (<1 mA), there’s almost no chance they were directly changing cortical behavior.

Nevertheless, low-output (<10 mA maximum average current) tES began to rise again. New methods like HD-tDCS, transcranial Alternating Current Stimulation (tACS), transcranial Random Noise Stimulation (tRNS), and transcranial Sinusoidal Direct Current Stimulation (tSDCS) began popping up. The hottest new tES method as of writing is probably Temporal Interference (TI). Originally proposed by Soviet scientists in 1967, it was resurrected/rediscovered in 2017 as a way to potentially get current into the deep brain.

But the most interesting part of the low-output tES resurgence isn’t the methods themselves, but the DIY culture that sprang up around them.

Prof. Anna Wexler tells this story better than I can. But in short, in 2006 a few articles about tDCS make it into the popular press. From there it gets its first Wikipedia article. It’s on Shroomery by 2007. Then on rotten.com. By 2011 it’s got its own DIY subreddit.

D2C tDCS kits start being sold in 2012. By 2014 there were two low-output tES startups: Halo and Thync. Both are still around, and have been joined by many others. While a much smaller percentage of people self-treat with tES now than in 1900, the DIY/D2C tES movement remains strong.

Which raises the question: where is the FDA in all this?

Interlude: How medical devices are regulated in the US

• Since 1976, if you want to sell a medical device in the US, the FDA has to approve it.
• The FDA decides what counts as a medical device based on its “intended use”.
  • Teeeechnically, a medical device is any man-made artifact that is intended to impinge upon a human in any way (and isn’t a drug). (21 USC 321(h)(1))
  • But since that definition is absurdly broad, the FDA mostly just pays attention to products that explicitly claim to have a medical purpose, though implicit claims can count too.
    • Famously, a dumbbell is both a medical device and not, depending on the ad copy. Same with EEG.
    • Wellness aids are a notorious gray area. Safer to market your device as a sex toy.
• There are 3 classes of medical device.
  • Class I are low-risk, like tongue depressors. 47% of devices are Class I.
  • Class II are moderate-risk, like ventilators. 43% of devices are Class II.
  • Class III are high-risk, like pacemakers. 10% of devices are Class III.
• All new devices are automatically Class III until the FDA says otherwise.
  • You can petition for your device to be reclassified.
• The higher the class, the more work you have to do to get FDA approval.
  • Here’s a flowchart for the whole process.
  • Class III (“PMA”) approval is way harder than Class II, requiring years and millions of dollars. People work hard to get reclassified down.
• But, if your device is similar enough to an existing Class I or II (or sometimes III) device, you don’t have to go through the whole approval process again. You can just prove “substantial equivalence” to the existing device.
  • This is called a “510(k)”.
  • Protip: if you see a device that’s “FDA Approved”, it went through a PMA. If it’s “FDA Cleared”, it went through a 510(k).
• Devices are also regulated by other agencies, even if the FDA doesn’t consider yours a medical device.
  • The FTC takes action against “hazardous…products…without adequate disclosures”.
  • The CPSC takes action against “unreasonable risks of injury”, though not in the FDA’s remit.
  • The FCC regulates devices that emit RF signals.

The FDA has cleared a number of medium- and high- (>10 mA maximum average current) tES devices, like the Cefaly device mentioned above and ECT devices. But D2C and DIY low-output tES companies have so far avoided language that would lead the FDA to classify them as medical devices. The field has thought about what it would take to get devices FDA cleared or approved. And some of these companies have sought guidance from the FDA and even done clinical trials. But none have yet sought approval for big indications like depression.

Is this because they can’t demonstrate effectiveness? Because goals like cognitive enhancement aren’t considered therapeutic indications by the FDA? Because clinical trials cost too much and there’s likely not much chance of insurance reimbursement to pay them off? Probably a bit of all of these.

To date, neither the FDA nor FTC has made sweeping moves regarding tES devices. Just issued some guidance. Some expect a big reckoning to come for the D2C tES market after the FDA has some bandwidth post-COVID. The way this would probably happen is them saying, “Hey, we the FDA have decided that the way you’re describing these devices makes them medical devices. And we haven’t given you permission to sell them, so shut it down.”

But it’s not in the FDA’s jurisdiction to actually outlaw DIY tES, or any neurotech, even if they wanted to. In fact, it’s not really in any government agency’s. If the government wanted to treat DIY neurotech the way they treat DIY cannabis, they would need to pass legislation like the Controlled Substances Act. If someday soon companies start selling neurotech devices D2C that are, in some sense, as “potent” as cannabis…that’ll be an interesting day.

“What did we learn, Palmer?”

You may have noticed I’ve been avoiding the question of whether any of these methods work for anything.

Besides ECT, which obviously works for depression, schizophrenia, and other indications, the answer is that I don’t know what tES methods work for what purposes, and neither does anyone else, really. Even in 2021, most tES conferences still include a healthy number of papers asking, “Do these methods actually do anything?”

When a neurotechnology is a credence good, like a treatment for stroke rehabilitation, Alzheimer’s, or catatonia, it’s worth resolving such confusion over efficacy with careful study. And of course establishing safety is critical.

But many if not most neurotechnologies are not credence goods. They’re experience goods: goods for which if the user is asking whether it’s working, it’s not working. If you can’t detect an effect in yourself, you either aren’t experiencing one, or you’re not keeping track of your own mentation, mood, behavior, or performance enough for the technology to matter.

The inverse of course isn’t true. If a user says a technology is working, it might be working for reasons we don’t care about like the placebo effect or regression to the mean. And subjects’ desire to please researchers with self-reports should not be underrated. Considering how hard it is to perform tDCS treatments correctly on one’s self (despite their seeming simplicity), it would not be surprising if the entire DIY tDCS movement was fueled by the placebo effect.

Even though this has no real downside given how cheap and safe tDCS devices are, the popularity of DIY tES without undeniable efficacy is something anyone interested in neurotech should reflect on carefully.

Have feedback? Find a mistake? Please let me know!

• Technological arbitrage.
  • Giving people 10-year-old technology is valuable if they’re currently using 40-year-old technology.
• University lawyers often have unrealistic opinions about the IP their staff will generate.
• Scientists have an unreasonable affection for (their underlings’) suffering.
  • They won’t buy tools to make their (underlings’) lives easier — they want to buy tools to do things they currently can’t.

Maybe as a legacy of the Manhattan Project, the field of physics is capable of large-scale collaboration to build research tools for themselves. CERN, LIGO, NIF, Arecibo — the list goes on.

Other fields of science struggle to do the same.

A recent exception is the Neuropixels project started by Prof. Tim Harris while he was at Janelia.

Tim noticed that dozens of labs were recording electrical activity in the brains of live mice for their research projects, but they were doing so with subpar electrode probes that each lab (tried to) manufacture for itself. He decided that was bogus; led an effort to make a reliable, reusable, 10x-better-than-state-of-the-art probe; and made it available to researchers at-cost (~$1,000).

As of 2021, over 5000 Neuropixels have been shipped to labs around the world. Tim’s too modest to say Neuropixels has revolutionized microelectrode recording in animals, but just about everyone else you ask will say it has.

So, how did he do it?

Tim showed up to Janelia in 2008, having never read a single primary neuroscience paper. His colleagues’ probes at the time had about 64 sensors apiece, and everyone wanted more. He made some obvious improvements (his words) to his colleagues’ probes, but he knew he could do 10x better with the right technology.

Luckily the right technology wasn’t new: it was the same CMOS fabrication technology that goes into making modern computer chips. It’s just that neuroscience labs aren’t set up to fabricate electronics at the level of TSMC. But neuroscience didn’t need the fancy stuff. As Tim put it, he just needed to bring probes “from 40-year-old technology to 20-year-old technology.”

In 2013 he started hunting for a fabrication facility to build that technology. His opening offer to every one was “I don’t need a paper, I need a probe, and if you won’t manufacture them for everyone who wants one when we’re done, I’m going elsewhere.” Eventually he found IMEC in Belgium, who after some back-and-forth about the technical specifications said, “Sure, we’ll do it. That’ll be €5.5M.”

This was considerably more money than Tim had at his discretion, so he asked for some from his boss at Janelia, the legendary Prof. Gerry Rubin. Gerry did have that much money, but he told Tim, “If this project is as important as you think it is, you should be able to find a partner.” This wasn’t Gerry being withholding. It was that when you’re spending millions on a tool, you can only make one tool, so you need a strong, international consensus on what to build. And there’s no better way to get buy-in on a decision than…well…literally having people buy in.

Enter the lawyers

Two days after talking to Gerry, Tim heard that Christof Koch was leaving Caltech to go to the Allen Institute and that he wanted a better probe. Tim called Christof, and a few days later Allen signed on as a partner on the project. After some more phonecalls, Tim got Wellcome and researchers at UCL’s Gatsby interested.

Altogether that meant making a 5-way international multiparty contract between Janelia, Allen, UCL, Wellcome, and IMEC. Everyone understood IMEC would keep all the IP on the engineering and fabrication side, since they were doing all the work there. But Tim needed to make sure researchers would be free to publish papers about whatever they used Neuropixels to discover. Without that guarantee, the project was pointless. And he was insistent that “nobody gets to make money, and nobody gets to go first [once the probes were fabricated].”

At this point in the story, Tim gives all the credit to Janelia’s lawyers. We’ll have to use our imaginations about what went down, but the upshot is that thanks to “real champions” at the various institutions to curtail the “unrealistic views of academic lawyers and their opinions about the IP their faculty [would] generate,” all five parties eventually signed an agreement. The four academic partners agreed to pay IMEC to design probes that weren’t fully open-source, but were documented well enough for any researcher to use, and once completed they would be distributed by lottery to researchers, at-cost, with no IP claims on downstream discoveries.

The first 30 probes were shipped in 2018, five years after the project began. Each carries 960 sensors, of which a researcher can select any 384 to simultaneously record from. A 2.0 version of Neuropixels was released in April 2021, and the team is now working on a human-usable version.

Product design for scientists

Tim gives the architecture of Janelia a lot of credit for making Neuropixels happen. It’s designed for serendipitous meetings: all glass walls, a single good coffee pot, and a single cafeteria that only serves lunch for 90 minutes. He was submerged in people who needed the tool he was building, and every day he could ask people, “Well, what if we did this?” And, critically, he could tell just by walking around who was forward-thinking and who had their heads down.

The initial Neuropixels proposal was made with the backing of only two such forward-thinking Janelia researchers. They were two of the top researchers in the field, so their backing meant something. But still, Tim is clear that he had to overcome a lot of conservatism. If you don’t offer neuroscientists a big step forward, they’re not going to change what they’re doing. Their projects already contain enough scientific risk; there’s no desire for additional risk of equipment failure. Moreover, scientists have an unreasonable affection for suffering (or for their underlings’ suffering), so you have to prove to them they’ll be able to do things they couldn’t do before, rather than just making their (underlings’) lives easier.

Once Tim got a critical mass of support, he claims his main contribution was consensus building, i.e. keeping the neuroscientists from screwing up the engineering by constantly changing specifications. The neuroscientists would change their mind about what they wanted every week, and he condensed that into small alterations given to IMEC every 6 months.

His main tip for consensus building is that meetings (1) should never last more than 30 minutes, (2) should have an agenda of very specific questions, and (3) should have cookies provided by you.

Have feedback? Find a mistake? Please let me know!

Special thanks to Prof. Tim Harris for wasting an hour of his day talking to me, and to Prof. Nick Steinmetz for his email responsiveness.

If I’m missing your favorite market failure, please let me know.

Contents

Information Asymmetries

Knowledge monopolies

Scientists are incentivized to withhold as much information as possible.

Scientists are incentivized to, and often do, withhold as much information as possible about their innovations in their publications to maintain a monopoly over future innovations. This slows the overall progress of science.

For example, a chemist who synthesizes a new molecule will publish that they have done so in order to be rewarded for their work with a publication. But in the publication they will describe their synthesis method in as minimal detail as they can while still making it through peer review. This forces other scientists to invest time and effort to reproduce their work, giving them a head-start in developing the next, better synthesis.

This is not necessarily malicious. Effective communication of ideas requires effort that the scientist may rather spend doing something else. Most scientists will divulge details of their experiments when asked in personal communication, but their incentive is still to respond as slowly and cryptically as possible.

If there were a way to reward scientists proportional to the ease of replicating their results that would be better, but it would have to outweigh the reward of a monopoly on future publications.

Mathwashing

Scientists are incentivized to make their work as inscrutble as possible.

Scientists often use overly complex mathematical or statistical methods to obscure problems from reviewers. This works because most reviewers will not admit to not understanding details of work they are reviewing. The harder it is to spot an error, the less foolish you are for having failed to spot it.

Unreadable academic prose can serve a similar obfuscatory purpose, though this seems to be a bigger problem in the humanities.

Replication Crises

Scientists are rewarded more for false positives than for true negatives.

You already know about these, or if you don’t, read Stuart Ritchie’s book.

They tend not to be due to outright fraud, though that does happen.

Poor Credit Assignment

Credit is the primary reward for a scientist. Credit accrues into influence, which is valued in its own right, and is also convertible into funding.

Credit is given in science by priority: if you publish an idea or discovery first, you get the credit.

This system successfully incentivizes the creation of public goods, which is impressive. But it’s not perfect.

Scooping and duplicate work

Science wastes a lot of effort by not coordinating between groups.

When scientists don’t coordinate it can lead to duplicate work. If two groups are interested in the same project, ideally they would divide up the relevant work between their members, sharing the credit. This often doesn’t happen because groups typically only publicize what they’re working on after they’ve finished and published the work. This leads multiple groups to do nearly identical work simultaneously. The first group to publish, “scooping” the others, usually gets all or most of the credit. Even Darwin thought this was a bad system.

Obviously we want some work in science to be duplicated: the lack of duplicate work is a cause of the replication crises. But non-observational work like theory or method development gains almost nothing from groups scooping each other.

Why don’t scientists coordinate more? Partly because broadcasting what one is working on would let competitors free-ride on your ingenuity. They could even claim they were working on the same idea when they weren’t. Another reason is that it’s hard to evenly divide labor between groups. Plus the more authors are on a paper, the less credit each receives. If you publish before your competitor, you don’t have to share credit with them, though of course you might lose the race. At some career stages, having solo or majority credit for an idea matters a lot, like when working on a job-market paper. This can even prevent people from the same lab from collaborating with each other.

One way to prevent coordination failures are turf norms. Some fields like geology or parts of biology have norms about not working in areas (literally, for the geologists) that other people study. But forcing everyone to find a separate niche is inefficient: if everyone in a field thinks a particular idea is the most promising thing to work on, we want them all to be able to coordinate on doing so!

This paper suggests based on PDB submissions that duplicate work doesn’t have a huge impact on the careers of scientists themselves, despite the waste of scientific effort. The same authors went on to study the impacts of competition on research quality, again using PDB data, with structure resolution as a metric for quality. It’s a great idea, and they present a very detailed model. But PDB crystallography is also an unusual “minigame” in science where groups are all trying to produce the exact same thing. In most of science, when you get scooped you pivot (often in a direction you don’t really care about), or dress your work up to look unique, or try to tack more on for novelty. People almost never publish purely duplicate work, unless the projects are so contemporaneous that they make it into the same “release cycle”. I don’t know how one could quantify that kind of inefficiency.

One solution to this problem is working in public, but that opens up opportunities for flag-planting, and researchers often find working in public embarrassing since flaws aren’t hidden. Another option would be a trusted neutral party to whom everyone could tell what they’re working on (along with appropriate proof of investment in the idea) and groups duplicating each other’s work could be put in touch.

Author lists and contributions

A meritocracy is only as good as its ability to assign merit, and science is awful at this.

Every field has different conventions, but generally authors on scientific papers are listed from greatest to least contribution. The first author did the most work, and the last author (usually the principal investigator(s) supervising the project) did the least work.³ People who only helped a little don’t get authorship, but are thanked in an Acknowledgements section. First authorship matters more than any other position in the author list because papers are typically cited as “Lastname et al.” in text, so only the first author’s name is seen by many more readers.

This is not a great way to apportion credit. Partly because an ordering doesn’t tell you how much more work one author did than the next. But mainly because credit can’t be reduced to a scalar quantity. Contributions of different kinds aren’t comparable. What’s the exchange rate between hours writing prose and hours building apparatus? Even if one measured contribution purely in hours spent on specific project, how does one weight preparatory work done on a similar previous project, or an entire career of learning?

The market failure in all this is that a meritocracy is only as good as its ability to assign merit. Science is inefficient when talent is misjudged. This inefficiency is not hypothetical — every scientist has at least one colleague whose scientific skills are dwarfed by their ability to get their name on papers, and colleagues who are perennially undercredited.

Some authors now include a section detailing precisely what each author contributed to a paper. For example see the first footnote in this paper. This probably helps apportion credit in narrow groups of peers. Some authors take an absurdist approach and declare that their paper has 19 co-first authors. But I suspect as long as there is a top-line author list, most readers will apportion credit inaccurately. In this paper, even though the authors listed their names randomly, everyone in machine learning would agree that the first author has gained vastly more name recognition than the others.

Citation pile-ons

Scientists have little incentive to cite accurately or fairly.

What do you do when you’re writing a paper and need to cite an idea whose provenance you don’t know? You see whatever reference someone else cited and cite it too.

Sometimes this is the right citation, but often an idea is developed by multiple people across multiple publications. Citing all of them is tedious, so a winner-take-all reward accrues to just one of these publications. Often the “winning” publication is the one that coins a term rather than the discovery of an idea, or more often is the publication with the most famous author.

The worst case is when people cite references that don’t even contain the information they’re citing. Here’s a delightful paper on the subject. (HT Alexey Guzey)

There is a small incentive to finding the right citation: if you cite an older paper than the one readers are expecting, you look smart. But this rarely outweighs the ease of citing whom everyone else cited.

Public Goods

Work everyone agrees someone should do but which nobody does, because it’s inadequately rewarded in money or reputation.

• Better animal and in vitro models for disease (Too low credit : effort ratio for academics to work on, too hard to protect IP for industry to work on)
• “Unsexy” studies like exhaustive parameter sweeps, calibration curves, etc.
• Open-source scientific software (A lot still gets made, but undersupported given its importance)
• Open-access publication, at least pre-Sci-Hub
• Open-access dataset collection/curation, esp. highly paywalled ones like SciFinder and Reaxys
• Insufficiently profitable drugs
  • vaccines, at least pre-COVID
  • antimicrobials, at least until the Hyper-SARS Pandemic of 2028
  • therapies for rare diseases
• Shared fabrication facilities, MOSIS being an ideal example

Organizational inflexibility

Scientific career paths, organizational structures, and funding sources are too standardized.

It’s a bit suspicious that all PhD programs should take about the same amount of time, regardless of field. Or that it’s so hard to fund anyone who isn’t on the PhD-to-postdoc-to-prof path, like the high-level independent contributors common in the tech world.

Or that the entirety of a professor’s career should be planned years in advance to increase the odds of getting an NIH R01 grant. Or that universities should prevent scientists from hiring engineers to build critical infrastructure by setting low salary caps.

Someone invented the existing ways we do science, and we can invent better ways. And people are. The Overedge Catalog is a great collection of new types of research organizations breaking these molds.

Bureaucratic Bottlenecks

It takes too long to get permission and money.

“Drug lag” is the delay between discovery of a drug and its availability to patients. The largest contributor to drug lag is the time required to obtain regulatory approval to market the drug. The justification for regulatory control of drugs is that many drugs are credence goods, but the current system is certainly not at the Pareto frontier between minimizing drug lag and maintaining customer confidence in drugs.

Similar delays exist across biomedical science. Like delays for giving scientists the Investigational Device Exemptions (IDE) they need to do research on new medical devices. This contributes to the “1- to 3-year delays in the introduction of new device technologies into general clinical practice within the United States as compared with Europe.” Or delays for determining whether a new clinical trial protocol will convince the FDA of efficacy. And in general, as Fast Grants has shown, grantmaking is also much slower than it need be.

Changing from a pre-approval to an escrow-until-approval (HT Michael Sklar) or post-monitoring regulatory system could help. More explicit, quantitative regulatory standards could also help. E.g. rather than new clinical trial protocols being approved by qualitative judgement of the FDA’s statisticians, have the FDA state a false-positive rate cutoff that new protocols can prove they are below (in the absence of other issues).

Taxes

Administrative Burden

Scientists only spend 1/2 their time on active research.

A study of 11,167 principal investigators with active federal grants in 2018 found that only 55.7% of their time was spent on active research. The rest of it went to things like grant administration and satisfying Institutional Review Boards (IRBs) and other compliance requirements. Worse, an institution’s IRB is incentivized to disallow any risky (i.e. potentially valuable) research lest some lawsuit be raised.

This statistic is less worrying than it might seem, since most research is done by graduate students and postdocs who spend (much) more time doing active research than PIs. And there may be benefits to the seemingly wasteful process of grantwriting. But it’s clearly a deadweight loss to science.

Institutional Fees

Universities get paid by amount of grant money, not scientific output.

Universities get paid overhead when their scientists win grants, taking around 1/3 of what comes in the door. In return scientists get facilities, admin help, a parking spot, etc. Unsurprisingly, this leads to perverse outcomes like incentivizing universities to grant tenure based on grant size rather than scientific merit.

Course Teaching Requirements

Teaching is its own skill, distinct from research, and most scientists are bad at it. We shouldn’t force them to do it.

Most training relevant to actually doing research occurs in the lab, where students learn from more senior students and occasionally from their advisors. But many researchers are forced to spend inordinate amounts of time teaching subjects to general student populations.

Pushing the frontiers of a field of science does not, contrary to what many seem to think, make one better informed about or better at teaching the basics of that field. While many researchers enjoy teaching, many more are bad at it. (And often these are the same people.) Information dissemination is obviously a good thing, but forcing research-oriented professors to disseminate basic information through the medium of lecture-based courses is wildly inefficient.

Traditional lecture-based courses are themselves, of course, far from the least efficient way of imparting knowledge.

Small Market for Research Tools

Experimental apparatus for science is mostly awful since there’s little money to be made selling it.

Most experimental science relies on specialty equipment. Some equipment like petri dishes or high-performance computers have large enough customer bases for them to be available reliably and cheaply to scientists. But newer and less-popular niches of science require equipment provided by a few companies, if any at all.

Most small businesses providing scientific equipment do it as a labor of love, since the markets involved are too small for grandeur. But small markets mean these companies can’t invest in product development. Bad products, bad service, and slow innovation follow. And occasionally, among less generous companies, oligopolistic price hiking ensues.

I’m not sure what solutions there are besides helping research equipment companies capture more value from work done with their tools. Giving research equipment companies equity in developments done with the tools might help. Research equipment (a.k.a. “platform”) companies in biotech often do this. Advance market commitments might also help.

Switching Costs

When scientists publish in new areas, their work is less impactful.

To the extent it’s because their work is worse, it’s bad for the adaptability of science. To the extent it’s because their work is undervalued, it’s bad for scientific progress.

Graduate-student Illiquidity

Once they pick an advisor, grad students are mostly stuck with them. Obvious problems arise.

It’s extremely hard for graduate students to change advisors or work without an advisor. This labor market illiquidity is due to the advisor’s monopoly over graduate students’ funding and future career prospects. Graduate students are funded by their advisor, except for a small minority of students who receive personal fellowships like the NSF’s GRFP. And a graduate student’s (or postdoc’s) advisor’s reference letter is by far the most important factor in future academic employability.

The negative effects of this illiquidity include advisors taking excess credit for their students’ work, advisors forcing students to work on projects they don’t care about or do jobs that benefit the advisor but not the student, and students being trapped into working under abusive advisors.

Funding students directly rather than funding advisors could ameliorate some of this problem. (Can’t find original citations for this, but I’ve heard it multiple times.)

Bias by Funding Source

Scientists are incentivized to produce results their funders will like.

For example, tobacco companies sponsoring research on how smoking isn’t unhealthy. Or Berkeley’s controversial agreement with Novartis. These days scientists have to disclose their funding sources, but it’s not hard for funders to hide themselves.

Bad motivation

The cooler science becomes, the more people become scientists for the wrong reasons.

Science relies on an obsession with truth-seeking to hold scientists to norms of good conduct. But the increasing status of scientists in society may have increased the number of scientists who are motivated by status-seeking rather than truth-seeking.

It’s not clear how much fraud, p-hacking, strategic citation, authorship-jockying, academic politics, etc. occurred when science was less cool. But I certainly know a lot of people who do science for status and not truth. Maybe it’s worth making science less cool. And yes, it’s a stretch to call this a market failure.

Useful References

• Microeconomics of Market Failures
• How Economics Shapes Science
• New Things Under the Sun
• Ben Reinhardt’s List of Academic Constraints
• José’s reminder that we don’t know if fixing any of the above will help

Thanks to Sci-Hub for unspecified services.

Have feedback? Find a mistake? Please let me know!

I’m glad aircraft have to be pre-approved by the FAA before I can fly in them, rather than letting Boeing release whatever planes they want and recalling any that end up not working. And on the other hand, we don’t expect the FDA to pre-approve every new culinary delight the food industry graces us with.

But of all society’s pre-approval vs. post-monitoring decisions, the most consequential to science was the creation of the Institutional Review Board.

An Institutional what?

An IRB is a group of people that decides whether a research project is ethical. In the U.S. and most other countries, it’s practically impossible to do research involving humans unless an IRB (or equivalent body) pre-approves your research plan.

“Involving humans” really is as general as it sounds: from surgeries to surveys, if any information about any human is being collected, an IRB approved it first.

IRBs are a close 2nd to paywalled journals in the rankings of things scientists complain about most. This isn’t because scientists want to do unethical research. It’s because IRBs are slow and frustrating.

How bad can it be?

Anecdotes abound. Scott Alexander’s IRB Nightmare is the best known, in which our hero tries for over a year and eventually fails to get permission to ask patients questions they’re already being asked. (Follow-up discussion here.) Andrew Gelman has some good ones, and you can find endless more on reddit. The moral monsters at Give Directly had to wait a year when they once tried to give people free money.

A professor I know spent 3 years (years!) getting approval to try to wake a coma patient by applying ultrasound at already-FDA-approved intensities to the patient’s head. It worked, by the way, though the real miracle may be that the patient survived the 3-year wait.

There’s also no shortage of legal and moral critiques of IRBs.

But the worst stories always get the most attention. And every regulation has haters. How bad are IRBs for science really, overall?

Quite bad, is the answer.

In the most comprehensive survey of 11,167 researchers with active federal grants in 2018, ~66% of researchers reported that petitioning IRBs was a “substantial” part of their workload. Unfortunately the study didn’t ask for specific time estimates. But researchers also reported spending ~44% of their research time on administrative tasks other than active research, including petitioning IRBs. And a substantial amount of 70% of 44% is still substantial.

More directly, ~33% of researchers said improving IRBs was a high priority for reducing unnecessary administrative burden.

One can see why. At Berkeley, getting permission to email out a social science survey takes at least 3 to 4 weeks, while “expedited” review takes 6-8 weeks. The University of Illinois at Chicago reports a median of 70 days for full IRB review and 33 days for expedited review. The University of Arkansas for Medical Sciences claims to be quicker, reporting an impressive median of 30 days for full review and 5 days to confirm “exempt” studies. Less impressively, a VA hospital reports a median of 131 days for full review and 82 days for exempt review. Weirdly, a regional IRB in the UK reports taking 30 days to do expedited reviews but only 25 days to do full reviews.

The financial costs of IRBs aren’t as outrageous, though of course they’re not free. 2003 estimate were between ~$70,000/yr and ~$770,000/yr depending on the institution size. A 2007 estimate of that range was from ~$100k/yr to ~$4.5M/yr.

More worrying is how wildly inconsistent IRB rulings are between institutions. This inconsistency means researchers at institutions with more permissive IRBs often get to publish first, an unfairness that has not gone unnoticed nor unexploited by researchers. IRBs also often mandate changes that drastically increase the costs of studies, like requiring researchers to hire a doctor to be present for trivial medical procedures. And IRBs make it challenging for scientists at different institutions to collaborate, since e.g. for medical device studies if any one institution’s IRBs does not approve a study, the study has to go all the way to the FDA to get approval.

Sadly, comprehensive data on IRB performance isn’t collected, so we don’t have good estimates for the two most important quantities: how many worthwhile studies are killed by IRBs, and how many unethical studies IRBs have stopped or fixed. We know the first quantity is nonzero. The second quantity may well be zero by the usual prohibition logic: people who want to do unethical research just do it anyway, clandestinely or overseas where it’s legal.

Of course scientists don’t just sit idle while waiting for IRB approval. They can do other things while they wait. And the data suggest that patient recruitment and fundraising consume more researcher time than interacting with the IRB.

But the fundamental problem with IRBs is not the time they take. It’s that the more novel an experiment, the less likely an IRB is to allow it, which is precisely the opposite of what we want to incentivize in science. Novelty is not inherently unethical. It’s only unethical if the benefits don’t out weight the risks. And when it comes to trying truly new, high-risk-high-reward technologies in the clinic, IRBs are by far the biggest bottleneck.

How did we get here?

There’s no question that the creation of IRBs was well-meaning.

For a complete history you can read this book. But in brief: the 20th century was full of awful, unethical science. The worst crimes were (and probably still are) committed by totalitarian regimes like the Nazis, but the U.S. government was far from innocent.

In particular, in 1932 the U.S. Public Health Service (PHS) and the CDC started the infamous Tuskegee Syphilis Study. The Tuskegee study wasn’t remotely as bad as other studies the PHS ran, with which it is often conflated. But it was certainly unethical, and it continued for 40 years until PHS employee Peter Buxton blew the whistle on it in 1972.

National outrage at Tuskegee led congress to pass the National Research Act in 1974, which said, “Make sure nothing like Tuskegee happens again.” There followed a series of commissions and reports to figure out how to accomplish this. And the reports’ main inspiration was the NIH.

Why the NIH? Flashback to the 1950s when, unrelated to Tuskegee, doctors at the NIH had started trying a newfangled idea: doing research on healthy people instead of just sick people. This was great for science, but the doctors doing the research were nervous about bad PR and lawsuits, since they were only used to treating patients and not “Normals,” as they called them. So in 1953 the NIH assembled a few senior doctors into a “Clinical Research Committee” to review (or rubber-stamp) all research on Normals. This was the first proto-IRB.

By 1962 the NIH had also started requiring signed consent forms from all research participants.¹ NIH doctors hated this rule and mostly ignored it, since they thought the whole point of having a professional code of ethics was to avoid that stuff. But NIH lawyers loved it because it helped them defend lawsuits.

Then the NIH started getting sued not just for their own research but for research they funded at other institutions. So in 1966 the NIH mandated that their grantees all use proto-IRBs and consent forms too. This especially bummed psychologists, since it limited the use of deception in research, which at least for me would have been the main perk of being a psychologist. (The Milgram experiment was done in 1961, if you’re wondering.)

So, returning to the Tuskegee side of the story: in 1974 when the government was looking for ways to comply with the National Research Act, they saw that NIH had been doing ethical-looking things for decades, and more-or-less copy-pasted the NIH’s system into official regulations with the memorable name 45 CFR part 46. These regulations were so good that by 1991 15 other agencies² had copy-pasted HHS’s regulations into their own regulations. Since then, everyone calls these regulations the Common Rule, since they’re common to nearly every government agency. And the centerpiece of the Common Rule is the requirement that all research be approved by an IRB.³

Thus the modern research regulatory environment was born. (At least in the U.S., though most of the world has copied the system too.)

Reminder: How the U.S. government decides what you’re allowed to do

• Congress makes laws, a.k.a. statutes, a.k.a. acts, a.k.a. legislation.
  • Laws are collected in the United States Code.
• Government agencies (FDA, DEA, EPA, etc.) make regulations, a.k.a. rules.
  • Regulations are collected in the Code of Federal Regulations.
  • The live feed of regulations being debated and finalized is called the Federal Register, which you can subscribe to.
  • Agencies are allowed to make regulations because congress passes a special law giving them permission.
    • And of course there’re also laws about how they’re allowed to make regulations.
• Regulations aren’t laws, but they have the “force of law.”
  • There is about 5x more regulation than law by word count.
• The courts get to decide whether laws and/or regulations contradict each other.
  • The Constitution takes precedence over laws, and laws take precedence over regulations.
• There are also state and local versions of all the above.

Does this affect my research?

The Common Rule says you have to use an IRB if your research is supported in any way by HHS or another Common Rule agency. The Office of Human Research Protections in HHS is the group that actually enforces all this.

“Well then,” you might say, “I’ll just get private funding for all my research, and then I don’t have to use an IRB.” That’s absolutely true, and a lot of research is done that way, like political polls (though not the U.S. Census, because that’s funded by the Department of Commerce, a Common Rule agency).

Except that IRBs are also required by basically any agency if you ever want to submit the results of your research to them. For example, if you’re a biotech company, and you’d like to get a drug approved by the FDA, which you have to do in order to market it, all the research you do to prove the drug is safe and effective has to have been IRB-approved.

And even if you don’t care about the FDA, if you want to publish your research in any serious journal you’ll find they require IRB approval, too. Oh, and did you want money to do your research? Because many grantmakers have IRBs — apparently just for PR reasons, since I can’t find a regulation requiring this.

And even if you just want to throw your research on your blog, most institutions mandate that all work done with their money or equipment go through their IRB. Don’t need their equipment? Most federally funded datasets require you to have an IRB to use them, too.

Oh, and in case you were hoping you could at least do some self-experimentation without IRB approval, you can’t at most institutions, even if it wins you a Nobel Prize. You’ll have to do it in your free time.

Can we fix this?

Obviously the IRB mandate could be removed by regulation or legislation, but I’m not holding my breath.

Could we make a new IRB that works better and convince researchers to use it instead? That’s already been done. Private, for-profit IRBs exist, costing a few thousand dollars for review with a week-or-so turnaround. These private IRBs are how startups and other small organizations get the required IRB clearance described in the previous section. But most universities don’t allow researchers to use private IRBs in lieu of the university’s IRB, so private IRBs can’t save us.

What about going overseas? Most of the rest of the world uses an IRB-like system, but not everywhere. There are parts of the world where you can do basically whatever research you want, for better or worse. This isn’t a full solution for the same reason private IRBs aren’t. But some researchers have developed a clever strategy that could be scaled up: they do preliminary, IRB-overhead-free studies overseas and then re-run the successful ones in the U.S. There may be an outsourcing solution here.

Could we at least help researchers navigate the IRB system better? Yes. Experienced researchers learn lots of IRB tricks that aren’t taught to students. For example, you can get a very generic IRB approval – called an “umbrella” IRB – and smuggle as much research under its remit as possible. Cover yourself with enough umbrellas, and you never have to go to the IRB again. Training scientists to navigate a broken system is a depressing solution, but it’s practical.

None of these are great solutions though. If you have a better idea, please get in touch.

Have feedback? Find a mistake? Please let me know!

Everyone knows there’s been a revolution in machine learning (ML) over the past decade. But they tend to not know that the revolution has been spiky, not smooth. ML hasn’t improved a large amount at all tasks — rather it’s improved a flabbergasting amount at a few tasks, with other capabilities sometimes coming along for the ride.

In lieu of me telling my next interlocutor why sprinkling some sweet machine learning dust on their problem won’t help, I figured I should write down exactly where I think progress in ML has happened and let them decide whether it’s relevant to their problem. (I would point them toward various journalistic analyses of the field, except I haven’t found one that isn’t worryingly credulous.)

If the problem you’re trying to solve isn’t in an area where there’s been a revolutionary advance, you should assume ML is no more relevant for solving your problem than it was in 2012.

Revolutionary Advances

Large-scale model training

ML has figured out how to get mostly-non-diminishing returns to scale.

If you have a massive dataset (or the equivalent of one, like a simulator) then scaling up the amount of compute and model parameters you throw at a problem will get you better performance. Gwern’s analysis in The Scaling Hypothesis says it better than I can.

Specific advances that fall under this heading include GPU-filling self-attention architectures like transformers, better optimization algorithms like Adam, and better training schemes like GANs, MLM, PPO, etc.

Software and hardware infrastructure

Related, but not identical to, large-scale model training are advances in ML infrastructure.

If you have an ML idea you want to try, there is almost certainly a well-written software library with which you can try it, and easily accessible cloud hardware to run it on. This is true for small models as well as big ones.

Pretraining and transfer learning

Perhaps more a discovery than an advance, but if you want to perform a task that’s similar to one a pretrained model can do, you can often adapt the pretrained model to your task with little additional data and compute.

Adversarial attacks

It’s possible to break modern deep learning systems in just about every way you can imagine, and many you can’t.

Areas where progress has been made, but no silver bullets yet

• Robustness against adversarial attacks
• Explainability
• Bayesian optimization
• Incorporating expert knowledge/domain-specific models (i.e. imbuing models with inductive biases based on physics, chemistry, source code, etc.)
  • I could be convinced that AlphaFold is an example of this, but it would still be the only example I know of
• ML for robotics
• Multi-agent RL
• Exploration and intrinsically motivated learning (except when derived from large-scale model training)
• AutoML
  • Mostly advances in what you can buy from cloud providers
• Bayesian inference
• Causal inference

Areas where, despite sometimes attention and hype, not much progress has been made

• Tabular machine learning
• Few-shot and meta learning (except when derived from large-scale model training)
• Uncertainty quantification
• Time-series forecasting

If you have a strong opinion that this categorization is wrong, please let me know! This is a hard assessment to make, and I don’t have a perfect view of the field by any means.

This is surprising given that the answer is obviously “yes." In fact the arguments for “no” are so unimpressive I’m relegating them to an Appendix.

A better question is “should I work on human longevity, and if so, how?”¹

This is a personal question, but answering it well involves a shedload of strategic and economic considerations. I couldn’t find a comprehensive guide to them, so here’s my attempt at one.

Contents

Tractability and Strategy

Y Combinator sums up the opinion of the tech world today:

“We now have the understanding of biology to work on therapies that directly address the root causes of aging.”

Is this true? And if so what’s the best way to go about it? There are (at least) three big considerations.

Do we know the causes of aging, and do we even need to?

I’d say “not yet” and “maybe not.”

Here’s a summary of what experts think causes aging in mammals and how they figured them out.

Nintil’s Longevity FAQ is a great starting point to understand the biology behind most of these proposed causes if you’re unfamiliar.

Let’s first admit that it’s dubious to break down aging into discrete causes like in the table above. Here’s a diagram giving a much more detailed view of our current understanding of aging-related processes. (In fact that diagram isn’t even complete since aging varies between different parts of the body at different times.) Based that diagram (seriously, click the link, it’s worth it), understanding the causality of aging may be less about discovering new causes and more about disentangling what we already semi-understand, if that’s even possible. Though it might still be true that there really are only a few root causes of aging behind all these.

Nevertheless, from the outside view the above table doesn’t really suggest that we’ve finished discovering fundamental causes of aging. The pattern — if small-N history can have patterns — seems to be that every decade a couple biological processes are associated with aging and then 1+ decades later, if ever, a causal connection is established. The table only includes causes of aging that have made their way into big general reviews, so we wouldn’t expect to, and don’t, see any with initial discovery dates past 2000. Biological processes whose heydays came after 2000 that might someday make it into this table include microbiome changes and RNAi dysregulation.

You might argue that since investment into aging research and biotech in general has increased over the years, but the rate of discoveries hasn’t changed, that means we’re nearly done discovering things. But, while we clearly know more about aging than we ever have, we still don’t have much of a causal understanding of aging.

But how much do we need to understand?

The standard playbook for making a new therapeutic (the catch-all name for a drug or medical treatment) is called a target-based approach, and goes like this:

1. Identify a specific biological process that’s causing the disease you want to treat (e.g. plaque buildup, the expression of a single gene, a protein key to a virus’s function)
2. Develop a way to interfere with it (e.g. a synthetic molecule, an engineered antibody, a surgical intervention) in vitro or in an animal model
3. Adapt that treatment so it’s safe and effective in humans

It’s not clear this standard approach is well-suited to longevity therapeutic development.

Target-based approaches work best when we thoroughly understand the cause(s) of a disease. Bacterial infections and single-gene disorders are good examples: we really know what biological process we’re trying to interfere with to get the outcome we want. But as discussed above, we don’t understand the causes of aging, and our current understanding it that it’s massively multi-causal.

An alternative to the target-based approach for therapeutic development is the phenotypic approach, which goes like this:

1. Set up an experiment where you can cheaply try a gajillion treatments and find one that affects the phenotype you care about (e.g. the phenotype of living longer)
2. Adapt that treatment so it’s safe and effective in humans

In reality all therapeutic development takes place on a spectrum between target-based and phenotypic. Some understanding of causality is necessary for restricting the space of treatments screened in phenotypic approaches. And many causes of phenotypes are discovered via screening approaches.

Therapeutic development today generally leans toward target-based. This is despite the fact that more first-in-class small-molecule drugs are developed with phenotypic approaches. And despite the fact that many effective longevity therapies (in mice) had no mechanistic hypothesis when they were discovered, including parabiosis (and potentially neutral blood exchange) and caloric restriction.

Phenotypic approaches also seem better suited to finding effective combinations of therapies, for the inelegant reason that biological systems are so interconnected that to find an effective combination therapy we just have to try a lot of combinations. This is important for longevity, since it’s not clear how much partial credit we get for solving any single cause of aging. It may be that each one cured buys us more lifespan and eventually we reach longevity escape velocity. But since dying only requires one cause of death, we may need therapies that simultaneously address all or most causes of aging — even ones we don’t know of yet — before seeing benefits. The idea that phenotypic approaches might be better for targeting multi-causal biological phenomena like aging is not new. Companies working on them (not specifically on longevity) include Recursion, Janssen, and Insitro.

Phenotypic screening has plenty of challenges, of course. One is the gajillion factor: you need a way to try lots of different treatments, and the space of possible treatments is astronomical. Another is that you can’t do a phenotypic screen without a phenotype, and waiting for animals to die is pretty inefficient. Developing biomarkers (a.k.a. quantitative phenotypic measurements) for aging, like epigenetic clocks, is critical to enable better phenotypic screens. Gordian, Nemalife, Magnitude, InVivo, Vium, and Spring are among the companies working on these problems.

Do we have the technology?

It seems inarguable that humans can live forever in the limit of arbitrarily fancy technology, though some disagree.² But whether death is conquerable not just eventually, but with near-term tools and understanding, is not clear. To paraphrase Lincoln: we don’t know whether we’ve sharpened the biotechnological axe enough to cut down the death tree.

This is a key question for longevity strategy. The more confident we are that existing therapeutic technologies can make a dent in aging, the more we should invest in using them. The less confident we are, the more we need to make new tools.

One relevant datapoint is what modalities (a.k.a. types of therapeutics) are being used by companies currently working on longevity. As far as we know, it’s about 51% small-molecule drugs, 16% biologics, 18% cell therapies, and 15% gene therapies. This skews more toward non-small-molecule therapeutics than the industry average, which is what we might expect given that most of the causes of aging in the table above don’t have obvious targets for small-molecule inhibition. It also suggests that new tools are needed, since non-small-molecule therapeutics are newer and less straightforward to develop than small-molecule drugs (as if those were straightforward to develop in the first place).

Another datapoint is what we’ve managed to accomplish so far in mice. Here’s a table mostly derived from a table of Laura Deming’s showing the greatest median lifespan increase that’s been achieved in mice by different therapeutic modalities.

Note that median lifespan is not maximum lifespan, and lifespan often increased for only one sex of mouse.

The top two modalities in mice, spontaneous mutation and GGT, can’t be used (legally) in humans. And I’d argue diet is off the table for practical reasons; we’re talking extreme diets. The remaining modalities are used in some form in humans. I left surgeries off the table since it’s too broad a category, but the ones relevant to humans give <= 10% median lifespan increase. Also note that just because a lab chose to treat a mouse with one modality doesn’t mean another modality can’t accomplish the same effect. Developing small-molecule drugs to target proteins encoded by genes discovered by GGT in mice is standard practice.

You can read this table as showing that human-amenable modalities are less potent than they could be, or you can read it as a selection effect of what academics find easiest to do in the lab. The former is a big claim, and I’d need more evidence to believe it. The latter isn’t news to anyone. Synthesizing candidate drug molecules is slow. Producing immunotherapeutics is slow and hard. Gene therapy has a ways to go.

Altogether it seems clear that our therapeutic armamentarium is lacking, whether in potency or ease of use.

What are the biggest leverage points?

Bringing a medical treatment to market is extremely slow and expensive. It takes around a decade and $1 billion. A big reason for this is the 20% success rate for non-cancer drugs once they enter clinical trials, not to mention the definitely-less-than-100% success rate for making it to clinical trials in the first place.

Here’s a figure from this paper breaking down the costs, timelines, and success rates in a traditional target-based drug discovery campaign.

Paraphrased caption from the paper: Numbers are in millions of USD and based on data from Eli Lilly and Company. p(TS) is the probability of successful transition from one stage to the next. WIP is work in process, the number of candidate drugs being worked on. Lighter-shaded boxes show calculated values based on assumed inputs. Costs do not include investments for exploratory discovery research, post-launch expenses like marketing, or non-R&D overheads like non-scientist salaries.

Worth noting in the figure is that most of the cost is in the clinical trials (the blue parts), and that the step with the lowest success rate is Phase 2 clinical trials (the first time the therapy’s efficacy is tested in humans, usually).

What does this mean for longevity? Barring an adventure in regulatory arbitrage or medical tourism, any longevity treatments will have to get through this process before they can get to you or me.³ So it may well be that the best way to pursue longevity is to improve the drug development process, depending on how tractable and neglected various improvements are.

Diagnosing the problems in the drug development process is a lot like studying aging: everything you look at is broken, but we have no idea about the root causes or where best to intervene.

At the lab level

Animal models for disease are fraught with difficulty. Those for aging are particularly challenging. How challenging? So challenging that the longest-lived strain of mouse known to biology used to be used as a model for rapid aging, because the mice were being handled wrong and dying young.

Of course we could dispense with animals models and focus on human trials, but a trial that waits around for people to die will take a long time. See for example the TAME Trial, which plans to run for six years. As mentioned above, better biomarkers could help with this. Or one can follow George Church’s advice and ignore therapies for slowing down aging in lieu of those with undeniable, immediately noticable effects.

A big change in lab operations over the last two decades has been the “virtualization” of biotech, which is a fancy word for outsourcing. Work once done in-house, like synthesizing molecules or running toxicity assays, is now usually done by Contract Research Organizations (CROs). Outsourcing has benefits: increased specialization, lower prices, etc.

But outsourcing is also inefficient: CRO management is the biggest headache for many biotech companies. It’s still a lot easier to do drug development when everyone’s in the same building, or so says every scientist I’ve asked about it. This is not the case in all industries. Solutions could come in the form of automation — Emerald Cloud Lab and Transcriptic are two companies working on that — or better outsourcing offerings. One thing the field probably is good on is electronic lab notebooks. Thanks though.

At the community level

Longevity research, and medicine in general, is bad at integrating its knowledge and learning from failures. Between 25% and 50% of clinical trial results remain unpublished even several years after completion. This isn’t surprising: why publish your clinical data when there’s zero benefit to you for doing so, and when there’s always some chance you might want to repurpose the therapy in the future? There’s a similar market failure around improving animal models: animal models aren’t easy to defend with patents and are hard to keep secret, which means they get relatively little private R&D investment. All this means a potential downside of moving longevity efforts from academia to commercial therapeutic development is that the field may move more slowly.

A related problem is that because liquidity events can happen far before convincing clinical data, biotechs are incentivized to push their risky studies until as late as possible (ideally after the employees get their money) and to gussy them up to look better than they really are. A fail-fast-and-publish-honestly mentality would of course be better for longevity overall.

Then again, it’s not like academic research is necessarily better. Medical research results in general are apparently unreproducible up to 50% of the time. Plus private biotech can afford to do many things that academic labs can’t. Fortunately, some private funders are working to fill this gap.

At the regulatory level

Before it can conquer death, a longevity treatment will have to conquer the U.S. FDA’s clinical trial process, at least if it wants to become part of the mainstream U.S. healthcare system. The FDA doesn’t consider aging to be a medical indication (a.k.a. something a company can advertise a medicine to fix). This means longevity companies have to choose a specific age-related indication like Alzheimer’s against which to demonstrate their treatment’s efficacy.

More optimistically, regulatory changes like dual-track clinical trials or eliminating phase 3 trials altogether might entail a massive acceleration in longevity therapeutic development. Outsourcing or decentralizing clinical trials is also an exciting option. Science 37 is one company working on this.

It’s also possible companies could avoid the FDA altogether, though it’s not clear how this would go down in practice. A company can sell therapeutics-in-all-but-name as supplements or general wellness devices without getting FDA approval for them, and doctors can prescribe more-or-less anything off-label, but the FDA can, and often does, decide to crack down on such gray-area activities. Another option in the near term is focusing on longevity in pets as are Rejuvenate, Loyal, and GenFlow. This may be a great way to get a lot of good science done with relatively less regulation before trying to clear the FDA hurdles.

At the macro level

The high costs and failure rates of drug development make it a tough space for entrepreneurial disruption. (The first chapters here are illustrative.)

This is partly because high risk makes behavior change hard. There’s natural resistance to changing any part of the playbook. As one medicinal chemist put it to me “if you make a service 2x better, nobody cares. If you make it 10x better, or tell them to try a new approach, they won’t believe you. 4x or 5x better with their existing process is optimal.” Not exactly culture of disruptive innovation. One exception is the genetic sequencing sector, which has been extremely disruptive and innovative. I’d like to better understand what degree of price decrease and reliability was required for that.

Another reason biotech entrepreneurship is hard is that the payoffs take so long to arrive, way after all the research is done. This means platforms and services for drug development aren’t bought and sold by biotechs as much as bartered for with equity in whatever future drugs they hopefully produce. And this in turn makes it more complicated and expensive to start companies offering platforms and services to biotechs compared to other industries.

As if all the above wasn’t good enough, the time and cost required to make a drug are increasing over time, despite lots of technological advancement. There’s even a cute name for it: Eroom’s law. Nobody knows why it’s happening, but if the causes of Eroom’s law are exogenous, we should launch more longevity therapeutic ventures sooner while it’s relatively cheaper.

So, which has more marginal benefit, improving the drug discovery system or going after longevity with the existing system? Obviously I have no idea, but the point is that it’s not obviously the latter. We should also appreciate how many important problems there are to be solved outside the lab and with non-biology skills.

Externalities

The pursuit of longevity doesn’t happen in a vacuum. A full answer to “should we pursue human longevity” needs to take all externalities into account.

A big externality is the aspirational value of longevity as a brand/meme for biotech in general. Some might argue that society is already working on longevity with every new medicine it develops. If we keep curing things that kill us, eventually we stop dying, right? Who needs the new branding? Well, a lot of longevity investment is not coming from traditional biotech/pharma investors, but from Silicon Valley. That may be attributable to the aspiration value. And even if the aspirational call of longevity does nothing more than encourage talented people to work on biotech, that may be well worth it.

Another possibility is that the promise of death-postponement could spur funding increases or regulatory reforms for drug discovery. This is an advantage of the aforementioned efforts focused on longevity in charismatic species like household pets. If everyone’s dogs were suddenly living to age 30, people would hopefully start getting jealous.

On the other hand, if a field over-promises and under-delivers it can go through a winter. This wasn’t great for the field of AI, and it would be nice if longevity can avoid this. There’s also brand risk. Gene therapy was delayed a lot by the death of a teenager in 1999. Psychedelics were delayed by 50+ years after society turned against them, despite being widely accepted among the wealthy and well-educated in the mid-20th century. If longevity therapies develop brand recognition — I’m looking at you, senolytics — and there’s a major adverse incident, or even just embarrassments, it might have been better to keep a low profile.

Then again, the same fears of death that make longevity prone to pseudoscience might act as a countervailing force to a winter. People don’t really care about AI research. They do care about staying alive.

Opportunity Costs

Why do we care about longevity? Because death is bad.⁴ There are four problems with it:

1. You suffer through aging and dying
2. Others suffer as they empathize and care for you
3. It precludes future utility for you
4. It precludes future utility for others, including the loss of your contributions to society

But longevity isn’t the only solution to these problems. And these problems aren’t the only causes of suffering in the world that we might care about. This means we have two opportunity cost calculations to make.

Is longevity the best solution to these problems?

Humanity has made a massive dent in problems 1 and 2 with analgesics. Analgesics get short shrift in the history of technology, but they’re one of the greatest medical victories ever won. They also have lots of room for improvement, though that improvement may be hard to attain if the history of analgesic development is any indication. Then again, it’s hard to imagine it being any harder than longevity therapeutic development.

Physical pain is hardly the only kind of suffering implicated in problems 1 and 2, of course. Aging and death, and witnessing them, entail significant existential suffering and anguish. (Citation probably unnecessary.) If the psychonauts are to be believed, we may be close to treatments for those, too: clinical results so far are promising. But it’s hard to imagine psychedelics making us fully disinterested in death. Doing so would require significant improvements to analgesia, neuromodulation techniques, and cultural attitudes toward medication and death.

Some readers may feel that pain- or mood-manipulation are inauthentic ways to defy death. I’d gently remind such readers that we’re comparing these strategies against making humans immortal, so you may want to check your naturalistic fallacy at the door.

Problems 3 and 4 run up against hairy philosophical issues about the continuity of an individual life and whether it has inherent value. As a personal diagnostic, you can ask yourself whether you think one person living 100 healthy years is different to two people living 50 healthy years each, all things equal. If you think they’re the same, and you’re not an antinatalist, problems 3 and 4 aren’t problems for you at all.

Are these the best problems to work on?

I hate this question, because the right answer to “should we fund longevity or something else?” is “can we seriously not fund both?!” It’s also suspiciously close to the fallacy of relative privation, which longevity detractors fall prey to whenever they argue against longevity research in principle due to the existence of other big problems in the world.

But the fallacy of relative privation isn’t a fallacy when making a cost-benefit analysis — something defenders of longevity can forget. Society does have to make choices about where it spends its resources. Even if its choice is to spend them on tokenized digital cats.

I’ll leave debating those choices to the experts. And what you should work on comes down to personal fit. Emanuele Ascani, Matthew Barnett, and their commenters have gotten some nice conversations started on the topic. But suffice it to say there are no decisive arguments that longevity research is more or less important than, say, existential risk research, or promoting pronatalism.⁵ Unless, that is, you have some unusual conceptions of value being inherently tied to human lifespans and not human lived experience itself.⁶

Conclusions

We shouldn’t overestimate how much we understand aging, and we shouldn’t underestimate how much progress we can make without full understanding.

On the margin I agree with Open Phil that basic research and engineering for biotech tools are likely to have the biggest impact on longevity, whether this research is called “longevity research” or not. I’d say this is especially true for tools like biomarkers to enable large-scale phenotypic assays. The hard(er) part of building the atomic bomb wasn’t the nuclear physics, it was building the bomb, and I suspect longevity is similar.

There are a million problems to be solved in the larger biotech ecosystem that will help achieve human longevity. Solving them may accomplish more for longevity than any direct work on aging at this point.

Many problems in the larger biotech ecosystem don’t require primary expertise in biological science. And most of them aren’t specific to longevity and require no explicit association with longevity. This is useful for decoupling longevity progress from the longevity brand.

Despite how underrated longevity research is, neuro- and bio-technologies addressing suffering at its source in consciousness and the brain are underrated even more, despite the tech world’s recent interest in psychedelic therapy and pharma’s historic interest in non-addictive painkillers. Neuromodulation, mood adjustment, and new forms of analgesia should be getting as much or more attention than longevity.

Have feedback? Find a mistake? Please let me know!

Many thanks to Egan Peltan, Stephen Malina, Gavin Taylor, Alexey Guzey, Alex K Chen, William Eden, Karl Pfleger, Nathan Cheng, and Prof. João Pedro Magalhães for conversations and feedback.

And an extra huge thank you to Sarah Constantin and Elizabeth Van Nostrand (whom you can and should hire to do research, by the way) for their help.

Appendix: Bad arguments against human longevity

• “Death makes life meaningful”
  • You could make the same argument about the opposite of any good thing (we need pain to appreciate pleasure, etc.). This is not usually considered grounds for not working on them.
  • People will still die from accidents, so the “thrill of living” will still exist.
    • It might even increase, since unexpected death would interrupt the grander plans of those who expecting to live forever.
  • You can always opt into death.
• “People will just suffer forever”
  • A.k.a. the Tithonus Error. Nobody’s talking about this when they talk about longevity.
  • It is true though that there’s no guarantee a longevity therapy will increase healthspan rather than just increasing morbidity, so care should be taken here.
• “People will get bored,” or “society will stagnate”
  • Again, anyone’s welcome to get off the ride anytime they want.
• Inequality: “Some people will get treatments first”
  • We already live like this, just compare lifespans across countries.
  • A net increase in wellbeing is worth doing, even if it’s unequally distributed.
    • It’s important to factor in the negative wellbeing associated with being on the losing end of an unequal split, but still net positive.
• “Incumbents won’t ever die”
  • Already a problem in the real world w.r.t. regimes instead of people. Longevity doesn’t materially change this.
• “Something something Malthus”
  • The worst versions of these arguments go “if nobody dies, our population will explode, and we’ll all starve to death!”
    • This could only happen if we developed true immortality. Otherwise is just slightly increases the population, since people are dying at the same rate, just later.
    • If we do develop true immortality, we would also almost certainly have the technology to be fertile at any age. (Not to mention sufficient technology to feed everyone.) This would reduce the urgency to have children in the unlikely case of a Malthusian scenario.
  • There are more realistic ones though (perhaps not technically Malthusian). Suppose a senolytic cocktail was approved in 2040 that extended human lifespan by median 50 years but didn’t ameliorate various nonterminal dementias, which set in at 100. And suppose these dementias resisted treatment until 2050, 2060, or later. This future would likely be net positive for the world, but practically speaking it would indeed amount to an economic and ethical crisis.
• “You can’t go to heaven if you don’t die”
  • The sorts of secular people who write articles promoting longevity research always seem to leave this one out.
  • Nevertheless, if there’s a heaven (or hell) awaiting you, you’ll get there eventually. What’s the rush?
• “People just should die”
  • This is the naturalistic fallacy. Apply Bostrom’s reversal test.
• Other, better rebuttals:
  • Nice summary of common objections
  • Another nice summary
  • Good FAQ
  • Nice summary of logical fallacies against longevity
  • A classic: The Fable of the Dragon-Tyrant

Notes

• meditation
  • concentrative
    • focusing on a specific sensation or thought, like the breath
    • metta
    • gratitude practices
  • open monitoring
• psychotherapy
  • cognitive behavioral therapy
• neurofeedback
• hypnosis
• breathwork
• brain training
  • Akili Interactive targeting ADHD
  • action video games to reduce depression
• conditioning
  • disulfiram
• social engagement
  • friendships
  • romance
  • group activities (games, sports, dance)

Peripheral

• activities
  • walking
  • exercise
  • eating
  • breathing techniques
  • sauna
  • cold plunge
• sensory stimulation
  • music
  • binaural beats
  • auditory entrainment, e.g. Elemind
  • audiovisual entrainment, e.g. Cognito (currently in Breakthrough Designation Phase 3 for Alzheimer’s)
  • vibrotactile stimulation
• sensory deprivation
• peripheral nerve stimulation
  • vagus nerve stimulation (FDA-approved for depression, epilepsy)
  • trigeminal nerve stimulation
  • auricular nerve stimulation
  • percutaneous nerve field stimulation
  • cochlear implants
  • auditory brainstem implant (technically not peripheral, but right on the border)
  • retinal implants
  • science.xyz

Biologic

These are currently uncommon due to the difficulty of getting biologics across the blood-brain barrier, but there are many in the pipeline. Interesting targets include FAAH-OUT and rs324420 for potential pain insensitivity and short-sleeper genes.

• antibodies
  • CGRP antagonists for migraine exist but act on peripheral nerves
  • endogenous antibodies
    • anti-ganglioside antibodies
    • anti-glutamate receptor antibodies
    • voltage-gated potassium channels antibodies
• proteins and peptides
  • Peptides for stroke. Guess that’s one way around the BBB…
  • Peptides for Alzheimer’s. Unknown how they cross the BBB.
• nucleic acids
  • in-vivo AAV gene therapy to increase BDNF
  • antisense oligonucleotides
• live cells
  • cell therapy for depression and potentially other things
  • ex-vivo modified autologous fibroblasts to produce nerve growth factor

Chemical

There are too many chemicals to list, so instead here are the key mechanisms and effects by which we categorize chemicals, followed by some examples.

Mechanisms of action of psychoactive substances

This is a mostly comprehensive list. Most chemicals act by multiple mechanisms.

• receptor modulation
  • agonism (activates a receptor on a cell)
    • It’s actually more complicated due to biased agonism
  • antagonism (binds to but doesn’t activate a receptor, a “blocker”)
  • inverse agonism (binds a receptor but produces an opposite effect)
  • allosteric modulation (binds somewhere other than the active site and increases/decreases activity)
• enzyme modulation
  • induction
  • inhibition
• ion channel modulation
  • opening
  • blocking
• transporter modulation
  • neurotransmitter release enhancement
  • neurotransmitter release inhibition
  • reuptake enhancement
  • reuptake inhibition

Neurological and psychological effects of psychoactive substances

This is definitely not a comprehensive list, just commonly used terminology.

• depressants
  • anxiolytics/sedatives (reduce anxiety)
  • hypnotics (make you sleep)
• hallucinogens (alter perception)
  • psychedelics
  • dissociatives
  • anesthetics
  • deliriants
  • empathogens/entactogens (emotional openness, empathy, sympathy)
• stimulants (may cause euphoria, but do not affect perception)
• mood stabilizers
• antidepressants
• antipsychotics

Examples

• caffeine: stimulant; adenosine receptor antagonist (among other things)
• fluoxetine (prozac): antidepressant; selective serotonin reuptake inhibitor (SSRI)
• haloperidol: antipsychotic; dopamine receptor antagonist
• clozapine: antipsychotic; 5-HT2A antagonist
• cocaine: stimulant; serotonin, norepinephrine, and dopamine reuptake inhibitor
• nicotine: stimulant and anxiolytic; nicotinic acetylcholine receptor agonist/antagonist
• phenethylamines
  • substituted amphetamines
    • amphetamine (adderall): stimulant; TAAR1 agonist (among other things)
    • MDMA: stimulant and empathogen; serotonin, norepinephrine, and dopamine reuptake inhibitor
  • mescaline: psychedelic; 5-HT2A agonist
• tryptamines
  • psilocybin: psychedelic; 5-HT2A agonist
  • N,N-Dimethyltryptamine (DMT): psychedelic, nonselective 5HT and dopamine (ant)agonist
  • melatonin: hypnotic; melatonin receptor agonist
• arylcyclohexylamines
  • ketamine: dissociative anesthetic; NMDA receptor antagonist
  • phencyclidine (PCP): dissociative anesthetic; NMDA receptor antagonist
• elements
  • Lithium: mood stabilizer, multifarious enzyme modulator
  • Xenon: dissociative anaesthetic, unknown mechanism
• nitrous oxide: dissociative anesthetic; ion channel blocker (among other things)
• morphine: anesthetic; opioid receptor agonist
• benzodiazepines
  • diezepam (valium): anxiolytic and hypnotic; positive allosteric GABA-A receptor modulator
  • alprazolam (xanax): anxiolytic; positive allosteric GABA-A receptor modulator
• valproate: mood stabilizer, anticonvulsant; voltage-gated sodium channel blocker, histone deacetylase (HDAC) inhibitor (nice notes on HDAC inhibitors here)

Subatomic

• electronic
  • electron beam therapy
  • beta-decay radiotherapy
• hadronic
  • proton therapy
  • fast neutron therapy
  • heavy-ion radiotherapy

Electromagnetic

• electroconvulsive therapy (ECT) (undeserved bad reputation, safe and often effective for depression when done under general anesthesia)
• microelectrode stimulation
  • deep brain stimulation (DBS)
  • cortical stimulation (RNS Therapy, Nia Therapeutics, etc.)
  • intracortical arrays (braingate, paradromics, neuralink, etc.)
  • stent-mounted array (stendrode)
• transcranial electrical stimulation (tES) (running currents through the brain)
  • transcranial direct current stimulation (tDCS) (less effective than popularly portrayed, but useful for specific indications)
  • transcranial random noise stimulation (tRNS)
  • transcranial alternating current stimulation (tACS)
  • temporal interference
• induction with magnetic fields
  • transcranial magnetic stimulation (TMS)
    • FDA-approved for depression, migraines, OCD, and smoking cessation
  • low field magnetic stimulation
    • Company blew their clinical trial, but that was probably due to rushing the trial and not inefficacy
• photonic
  • photobiomodulation/low-level light therapy
    • Big review of light penetrance in the brain
  • x-ray
  • gamma knife
  • gamma-decay radiotherapy

Mechanical

• invasive surgery
  • lobotomy
  • corpus callosotomy
  • invasive thalamotomy
  • many other -tomys
• ultrasound
  • high-intensity focused ultrasound (HIFU) (for ablation)
  • blood-brain barrier opening
  • transcranial focused ultrasound stimulation (TUS) (direct neural activity modulation by low-intensity ultrasound)

Combination

• environmental
  • nutrition
  • genetic makeup
  • social life
  • prenatal environment
  • gut-brain axis/microbiome
• optogenetics
  • and optogenetic cell therapy
• chemogenetics
• sonogenetics
• sonothermogenetics
• sonomagnetics
• antibody-drug conjugates
• ultrasound blood-brain barrier opening for drug delivery

AI is quickly getting better at many things. This excites many people, and justly so.

But an AI’s work doesn’t get shipped unless it’s checked, assuming you don’t want to go out of business.¹ So it’s important to remember in this age of AI hype that:

The time it takes the AI to do the task and for you to check its work has to be less than the existing solution, otherwise the AI isn’t useful.

People seem inclined to forget this when prognosticating about AI. But you have to do the full cost accounting for how the AI will get used in order to assess the impact.

Of course if the task your AI is doing is something humans can’t do at all, then no matter how long the AI takes, it might be worth it. And if your AI is really good, then you may only have to check its work once, or occasionally. But both these situations are really quite rare in modern AI applications.² Definitely more rare than much media lets on.

This is why AI makes better minions than prophets, in general. A minion repeatedly performs a task very similar to the one it was validated on. A prophet tells you something you don’t know, can’t check, and maybe will never be able to check. It’s almost always easier to check the former’s work than the latter’s.

Also, it goes without saying, this is not the only requirement for AI to be useful in the real world. Just the one I see people miss most and that I’m ranting against here.

Case studies

Good example: code generation for front-end component (as of 2020)

Existing solution:

1. Non-expert human writes code (potentially ages)
2. Non-expert human checks that rendered component works (2 mins)

AI solution:

1. AI generates code (instant)
2. Non-expert human checks that rendered component works (2 mins)

Bad example: code generation for back-end code (as of 2020)

Existing solution:

1. Non-expert human writes code (potentially ages)
2. Non-expert human checks code correctness (also potentially ages)

AI solution:

1. AI generates code (instant)
2. Non-expert human reads, fully understands, corrects, and tests code, or hires someone to do all this (ages x 2)

Good example: automatically writing legal petitions, as done by DoNotPay.com

Existing solution:

1. Human writes a letter, checking it while they write (20 mins, or way more if they’ve never written a legal letter)
2. Human proofreads letter (1 min)

AI solution:

1. AI generates letter Time: instant
2. Human proofreads letter to make sure it isn’t crazy (1 min)

Good example: generate stock media, as done by Rosebud.ai

Existing solution:

1. Human goes out and produces a photoshoot/videoshoot, etc. (hours to days) or buys stock media (not cheap)
2. Human verifies that media looks okay (2 mins)

AI solution:

1. AI generates media (instant)
2. Human verifies that media looks okay (2 mins)

Questionable example: checkout-free stores

It’s not clear how much “AI” Amazon Go and its various competitors use. And to the extent that they do, it’s not clear how accurate they’re going to be able to make it. There’s a lot of innovation involved here that’ll be interesting to watch.

But bottom line: they’re going to have to spend some money to pay human-in-the-loop operators at the beginning, plus the usual human duties of cleaning and stocking the store. So we’ll see whether all the factors net out to a cost lower than that of just having employees in the store do the usual job.

Why bother?

Progress in mathematics is upstream of nearly all progress in science and technology. But unfortunately, progress in math is getting harder to make. Humanity’s mathematical knowledge is getting larger, more complex, and more Balkanized¹ by the year.

Better tools can help. Calculators, Wolfram Alpha and the like, Wikipedia, and even humble blogs play a huge role in modern mathematical research. But they don’t directly aid in the core task of math research: assembling ideas into a logical sequence that proves something interesting. Scratch paper, whiteboards, and word processors remain the only aids mathematicians have for this task.

In other words: mathematics today is like programming before IDEs and GitHub. It’s time we build equivalent tools for math research.

What exactly am I suggesting we build?

• A Proof Development Environment (PDE) web app
• A knowledge base linking together proofs written and published using the PDE
• Tight integration between prose proofs, formal proofs, and automated theorem provers

Proof Development Environment

A mathematical proof is a sequence of statements that leads logically from some premises to some conclusion. Each statement in a proof is justified by some combination of previous statements in the sequence.

Usually they’re written in prose and algebra, but here’s a visual example:

In this diagram, boxes are statements, and the sequence (or partially ordered set if you’re being fancy) of statements that makes up the proof is shown by the arrows: each statement logically follows from the statements pointing to it. Dotted-line boxes are statements that are themselves the result of proofs; the proof of the statement is inside the dotted-line box.

Nobody publishes proofs in a visual format like the above diagram. But behind every proof is a diagram like this, even if it only exists in the readers’ and writers’ heads.

Of course, the hard part of mathematics isn’t writing up a finished proof for publication. The hard part is coming up with the proof itself. Some people who develop proofs, like students, know exactly what premises they’re starting from and what conclusion they need to reach. But working mathematicians need to alternate between trying to prove things and trying to find interesting things they’re able to prove.

The process might proceed like this:

So, concretely: the proof development environment I’d like to build is an online editor for making directed acyclic graphs with expandable nodes, editable edges, and other nice features like

• LaTeX support for the text in the nodes
• Collaborative real-time editing
• Version history
• Contribution tracking
• Tags and comments
• Ability to save and share views (subsets of statements arranged in a certain way)
• Easily search, cite, and link to other proofs
• Ability to write statements in formal notation and use built-in proof checkers and proof assistants on statements so written

Proofscape is an underappreciated project that’s very close to what I’d like in a PDE. It’s built for presenting and explaining finished mathematical proofs, so some features would have to be added to make it useful for the proof development process. But it’s excellent. Rationale is a tool for argument mapping that’s similar in spirit and has a nice UI.

Mathematical Knowledge Base

“But it is not complicated. There’s just a lot of it!” - Richard Fenyman

If you’re reading some source code and encounter a function whose behavior you don’t understand, you can easily hunt down the code defining that function and read it. And if that code is itself confusing, you can hunt down the code for the functions used therein, etc., etc., until you understand what’s happening.

Mathematics, in principle, works the same way: you can justify every deduction made in the course of a (correct) mathematical proof, and you can justify every justification, etc., etc., all the way down until you hit axiomatic bedrock.

Unfortunately, unlike source code, most mathematical knowledge is not written in a complete, internally consistent form. Mathematical knowledge is scattered across textbooks, articles, websites like Wikipedia, Wolfram MathWorld, ProofWiki, and Math Overflow, and of course in mathematicians’ brains. This makes makes math hard to learn and understand, slows down mathematical progress, and in the worst case leads to mistakes.

A centralized knowledge base for mathematics would have many benefits: it would simplify search and learning, facilitate discovery, reduce duplicate effort, and be more amenable to computer augmentation. So it’s not surprising that people have tried to make one before. Notable attempts include the Lean mathematical libary (currently the most active), the Mizar Mathematical Library, and the Isabelle Archive of Formal Proofs.

I’m a huge fan of these projects and projects like them. But they haven’t been adopted by most mathematicians because their benefits do not outweigh the switching costs. They require a mathematician to learn a new formal language and replace their existing workflow with a proof-checker-based workflow, and the only immediate, obvious reward the mathematician gets is a guarantee of correctness for a proof they were pretty sure was correct already.

To get mathematicians to centralize their knowledge, we need to get them to adopt a tool that makes their current work easier, and build the knowledge base around that tool. To facilitate this, the online system that hosts the Proof Development Environment could also act as a publishing platform. Similar to GitHub’s public/private repo model, users can work on their proofs in private, but they can also make them public with one click. The network of published proofs, linked by citations, becomes the knowledge base.

How to solve the bootstrapping problem for the knowledge base? Hopefully we can convince a few true believers to produce an initial base of theorems, perhaps poached from spiritually similar projects like the Polymath Projects. It’s also important to make it easier for authors to cite proofs from within the knowledge base than from external sources. A well-designed citation feature in the PDE could allow inline search by name of proofs in knowledge base. And it’s important to make it easy for new users to contribute. In a PDE proof it’s easy to visually flag any theorems that are missing a citation, like Wikipedia’s red links, making it clear to users new and old where gaps are that need to be filled.

The most difficult issue in curating a mathematical knowledge base is handling multiple statements of the same theorem or slight variations thereof. Nobody wants a knowledge base clogged with restatements of the same ideas, but at the same time it’s useful to include multiple proofs of the same claim, or simpler versions of general theorems. This issue may solve itself by winner-take-all dynamics among citations, but other approaches may be necessary.

Attribution could be another issue. For proofs created entirely in the PDE, attribution should be as easy as looking at the edit history. But for proofs originally developed off-platform, one has to credit the original author of the proof and the person who added it.

Integration with Formal Mathematics tools

The examples I’ve given have all dealt with human-readable proofs (written in prose and LaTeX algebra) because enhancing mathematicians’ existing workflows is key to adoption. But in the longer term, I’m aligned with the QED Manifesto: we should get mathematics into formal notation and let computers loose on it.

Formalized mathematics is math written in a programming language rather than prose. A formal proof carries the exact same content as a prose proof, but it’s machine readable. This means it can be checked for correctness with proof verification software and, more importantly, someday expand upon itself autonomously via machine-intelligence-powered automated theorem provers. (Expanding the amount of training data for such a system is a major motivation for this project.) But, as mentioned above, almost no mathematician does their work in formalized notation. The costs to switching workflows outweighs the benefits.

Proofs written in the Proof Development Environment are a step closer to being formalized than standard prose proofs. The sequence of deductions denoted by the arrows in a PDE proof are an outline of a formal proof, mapping closely to ideas from Leslie Lamport and Freek Wiedijk. To encourage mathematicians to go all the way and formalize their proof, I’d like the PDE to be integrated with formal math languages like Lean, which has a burgeoning library of formally-proven mathematics of its own that could serve as a basis for the Knowledge Graph described above. Users can then choose to write statements in the PDE in these languages, and the PDE will run these languages’ verifiers automatically on the series of statements, visually displaying the state of correctness of the proof. Statements in a PDE proof will contain layers for multiple languages, so the same statement can be written in prose or in a proof language. In the longer term other features of formal math systems, like proof assistants, could be incorporated, effectively making the PDE a GUI for formalized mathematics.

Not all mathematics can or should be written in formal notation. Proof sketches and intuitions written in prose are valuable, and the PDE would support them. But formalized ideas are clearer and stronger, and they open up possibilities for intelligence augmentation for mathematicians.

Further Reading

• Augmenting Human Intellect: A Conceptual Framework (Douglas Englebart)
• Argument Maps (similar to Proof Development Environments)
• Polymath Projects (collaborative mathematics)
• Formal Proof Sketches (Freek Wiedijk)
• How to Write a 21st Century Proof (Leslie Lamport)
• The QED Manifesto
• Automated Reasoning (Stanford EOP)
• Has anyone thought about creating a formal proof wiki with verifier? (Math Overflow)
• Using Artificial Intelligence to Augment Human Intelligence (Shan Carter, Michael Nielsen)
• The Existential Risk of Mathematical Error (Gwern)

2020-04 Note: I think an MVP could be built on top of Roam + CSS styling + Lean. If you’d like to collaborate, please get in touch: mwcvitkovic@gmail.com. I can provide funding.

• Don’t attend any talk that’s being recorded.
  • You’ll get all the same info in 10% the time by skipping through the video later. Use the time to go meet people.
  • Caveat: if attending the talk will let you meet the speaker (usually it won’t), and you’re dying to meet them, you should go.
  • What if you have a question for the speaker? Email them after you watch the video. Then you get to correspond with them instead of being a faceless voice in their audience.
• Don’t eat any meal alone.
  • Ideally, get food with a friend and a bunch of their friends whom you don’t know. But eating with strangers is good too.
  • Book your lodging and travel so you can easily attend social events in the evenings, including the evening after the conference ends.
• Go to the poster/session/event where you can spend the most time in the smallest groups.
• Use poster sessions for free tutoring.
  • Find a poster with no audience on a topic you’d like to understand better, and buttonhole its owner into teaching it to you at your pace. Though obviously don’t monopolize them if other people show up at their poster.
• If you’re presenting, leave more time for questions than you’re supposed to.
  • Fire out all your interesting ideas as fast as possible (it never takes the whole allotted time to do this), then open it up for questions.
    • The organizers won’t stop you.
    • You and the audience will learn more.
  • Example: the last talk I gave was supposed to be 17 minutes with 3 minutes for questions. Instead I gave a 5 minute talk with 15 minutes for questions. It was vastly more productive for everyone.
  • Note: you still need to make a full-talk’s-worth of slides so you have visual aids to answer people’s questions.

Connections are easier to maintain remotely than to start remotely, so:

• Once you’re made a meaningful connection with someone, find someone else to talk to.
  • To be clear: I’m not saying you should flit around trying to collect as many business cards as possible. I’m saying you probably shouldn’t talk to one person for more than an hour. If you’ve talked that long, you’ll be able to continue the conversation after the conference, so go meet some new folks.

All that Psych 101 stuff matters, so:

• Prioritize invite-only or apply-to-attend events.
  • In-group bias
• Wait to contact people until at least a week after the conference.
  • By then they’ll have dealt with the deluge of emails from other people they met at the conference: Salience bias

The world could use fewer buttheads, so:

• Don’t try to impress people.
  • The ones worth impressing will see through it, and they always find it annoying. Just self-assuredly express your ignorance - everyone likes this.
  • Forcing yourself to only ask questions is a fun game to play at conferences.
• Focus on their problem.
  • If you’re at a technical conference, make your goal in every conversation to discover and understand a big problem the other person is trying to solve right now. And, if you can, see if you can help them solve it.
  • Outside of technical conferences, don’t overdo this. Make sure the person is in a problem-solving mood first.
• If you spot a wallflower, ask them to join your conversation.
  • If you can’t think of a natural way to do this, just say: “Hey, sorry - I have a rule that when I see someone who looks like they’re searching for a conversation to join, I always ask if they want to join mine. Want to?” (HT Dan Ariely)