‘x-risk’ directory
- See Also
- Gwern
-
Links
- “A US Engineer Had a Shocking Plan to Improve the Climate—Burn All Coal on Earth ”, Moynihan 2024
- “Remarks by Vice President Harris on the Future of Artificial Intelligence London, United Kingdom ”, Harris 2023
- “Will Releasing the Weights of Large Language Models Grant Widespread Access to Pandemic Agents? ”, Gopal et al 2023
- “Lessons For Humanity From The Extinction Of The Dinosaurs ”, Moynihan 2023
- “One Small Step for Man, One Giant Leap for Moon Microbes? Interpretations of Risk and the Limits of Quarantine in NASA’s Apollo Program ”, Degroot 2023
- “ChemCrow: Augmenting Large-Language Models With Chemistry Tools ”, Bran et al 2023
- “Impact of the Tambora Volcanic Eruption of 1815 on Islands and Relevance to Future Sunlight-Blocking Catastrophes ”, Wilson et al 2023
- “Is It Possible to Prepare for a Pandemic? ”, Omberg & Tabarrok 2022
- “DeepMind: The Podcast—Excerpts on AGI ”, Kiely 2022
- “Joan Rohlfing on How to Avoid Catastrophic Nuclear Blunders: The Interaction between Nuclear Weapons and Cybersecurity ”, Wiblin & Rohlfing 2022
- “Results of a 2020 Survey on Reporting Requirements and Practices for Biocontainment Laboratory Accidents ”, Manheim 2021
- “Late-Time Small Body Disruptions for Planetary Defense ”, King et al 2021
- “Synthetic Fat from Petroleum As a Resilient Food for Global Catastrophes: Preliminary Techno-Economic Assessment and Technology Roadmap ”, Martínez et al 2021
- “A Global Nucleic Acid Observatory for Biodefense and Planetary Health ”, Consortium 2021
- “Photovoltaic-Driven Microbial Protein Production Can Use Land and Sunlight More Efficiently Than Conventional Crops ”, Leger et al 2021
- “Goal Misgeneralization in Deep Reinforcement Learning ”, Koch et al 2021
- “Putting the Humanity into Inhuman Systems: How Human Factors and Ergonomics Can Be Used to Manage the Risks Associated With Artificial General Intelligence ”, Salmon et al 2020
- “The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare ”, Snyder-Beattie et al 2020
- “Possible Takeaways from the Coronavirus Pandemic for Slow AI Takeoff ”, Vika 2020
- “Counterproductive Altruism: The Other Heavy Tail ”, Kokotajlo & Oprea 2020
- “What Intellectual Progress Did I Make In The 2010s? ”, Alexander 2020
- “2019 AI Alignment Literature Review and Charity Comparison ”, Larks 2019
- “The Vulnerable World Hypothesis ”, Bostrom 2019
- “DeepMind and Google: the Battle to Control Artificial Intelligence. Demis Hassabis Founded a Company to Build the World’s Most Powerful AI. Then Google Bought Him Out. Hal Hodson Asks Who Is in Charge ”, Hodson 2019
- “Forecasting Transformative AI: An Expert Survey ”, Gruetzemacher et al 2019
- “The Psychology of Existential Risk: Moral Judgments about Human Extinction ”, Schubert et al 2019
- “Dissolving the Fermi Paradox ”, Sandberg et al 2018
- “The Silurian Hypothesis: Would It Be Possible to Detect an Industrial Civilization in the Geological Record? ”, Schmidt & Frank 2018
- “The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities ”, Lehman et al 2018
- “Multiverse-Wide Cooperation via Correlated Decision Making ”, Oesterheld 2018
- “There’s No Fire Alarm for Artificial General Intelligence ”, Yudkowsky 2017
- “Some Key Ways Which I'Ve Changed My Mind Over The Last Several Years ”, Karnofsky 2016
- “The Singularity: A Philosophical Analysis ”, Chalmers 2016
- “Singularity and Inevitable Doom ”, Prinz 2013
- “Doing Enough ”
- “Metamagical Themas: Sanity and Survival ”, Hofstadter 2012
- “Bayesian Analysis of the Astrobiological Implications of Life’s Early Emergence on Earth ”, Spiegel & Turner 2012
- “Intelligence Explosion: Evidence and Import ”, Muehlhauser & Salamon 2012
- “Chapter 13: Corporations ”
- “Aum Shinrikyo: Insights Into How Terrorists Develop Biological and Chemical Weapons ”, Danzig et al 2011
- “On the Origin and Evolution of Life in the Galaxy ”, McCabe & Lucas 2011
- “Alexander Kruel’s AI Risk Interview Series (2011–2013) ”, Kruel 2011
- “Anthropic Shadow: Observation Selection Effects and Human Extinction Risks ”, Ćirković et al 2010
- “Myopic Voters and Natural Disaster Policy ”, Healy & Malhotra 2009
- “The Unabomber Was Right ”, Kelly 2009
- “The Physical Principles of Thermonuclear Explosives, Inertial Confinement Fusion, and the Quest for Fourth Generation Nuclear Weapons ”, Gsponer & Hurni 2009
- “Probing the Improbable: Methodological Challenges for Risks With Low Probabilities and High Stakes ”, Ord et al 2008
- “Implications of an Anthropic Model of Evolution for Emergence of Complex Life and Intelligence ”, Watson 2008
- “The Optimistic Thought Experiment ”, Thiel 2008
- The Nuclear Taboo: The United States and the Non-Use of Nuclear Weapons Since 1945, Tannenwald 2008
- “Five or Six Step Scenario for Evolution? ”, Carter 2007
- “Aberrant Innate Immune Response in Lethal Infection of Macaques With the 1918 Influenza Virus ”, Kobasa et al 2007
- “Forbidden Knowledge ”, Kempner et al 2005
- “Must Early Life Be Easy? The Rhythm of Major Evolutionary Transitions ”, Hanson 1998
- “Mathematics on a Distant Planet ”
- “The Sverdlovsk Anthrax Outbreak of 1979 ”, Meselson et al 1994
- “Implications of the Copernican Principle for Our Future Prospects ”, III 1993
- “The Logic of Failure [And Discussion] ”, Dörner et al 1990
- “Nuclear Fission: Reaction to the Discovery in 1939 ”, Badash et al 1986
- “Metamagical Themas: Sanity and Survival ”, Hofstadter 1985
- “The Anthropic Principle and Its Implications for Biological Evolution ”, Carter 1983
- “The Loving Parent Meets the Selfish Gene ”, Gray & Wolfe 1980
- “Rejoinder to Gray and Wolfe ”, Pascal 1980
- “Human Tragedy and Natural Selection ”, Pascal 1978
- “Large Number Coincidences and the Anthropic Principle in Cosmology ”, Carter 1974
- “All You Ever Wanted to Know about MIRV and ICBM Calculations but Were Not Cleared to Ask ”, Davis & Schilling 1973
- “Speculations Concerning the First Ultraintelligent Machine ”, Good 1966
- “Can We Survive Technology? ”, Neumann 1955
- “Don’t Worry—It Can’t Happen ”, Harrington 1940
- “Alleged Sins of Science ”, Millikan 1930
- “David Denkenberger on Using Paper Mills and Seaweed to Feed Everyone in a Catastrophe, Ft Sahil Shah ”
- “2022 Expert Survey on Progress in AI ”
- “Looking Back at the Future of Humanity Institute ”
- “Book Review: Barriers to Bioweapons ”
- “Detecting Genetically Engineered Viruses With Metagenomic Sequencing ”
- “1972 Talk at CERN on Scientific Research ”, Grothendieck 2025
- “2017: Universal Paperclips ”
- “Homepage of Paul F. Christiano ”, Christiano 2025
- “Envisioning a World Immune to Global Catastrophic Biological Risks ”
- “MIT Researchers Ordered and Combined Parts of the 1918 Pandemic Influenza Virus. Did They Expose a Security Flaw? ”
- “Thomas Moynihan—Homepage ”, Moynihan 2025
- “Apocalyptic Log ”
- “The Dangers of Mirrored Life ”
- “Your Book Review: The Family That Couldn’t Sleep ”
- “Forget about Drones, Forget about Dystopian Sci-Fi—A Terrifying New Generation of Autonomous Weapons Is Already Here. Meet the Small Band of Dedicated Optimists Battling Nefarious Governments and Bureaucratic Tedium to Stop the Proliferation of Killer Robots And, Just Maybe, save Humanity from Itself. ”
- “Optimality Is the Tiger, and Agents Are Its Teeth ”
- “Liability Regimes for AI ”
- “We Know Lab Leaks Are Possible, and One Could Start a New Pandemic ”
- Sort By Magic
- Wikipedia
- Miscellaneous
- Bibliography
See Also
Gwern
“Why Tool AIs Want to Be Agent AIs ”, Gwern 2016
Links
“A US Engineer Had a Shocking Plan to Improve the Climate—Burn All Coal on Earth ”, Moynihan 2024
A US engineer had a shocking plan to improve the climate—burn all coal on Earth
“Remarks by Vice President Harris on the Future of Artificial Intelligence London, United Kingdom ”, Harris 2023
Remarks by Vice President Harris on the Future of Artificial Intelligence London, United Kingdom :
“Will Releasing the Weights of Large Language Models Grant Widespread Access to Pandemic Agents? ”, Gopal et al 2023
Will releasing the weights of large language models grant widespread access to pandemic agents?
“Lessons For Humanity From The Extinction Of The Dinosaurs ”, Moynihan 2023
“One Small Step for Man, One Giant Leap for Moon Microbes? Interpretations of Risk and the Limits of Quarantine in NASA’s Apollo Program ”, Degroot 2023
“ChemCrow: Augmenting Large-Language Models With Chemistry Tools ”, Bran et al 2023
ChemCrow: Augmenting large-language models with chemistry tools
“Impact of the Tambora Volcanic Eruption of 1815 on Islands and Relevance to Future Sunlight-Blocking Catastrophes ”, Wilson et al 2023
“Is It Possible to Prepare for a Pandemic? ”, Omberg & Tabarrok 2022
“DeepMind: The Podcast—Excerpts on AGI ”, Kiely 2022
“Joan Rohlfing on How to Avoid Catastrophic Nuclear Blunders: The Interaction between Nuclear Weapons and Cybersecurity ”, Wiblin & Rohlfing 2022
“Results of a 2020 Survey on Reporting Requirements and Practices for Biocontainment Laboratory Accidents ”, Manheim 2021
“Late-Time Small Body Disruptions for Planetary Defense ”, King et al 2021
“Synthetic Fat from Petroleum As a Resilient Food for Global Catastrophes: Preliminary Techno-Economic Assessment and Technology Roadmap ”, Martínez et al 2021
“A Global Nucleic Acid Observatory for Biodefense and Planetary Health ”, Consortium 2021
A Global Nucleic Acid Observatory for Biodefense and Planetary Health
“Photovoltaic-Driven Microbial Protein Production Can Use Land and Sunlight More Efficiently Than Conventional Crops ”, Leger et al 2021
“Goal Misgeneralization in Deep Reinforcement Learning ”, Koch et al 2021
“Putting the Humanity into Inhuman Systems: How Human Factors and Ergonomics Can Be Used to Manage the Risks Associated With Artificial General Intelligence ”, Salmon et al 2020
“The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare ”, Snyder-Beattie et al 2020
The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare
“Possible Takeaways from the Coronavirus Pandemic for Slow AI Takeoff ”, Vika 2020
Possible takeaways from the coronavirus pandemic for slow AI takeoff
“Counterproductive Altruism: The Other Heavy Tail ”, Kokotajlo & Oprea 2020
“What Intellectual Progress Did I Make In The 2010s? ”, Alexander 2020
“2019 AI Alignment Literature Review and Charity Comparison ”, Larks 2019
“The Vulnerable World Hypothesis ”, Bostrom 2019
“DeepMind and Google: the Battle to Control Artificial Intelligence. Demis Hassabis Founded a Company to Build the World’s Most Powerful AI. Then Google Bought Him Out. Hal Hodson Asks Who Is in Charge ”, Hodson 2019
“Forecasting Transformative AI: An Expert Survey ”, Gruetzemacher et al 2019
“The Psychology of Existential Risk: Moral Judgments about Human Extinction ”, Schubert et al 2019
The Psychology of Existential Risk: Moral Judgments about Human Extinction
“Dissolving the Fermi Paradox ”, Sandberg et al 2018
“The Silurian Hypothesis: Would It Be Possible to Detect an Industrial Civilization in the Geological Record? ”, Schmidt & Frank 2018
“The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities ”, Lehman et al 2018
“Multiverse-Wide Cooperation via Correlated Decision Making ”, Oesterheld 2018
“There’s No Fire Alarm for Artificial General Intelligence ”, Yudkowsky 2017
“Some Key Ways Which I'Ve Changed My Mind Over The Last Several Years ”, Karnofsky 2016
Some Key Ways Which I've Changed My Mind Over The Last Several Years :
View PDF:
“The Singularity: A Philosophical Analysis ”, Chalmers 2016
“Singularity and Inevitable Doom ”, Prinz 2013
Singularity and Inevitable Doom :
View PDF:
“Doing Enough ”
“Metamagical Themas: Sanity and Survival ”, Hofstadter 2012
“Bayesian Analysis of the Astrobiological Implications of Life’s Early Emergence on Earth ”, Spiegel & Turner 2012
Bayesian analysis of the astrobiological implications of life’s early emergence on Earth
“Intelligence Explosion: Evidence and Import ”, Muehlhauser & Salamon 2012
“Chapter 13: Corporations ”
“Aum Shinrikyo: Insights Into How Terrorists Develop Biological and Chemical Weapons ”, Danzig et al 2011
Aum Shinrikyo: Insights Into How Terrorists Develop Biological and Chemical Weapons
“On the Origin and Evolution of Life in the Galaxy ”, McCabe & Lucas 2011
“Alexander Kruel’s AI Risk Interview Series (2011–2013) ”, Kruel 2011
“Anthropic Shadow: Observation Selection Effects and Human Extinction Risks ”, Ćirković et al 2010
Anthropic Shadow: Observation Selection Effects and Human Extinction Risks
“Myopic Voters and Natural Disaster Policy ”, Healy & Malhotra 2009
“The Unabomber Was Right ”, Kelly 2009
“The Physical Principles of Thermonuclear Explosives, Inertial Confinement Fusion, and the Quest for Fourth Generation Nuclear Weapons ”, Gsponer & Hurni 2009
“Probing the Improbable: Methodological Challenges for Risks With Low Probabilities and High Stakes ”, Ord et al 2008
Probing the Improbable: Methodological Challenges for Risks with Low Probabilities and High Stakes
“Implications of an Anthropic Model of Evolution for Emergence of Complex Life and Intelligence ”, Watson 2008
Implications of an Anthropic Model of Evolution for Emergence of Complex Life and Intelligence
“The Optimistic Thought Experiment ”, Thiel 2008
The Nuclear Taboo: The United States and the Non-Use of Nuclear Weapons Since 1945, Tannenwald 2008
The Nuclear Taboo: The United States and the Non-Use of Nuclear Weapons Since 1945
“Five or Six Step Scenario for Evolution? ”, Carter 2007
“Aberrant Innate Immune Response in Lethal Infection of Macaques With the 1918 Influenza Virus ”, Kobasa et al 2007
Aberrant innate immune response in lethal infection of macaques with the 1918 influenza virus
“Forbidden Knowledge ”, Kempner et al 2005
“Must Early Life Be Easy? The Rhythm of Major Evolutionary Transitions ”, Hanson 1998
Must Early Life Be Easy? The Rhythm of Major Evolutionary Transitions
“Mathematics on a Distant Planet ”
“The Sverdlovsk Anthrax Outbreak of 1979 ”, Meselson et al 1994
“Implications of the Copernican Principle for Our Future Prospects ”, III 1993
Implications of the Copernican principle for our future prospects
“The Logic of Failure [And Discussion] ”, Dörner et al 1990
“Nuclear Fission: Reaction to the Discovery in 1939 ”, Badash et al 1986
“Metamagical Themas: Sanity and Survival ”, Hofstadter 1985
“The Anthropic Principle and Its Implications for Biological Evolution ”, Carter 1983
The anthropic principle and its implications for biological evolution
“The Loving Parent Meets the Selfish Gene ”, Gray & Wolfe 1980
“Rejoinder to Gray and Wolfe ”, Pascal 1980
“Human Tragedy and Natural Selection ”, Pascal 1978
“Large Number Coincidences and the Anthropic Principle in Cosmology ”, Carter 1974
Large Number Coincidences and the Anthropic Principle in Cosmology
“All You Ever Wanted to Know about MIRV and ICBM Calculations but Were Not Cleared to Ask ”, Davis & Schilling 1973
All You Ever Wanted to Know about MIRV and ICBM Calculations but Were Not Cleared to Ask :
View PDF:
“Speculations Concerning the First Ultraintelligent Machine ”, Good 1966
“Can We Survive Technology? ”, Neumann 1955
“Don’t Worry—It Can’t Happen ”, Harrington 1940
“Alleged Sins of Science ”, Millikan 1930
View PDF:
“David Denkenberger on Using Paper Mills and Seaweed to Feed Everyone in a Catastrophe, Ft Sahil Shah ”
David Denkenberger on using paper mills and seaweed to feed everyone in a catastrophe, ft Sahil Shah :
“2022 Expert Survey on Progress in AI ”
“Looking Back at the Future of Humanity Institute ”
“Book Review: Barriers to Bioweapons ”
“Detecting Genetically Engineered Viruses With Metagenomic Sequencing ”
Detecting Genetically Engineered Viruses With Metagenomic Sequencing :
“1972 Talk at CERN on Scientific Research ”, Grothendieck 2025
“2017: Universal Paperclips ”
“Homepage of Paul F. Christiano ”, Christiano 2025
“Envisioning a World Immune to Global Catastrophic Biological Risks ”
Envisioning a world immune to global catastrophic biological risks :
“MIT Researchers Ordered and Combined Parts of the 1918 Pandemic Influenza Virus. Did They Expose a Security Flaw? ”
“Thomas Moynihan—Homepage ”, Moynihan 2025
“Apocalyptic Log ”
“The Dangers of Mirrored Life ”
“Your Book Review: The Family That Couldn’t Sleep ”
“Forget about Drones, Forget about Dystopian Sci-Fi—A Terrifying New Generation of Autonomous Weapons Is Already Here. Meet the Small Band of Dedicated Optimists Battling Nefarious Governments and Bureaucratic Tedium to Stop the Proliferation of Killer Robots And, Just Maybe, save Humanity from Itself. ”
“Optimality Is the Tiger, and Agents Are Its Teeth ”
“Liability Regimes for AI ”
“We Know Lab Leaks Are Possible, and One Could Start a New Pandemic ”
We know lab leaks are possible, and one could start a new pandemic
Sort By Magic
Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.
Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.
risk-assessment
technological-singularity
biological-dynamics
existential-implications
Wikipedia
Miscellaneous
-
/doc/existential-risk/2017-barrett.pdf:View PDF:
-
/doc/existential-risk/2016-lipsitch.pdf:View PDF:
-
/doc/existential-risk/2011-05-10-givewell-holdenkarnofskyjaantallinn.doc:View Word document:
/doc/existential-risk/2011-05-10-givewell-holdenkarnofskyjaantallinn.doc -
/doc/fiction/science-fiction/1990-dansimmons-thefallofhyperion-ch41-ummonquotesoceanus.png -
https://80000hours.org/podcast/episodes/tom-moynihan-prior-generations/:View External Link:
https://80000hours.org/podcast/episodes/tom-moynihan-prior-generations/ -
https://antiapocalyptus.substack.com/p/interview-thomas-moynihan-the-discovery -
https://blog.nuclearsecrecy.com/2012/09/12/in-search-of-a-bigger-boom/: -
https://bounded-regret.ghost.io/base-rates-of-catastrophes/: -
https://constructionphysics.substack.com/p/could-we-stop-yellowstone-from-erupting -
https://www.newyorker.com/news/annals-of-a-warming-planet/what-to-do-with-climate-emotions -
https://www.openphilanthropy.org/research/detection-of-near-earth-asteroids/ -
https://www.reddit.com/r/unsong/comments/zckyg0/completing_edward_tellers_atom_alphabet_with/ -
https://www.thelancet.com/journals/lanmic/article/PIIS2666-5247(23)00319-1/fulltext -
https://www.wired.com/story/synthetic-dna-us-biden-regulation/:View External Link:
https://www.wired.com/story/synthetic-dna-us-biden-regulation/ -
View External Link:
Bibliography
-
https://www.lesswrong.com/posts/SbAgRYo8tkHwhd9Qx/deepmind-the-podcast-excerpts-on-agi: “DeepMind: The Podcast—Excerpts on AGI ”, -
https://80000hours.org/podcast/episodes/joan-rohlfing-avoiding-catastrophic-nuclear-blunders/#the-interaction-between-nuclear-weapons-and-cybersecurity-011018: “Joan Rohlfing on How to Avoid Catastrophic Nuclear Blunders: The Interaction between Nuclear Weapons and Cybersecurity ”, -
https://www.liebertpub.com/doi/full/10.1089/ast.2019.2149: “The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare ”, -
https://www.lesswrong.com/posts/SmDziGM9hBjW9DKmf/2019-ai-alignment-literature-review-and-charity-comparison: “2019 AI Alignment Literature Review and Charity Comparison ”, -
https://www.economist.com/1843/2019/03/01/deepmind-and-google-the-battle-to-control-artificial-intelligence: “DeepMind and Google: the Battle to Control Artificial Intelligence. Demis Hassabis Founded a Company to Build the World’s Most Powerful AI. Then Google Bought Him Out. Hal Hodson Asks Who Is in Charge ”, -
1985-hofstadter: “Metamagical Themas: Sanity and Survival ”, -
2011-danzig.pdf: “Aum Shinrikyo: Insights Into How Terrorists Develop Biological and Chemical Weapons ”, -
2005-kempner.pdf: “Forbidden Knowledge ”, -
1966-good.pdf: “Speculations Concerning the First Ultraintelligent Machine ”, -
https://paulfchristiano.com/: “Homepage of Paul F. Christiano ”,