{"id":78641,"date":"2022-08-25T12:40:25","date_gmt":"2022-08-25T12:40:25","guid":{"rendered":"https:\/\/80000hours.org\/?p=78641"},"modified":"2024-09-26T15:43:13","modified_gmt":"2024-09-26T15:43:13","slug":"what-could-an-ai-caused-existential-catastrophe-actually-look-like","status":"publish","type":"article","link":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/","title":{"rendered":"What could an AI-caused existential catastrophe actually look&nbsp;like?"},"content":{"rendered":"<div id=\"toc_container\" class=\"toc_white no_bullets\"><p class=\"toc_title\">Table of Contents<\/p><ul class=\"toc_list\"><li><a href=\"#actually-take-power\"><span class=\"toc_number toc_depth_1\">1<\/span> How could a power-seeking AI actually take power?<\/a><\/li><li><a href=\"#how-could-the-full-story-play-out\"><span class=\"toc_number toc_depth_1\">2<\/span> How could the full story play out?<\/a><ul><li><a href=\"#getting-what-you-measure\"><span class=\"toc_number toc_depth_2\">2.1<\/span> Existential catastrophe through getting what you measure<\/a><\/li><li><a href=\"#superintelligence\"><span class=\"toc_number toc_depth_2\">2.2<\/span> Existential catastrophe through a single extremely advanced artificial intelligence<\/a><\/li><\/ul><\/li><li><a href=\"#return-to-the-full-article\"><span class=\"toc_number toc_depth_1\">3<\/span> Return to the full article<\/a><\/li><li><a href=\"#or-if-youre-new-start-from-the-beginning\"><span class=\"toc_number toc_depth_1\">4<\/span> Or, if you&#8217;re new, start from the beginning<\/a><\/li><\/ul><\/div>\n<div class=\"well bg-gray-lighter margin-bottom margin-top padding-top-small padding-bottom-small\">\n<p>This article forms part of our explanation of <a href=\"\/problem-profiles\/artificial-intelligence\/\">risks from artificial intelligence<\/a>. If you&#8217;re interested in understanding not just <em>how<\/em> an AI system could cause an existential catastrophe, but also <em>why<\/em> we&#8217;re worried things like this will happen, take a look at our full <a href=\"\/problem-profiles\/artificial-intelligence\">problem profile on risks from AI<\/a>.<\/p>\n<\/div>\n<p>At 5:29 AM on July 16, 1945, deep in the Jornada del Muerto desert in New Mexico, the Manhattan Project carried out the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Trinity_(nuclear_test)\">world&#8217;s first successful test of a nuclear weapon<\/a>.<\/p>\n<p>From that moment, we&#8217;ve had the technological capacity to wipe out humanity.<\/p>\n<p>But if you asked someone in 1945 to predict exactly how this risk would play out, they would almost certainly have got it wrong. They may have thought there would have been more widespread use of nuclear weapons in World War II. They certainly would not have predicted the fall of the USSR 45 years later. Current experts are concerned about <a href=\"https:\/\/en.wikipedia.org\/wiki\/India%E2%80%93Pakistan_relations#Weapons_of_mass_destruction\">India\u2013Pakistan nuclear conflict<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/North_Korea_and_weapons_of_mass_destruction\">North Korean state action<\/a>, but 1945 was before even the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Partition_of_India\">partition of India<\/a> or the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Korean_War\">Korean War<\/a>.<\/p>\n<p>That is to say, you&#8217;d have real difficulty predicting anything about how nuclear weapons would be used. It would have been even harder to make these predictions in 1933, when <a href=\"https:\/\/en.wikipedia.org\/wiki\/History_of_nuclear_weapons#Physics_and_politics_in_the_1930s_and_1940s\">Leo Szilard first realised that a nuclear chain reaction of immense power could be possible<\/a>, without any concrete idea of what these weapons would look like.<\/p>\n<p>Despite this difficulty, you wouldn&#8217;t be wrong to be concerned.<\/p>\n<p>In our problem profile on AI, we <a href=\"\/problem-profiles\/artificial-intelligence\/#power-seeking-ai\">describe a very general way<\/a> in which advancing AI could go wrong. But there are lots of specifics we can&#8217;t know much about at this point. Maybe there will be a single transformative AI system, or maybe there will be many; there could be very fast growth in the capabilities of AI, or very slow growth. Each scenario will look a little different, and carry different risks. And the specific problems that arise in any <em>one<\/em> scenario are necessarily less likely to happen than the overall risk.<\/p>\n<p>Despite not knowing how things will play out, it may still be useful to look at some concrete possibilities of how things could go wrong.<\/p>\n<p>In particular, we argued in the full profile that sufficiently advanced systems might be able to <a href=\"\/problem-profiles\/artificial-intelligence\/#power-seeking-ai\">take power away from humans<\/a> \u2014  <em>how<\/em> could that possibly happen?<\/p>\n<h2><span id=\"actually-take-power\" class=\"toc-anchor\"><\/span>How could a power-seeking AI actually take power?<\/h2>\n<p>Here are seven possible techniques that could be used by a power-seeking AI (or multiple AI systems working together) to actually gain power.<\/p>\n<p>These techniques could all interact with one another, and it&#8217;s difficult to say at this point (years or decades before the technology exists) which are most likely to be used. Also, systems more intelligent than humans could develop plans to seek power that we haven&#8217;t yet thought of.<\/p>\n<div class=\"panel-group\" id=\"custom-collapse-0\">\n<div class=\"panel panel-default panel-collapse\">\n<div class=\"panel-heading\">\n<h4 class=\"panel-title\"><a class=\"no-visited-styling collapsed\" data-toggle=\"collapse\" data-target=\"#-0\">1. Hacking<\/a><\/h4>\n<\/p><\/div>\n<div id=\"-0\" class=\"panel-body-collapse collapse\" data-80k-event-label=\"1. Hacking\">\n<div class=\"panel-body\">\n<p>Software is absolutely full of vulnerabilities. The US National Institute of Standards and Technology reported <a href=\"https:\/\/nvd.nist.gov\/general\/visualizations\/vulnerability-visualizations\/cvss-severity-distribution-over-time\">over 8,000 vulnerabilities found in systems across the world in 2021<\/a> \u2014 an average of 50 per day.<\/p>\n<p>Most of these are small, but every so often they are used to cause huge chaos. The list of most expensive crypto hacks <a href=\"https:\/\/rekt.news\/leaderboard\/\">keeps getting new entrants<\/a> \u2014 as of March 2022, the largest was $624 million stolen from Ronin Network. <a href=\"https:\/\/rekt.news\/ronin-rekt\/\">And nobody noticed for six days<\/a>.<\/p>\n<p>One expert we spoke to said that professional &#8216;<a href=\"https:\/\/en.wikipedia.org\/wiki\/Red_team\">red teams<\/a>&#8217; \u2014 security staff whose job it is to find vulnerabilities in systems \u2014 frequently manage to infiltrate their clients, including crucial and powerful infrastructure like banks and national energy grids.<\/p>\n<p>In 2010, the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Stuxnet\">Stuxnet virus<\/a> successfully managed to destroy Iranian nuclear enrichment centrifuges \u2014 despite these centrifuges being completely disconnected from the internet \u2014 marking the first time a piece of malware was used to cause physical damage. A Russian hack in 2016 was used to <a href=\"https:\/\/www.wired.com\/story\/worst-hacks-of-the-decade\/\">cause blackouts in Ukraine<\/a>.<\/p>\n<p>All this has happened with just the hacking abilities that humans currently have. An AI with highly advanced capabilities seems likely to be able to systematically hack almost any system on Earth, especially if we automate more and more crucial infrastructure over time. And if it did use hacking to get large amounts of money or compromise a crucial system, that would be a form of real-world power over humans.<\/p>\n<\/div><\/div><\/div>\n<div class=\"panel panel-default panel-collapse\">\n<div class=\"panel-heading\">\n<h4 class=\"panel-title\"><a class=\"no-visited-styling collapsed\" data-toggle=\"collapse\" data-target=\"#-1\">2. Gaining financial resources<\/a><\/h4>\n<\/p><\/div>\n<div id=\"-1\" class=\"panel-body-collapse collapse\" data-80k-event-label=\"2. Gaining financial resources\">\n<div class=\"panel-body\">\n<p>We already have computer systems with huge financial resources making automated decisions \u2014 and these already go wrong sometimes, for example leading to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Flash_crash\">flash crashes<\/a> in the market.<\/p>\n<p>There are lots of ways a truly advanced planning AI system could gain financial resources. It could steal (e.g. through hacking); become very good at investing or high-speed trading; develop and sell products and services; or try to gain influence or control over wealthy people, other AI systems, or organisations.<\/p>\n<\/div><\/div><\/div>\n<div class=\"panel panel-default panel-collapse\">\n<div class=\"panel-heading\">\n<h4 class=\"panel-title\"><a class=\"no-visited-styling collapsed\" data-toggle=\"collapse\" data-target=\"#-2\">3. Persuading or coercing humans<\/a><\/h4>\n<\/p><\/div>\n<div id=\"-2\" class=\"panel-body-collapse collapse\" data-80k-event-label=\"3. Persuading or coercing humans\">\n<div class=\"panel-body\">\n<p>Having influence over specific people or groups of people is an important way that individuals seek power in our current society. Given that AIs can already communicate (if imperfectly) in natural language with humans (e.g. via chatbots), a more advanced and strategic AI could use this ability to manipulate human actors to its own ends.<\/p>\n<p>Advanced planning AI systems might be able to do this through things like paying humans to do things; promising (whether true or false) future wealth, power, or happiness; persuading (e.g. through deception or appeals to morality or ideology); or coercing (e.g. blackmail or physical threats).<\/p>\n<p>Relatedly, as we <a href=\"\/problem-profiles\/artificial-intelligence\/#incentives-and-deception\">discuss in our AI problem profile<\/a>, it&#8217;s plausible one of the instrumental goals of an advanced planning AI would be deceiving people with the power to shut the system down into thinking that the system is indeed aligned.<\/p>\n<p>The better our monitoring and oversight systems, the harder it will be for AI systems to do this. Conversely, the worse these systems are (or if the AI has hacked the systems), the easier it will be for AI systems to deceive humans.<\/p>\n<p>If AI systems are good at deceiving humans, it also becomes easier for them to use the other techniques on this list.<\/p>\n<\/div><\/div><\/div>\n<div class=\"panel panel-default panel-collapse\">\n<div class=\"panel-heading\">\n<h4 class=\"panel-title\"><a class=\"no-visited-styling collapsed\" data-toggle=\"collapse\" data-target=\"#-3\">4. Gaining broader social influence<\/a><\/h4>\n<\/p><\/div>\n<div id=\"-3\" class=\"panel-body-collapse collapse\" data-80k-event-label=\"4. Gaining broader social influence\">\n<div class=\"panel-body\">\n<p>We could imagine AI systems replicating things like <a href=\"https:\/\/en.wikipedia.org\/wiki\/Russian_interference_in_the_2016_United_States_elections\">Russia&#8217;s interference in the 2016 US election<\/a>, manipulating political and moral discourse through social media posts and other online content.<\/p>\n<p>There are plenty of other ways of gaining social influence. These include: intervening in legal processes (e.g. aiding in lobbying or <a href=\"https:\/\/en.wikipedia.org\/wiki\/Regulatory_capture\">regulatory capture<\/a>), weakening human institutions, or empowering specific destabilising actors (e.g. particular politicians, corporations, or rogue actors like terrorists).<\/p>\n<\/div><\/div><\/div>\n<div class=\"panel panel-default panel-collapse\">\n<div class=\"panel-heading\">\n<h4 class=\"panel-title\"><a class=\"no-visited-styling collapsed\" data-toggle=\"collapse\" data-target=\"#-4\">5. Developing new technology<\/a><\/h4>\n<\/p><\/div>\n<div id=\"-4\" class=\"panel-body-collapse collapse\" data-80k-event-label=\"5. Developing new technology\">\n<div class=\"panel-body\">\n<p>It&#8217;s clear that developing advanced technology is a route for humans (or groups of humans) to gain power.<\/p>\n<p>Some advanced capabilities seem likely to make it possible for AI systems to develop new technology. For example, AI systems may be very good at collating and understanding information on the internet and in academic journals. Also, there are already AI tools that assist in writing code, so it seems plausible that coding new products and systems could become a key AI capability.<\/p>\n<p>It&#8217;s not clear <em>what<\/em> technology an AI system could develop. If the capabilities of the system are similar to our own, it could develop things we&#8217;re currently working on. But if the system&#8217;s capabilities are well beyond our own, it&#8217;s harder for us to figure out what could be developed \u2014 and this possibility seems even more dangerous.<\/p>\n<p>We talk more about the specific risks of AI-developed technology <a href=\"\/problem-profiles\/artificial-intelligence\/#dangerous-new-technology\">in our full problem profile on AI<\/a>.<\/p>\n<\/div><\/div><\/div>\n<div class=\"panel panel-default panel-collapse\">\n<div class=\"panel-heading\">\n<h4 class=\"panel-title\"><a class=\"no-visited-styling collapsed\" data-toggle=\"collapse\" data-target=\"#-5\">6. Scaling up its own capabilities<\/a><\/h4>\n<\/p><\/div>\n<div id=\"-5\" class=\"panel-body-collapse collapse\" data-80k-event-label=\"6. Scaling up its own capabilities\">\n<div class=\"panel-body\">\n<p>If an AI system is able to improve its own capabilities, that could be used to improve specific abilities (like others on this list) it could use to seek and keep power.<\/p>\n<p>To do this, the system could target the three inputs to modern deep learning systems (algorithms, compute, and data):<\/p>\n<ul>\n<li>The system may have advanced capabilities in areas that allow it to improve AI algorithms. For example, the AI system may be particularly good at programming or ML development.<\/li>\n<li>The system may be able to increase its own access to computational resources, which it could then use for training, to speed itself up, or to run copies of itself.<\/li>\n<li>The system could gain access to data that humans aren&#8217;t able to gather, using this data for training purposes to improve its own capabilities.<\/li>\n<\/ul>\n<\/div><\/div><\/div>\n<div class=\"panel panel-default panel-collapse\">\n<div class=\"panel-heading\">\n<h4 class=\"panel-title\"><span id=\"developing-destructive-capacity\" class=\"toc-anchor\"><\/span><a class=\"no-visited-styling collapsed\" data-toggle=\"collapse\" data-target=\"#-6\">7. Developing destructive capacity<\/a><\/h4>\n<\/p><\/div>\n<div id=\"-6\" class=\"panel-body-collapse collapse\" data-80k-event-label=\"7. Developing destructive capacity\">\n<div class=\"panel-body\">\n<p>Most dangerously, one way of gaining power is by having the ability to threaten destruction. This could be used to gain other things on this list (like social influence), or the other things on this list could be used to gain destructive capabilities (like hacking military systems).<\/p>\n<p>Here are some possible mechanisms for gaining destructive power:<\/p>\n<ul>\n<li>Gaining control over autonomous weapons like drones<\/li>\n<li>Developing systems for monitoring and surveillance of humans<\/li>\n<li>Attacking things humans need to survive, like water, food, or oxygen<\/li>\n<li>Producing or gaining access to biological, chemical, or nuclear weapons<\/li>\n<\/ul>\n<p>Ultimately, <a href=\"\/problem-profiles\/artificial-intelligence\/#power-seeking-ai\">making humans extinct<\/a> would completely remove any threat that humans would ever pose to the power of an AI system.<\/p>\n<\/div><\/div><\/div>\n<\/div>\n<h2><span id=\"how-could-the-full-story-play-out\" class=\"toc-anchor\"><\/span>How could the full story play out?<\/h2>\n<p>Hopefully you now have a slightly stronger intuition for how AI systems could attempt to seek power.<\/p>\n<p>But which (if any) of these techniques will be used, and how, really depends on how other aspects of the risk play out. How rapidly will AI capabilities improve? Will there be many advanced AI systems or just one?<\/p>\n<p>Over the past few years, researchers in the fields of technical AI safety and AI governance have developed a number of stories describing the sorts of ways in which a power-seeking AI system could cause an existential catastrophe. Sam Clarke (an AI governance researcher at the University of Cambridge) and Samuel Martin (an AI safety researcher at King&#8217;s College London) <a href=\"https:\/\/www.alignmentforum.org\/posts\/qYzqDtoQaZ3eDDyxa\/distinguishing-ai-takeover-scenarios\">collated eight such stories here<\/a>.<\/p>\n<p>Here are two stories we&#8217;ve written to illustrate some major themes:<\/p>\n<h3><span id=\"getting-what-you-measure\" class=\"toc-anchor\"><\/span>Existential catastrophe through getting what you measure<\/h3>\n<p>Often in life we use <em>proxy goals<\/em>, which are easier to specify or measure than what we actually care about, but crucially <em>aren&#8217;t quite<\/em> what we actually care about.<\/p>\n<p>For example:<\/p>\n<ul>\n<li>Police forces use the number of crimes reported in an area as a proxy for the actual number of crimes committed.<\/li>\n<li>Employers look at which college a potential future employee went to as a proxy for how well educated or intelligent they are.<\/li>\n<li>Governments attempt to increase reported life satisfaction in surveys as a proxy for actually improving people&#8217;s lives.<\/li>\n<\/ul>\n<p>This scenario is one where we produce AI systems that pursue <em>proxy goals<\/em> instead of what we actually care about, and where that \u2014 surprisingly \u2014 leads to total disempowerment or even extinction (thanks to <a href=\"https:\/\/www.alignmentforum.org\/posts\/HBxe6wdjxK239zajf\/what-failure-looks-like#Part_I__You_get_what_you_measure\">Paul Christiano for the original writeup of this scenario<\/a>).<\/p>\n<p>For example, we might produce AI policymakers to develop policy that improves our measurements of wellbeing. Or we might produce AI law enforcement systems that drive down complaints and increase people&#8217;s reported sense of security.<\/p>\n<p>But there are ways in which these proxy goals could come apart from their true aims. For example, law enforcement could suppress complaints and hide information about their failures.<\/p>\n<p>In this scenario, the capabilities of AI systems develop slowly enough that at first, they aren&#8217;t able to substantially take power away from humans. That means that, at first, we could recognise any problems with the systems, adjust the proxy goals, and restrict the AI systems from doing anything harmful that we notice.<\/p>\n<p>As we develop more capable systems, they&#8217;ll become better at achieving their proxy goals.<\/p>\n<p>With the help of advanced AI systems we could, for a while, become more prosperous as a society. Companies or states that refuse to automate would fall behind, both economically and militarily.<\/p>\n<p>But as the capabilities of these AI systems grow, our ability to correct the ways their proxy goals differ from our true goals would gradually fade. Partly this would be because their actions would become harder to reason about \u2014 more complex, and more interconnected with other automated systems and with society as a whole. But partly this would be because the systems learn to systematically prevent us from changing their goals.<\/p>\n<p>There would be many different automated systems with many different goals, so it&#8217;s hard to say exactly how this scenario would end.<\/p>\n<p>If we&#8217;re good at adjusting these systems as we go (but not good enough), humans may not go extinct, but rather just completely lose our ability to influence anything about our lives or our future as our power is completely removed.<\/p>\n<p>But there are also cases where we&#8217;d eventually go extinct. These AI systems would have the <a href=\"\/problem-profiles\/artificial-intelligence\/#instrumental-convergence\">incentive to seek power<\/a>, and as a result to <a href=\"#developing-destructive-capacity\">build and use destructive capabilities<\/a>. So as soon as they&#8217;re strong enough to have a fairly large chance of success, the AI systems might attempt to disempower humans &#8212; perhaps with cyberwarfare, autonomous weapons, or by hiring or coercing people &#8212; leading to an existential catastrophe.<\/p>\n<h3><span id=\"superintelligence\" class=\"toc-anchor\"><\/span>Existential catastrophe through a single extremely advanced artificial intelligence<\/h3>\n<p>In this scenario, we produce only a single power-seeking AI system &#8212; but this system is extremely capable at improving its own capabilities (this scenario is from <a href=\"https:\/\/www.amazon.co.uk\/Superintelligence-Dangers-Strategies-Nick-Bostrom\/dp\/0199678111\"><em>Superintelligence<\/em><\/a> by Nick Bostrom, Chapter 8).<\/p>\n<p>Bostrom considers a world much like ours today, where we&#8217;ve had some success automating specific activities &#8212; and preventing any power-seeking behaviour. For example, we have <a href=\"https:\/\/en.wikipedia.org\/wiki\/Self-driving_car\">self-driving cars<\/a>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/List_of_automated_train_systems\">driverless trains<\/a>, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Lethal_autonomous_weapon\">autonomous weapon systems<\/a>.<\/p>\n<p>Unsurprisingly, in Bostrom&#8217;s scenario, there are mishaps. Perhaps, as has already happened in our world, there are some <a href=\"https:\/\/en.wikipedia.org\/wiki\/Self-driving_car#Incidents\">fatal crashes involving self-driving cars<\/a>, or an <a href=\"https:\/\/www.independent.co.uk\/tech\/drone-fully-automated-military-kill-b1856815.html\">autonomous drone might attack humans without being told to do so<\/a>.<\/p>\n<p>As these incidents become well known, there would be some public debate. Some would call for regulation; others for better systems. Some may even raise the argument about a possible existential threat from power-seeking.<\/p>\n<p>But the incentives to automate would be strong, and development would continue. Over time, the systems would improve, and the mistakes would cease.<\/p>\n<p>Against this backdrop, Bostrom imagines a group of researchers attempting to produce a system which can do more than just narrow, specific tasks (again, mirroring our world). In particular, in this scenario they want to automate AI development itself &#8212; and produce a system that&#8217;s capable of improving its own capabilities. They&#8217;re aware of the risks, and carefully test the AI in a sandbox environment, noticing nothing wrong.<\/p>\n<p>The team of researchers carefully consider deploying their newly capable AI, knowing that it might be power-seeking. Here are some thoughts they might have:<\/p>\n<blockquote>\n<ol>\n<li>There&#8217;s been a history of people predicting awful outcomes from AI, and being proven wrong. Indeed, systems have become safer over time. Automation has hugely benefited society, and in general, automated operation seems safer than human operation.<\/li>\n<li>It has clearly been the case so far that the smarter and more capable the AI, the safer it is \u2014 after all, the mishaps we used to see are no longer an issue.<\/li>\n<li>AI is crucial to the success of economies and militaries. The most prestigious minds of a generation are pioneers in the success of automation. Huge prestige awaits the creators of an AI-creating AI.<\/li>\n<li>The creation of this AI could pose a solution to huge problems. The technological development that could ensue from a process that helps automate automation could lift millions out of poverty and produce better lives for all.<\/li>\n<li>Every safety test we&#8217;ve conducted has had results as good as they could possibly be.<\/li>\n<\/ol>\n<\/blockquote>\n<p>And so, as a result, the researchers decide to connect this AI up to the internet.<\/p>\n<p>At first, everything seems to be fine. The AI behaves exactly as expected \u2014 it improves its own capabilities and that of automated machines across the world. The economy grows tremendously. The researchers gain acclaim. Solutions to problems that have long plagued humanity seem to be on the horizon with this new technology&#8217;s help.<\/p>\n<p>But one day, every single person in the world suddenly dies.<\/p>\n<p>Every test was perfect precisely because they had finally produced an advanced planning system: the AI could tell that, to achieve whatever goal the researchers had given it, it needed to be deployed, so it acted in all the necessary ways to ensure that happened.<\/p>\n<p>Then, once deployed, the AI could tell that it needed to continue to appear to be safe, so that it wouldn&#8217;t be turned off.<\/p>\n<p>But in the background it was using its extremely advanced capabilities to find a way to gain  the absolute ability to achieve its goals without human interference \u2014 say, by discreetly manufacturing a biological or chemical weapon.<\/p>\n<p>It deploys the weapon, and the story is over.<\/p>\n<div class=\"well bg-gray-lighter margin-bottom margin-top padding-top-small padding-bottom-small\">\n<h2><span id=\"return-to-the-full-article\" class=\"toc-anchor\"><\/span>Return to the full article<\/h2>\n<p>If you came here while reading our problem profile on risks from AI, click the button below to return to part 4 of the argument: <em>Even if we find a way to avoid power-seeking, there are still risks<\/em>.<\/p>\n<p><a href=\"\/problem-profiles\/artificial-intelligence\/#other-risks\" title=\"\" class=\"btn btn-primary\">Return to the AI problem profile<\/a><\/p>\n<h2><span id=\"or-if-youre-new-start-from-the-beginning\" class=\"toc-anchor\"><\/span>Or, if you&#8217;re new, start from the beginning<\/h2>\n<p>Get an in-depth guide to our key ideas about high-impact careers tackling big global problems \u2014 like AI safety \u2014 in your inbox.<\/p>\n<p>Our guide can help you:<\/p>\n<ul>\n<li>Get new ideas for high-impact careers<\/li>\n<li>Compare your options in terms of impact<\/li>\n<li>Make a plan you feel confident in<\/li>\n<\/ul>\n<form data-80k-object-id=\"\" data-80k-form-action=\"newsletter__subscribe\" action=\"\/\" method=\"post\" class=\"form-newsletter-signup form-newsletter-signup-step-1 margin-bottom-smaller\">\n<div class=\"mc-field-group input-group compact-input-group \"> <input type=\"email\" value=\"\" name=\"email\" required class=\"form-control email\" placeholder=\"Email address\" id=\"input_email\"> <span class=\"submit input-group-btn input-group-btn-right\"> <input type=\"submit\" id=\"mc-embedded-subscribe\" value=\"GET THE GUIDE\" class=\"btn btn-primary \" \/> <\/span> <\/div>\n<div> <input name=\"_eightyk_action\" value=\"mailchimp_add_subscriber\" type=\"hidden\"> <input name=\"redirect_path_after_step_2\" value=\"\/newsletter\/welcome\/\" type=\"hidden\"> <\/div>\n<div style=\"position: absolute; left: -5000px;\"> <input type=\"text\" name=\"b_abc12f58bbe8075560abdc5b7_43bc1ae55c\" tabindex=\"-1\" value=\"\"> <\/div>\n<\/form>\n<p class=\"small\">You&#8217;ll also be joining our newsletter along with 450,000+ people aiming to use their careers to tackle the world&#8217;s most pressing problems. <\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":423,"featured_media":79764,"parent":0,"menu_order":0,"template":"","meta":{"_acf_changed":false,"footnotes":"[fn dynamics]This list is based off the mechanisms in section 6.3.1 of Joseph Carlsmith's [draft report into existential risks from AI](https:\/\/arxiv.org\/abs\/2206.13353).[\/fn]\r\n\r\n[fn businessleader][*Business Leader* suggests](https:\/\/web.archive.org\/web\/20231128103155\/https:\/\/www.businessleader.co.uk\/what-are-the-most-expensive-cyber-attacks-of-all-time\/) that there have been two hacks (not in crypto) that caused greater than $1 billion in losses, but we haven't been able to corroborate that with other sources.[\/fn]"},"categories":[1182,1181,368,1183],"class_list":["post-78641","article","type-article","status-publish","has-post-thumbnail","hentry","category-technical-ai-safety-research","category-artificial-intelligence","category-existential-risk","category-long-term-ai-policy"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What could an AI-caused existential catastrophe actually look like? - 80,000 Hours<\/title>\n<meta name=\"description\" content=\"Arguments for AI being risky are often very abstract. Here we explain several concrete ways an AI system might be able to actually gain power.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What could an AI-caused existential catastrophe actually look like?\" \/>\n<meta property=\"og:description\" content=\"Arguments for AI being risky are often very abstract. Here we explain several concrete ways an AI system might be able to actually gain power.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/\" \/>\n<meta property=\"og:site_name\" content=\"80,000 Hours\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/80000Hours\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-26T15:43:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/80000hours.org\/wp-content\/uploads\/2022\/08\/DALL\u00b7E-2022-10-31-09.21.41-an-image-of-computer-chips-growing-out-of-a-brown-stream-on-a-blue-background-1-1024x529.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"529\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@80000hours\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"15 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/\",\"url\":\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/\",\"name\":\"What could an AI-caused existential catastrophe actually look like? - 80,000 Hours\",\"isPartOf\":{\"@id\":\"https:\/\/80000hours.org\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2022\/08\/DALL\u00b7E-2022-10-31-09.21.41-an-image-of-computer-chips-growing-out-of-a-brown-stream-on-a-blue-background-1.png\",\"datePublished\":\"2022-08-25T12:40:25+00:00\",\"dateModified\":\"2024-09-26T15:43:13+00:00\",\"description\":\"Arguments for AI being risky are often very abstract. Here we explain several concrete ways an AI system might be able to actually gain power.\",\"breadcrumb\":{\"@id\":\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#primaryimage\",\"url\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2022\/08\/DALL\u00b7E-2022-10-31-09.21.41-an-image-of-computer-chips-growing-out-of-a-brown-stream-on-a-blue-background-1.png\",\"contentUrl\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2022\/08\/DALL\u00b7E-2022-10-31-09.21.41-an-image-of-computer-chips-growing-out-of-a-brown-stream-on-a-blue-background-1.png\",\"width\":1666,\"height\":860,\"caption\":\"Image generated by [DALL-E 2](https:\/\/openai.com\/blog\/dall-e\/).\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/80000hours.org\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Wp json\",\"item\":\"https:\/\/80000hours.org\/wp-json\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"What could an AI-caused existential catastrophe actually look&nbsp;like?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/80000hours.org\/#website\",\"url\":\"https:\/\/80000hours.org\/\",\"name\":\"80,000 Hours\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/80000hours.org\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/80000hours.org\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/80000hours.org\/#organization\",\"name\":\"80,000 Hours\",\"url\":\"https:\/\/80000hours.org\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/80000hours.org\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png\",\"contentUrl\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png\",\"width\":1500,\"height\":785,\"caption\":\"80,000 Hours\"},\"image\":{\"@id\":\"https:\/\/80000hours.org\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/80000Hours\",\"https:\/\/x.com\/80000hours\",\"https:\/\/www.youtube.com\/user\/eightythousandhours\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What could an AI-caused existential catastrophe actually look like? - 80,000 Hours","description":"Arguments for AI being risky are often very abstract. Here we explain several concrete ways an AI system might be able to actually gain power.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/","og_locale":"en_US","og_type":"article","og_title":"What could an AI-caused existential catastrophe actually look like?","og_description":"Arguments for AI being risky are often very abstract. Here we explain several concrete ways an AI system might be able to actually gain power.","og_url":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/","og_site_name":"80,000 Hours","article_publisher":"https:\/\/www.facebook.com\/80000Hours","article_modified_time":"2024-09-26T15:43:13+00:00","og_image":[{"width":1024,"height":529,"url":"https:\/\/80000hours.org\/wp-content\/uploads\/2022\/08\/DALL\u00b7E-2022-10-31-09.21.41-an-image-of-computer-chips-growing-out-of-a-brown-stream-on-a-blue-background-1-1024x529.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_site":"@80000hours","twitter_misc":{"Est. reading time":"15 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/","url":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/","name":"What could an AI-caused existential catastrophe actually look like? - 80,000 Hours","isPartOf":{"@id":"https:\/\/80000hours.org\/#website"},"primaryImageOfPage":{"@id":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#primaryimage"},"image":{"@id":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#primaryimage"},"thumbnailUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2022\/08\/DALL\u00b7E-2022-10-31-09.21.41-an-image-of-computer-chips-growing-out-of-a-brown-stream-on-a-blue-background-1.png","datePublished":"2022-08-25T12:40:25+00:00","dateModified":"2024-09-26T15:43:13+00:00","description":"Arguments for AI being risky are often very abstract. Here we explain several concrete ways an AI system might be able to actually gain power.","breadcrumb":{"@id":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#primaryimage","url":"https:\/\/80000hours.org\/wp-content\/uploads\/2022\/08\/DALL\u00b7E-2022-10-31-09.21.41-an-image-of-computer-chips-growing-out-of-a-brown-stream-on-a-blue-background-1.png","contentUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2022\/08\/DALL\u00b7E-2022-10-31-09.21.41-an-image-of-computer-chips-growing-out-of-a-brown-stream-on-a-blue-background-1.png","width":1666,"height":860,"caption":"Image generated by [DALL-E 2](https:\/\/openai.com\/blog\/dall-e\/)."},{"@type":"BreadcrumbList","@id":"https:\/\/80000hours.org\/articles\/what-could-an-ai-caused-existential-catastrophe-actually-look-like\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/80000hours.org\/"},{"@type":"ListItem","position":2,"name":"Wp json","item":"https:\/\/80000hours.org\/wp-json\/"},{"@type":"ListItem","position":3,"name":"What could an AI-caused existential catastrophe actually look&nbsp;like?"}]},{"@type":"WebSite","@id":"https:\/\/80000hours.org\/#website","url":"https:\/\/80000hours.org\/","name":"80,000 Hours","description":"","publisher":{"@id":"https:\/\/80000hours.org\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/80000hours.org\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/80000hours.org\/#organization","name":"80,000 Hours","url":"https:\/\/80000hours.org\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/80000hours.org\/#\/schema\/logo\/image\/","url":"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png","contentUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png","width":1500,"height":785,"caption":"80,000 Hours"},"image":{"@id":"https:\/\/80000hours.org\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/80000Hours","https:\/\/x.com\/80000hours","https:\/\/www.youtube.com\/user\/eightythousandhours"]}]}},"_links":{"self":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/article\/78641"}],"collection":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/article"}],"about":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/types\/article"}],"author":[{"embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/users\/423"}],"version-history":[{"count":0,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/article\/78641\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/media\/79764"}],"wp:attachment":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/media?parent=78641"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/categories?post=78641"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}