{"id":74400,"date":"2023-06-19T10:28:33","date_gmt":"2023-06-19T10:28:33","guid":{"rendered":"https:\/\/80000hours.org\/?post_type=career_profile&#038;p=74400"},"modified":"2024-11-25T18:32:18","modified_gmt":"2024-11-25T18:32:18","slug":"ai-safety-researcher","status":"publish","type":"career_profile","link":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/","title":{"rendered":"AI safety technical research"},"content":{"rendered":"<p>Progress in AI \u2014 while it could be hugely beneficial \u2014 comes with significant risks. Risks that we&#8217;ve argued <a href=\"\/problem-profiles\/artificial-intelligence\/\">could be existential<\/a>.<\/p>\n<p>But these risks <em>can<\/em> be tackled.<\/p>\n<p>With further progress in AI safety, we have an opportunity to develop AI for good: systems that are safe, ethical, and beneficial for everyone.<\/p>\n<p>This article explains how you can help.<br \/>\n<!--more--><\/p>\n<div id=\"toc_container\" class=\"toc_white no_bullets\"><p class=\"toc_title\">Table of Contents<\/p><ul class=\"toc_list\"><li><a href=\"#why-ai-safety-technical-research-is-high-impact\"><span class=\"toc_number toc_depth_1\">1<\/span> Why AI safety technical research is high impact<\/a><ul><li><a href=\"#want-to-learn-more-about-risks-from-ai-read-the-problem-profile\"><span class=\"toc_number toc_depth_2\">1.1<\/span> Want to learn more about risks from AI? Read the problem profile.<\/a><\/li><\/ul><\/li><li><a href=\"#what-does-this-path-involve\"><span class=\"toc_number toc_depth_1\">2<\/span> What does this path involve?<\/a><ul><li><a href=\"#what-does-work-in-the-empirical-ai-safety-path-involve\"><span class=\"toc_number toc_depth_2\">2.1<\/span> What does work in the empirical AI safety path involve?<\/a><\/li><li><a href=\"#what-does-work-in-the-theoretical-ai-safety-path-involve\"><span class=\"toc_number toc_depth_2\">2.2<\/span> What does work in the theoretical AI safety path involve?<\/a><\/li><li><a href=\"#exciting-approaches\"><span class=\"toc_number toc_depth_2\">2.3<\/span> Some exciting approaches to AI safety<\/a><\/li><\/ul><\/li><li><a href=\"#what-are-the-downsides-of-this-career-path\"><span class=\"toc_number toc_depth_1\">3<\/span> What are the downsides of this career path?<\/a><\/li><li><a href=\"#how-much-do-ai-safety-technical-researchers-earn\"><span class=\"toc_number toc_depth_1\">4<\/span> How much do AI safety technical researchers earn?<\/a><\/li><li><a href=\"#examples-of-people-pursuing-this-path\"><span class=\"toc_number toc_depth_1\">5<\/span> Examples of people pursuing this path<\/a><\/li><li><a href=\"#how-to-predict-your-fit\"><span class=\"toc_number toc_depth_1\">6<\/span> How to predict your fit in advance<\/a><\/li><li><a href=\"#how-to-enter\"><span class=\"toc_number toc_depth_1\">7<\/span> How to enter<\/a><ul><li><a href=\"#learn-the-basics\"><span class=\"toc_number toc_depth_2\">7.1<\/span> Learning the basics<\/a><\/li><li><a href=\"#should-you-do-a-phd\"><span class=\"toc_number toc_depth_2\">7.2<\/span> Should you do a PhD?<\/a><\/li><li><a href=\"#getting-a-job-empirical-research\"><span class=\"toc_number toc_depth_2\">7.3<\/span> Getting a job in empirical AI safety research<\/a><\/li><li><a href=\"#getting-a-job-theoretical-research\"><span class=\"toc_number toc_depth_2\">7.4<\/span> Getting a job in theoretical AI safety research<\/a><\/li><li><a href=\"#key-organisations\"><span class=\"toc_number toc_depth_2\">7.5<\/span> Key organisations<\/a><\/li><\/ul><\/li><li><a href=\"#want-one-on-one-advice-on-pursuing-this-path\"><span class=\"toc_number toc_depth_1\">8<\/span> Want one-on-one advice on pursuing this path?<\/a><\/li><li><a href=\"#find-a-job-in-this-path\"><span class=\"toc_number toc_depth_1\">9<\/span> Find a job in this path<\/a><\/li><li><a href=\"#learn-more-about-ai-safety-technical-research\"><span class=\"toc_number toc_depth_1\">10<\/span> Learn more about AI safety technical research<\/a><ul><li><a href=\"#top-recommendations\"><span class=\"toc_number toc_depth_2\">10.1<\/span> Top recommendations<\/a><\/li><li><a href=\"#further-recommendations\"><span class=\"toc_number toc_depth_2\">10.2<\/span> Further recommendations<\/a><\/li><\/ul><\/li><\/ul><\/div>\n<div class=\"panel clearfix no-padding-bottom\">\n<p><strong>In a nutshell:<\/strong> Artificial intelligence will have transformative effects on society over the coming decades, and could bring huge benefits \u2014 but we also think there&#8217;s a substantial risk. One promising way to <a href=\"https:\/\/80000hours.org\/problem-profiles\/artificial-intelligence\/\">reduce the chances of an AI-related catastrophe<\/a> is to find technical solutions that could allow us to prevent AI systems from carrying out dangerous behaviour.<\/p>\n<div class=\"row padding-top bg-off-white\">\n<div class=\"col-sm-6\">\n<h4>Pros<\/h4>\n<ul>\n<li>Opportunity to make a significant contribution to a hugely important area of research<\/li>\n<li>Intellectually challenging and interesting work<\/li>\n<li>The area has a strong need for skilled researchers and engineers, and is highly neglected overall<\/li>\n<\/ul>\n<\/div>\n<div class=\"col-sm-6\">\n<h4>Cons<\/h4>\n<ul>\n<li>Due to a shortage of managers, it&#8217;s difficult to get jobs and might take you some time to build the required career capital and expertise<\/li>\n<li>You need a strong quantitative background<\/li>\n<li>It might be very difficult to find solutions<\/li>\n<li>There&#8217;s a real risk of doing harm<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<div class=\"row padding-top padding-bottom bg-off-white\">\n<div class=\"col-sm-12\">\n<h4>Key facts on fit<\/h4>\n<p>You&#8217;ll need a quantitative background and should probably enjoy programming. If you&#8217;ve never tried programming, you may be a good fit if you can break problems down into logical parts, generate and test hypotheses, possess a willingness to try out many different solutions, and have high attention to detail.<\/p>\n<p>If you already:<\/p>\n<ul>\n<li>Are a strong software engineer, you could apply for empirical research contributor roles right now (even if you don&#8217;t have a machine learning background, although that helps)<\/li>\n<li>Could get into a top 10 machine learning PhD, that would put you on track to become a research lead<\/li>\n<li>Have a very strong maths or theoretical computer science background, you&#8217;ll probably be a good fit for theoretical alignment research<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"border tw--rounded-md tw--mb-8\" >\n<div class=\"tw--bg-off-white tw--px-3.5 tw--py-5\">\n<h3 class=\"no-toc\"> Recommended<\/h3>\n<p>If you are well suited to this career, it may be the best way for you to have a social impact.\n<\/p><\/div>\n<div class=\"tw--px-3.5 tw--py-5\">\n<h4 class=\"tw--text-base\">Review status<\/h4>\n<p>Based on a medium-depth investigation&nbsp;<i class=\n                  \"fas fa-question-circle text-primary icon-tooltip career-tooltip\" data-placement=\"right\"\n                  data-toggle=\"tooltip\" title=\n                  \"This review is informed by people with expertise about this path, an understanding of the best existing advice, and an in-depth investigation into at least one of our key uncertainties concerning this path. Some of our views will be thoroughly researched, though it&#039;s likely there remains some gaps in our understanding.\"><br \/>\n                  <\/i><\/p><\/div><\/div>\n<p><em>Thanks to Adam Gleave, Jacob Hilton and Rohin Shah for reviewing this article. And thanks to Charlie Rogers-Smith for his help, and his article on the topic \u2014 <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/7WXPkpqKGKewAymJf\/how-to-pursue-a-career-in-technical-ai-alignment\"><em>How to pursue a career in technical AI alignment<\/em><\/a>.<\/em><\/p>\n<h2><span id=\"why-ai-safety-technical-research-is-high-impact\" class=\"toc-anchor\"><\/span>Why AI safety technical research is high impact<\/h2>\n<p><a href=\"https:\/\/80000hours.org\/problem-profiles\/positively-shaping-artificial-intelligence\/\">As we&#8217;ve argued<\/a>, in the next few decades, we might see the development of hugely powerful machine learning systems with the potential to transform society. This transformation could bring huge benefits \u2014 but only if we avoid the risks.<\/p>\n<p>We think that the worst-case risks from AI systems arise in large part because AI systems could be <em>misaligned<\/em> \u2014 that is, they will aim to do things that we don&#8217;t want them to do. In particular, we think they could be misaligned in such a way that they develop (and execute) plans that pose risks to humanity&#8217;s ability to influence the world, even when we don&#8217;t want that influence to be lost.<\/p>\n<p>We think this means that these future systems pose an <a href=\"https:\/\/80000hours.org\/articles\/existential-risks\/\">existential threat<\/a> to civilisation.<\/p>\n<p>Even if we find a way to avoid this power-seeking behaviour, there are still <a href=\"https:\/\/80000hours.org\/problem-profiles\/artificial-intelligence\/#other-risks\">substantial risks<\/a> \u2014 such as misuse by governments or other actors \u2014 which could be existential threats in themselves.<\/p>\n<div class=\"well bg-gray-lighter margin-bottom margin-top padding-top-small padding-bottom-small\">\n<h3><span id=\"want-to-learn-more-about-risks-from-ai-read-the-problem-profile\" class=\"toc-anchor\"><\/span>Want to learn more about risks from AI? Read the problem profile.<\/h3>\n<p>We think that technical AI safety could be among the highest-impact career paths we&#8217;ve identified to date. That&#8217;s because it seems like one of the most promising ways of reducing risks from AI. We&#8217;ve written an entire article about what those risks are and why they&#8217;re so important.<\/p>\n<p><a href=\"\/problem-profiles\/artificial-intelligence\/\" title=\"\" class=\"btn btn-primary\">Read more about preventing an AI-related catastrophe<\/a><\/p>\n<\/div>\n<p>There are many ways in which we could go about reducing the risks that these systems might pose. But one of the most promising may be researching technical solutions that prevent unwanted behaviour \u2014 including misaligned behaviour \u2014 from AI systems. (Finding a technical way to prevent misalignment in particular is known as the <em>alignment problem<\/em>.)<\/p>\n<p>In the past few years, we&#8217;ve seen more organisations start to take these risks more seriously. Many of the leading companies developing AI \u2014 including <a href=\"https:\/\/deepmindsafetyresearch.medium.com\/building-safe-artificial-intelligence-52f5f75058f1\">Google DeepMind<\/a> and <a href=\"https:\/\/openai.com\/alignment\/\">OpenAI<\/a> \u2014 have teams dedicated to finding these solutions, alongside academic research groups including at <a href=\"https:\/\/people.csail.mit.edu\/dhm\/\">MIT<\/a>, <a href=\"https:\/\/www.davidscottkrueger.com\/\">Cambridge<\/a>, <a href=\"https:\/\/www.cs.cmu.edu\/~focal\/\">Carnegie Mellon University<\/a>, and <a href=\"https:\/\/humancompatible.ai\/\">UC Berkeley<\/a>.<\/p>\n<p>That said, the field is still very new. We estimated in 2022 that there were only around 300 people working on technical approaches to reducing existential risks from AI systems, making this a highly neglected field.<\/p>\n<p>Finding technical ways to reduce this risk could be quite challenging. Any practically helpful solution must retain the usefulness of the systems (remaining economically competitive with less safe systems), and continue to work as systems improve over time (that is, it needs to be &#8216;scalable&#8217;). As we argued in our <a href=\"https:\/\/80000hours.org\/problem-profiles\/artificial-intelligence\/#instrumental-convergence\">problem profile<\/a>, it seems like it might be difficult to find viable solutions, particularly for modern ML (machine learning) systems.<\/p>\n<p><em>(If you don&#8217;t know anything about ML, we&#8217;ve written a <a href=\"\/problem-profiles\/artificial-intelligence\/#what-is-deep-learning\">very very short introduction to ML<\/a>, and we&#8217;ll go into more detail on how to learn about ML <a href=\"#learn-the-basics\">later in this article<\/a>. Alternatively, if you do have ML experience, <a href=\"\/speak-with-us\/\">talk to our team<\/a> \u2014 they can give you personalised career advice, make introductions to others working on these issues, and possibly even help you find jobs or funding opportunities.)<\/em><\/p>\n<p>Although it seems hard, there are lots of avenues for more research \u2014 and the field really is very young, so there are new promising research directions cropping up all the time. So we think it&#8217;s moderately tractable, though we&#8217;re highly uncertain.<\/p>\n<p>In fact, we&#8217;re uncertain about <em>all<\/em> of this and have written extensively about <a href=\"\/problem-profiles\/artificial-intelligence\/#best-arguments-against-this-problem-being-pressing\">reasons we might be wrong about AI risk<\/a>.<\/p>\n<p>But, overall, we think that \u2014 if it&#8217;s a <a href=\"\/career-guide\/personal-fit\/\">good fit for you<\/a> \u2014 going into AI safety technical research may just be the highest-impact thing you can do with your career.<\/p>\n<h2><span id=\"what-does-this-path-involve\" class=\"toc-anchor\"><\/span>What does this path involve?<\/h2>\n<p>AI safety technical research generally involves working as a scientist or engineer at major AI companies, in academia, or in independent nonprofits.<\/p>\n<p>These roles can be very hard to get. You&#8217;ll likely need to build up <a href=\"\/career-guide\/career-capital\">career capital<\/a> before you end up in a high-impact role (more on this later, in the section on <a href=\"#how-to-enter\">how to enter<\/a>). That said, you may not need to spend a long <em>time<\/em> building this career capital \u2014 we&#8217;ve seen exceptionally talented people move into AI safety from other quantitative fields, sometimes in less than a year.<\/p>\n<p>Most AI safety technical research falls on a spectrum between empirical research (experimenting with current systems as a way of learning more about what will work), and theoretical research (conceptual and mathematical research looking at ways of ensuring that future AI systems are safe).<\/p>\n<p>No matter where on this spectrum you end up working, your career path might look a bit different depending on whether you want to aim at becoming a research lead \u2014 proposing projects, managing a team and setting direction \u2014 or a contributor \u2014 focusing on carrying out the research.<\/p>\n<p>Finally, there are two slightly different roles you might aim for:<\/p>\n<ul>\n<li>In academia, research is often led by professors \u2014 the key distinguishing feature of being a professor is that you&#8217;ll also teach classes and mentor grad students (and you&#8217;ll definitely need a PhD).<\/li>\n<li>Many (but not all) contributor roles in empirical research are also engineers, often software engineers. Here, we&#8217;re focusing on software roles that directly contribute to AI safety research (and which often require some ML background) \u2014 we&#8217;ve written about software engineering more generally in a <a href=\"\/career-reviews\/software-engineering\">separate career review<\/a>.<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/Type-of-research.png\" alt=\"4 kinds of AI safety role: empirical lead, empirical contributor, theoretical lead and theoretical contributor\" \/><\/p>\n<p>We think that research lead roles are probably higher-impact in general. But overall, the impact you could have in any of these roles is likely primarily determined by your <a href=\"\/career-guide\/personal-fit\">personal fit<\/a> for the role \u2014 see the section on <a href=\"#how-to-predict-your-fit\">how to predict your fit in advance<\/a>.<\/p>\n<p>Next, we&#8217;ll take a look at what working in each path might involve. <a href=\"#how-to-enter\">Later, we&#8217;ll go into how you might enter each path.<\/a><\/p>\n<h3><span id=\"what-does-work-in-the-empirical-ai-safety-path-involve\" class=\"toc-anchor\"><\/span>What does work in the empirical AI safety path involve?<\/h3>\n<p>Empirical AI safety tends to involve teams working directly with ML models to identify any risks and develop ways in which they might be mitigated.<\/p>\n<p>That means the work is focused on current ML techniques and techniques that might be applied in the very near future.<\/p>\n<p>Practically, working on empirical AI safety involves lots of programming and ML engineering. You might, for example, come up with ways you could test the safety of existing systems, and then carry out these empirical tests.<\/p>\n<p>You can find roles in empirical AI safety in industry and academia, as well as some in AI safety-focused nonprofits.<\/p>\n<p>Particularly in academia, lots of relevant work isn&#8217;t explicitly labelled as being focused on existential risk \u2014 but it can still be highly valuable. For example, work in <a href=\"https:\/\/arxiv.org\/pdf\/2207.13243.pdf\">interpretability<\/a>, <a href=\"https:\/\/ieeexplore.ieee.org\/abstract\/document\/8842604\">adversarial examples<\/a>, <a href=\"https:\/\/arxiv.org\/pdf\/2302.10894.pdf\">diagnostics<\/a> and <a href=\"https:\/\/arxiv.org\/pdf\/2206.12654.pdf\">backdoor learning<\/a>, among other areas, could be highly relevant to reducing the chance of an AI-related catastrophe.<\/p>\n<p>We&#8217;re also excited by experimental work to develop safety standards that AI companies might adhere to in the future \u2014 for example, the work being carried out by <a href=\"https:\/\/metr.org\/\">METR<\/a>.<\/p>\n<p>To learn more about the sorts of research taking place at companies and labs focused on empirical AI safety, take a look at:<\/p>\n<ul>\n<li><a href=\"https:\/\/openai.com\/blog\/our-approach-to-alignment-research\">OpenAI&#8217;s approach to alignment research<\/a> <\/li>\n<li><a href=\"https:\/\/www.anthropic.com\/index\/core-views-on-ai-safety\">Anthropic&#8217;s views on AI safety<\/a><\/li>\n<li><a href=\"https:\/\/www.redwoodresearch.org\/research\">Redwood Research&#8217;s recent research highlights<\/a> <\/li>\n<li><a href=\"https:\/\/www.deepmind.com\/tags\/safety\">Publications from Google DeepMind&#8217;s safety team<\/a> <\/li>\n<\/ul>\n<p>While programming is central to all empirical work, generally, research lead roles will be <em>less<\/em> focused on programming; instead, they need stronger research taste and theoretical understanding. In comparison, research contributors need to be very good at programming and software engineering.<\/p>\n<h3><span id=\"what-does-work-in-the-theoretical-ai-safety-path-involve\" class=\"toc-anchor\"><\/span>What does work in the theoretical AI safety path involve?<\/h3>\n<p>Theoretical AI safety is much more heavily conceptual and mathematical. Often it involves careful reasoning about the hypothetical behaviour of future systems.<\/p>\n<p>Generally, the aim is to come up with properties that it would be useful for safe ML algorithms to have. Once you have some useful properties, you can try to develop algorithms with these properties (bearing in mind that to be practically useful these algorithms will have to end up being adopted by industry). Alternatively, you could develop ways of <em>checking<\/em> whether systems have these properties. These checks could, for example, help hold future AI products to high safety standards.<\/p>\n<p>Many people working in theoretical AI safety will spend much of their time proving theorems or developing new mathematical frameworks. More conceptual approaches also exist, although they still tend to make heavy use of formal frameworks.<\/p>\n<p>Some examples of research in theoretical AI safety include:<\/p>\n<ul>\n<li><a href=\"https:\/\/arxiv.org\/abs\/1906.01820\"><em>Risks from learned optimisation in advanced machine learning systems<\/em><\/a> by Hubinger et al.<\/li>\n<li><a href=\"https:\/\/www.alignment.org\/blog\/arcs-first-technical-report-eliciting-latent-knowledge\/\"><em>Eliciting latent knowledge<\/em><\/a> by Christiano, Cotra and Xu.<\/li>\n<li><a href=\"https:\/\/arxiv.org\/abs\/2211.06738\"><em>Formalizing the presumption of independence<\/em><\/a> by Christiano, Neyman, and Xu<\/li>\n<li><a href=\"https:\/\/arxiv.org\/pdf\/2208.08345.pdf\"><em>Discovering agents<\/em><\/a> by Kenton et al.<\/li>\n<li><a href=\"https:\/\/arxiv.org\/pdf\/2303.00894.pdf\"><em>Active reward learning from multiple teachers<\/em><\/a> by Barnett et al.<\/li>\n<\/ul>\n<p>There are generally fewer roles available in theoretical AI safety work, especially as research contributors. Theoretical research contributor roles exist at nonprofits (primarily the <a href=\"https:\/\/www.alignment.org\/\">Alignment Research Center<\/a>), as well as at some labs (for example, Anthropic&#8217;s work on <a href=\"https:\/\/arxiv.org\/abs\/2302.00805\">conditioning predictive models<\/a> and the <a href=\"https:\/\/causalincentives.com\/\">Causal Incentives Working Group<\/a> at Google DeepMind). Most contributor roles in theoretical AI safety probably exist in academia (for example, PhD students in teams working on projects relevant to theoretical AI safety).<\/p>\n<h3><span id=\"exciting-approaches\" class=\"toc-anchor\"><\/span>Some exciting approaches to AI safety<\/h3>\n<p>There are lots of technical approaches to AI safety currently being pursued. Here are just a few of them:<\/p>\n<ul>\n<li><strong>Scalably learning from human feedback.<\/strong> Examples include <a href=\"https:\/\/www.youtube.com\/watch?v=v9M2Ho9I9Qo\">iterated amplification<\/a>, <a href=\"https:\/\/openai.com\/research\/debate\">AI safety via debate<\/a>, <a href=\"https:\/\/www.alignmentforum.org\/posts\/nd692YfFGfZDh9Mwz\/an-69-stuart-russell-s-new-book-on-why-we-need-to-replace\">building AI assistants that are uncertain about our goals and learn them by interacting with us<\/a>, and <a href=\"https:\/\/www.alignment.org\/blog\/arcs-first-technical-report-eliciting-latent-knowledge\/\">other ways to get AI systems to report truthfully what they know<\/a>.<\/li>\n<li><strong>Threat modelling.<\/strong> An example of this work would be demonstrating the possibility of (allowing us to study) dangerous capabilities, like deceptive or manipulative AI systems. This approach splits into work that evaluates whether a model has dangerous capabilities (like the work of <a href=\"https:\/\/metr.org\/\">METR<\/a> in <a href=\"https:\/\/arxiv.org\/pdf\/2303.08774.pdf\">evaluating GPT-4<\/a>), and work that evaluates whether a model would cause harm in practice (like <a href=\"https:\/\/twitter.com\/AnthropicAI\/status\/1604883576218341376\">Anthropic&#8217;s research into the behaviour of large language models<\/a> and <a href=\"https:\/\/arxiv.org\/abs\/2210.01790\">this paper on goal misgeneralisation<\/a>). It can also include work to <a href=\"https:\/\/www.alignmentforum.org\/posts\/ChDH335ckdvpxXaXX\/model-organisms-of-misalignment-the-case-for-a-new-pillar-of-1\">find &#8216;model organisms of misalignment&#8217;<\/a>, in the hope of better understanding their dangers.<\/li>\n<li>Work to figure out how to <strong>control<\/strong> powerful AI systems \u2013 preventing them from causing harm even if they are unsafe. Read more in <a href=\"https:\/\/www.alignmentforum.org\/posts\/kcKrE9mzEHrdqtDpE\/the-case-for-ensuring-that-powerful-ais-are-controlled\">this blogpost from the team at Redwood Research working on control<\/a>.<\/li>\n<li><strong>Interpretability research.<\/strong> This work involves studying <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/chris-olah-interpretability-research\/\">why AI systems do what they do<\/a> and trying to put it into human-understandable terms. For example, <a href=\"https:\/\/arxiv.org\/abs\/2111.09259\">this paper examined how AlphaZero learns chess<\/a>, and <a href=\"https:\/\/arxiv.org\/abs\/2212.03827\">this paper looked into finding latent knowledge in language models without supervision<\/a>. This category also includes  <a href=\"https:\/\/www.neelnanda.io\/mechanistic-interpretability\/\"><em>mechanistic interpretability<\/em><\/a> \u2014 for example, <a href=\"https:\/\/distill.pub\/2020\/circuits\/zoom-in\/\"><em>Zoom In: An Introduction to Circuits<\/em> by Olah et al.<\/a>. For more, see <a href=\"https:\/\/arxiv.org\/abs\/2207.13243\">this survey paper<\/a>, as well as Hubinger&#8217;s <a href=\"https:\/\/www.alignmentforum.org\/posts\/nbq2bWLcYmSGup9aF\/a-transparency-and-interpretability-tech-tree\">a transparency and interpretability tech tree<\/a>, and Nanda&#8217;s <a href=\"https:\/\/www.alignmentforum.org\/posts\/uK6sQCNMw8WKzJeCQ\/a-longlist-of-theories-of-impact-for-interpretability\"><em>A Longlist of Theories of Impact for Interpretability<\/em><\/a> for overviews of of how interpretability research could reduce existential risk from AI.<\/li>\n<li>Other <strong>anti-misuse research<\/strong> to reduce the risks of catastrophe caused by misuse of systems. For example, this work includes  training AIs so they&#8217;re hard to use for dangerous purposes. (Note there&#8217;s lots of overlap with the other work on this list). <\/li>\n<li><strong>Research to increase the robustness of neural networks.<\/strong> This work involves ensuring that the sorts of behaviour neural networks display when exposed to one set of inputs continues when exposed to inputs they haven&#8217;t previously been exposed to, in order to prevent AI systems changing to unsafe behaviour. See section 2 of <em><a href=\"https:\/\/arxiv.org\/pdf\/2109.13916.pdf\">Unsolved Problems in AI safety<\/a><\/em> for more.<\/li>\n<li><strong>Work to build cooperative AI.<\/strong>  Find ways to ensure that even if individual AI systems seem safe, they don&#8217;t produce bad outcomes through interacting with other sociotechnical systems. For more, see <a href=\"https:\/\/arxiv.org\/pdf\/2012.08630.pdf\">Open Problems in Cooperative AI<\/a> by Dafoe et al. or the <a href=\"https:\/\/www.cooperativeai.com\/\">Cooperative AI Foundation<\/a>. This seems particularly relevant for the reduction of &#8216;<a href=\"\/problem-profiles\/artificial-intelligence\">s-risks<\/a>.&#8217; <\/li>\n<li>More generally, there are some <strong>unified safety plans.<\/strong> For more, see Hubinger&#8217;s <a href=\"https:\/\/arxiv.org\/abs\/2012.07532\">11 possible proposals for building safe advanced AI<\/a>, or Karnofsky&#8217;s <a href=\"https:\/\/www.alignmentforum.org\/posts\/rCJQAkPTEypGjSJ8X\/how-might-we-align-transformative-ai-if-it-s-developed-very\">How might we align transformative AI if it&#8217;s developed very soon<\/a>.  <\/li>\n<\/ul>\n<p>It&#8217;s worth noting that there are many approaches to AI safety, and people in the field strongly disagree  on what will or won&#8217;t work.<\/p>\n<p>This means that, once you&#8217;re working in the field, it can be worth being charitable and careful not to assume that others&#8217; work is unhelpful just because it seemed so on a quick skim. You should probably be uncertain about your own research agenda as well.<\/p>\n<p>What&#8217;s more, as we mentioned earlier, lots of relevant work across all these areas isn&#8217;t explicitly labelled &#8216;safety.&#8217;<\/p>\n<p>So it&#8217;s important to think carefully about how or whether any particular research helps reduce the risks that AI systems might pose.<\/p>\n<h2><span id=\"what-are-the-downsides-of-this-career-path\" class=\"toc-anchor\"><\/span>What are the downsides of this career path?<\/h2>\n<p>AI safety technical research is not the only way to make progress on reducing the risks that future AI systems might pose. Also, there are <a href=\"\/problem-profiles\/\">many other pressing problems in the world<\/a> that <em>aren&#8217;t<\/em> the possibility of an AI-related catastrophe, and lots of careers that can help with them. If you&#8217;d be a <a href=\"\/career-guide\/personal-fit\">better fit<\/a> working on <a href=\"\/career-reviews\">something else<\/a>, you should probably do that.<\/p>\n<p>Beyond personal fit, there are a few other downsides to the career path:<\/p>\n<ul>\n<li>It can be very competitive to enter (although once you&#8217;re in, the jobs are well paid, and there are lots of backup options).<\/li>\n<li>You <em>need<\/em> quantitative skills \u2014 and probably programming skills.<\/li>\n<li>The work is geographically concentrated in just a few places (mainly the California Bay Area and London, but there are also opportunities in places with top universities such as Oxford, New York, Pittsburgh, and Boston). That said, remote work is increasingly possible at many research labs.<\/li>\n<li>It might not be very tractable to find good technical ways of reducing the risk. Although assessments of its difficulty vary, and while making progress is almost certainly possible, it may be quite hard to do so. This <a href=\"https:\/\/80000hours.org\/problem-profiles\/artificial-intelligence\/#this-could-be-extremely-difficult-to-solve\">reduces the impact that you could have working in the field<\/a>. That said, if you start out in technical work you might be able to transition to <a href=\"\/career-reviews\/ai-policy-and-strategy\/\">governance work<\/a>, since that often benefits from technical training and experience with the industry, which most people do not have.)<\/li>\n<li>Relatedly, there&#8217;s lots of disagreement in the field about what could work; you&#8217;ll probably be able to find at least some people who think what you&#8217;re working on is useless, whatever you end up doing.<\/li>\n<li>Most importantly, there&#8217;s some risk of <a href=\"\/articles\/accidental-harm\/\">doing harm<\/a>. While gaining <a href=\"\/career-guide\/career-capital\">career capital<\/a>, and while working on the research itself, you&#8217;ll have to make difficult decisions and judgement calls about whether you&#8217;re working on something beneficial (see our anonymous advice about <a href=\"\/articles\/should-you-work-on-ai-capabilities\/\">working in roles that advance AI capabilities<\/a>). There&#8217;s huge disagreement on which technical approaches to AI safety might work \u2014 and sometimes this disagreement takes the form of thinking that a strategy will actively <em>increase<\/em> existential risks from AI.<\/li>\n<\/ul>\n<p>Finally, we&#8217;ve written more about <a href=\"\/problem-profiles\/artificial-intelligence\/#best-arguments-against-this-problem-being-pressing\">the best arguments against AI being pressing<\/a> in our problem profile on preventing an AI-related catastrophe. If those are right, maybe you could have more impact working on a different issue.<\/p>\n<h2><span id=\"how-much-do-ai-safety-technical-researchers-earn\" class=\"toc-anchor\"><\/span>How much do AI safety technical researchers earn?<\/h2>\n<p>Many technical researchers work at companies or small startups that pay wages competitive with the Bay Area and Silicon Valley tech industry, and even smaller organisations and nonprofits will pay competitive wages to attract top talent. The median compensation for a software engineer in the San Francisco Bay area was $222,000 per year in 2020. (Read more about <a href=\"https:\/\/80000hours.org\/career-reviews\/software-engineering\/#how-much-do-software-engineers-earn\">software engineering salaries<\/a>).<\/p>\n<p>This $222,000 median may be an underestimate, as AI roles, especially in top AI companies that are rapidly scaling up their work in AI, often pay <em>better<\/em> than other tech jobs, and the same applies to safety researchers \u2014 even those in nonprofits.<\/p>\n<p>However, <a href=\"https:\/\/80000hours.org\/career-reviews\/academic-research\/#lower-salaries\">academia has lower salaries than industry in general<\/a>, and we&#8217;d guess that AI safety research roles in academia pay less than commercial labs and nonprofits.<\/p>\n<h2><span id=\"examples-of-people-pursuing-this-path\" class=\"toc-anchor\"><\/span>Examples of people pursuing this path<\/h2>\n<aside class=\"well well-person pull-right clearfix align-center  padding-top-small padding-bottom-small\">\n<p class=\"no-margin-bottom\"><img decoding=\"async\" class=\"img-circle well-person__portrait\" height=300 width=300 src=\"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/02\/Screenshot-2023-02-06-at-4.27.00-PM.png\" alt=\"Ethan Perez portrait\"><\/p>\n<h4 class=\"no-margin-top\">Ethan Perez<\/h4>\n<p>Ethan Perez was studying computer science when he came across 80,000 Hours, which convinced him that the <a href=\"\/problem-profiles\/artificial-intelligence\/\">risk from advanced artificial intelligence<\/a> was a highly pressing problem. After <a href=\"\/speak-with-us\/\">speaking with an 80,000 Hours advisor<\/a>, he decided to work full time on <a href=\"\/career-reviews\/ai-safety-researcher\/\">AI safety<\/a> instead of pursuing a career in self-driving car technology. He went on to write his PhD thesis on fixing undesirable behaviour in language models and took a role as a research scientist working on AI safety at <a href=\"https:\/\/jobs.80000hours.org\/organisations\/anthropic\">Anthropic<\/a>.<br \/>\n<a href=\"\/stories\/ethan-perez\/\" class=\"btn btn-tertiary\">Learn more<\/a><\/p>\n<\/aside>\n<aside class=\"well well-person pull-right clearfix align-center  padding-top-small padding-bottom-small\">\n<p class=\"no-margin-bottom\"><img decoding=\"async\" class=\"img-circle well-person__portrait\" height=300 width=300 src=\"https:\/\/80000hours.org\/wp-content\/uploads\/2021\/10\/catherine-ollson.jpeg\" alt=\"Catherine Olsson portrait\"><\/p>\n<h4 class=\"no-margin-top\">Catherine Olsson<\/h4>\n<p>Catherine started her PhD at NYU, working on computational models of human vision. Eventually, she decided to work directly on AI safety and got a job at <a href=\"https:\/\/openai.com\/\">OpenAI<\/a>, and then <a href=\"https:\/\/research.google\/teams\/brain\/\">Google Brain<\/a>, before moving to <a href=\"https:\/\/www.anthropic.com\/\">Anthropic<\/a>.<br \/>\n<a href=\"\/podcast\/episodes\/olsson-and-ziegler-ml-engineering-and-safety\/\" class=\"btn btn-tertiary\">Learn more<\/a><\/p>\n<\/aside>\n<aside class=\"well well-person pull-right clearfix align-center  padding-top-small padding-bottom-small\">\n<p class=\"no-margin-bottom\"><img decoding=\"async\" class=\"img-circle well-person__portrait\" height=300 width=300 src=\"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/05\/NeelNanda-e1684842910437-300x300.jpg\" alt=\"Neel Nanda portrait\"><\/p>\n<h4 class=\"no-margin-top\">Neel Nanda<\/h4>\n<p>Neel was doing an undergraduate degree in maths when he decided that he wanted to work in AI safety. Our team was able to introduce Neel to researchers in the field, and helped him secure internships in academic and industry research groups. Neel didn&#8217;t feel like he was a great fit for academia \u2014 he hates writing papers \u2014 so he applied to roles in commercial AI research labs. He&#8217;s now a researcher at <a href=\"https:\/\/www.deepmind.com\/\">DeepMind<\/a>, where he works on <a href=\"https:\/\/www.neelnanda.io\/mechanistic-interpretability\">mechanistic interpretability<\/a>.<br \/>\n<a href=\"https:\/\/80000hours.org\/stories\/neel-nanda\/\" class=\"btn btn-tertiary\">Learn more<\/a><\/p>\n<\/aside>\n<h2><span id=\"how-to-predict-your-fit\" class=\"toc-anchor\"><\/span>How to predict your fit in advance<\/h2>\n<p>You&#8217;ll generally need a quantitative background (although not necessarily a background in computer science or machine learning) to enter this career path.<\/p>\n<p>There are two main approaches you can take to predict your fit, and it&#8217;s helpful to do both:<\/p>\n<ul>\n<li><strong>Try it out<\/strong>: try out the first few steps in the <a href=\"#learn-the-basics\">section below on learning the basics<\/a>. If you haven&#8217;t yet, try learning some python, as well as taking courses in linear algebra, calculus, and probability. And if you&#8217;ve done that, try learning a bit about <a href=\"#basic-machine-learning\">deep learning<\/a> and <a href=\"#learning-about-AI-safety\">AI safety<\/a>. Finally, the best way to try this out for many people would be to actually get a job as a (non-safety) ML engineer (see more in the section on <a href=\"#how-to-enter\">how to enter<\/a>).<\/li>\n<li><strong>Talk to people about whether it would be a good fit for you<\/strong>: If you want to become a technical researcher, <a href=\"\/speak-with-us\/\">our team probably wants to talk to you<\/a>. We can give you 1-1 advice, for free. If you know anyone working in the area (or something similar), discuss this career path with them and ask for their honest opinion. You may be able to meet people through <a href=\"\/community\/\">our community<\/a>. Our <a href=\"\/speak-with-us\/\">advisors<\/a> can also help make connections.<\/li>\n<\/ul>\n<p>It can take some time to build expertise, and enjoyment can follow expertise \u2014 so be prepared to take some time to learn and practice before you decide to switch to something else entirely.<\/p>\n<p>If you&#8217;re not sure <a href=\"\/career-guide\/career-planning\/#have-a-longer-term-vision\">what roles you might aim for longer term<\/a>, here are a few rough ways you could make a guess about what to aim for, and whether you might be a good fit for various roles on this path:<\/p>\n<ul>\n<li><strong>Testing your fit as an empirical research contributor<\/strong>: In a <a href=\"https:\/\/www.alignmentforum.org\/posts\/nzmCvRvPm4xJuqztv\/deepmind-is-hiring-for-the-scalable-alignment-and-alignment\">blog post about hiring for safety researchers<\/a>, the Google DeepMind team said &#8220;as a rough test for the Research Engineer role, if you can reproduce a typical ML paper in a few hundred hours and your interests align with ours, we&#8217;re probably interested in interviewing you.&#8221;\n<ul>\n<li>Looking specifically at <em>software engineering<\/em>, one hiring manager at Anthropic said that if you could, with a few weeks&#8217; work, write a complex new feature or fix a very serious bug in a major ML library, they&#8217;d want to interview you straight away. (<a href=\"https:\/\/www.alignmentforum.org\/posts\/YDF7XhMThhNfHfim9\/ai-safety-needs-great-engineers\">Read more<\/a>.)<\/li>\n<\/ul>\n<\/li>\n<li><strong>Testing your fit for theoretical research<\/strong>: If you could have got into a top 10 maths or theoretical computer science PhD programme if you&#8217;d optimised your undergrad to do so, that&#8217;s a decent indication of your fit (and many researchers in fact have these PhDs). The <a href=\"https:\/\/www.alignment.org\/blog\/early-2022-hiring-round\/\">Alignment Research Center<\/a> (one of the few organisations that hires for theoretical research contributors, as of 2023) said that they were open to hiring people without any research background. They gave four tests of fit: creativity (e.g. you may have ideas for solving open problems in the field, like <a href=\"https:\/\/docs.google.com\/document\/d\/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8\/edit\">Eliciting Latent Knowledge<\/a>); experience designing algorithms, proving theorems, or formalising concepts; broad knowledge of maths and computer science; and having thought a lot about the AI alignment problem in particular.<\/li>\n<li><strong>Testing your fit as a research lead (or for a PhD)<\/strong>: The vast majority of research <em>leads<\/em> have a PhD. Also, many (but definitely not all) AI safety technical research roles will require a PhD \u2014 and if they don&#8217;t, having a PhD (or being the sort of person that could get one) would definitely help show that you&#8217;re a good fit for the work. To get into a top 20 machine learning PhD programme, you&#8217;d probably need to publish something like a first author workshop paper, as well as a third author conference paper at a major ML conference (like <a href=\"https:\/\/nips.cc\/\">NeurIPS<\/a> or <a href=\"https:\/\/icml.cc\/\">ICML<\/a>). (<a href=\"#should-you-do-a-PhD}\">Read more about whether you should do a PhD<\/a>).<\/li>\n<\/ul>\n<p>Read our article on <a href=\"\/career-guide\/personal-fit\/\">personal fit<\/a> to learn more about how to assess your fit for the career paths you want to pursue.<\/p>\n<h2><span id=\"how-to-enter\" class=\"toc-anchor\"><\/span>How to enter<\/h2>\n<p>You might be able to apply for roles right away \u2014 especially if you meet, or are near meeting, the tests we just looked at \u2014 but it also might take you some time, possibly several years, to skill up first.<\/p>\n<p>So, in this section, we&#8217;ll give you a guide to entering technical AI safety research. We&#8217;ll go through four key questions:<\/p>\n<ol>\n<li><a href=\"\/career-reviews\/ai-safety-researcher\/#learn-the-basics\">How to learn the basics<\/a><\/li>\n<li><a href=\"\/career-reviews\/ai-safety-researcher\/#should-you-do-a-phd\">Whether you should do a PhD<\/a><\/li>\n<li><a href=\"\/career-reviews\/ai-safety-researcher\/#getting-a-job-empirical-research\">How to get a job in empirical research<\/a><\/li>\n<li><a href=\"\/career-reviews\/ai-safety-researcher\/#getting-a-job-theoretical-research\">How to get a job in theoretical research<\/a><\/li>\n<\/ol>\n<p>Hopefully, by the end of the section, you&#8217;ll have everything you need to get going.<\/p>\n<h3><span id=\"learn-the-basics\" class=\"toc-anchor\"><\/span>Learning the basics<\/h3>\n<p>To get anywhere in the world of AI safety technical research, you&#8217;ll likely need a background knowledge of <a href=\"#learning-to-program\">coding<\/a>, <a href=\"#learning-the-maths\">maths<\/a>, and <a href=\"#basic-machine-learning\">deep learning<\/a>.<\/p>\n<p>You might also want to practice enough to become a decent ML engineer (although this is generally more useful for empirical research), and <a href=\"#basic-machine-learning\">learn a bit about safety techniques in particular<\/a> (although this is generally more useful for empirical research leads and theoretical researchers).<\/p>\n<p>We&#8217;ll go through each of these in turn.<\/p>\n<h4 id=\"learning-to-program\">Learning to program<\/h4>\n<p>You&#8217;ll probably want to learn to code in python, because it&#8217;s the most widely used language in ML engineering.<\/p>\n<p>The first step is probably just trying it out. As a complete beginner, you can write <a href=\"http:\/\/blog.udacity.com\/2015\/01\/remind-friend-take-break-python-program.html\">a Python program<\/a> in less than 20 minutes that reminds you to take a break every two hours. Don&#8217;t be discouraged if <a href=\"https:\/\/medium.com\/free-code-camp\/things-i-wish-someone-had-told-me-when-i-was-learning-how-to-code-565fc9dcb329\">your code doesn&#8217;t work the first time<\/a> \u2014 that&#8217;s what normally happens when people code!<\/p>\n<p>Once you&#8217;ve done that, you have a few options:<\/p>\n<ul>\n<li><strong>Teach yourself to program.<\/strong> Try working through a free beginner course like <a href=\"https:\/\/automatetheboringstuff.com\/2e\/chapter0\/\"><em>Automate the boring stuff with Python<\/em><\/a> by Al Seigart. There also are many great introductory computer science and programming courses online, including: <a href=\"https:\/\/www.udacity.com\/course\/intro-to-computer-science--cs101\">Udacity&#8217;s Intro to Computer Science<\/a>, <a href=\"http:\/\/ocw.mit.edu\/courses\/electrical-engineering-and-computer-science\/6-00sc-introduction-to-computer-science-and-programming-spring-2011\/\">MIT&#8217;s Introduction to Computer Science and Programming<\/a>, and <a href=\"http:\/\/videolectures.net\/stanfordcs106af07_programming_methodology\/\">Stanford&#8217;s Programming Methodology<\/a>. Then, try finding something you want to build, and building it \u2014 or getting involved in an open-source project. For interview practice, try <a href=\"https:\/\/leetcode.com\/\">leetcode<\/a> or <a href=\"https:\/\/www.topcoder.com\/\">TopCoder<\/a>, or the exercises in <a href=\"https:\/\/www.crackingthecodinginterview.com\/\"><em>Cracking the Coding Interview<\/em><\/a> by Gayle McDowell. <\/li>\n<li><strong>Take a college course.<\/strong> If you&#8217;re in university, this is a great option because it allows you to learn programming while the opportunity cost of your time is lower. You can even consider majoring in computer science (or another subject involving lots of programming). <\/li>\n<li><strong>Learn on the job.<\/strong> If you can find internships, you&#8217;ll gain practical experience and <a href=\"https:\/\/missing.csail.mit.edu\/\">skills you otherwise wouldn&#8217;t pick up<\/a> from academic degrees.<\/li>\n<li><strong>Go to a bootcamp.<\/strong> Coding bootcamps are focused on taking people with little knowledge of programming to as highly paid a job as possible within a couple of months \u2014 though some claim the long-term prospects are not as good because you lack a deep understanding of computer science. <a href=\"https:\/\/www.coursereport.com\/\">Course Report<\/a> is a great guide to choosing a bootcamp. Be careful to avoid <a href=\"http:\/\/web.archive.org\/web\/20170109105128\/https:\/\/www.bloomberg.com\/news\/features\/2016-12-06\/want-a-job-in-silicon-valley-keep-away-from-coding-schools\">low-quality bootcamps<\/a>. You can also find online bootcamps \u2014 for people completely new to programming \u2014 focused on ML, like Udemy&#8217;s <a href=\"https:\/\/www.udemy.com\/course\/python-for-data-science-and-machine-learning-bootcamp\/\"><em>Python for Data Science and Machine Learning Bootcamp<\/em><\/a>.<\/li>\n<\/ul>\n<p>You can read more about learning to program \u2014 and how to get your first job in software engineering (if that&#8217;s the route you want to take) \u2014 in our <a href=\"https:\/\/80000hours.org\/career-reviews\/software-engineering\/#how-to-enter\">career review on software engineering<\/a>.<\/p>\n<h4 id=\"learning-the-maths\">Learning the maths<\/h4>\n<p>The maths of deep learning relies heavily on calculus and linear algebra, and statistics can be useful too \u2014 although generally learning the maths is much less important than programming and basic, practical ML.<\/p>\n<p>We&#8217;d generally recommend studying a quantitative degree (like maths, computer science or engineering), most of which will cover all three areas pretty well.<\/p>\n<p>If you want to actually get good at maths, you <em>have<\/em> to be solving problems. So, generally, the most useful thing that textbooks and online courses provide isn&#8217;t their explanations \u2014 it&#8217;s a set of exercises to try to solve, in order, with some help if you get stuck.<\/p>\n<p>If you want to self-study (especially if you don&#8217;t have a quantitative degree) here are some possible resources:<\/p>\n<ul>\n<li><strong>Calculus<\/strong>: <a href=\"https:\/\/www.youtube.com\/playlist?list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr\">3blue1brown&#8217;s video series on calculus<\/a> could be a good place to start. You may also be able to follow recorded university courses: MIT&#8217;s <a href=\"https:\/\/ocw.mit.edu\/courses\/18-01sc-single-variable-calculus-fall-2010\/pages\/syllabus\/\">single variable calculus<\/a> (which requires only high school algebra and trigonometry) followed by MIT&#8217;s course in <a href=\"https:\/\/ocw.mit.edu\/courses\/18-02sc-multivariable-calculus-fall-2010\/pages\/syllabus\/\">vector and multivariable calculus<\/a>. <\/li>\n<li><strong>Linear algebra<\/strong>: Again, we&#8217;d suggest <a href=\"https:\/\/www.youtube.com\/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab\">3blue1brown&#8217;s video series on linear algebra<\/a> as a place to start. In his <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/7WXPkpqKGKewAymJf\/how-to-pursue-a-career-in-technical-ai-alignment\">post about technical alignment careers<\/a>, Rogers-Smith recommends <a href=\"https:\/\/linear.axler.net\/\"><em>Linear Algebra Done Right<\/em><\/a> by Sheldon Axler. Finally, if you prefer lectures, try <a href=\"https:\/\/ocw.mit.edu\/courses\/18-06-linear-algebra-spring-2010\/pages\/syllabus\/\">MIT&#8217;s undergraduate course in linear algebra<\/a> (although note that this course assumes knowledge of multivariate calculus). <\/li>\n<li><strong>Probability<\/strong>: Take a look at <a href=\"https:\/\/ocw.mit.edu\/courses\/18-600-probability-and-random-variables-fall-2019\/pages\/syllabus\/\">MIT&#8217;s undergraduate course in probability and random variables<\/a>. <\/li>\n<\/ul>\n<p>You might be able to find resources that cover all these areas, like Imperial College&#8217;s <a href=\"https:\/\/www.coursera.org\/specializations\/mathematics-machine-learning\">Mathematics for Machine Learning<\/a>.<\/p>\n<h4 id=\"basic-machine-learning\">Learning basic machine learning<\/h4>\n<p>You&#8217;ll likely need to have a decent understanding of how AI systems are currently being developed. This will involve learning about machine learning and neural networks, before diving into any specific subfields of deep learning.<\/p>\n<p>Again, there&#8217;s the option of covering this at university. If you&#8217;re currently at college, it&#8217;s worth checking if you can take an ML course even if you&#8217;re not majoring in computer science.<\/p>\n<p>There&#8217;s one important caveat here: you&#8217;ll learn a huge amount on the job, and the amount you&#8217;ll need to know in advance for any role or course will vary hugely! Not even top academics know <em>everything<\/em> about their fields. It&#8217;s worth trying to find out how much you&#8217;ll need to know for the role you want to do before you invest hundreds of hours into learning about ML.<\/p>\n<p>With that caveat in mind, here are some suggestions of places you might start if you want to self-study the basics:<\/p>\n<ul>\n<li><a href=\"https:\/\/www.youtube.com\/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi\">3blue1brown&#8217;s series on neural networks<\/a> is a really great place to start for beginners.<\/li>\n<li>When I was learning, I used <a href=\"http:\/\/neuralnetworksanddeeplearning.com\/\"><em>Neural Networks and Deep Learning<\/em><\/a> \u2014 it&#8217;s an online textbook, good if you&#8217;re familiar with the maths, with some helpful exercises as well.<\/li>\n<li>Online intro courses like <a href=\"https:\/\/course.fast.ai\/\">fast.ai<\/a> (focused on practical applications), <a href=\"https:\/\/fullstackdeeplearning.com\/\"><em>Full Stack Deep Learning<\/em><\/a>, and the various courses at <a href=\"https:\/\/www.deeplearning.ai\/courses\/\">deeplearning.ai<\/a>.<\/li>\n<li>For more detail, see university courses like MIT&#8217;s <a href=\"https:\/\/openlearninglibrary.mit.edu\/courses\/course-v1:MITx+6.036+1T2019\/course\/\">*Introduction to Machine Learning<\/a>, NYU&#8217;s <a href=\"https:\/\/github.com\/Atcold\/NYU-DLFL22\"><em>Deep Learning<\/em><\/a> for even more detail. We&#8217;d also recommend Google DeepMind&#8217;s <a href=\"https:\/\/www.youtube.com\/playlist?list=PLqYmG7hTraZCRwoyGxvQkqVrZgDQi4m-5\">lecture series<\/a>.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/pytorch.org\/tutorials\/beginner\/deep_learning_60min_blitz.html\">PyTorch<\/a> is a very common package used for implementing neural networks, and probably worth learning! When I was first learning about ML, my first neural network was a 3-layer <a href=\"https:\/\/cs231n.github.io\/convolutional-networks\/\">convolutional neural network<\/a> with <a href=\"https:\/\/cs231n.github.io\/neural-networks-2\/\">L2 regularisation<\/a> classifying characters from the <a href=\"https:\/\/en.wikipedia.org\/wiki\/MNIST_database\">MNIST database<\/a>. This is a pretty common first challenge, and a good way to learn PyTorch.<\/p>\n<h4 id=\"learning-about-AI-safety\">Learning about AI safety<\/h4>\n<p>If you&#8217;re going to work as an AI safety researcher, it usually helps to know about AI safety.<\/p>\n<p>This isn&#8217;t always true \u2014 some engineering roles won&#8217;t require much knowledge of AI safety. But even then, knowing the basics will probably help land you a position, and can also help with things like making difficult judgement calls and avoiding doing harm. And if you want to be able to identify and do useful work, you&#8217;ll need to learn about the field eventually.<\/p>\n<p>Because the field is still so new, there probably aren&#8217;t (yet) university courses you can take. So you&#8217;ll need to do some self-study. Here are some places you might start:<\/p>\n<ul>\n<li>Section 3 of our <a href=\"\/problem-profiles\/artificial-intelligence\/#power-seeking-ai\">problem profile about preventing an AI-related catastrophe<\/a> provides an introduction to the problems that AI safety attempts to solve (with a particular focus on alignment).<\/li>\n<li><a href=\"https:\/\/www.youtube.com\/c\/RobertMilesAI\">Rob Miles&#8217; YouTube channel<\/a> is full of popular and well-explained introductory videos that don&#8217;t need much background knowledge of ML.<\/li>\n<li><a href=\"https:\/\/axrp.net\/\">AXRP<\/a> \u2013 the AI X-risk Research Podcast \u2014 is full of in-depth (and enjoyable) conversations with researchers about their research.<\/li>\n<li>The <a href=\"https:\/\/www.arena.education\/\">ARENA<\/a> course and <a href=\"https:\/\/github.com\/callummcdougall\/ARENA_3.0\">curriculum<\/a> provide a strong foundation for empirical AI safety research and engineering.<\/li>\n<li>The courses from <a href=\"https:\/\/www.agisafetyfundamentals.com\/\">AGI Safety Fundamentals<\/a>, in particular the <a href=\"https:\/\/www.agisafetyfundamentals.com\/ai-alignment-curriculum\"><em>AI Alignment Course<\/em><\/a>, which provides an introduction to research on the alignment problem.<\/li>\n<li><a href=\"https:\/\/course.mlsafety.org\/about\"><em>Intro to ML Safety<\/em><\/a>, a course from the <a href=\"https:\/\/www.safe.ai\/\">Center for AI Safety<\/a> focuses on withstanding hazards (&#8220;robustness&#8221;), identifying hazards (&#8220;monitoring&#8221;), and reducing systemic hazards (&#8220;systemic safety&#8221;), as well as alignment.<\/li>\n<\/ul>\n<p>For more suggestions \u2014 especially when it comes to reading about the nature of the risks we might face from AI systems \u2014 take a look at the <a href=\"https:\/\/80000hours.org\/problem-profiles\/artificial-intelligence\/#top-resources-to-learn-more\">top resources to learn more<\/a> from our problem profile.<\/p>\n<h3><span id=\"should-you-do-a-phd\" class=\"toc-anchor\"><\/span>Should you do a PhD?<\/h3>\n<p>Some technical research roles will require a PhD \u2014 but many won&#8217;t, and PhDs aren&#8217;t the best option for everyone.<\/p>\n<p>The main benefit of doing a PhD is probably practising setting and carrying out your <em>own<\/em> research agenda. As a result, getting a PhD is practically the default if you want to be a research lead.<\/p>\n<p>That said, you can also become a research lead without a PhD \u2014 in particular, by transitioning from a role as a research contributor. At some large labs, the boundary between being a contributor and a lead is increasingly blurry.<\/p>\n<p>Many people find PhDs very difficult. They can be isolating and frustrating, and take a very long time (4&#8211;6 years). What&#8217;s more, both your quality of life and the amount you&#8217;ll learn will depend on your supervisor \u2014 and it can be really difficult to figure out in advance whether you&#8217;re making a good choice.<\/p>\n<p>So, if you&#8217;re considering doing a PhD, here are some things to consider:<\/p>\n<ul>\n<li><strong>Your long-term vision<\/strong>: If you&#8217;re aiming to be a research lead, that suggests you might want to do a PhD \u2014 the vast majority of research leads have PhDs. If you mainly want to be a contributor (e.g. an ML or software engineer), that suggests you might not. If you&#8217;re unsure, you should try doing something to <a href=\"\/career-guide\/personal-fit\/\">test your fit for each<\/a>, like trying a project or internship. You might try a pre-doctoral research assistant role \u2014 if the research you do is relevant to your future career, these can be good career capital, whether or not you do a PhD.<\/li>\n<li><strong>The topic of your research<\/strong>: It&#8217;s easy to let yourself become tied down to a PhD topic you&#8217;re not confident in. If the PhD you&#8217;re considering would let you work on something that seems useful for AI safety, it&#8217;s probably \u2014 all else equal \u2014 better for your career, and the research itself might have a positive impact as well.<\/li>\n<li><strong>Mentorship<\/strong>: What are the supervisors or managers like at the opportunities open to you? You might be able to find ML engineering or research roles in industry where you could learn much more than you would in a PhD \u2014 or vice versa. When picking a supervisor, try reaching out to the current or former students of a prospective supervisor and asking them some frank questions. (Also, see <a href=\"https:\/\/journals.plos.org\/ploscompbiol\/article?id=10.1371\/journal.pcbi.1009330\">this article<\/a> on how to choose a PhD supervisor.)<\/li>\n<li><strong>Your fit for the work environment<\/strong>: Doing a PhD means working on your own with very little supervision or feedback for long periods of time. Some people thrive in these conditions! But some really don&#8217;t and find PhDs extremely difficult.<\/li>\n<\/ul>\n<p>Read more in our more detailed (but less up-to-date) review of <a href=\"https:\/\/80000hours.org\/career-reviews\/machine-learning-phd\/\">machine learning PhDs<\/a>.<\/p>\n<p>It&#8217;s worth remembering that most jobs don&#8217;t <em>need<\/em> a PhD. And for some jobs, especially empirical research contributor roles, even if a PhD would be helpful, there are often better ways of getting the career capital you&#8217;d need (for example, working as a software or ML engineer). We&#8217;ve interviewed two ML engineers who have had hugely successful careers <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/olsson-and-ziegler-ml-engineering-and-safety\/\">without doing a PhD<\/a>.<\/p>\n<h4 id=\"phd-timelines\">Whether you should do a PhD doesn&#8217;t depend (much) on timelines<\/h4>\n<p>We think it&#8217;s plausible that we will develop AI that could be hugely transformative for society <a href=\"\/problem-profiles\/artificial-intelligence\/#when-can-we-expect-to-develop-transformative-AI\">by the end of the 2030s<\/a>.<\/p>\n<p>All else equal, that possibility could argue for trying to have an impact right away, rather than spending five (or more) years doing a PhD.<\/p>\n<p>Ultimately, though, <strong>how well you, in particular, are suited to a particular PhD<\/strong> is probably a much more important factor than when AI will be developed.<\/p>\n<p>That is to say, we think the <em>increase<\/em> in impact caused by choosing a path that&#8217;s a good fit for you is probably larger than any <em>decrease<\/em> in impact caused by delaying your work. This is in part because the spread in impact caused by the specific roles available to you, as well as your personal fit for them, is usually very large. Some roles (especially research lead roles) will just <em>require<\/em> having a PhD, and others (especially more engineering-heavy roles) won&#8217;t \u2014 and people&#8217;s fit for these paths varies quite a bit.<\/p>\n<p>We&#8217;re also highly uncertain about <a href=\"\/problem-profiles\/artificial-intelligence\/#when-can-we-expect-to-develop-transformative-AI\">estimates about when we might develop transformative AI<\/a>. This uncertainty reduces the <a href=\"\/articles\/expected-value\/\">expected<\/a> cost of any delay.<\/p>\n<p>Most importantly, we think PhDs <em>shouldn&#8217;t<\/em> be thought of as a pure delay to your impact. You can do useful work in a PhD, and generally, the first couple of years in any career path will involve a lot of learning the basics and getting up to speed. So if you have a good mentor, work environment, and choice of topic, your PhD work could be as good as, or possibly better than, the work you&#8217;d do if you went to work elsewhere early in your career. And if you suddenly receive evidence that we have less time than you thought, it&#8217;s relatively easy to drop out.<\/p>\n<p>There are lots of other considerations here \u2014 for a rough overview, and some discussion, see <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/jfLjsxcejCFDpo7dw\/whether-you-should-do-a-phd-doesn-t-depend-much-on-timelines\">this post by 80,000 Hours advisor Alex Lawsen<\/a>, as well as the <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/jfLjsxcejCFDpo7dw\/whether-you-should-do-a-phd-doesn-t-depend-much-on-timelines?commentId=oCxAANy3yoa3YZbwu\">comments<\/a>.<\/p>\n<p>Overall, we&#8217;d suggest that instead of worrying about a delay to your impact, think instead about which longer-term path you want to pursue, and how the specific opportunities in front of you will get you there.<\/p>\n<h4>How to get into a PhD<\/h4>\n<p>ML PhDs can be very competitive. To get in, you&#8217;ll probably need a few publications (as we said above, something like a first author workshop paper, as well as a third author conference paper at a major ML conference (like <a href=\"https:\/\/nips.cc\/\">NeurIPS<\/a> or <a href=\"https:\/\/icml.cc\/\">ICML<\/a>), and references, probably from ML academics. (Although publications also look good whatever path you end up going down!)<\/p>\n<p>To end up at that stage, you&#8217;ll need a fair bit of luck, and you&#8217;ll also need to find ways to get some research experience.<\/p>\n<p>One option is to do a master&#8217;s degree in ML, although make sure it&#8217;s a research masters \u2014 most ML master&#8217;s degrees primarily focus on preparation for industry.<\/p>\n<p>Even better, try getting an internship in an ML research group. Opportunities include <a href=\"https:\/\/riss.ri.cmu.edu\/\">RISS<\/a> at Carnegie Mellon University, <a href=\"https:\/\/www.imperial.ac.uk\/urop\/\">UROP<\/a> at Imperial College London, <a href=\"https:\/\/www.aalto.fi\/en\/aalto-science-institute-asci\/aalto-science-institute-international-summer-research-programme\">the Aalto Science Institute international summer research programme<\/a>, the <a href=\"https:\/\/data-science.llnl.gov\/dssi\">Data Science Summer Institute<\/a>, the <a href=\"https:\/\/www.ttic.edu\/visiting-student\/\">Toyota Technological Institute intern programme<\/a> and <a href=\"https:\/\/mila.quebec\/en\/supervision-requests\/\">MILA<\/a>. You can also try doing an internship specifically in AI safety, for example at <a href=\"https:\/\/humancompatible.ai\/jobs#internship\">CHAI<\/a>. However, there are sometimes disadvantages to doing internships specifically in AI safety directly \u2014 in general, it may be harder to publish and mentorship might be more limited.<\/p>\n<p>Another way of getting research experience is by asking whether you can work with researchers. If you&#8217;re already at a top university, it can be easiest to reach out to people working at the university you&#8217;re studying at.<\/p>\n<p>PhD students or post-docs can be more responsive than professors, but eventually, you&#8217;ll want a few professors you&#8217;ve worked with to provide references, so you&#8217;ll need to get in touch. Professors tend to get lots of cold emails, so try to get their attention! You can try:<\/p>\n<ul>\n<li>Getting an introduction, for example from a professor who&#8217;s taught you<\/li>\n<li>Mentioning things you&#8217;ve done (your grades, relevant courses you&#8217;ve taken, your GitHub, any ML research papers you&#8217;ve attempted to replicate as practice)<\/li>\n<li>Reading some of their papers and the main papers in the field, and mention them in the email<\/li>\n<li>Applying for <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/7WXPkpqKGKewAymJf\/how-to-pursue-a-career-in-technical-ai-alignment#Funding\">funding that&#8217;s available to students who want to work in AI safety<\/a>, and letting people know you&#8217;ve got funding to work with them<\/li>\n<\/ul>\n<p>Ideally, you&#8217;ll find someone who supervises you well and has time to work with you (that doesn&#8217;t necessarily mean the most famous professor \u2014 although it helps a lot if they&#8217;re regularly publishing at top conferences). That way, they&#8217;ll get to know you, you can impress them, and they&#8217;ll provide an amazing reference when you apply for PhDs.<\/p>\n<p>It&#8217;s very possible that, to get the publications and references you&#8217;ll need to get into a PhD, you&#8217;ll need to spend a year or two working as a research assistant, although these positions can also be quite competitive.<\/p>\n<p><a href=\"https:\/\/docs.google.com\/document\/d\/1RFo7_9JVmt0z8RPwUjB-mUMgCMoUQmsaj2CM5aHvxCw\/edit\">This guide by Adam Gleave<\/a> also goes into more detail on how to get a PhD, including where to apply and tips on the application process itself. We discuss ML PhDs in more detail in our <a href=\"https:\/\/80000hours.org\/career-reviews\/machine-learning-phd\/\">career review on ML PhDs<\/a> (though it&#8217;s outdated compared to this career review).<\/p>\n<h3><span id=\"getting-a-job-empirical-research\" class=\"toc-anchor\"><\/span>Getting a job in empirical AI safety research<\/h3>\n<p>Ultimately, the best way of learning to do empirical research \u2014 especially in contributor and engineering-focused roles \u2014 is to work somewhere that does both high-quality engineering and cutting-edge research.<\/p>\n<p>The top three companies are probably Google DeepMind (who offer <a href=\"https:\/\/ai-jobs.net\/job\/28354-research-engineer-intern-2023-london\/\">internships to students<\/a>), OpenAI (who have a 6-month <a href=\"https:\/\/openai.com\/blog\/openai-residency\">residency programme<\/a>) and Anthropic. (Working at a leading AI company carries with it some risk of doing harm, so it&#8217;s important to think carefully about your options. <a href=\"\/career-reviews\/working-at-an-AI-lab\/\">We&#8217;ve written a separate article going through the major relevant considerations.<\/a>)<\/p>\n<p>To end up working in an empirical research role, you&#8217;ll probably need to build some <a href=\"\/career-guide\/career-capital\">career capital<\/a>.<\/p>\n<p>Whether you want to be a research lead or a contributor, it&#8217;s going to help to become a really good software engineer. The best ways of doing this usually involve <a href=\"\/career-reviews\/software-engineering\/\">getting a job as a software engineer at a big tech company or at a promising startup<\/a>. (We&#8217;ve written an entire article about <a href=\"https:\/\/80000hours.org\/career-reviews\/software-engineering\/\">becoming a software engineer<\/a>.)<\/p>\n<p>Many roles will require you to be a good ML engineer, which means going further than just <a href=\"#basic-machine-learning\">the basics<\/a> we looked at above. The best way to become a good ML engineer is to <em>get a job doing ML engineering<\/em> \u2014 and the best places for that are probably <a href=\"\/career-reviews\/working-at-an-AI-lab\/\">leading AI companies<\/a>.<\/p>\n<p>For roles as a research lead, you&#8217;ll need relatively more research experience. You&#8217;ll either want to become a research contributor first, or enter through academia (for example by doing a PhD).<\/p>\n<p>All that said, it&#8217;s important to remember that you don&#8217;t need to know <em>everything<\/em> to start applying, as you&#8217;ll inevitably learn loads on the job \u2014 so do try to find out what you&#8217;ll need to learn to land the <em>specific<\/em> roles you&#8217;re considering.<\/p>\n<p>How much experience do you need to get a job? It&#8217;s worth reiterating the tests we looked at above for contributor roles:<\/p>\n<ul>\n<li>In a <a href=\"https:\/\/www.alignmentforum.org\/posts\/nzmCvRvPm4xJuqztv\/deepmind-is-hiring-for-the-scalable-alignment-and-alignment\">blog post about hiring for safety researchers<\/a>, the DeepMind team said &#8220;as a rough test for the Research Engineer role, if you can reproduce a typical ML paper in a few hundred hours and your interests align with ours, we&#8217;re probably interested in interviewing you.&#8221;<\/li>\n<li>Looking specifically at <em>software engineering<\/em>, one hiring manager at Anthropic said that if you could, with a few weeks&#8217; work, write a new feature or fix a serious bug in a major ML library, they&#8217;d want to interview you straight away. (<a href=\"https:\/\/www.alignmentforum.org\/posts\/YDF7XhMThhNfHfim9\/ai-safety-needs-great-engineers\">Read more<\/a>.)<\/li>\n<\/ul>\n<p>In the process of getting this experience, you might end up working in roles that advance AI capabilities. There are a variety of views on whether this might be harmful \u2014 so we&#8217;d suggest reading our <a href=\"\/career-reviews\/working-at-an-AI-lab\/\">article about working at leading AI companies<\/a> and our <a href=\"\/articles\/ai-capabilities\/\">article containing anonymous advice from experts about working in roles that advance capabilities.<\/a> It&#8217;s also worth <a href=\"\/speak-with-us\/\">talking to our team about any specific opportunities you have<\/a>.<\/p>\n<p>If you&#8217;re doing another job, or a degree, or think you need to learn some more before trying to change careers, there are a few good ways of getting more experience doing ML engineering that go beyond the basics we&#8217;ve already covered:<\/p>\n<ul>\n<li><strong>Getting some experience in software \/ ML engineering.<\/strong> For example, if you&#8217;re doing a degree, you might try an internship as a software engineer during the summer. DeepMind offer <a href=\"https:\/\/ai-jobs.net\/job\/28354-research-engineer-intern-2023-london\/\">internships<\/a> for students with at least two years of study in a technical subject, <\/li>\n<li><strong>Replicating papers.<\/strong> One great way of getting experience doing ML engineering, is to replicate some papers in whatever sub-field you might want to work in. Richard Ngo, an AI governance researcher at OpenAI, has written some <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/fRjj6nm9xbW4kFcTZ\/advice-on-pursuing-technical-ai-safety-research#2_1__Advice_on_paper_replication\">advice on replicating papers<\/a>. But bear in mind that replicating papers can be quite hard \u2014 take a look at <a href=\"http:\/\/amid.fish\/reproducing-deep-rl\">Amid Fish&#8217;s blog on what he learned replicating a deep RL paper<\/a>. Finally, <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/7WXPkpqKGKewAymJf\/how-to-pursue-a-career-in-technical-ai-alignment#How_to_pursue_research_contributor__ML_engineering__roles\">Rogers-Smith has some suggestions on papers to replicate<\/a>. If you do spend some time replicating papers, remember that when you get to applying for roles, it will be really useful to be able to prove you&#8217;ve done the work. So try uploading your work to GitHub, or writing a blog on your progress. And if you&#8217;re thinking about spending a long time on this (say, over 100 hours), try to get some feedback on the papers you might replicate before you start \u2014 you could even reach out to a lab or company you want to work for. <\/li>\n<li><strong>Taking or following a more in-depth course in empirical AI safety research.<\/strong> Redwood Research ran the <a href=\"https:\/\/www.redwoodresearch.org\/mlab\">MLAB<\/a> bootcamp, and you can apply for access to their curriculum <a href=\"https:\/\/airtable.com\/shrOOfZIZC4zZtAIr\">here<\/a>. You could also take a look at <a href=\"https:\/\/github.com\/jacobhilton\/deep_learning_curriculum\">this Deep Learning Curriculum<\/a> by Jacob Hilton, a researcher at the Alignment Research Center \u2014 although it&#8217;s probably very challenging without mentorship. The <a href=\"https:\/\/www.arena.education\/\">Alignment Research Engineer Accelerator<\/a> is a program that uses this curriculum. Mentors in the <a href=\"https:\/\/www.matsprogram.org\/\">ML Alignment &amp; Theory Scholars Program<\/a> primarily focus on empirical research.<\/li>\n<li><strong>Learning about a sub-field of deep learning.<\/strong> In particular, we&#8217;d suggest <em>natural language processing<\/em> (in particular transformers \u2014 see <a href=\"https:\/\/www.youtube.com\/watch?v=sNfkZFVm_xs\">this lecture<\/a> as a starting point) and <em>reinforcement learning<\/em> (take a look at <a href=\"https:\/\/karpathy.github.io\/2016\/05\/31\/rl\/\"><em>Pong from Pixels<\/em><\/a> by Andrej Karpathy, and OpenAI&#8217;s <a href=\"https:\/\/spinningup.openai.com\/en\/latest\/index.html\"><em>Spinning up in Deep RL<\/em><\/a>). Try to get to the point where you know about the most important recent advances.<\/li>\n<\/ul>\n<p>Finally, <a href=\"https:\/\/researchathena.org\/\">Athena<\/a> is an AI alignment mentorship program for women with a technical background looking to get jobs in the alignment field.<\/p>\n<h3><span id=\"getting-a-job-theoretical-research\" class=\"toc-anchor\"><\/span>Getting a job in theoretical AI safety research<\/h3>\n<p>There are fewer jobs available in theoretical AI safety research, so it&#8217;s harder to give concrete advice. Having a maths or theoretical computer science PhD isn&#8217;t always necessary, but is fairly common among researchers in industry, and is pretty much <em>required<\/em> to be an academic.<\/p>\n<p>If you do a PhD, ideally it&#8217;d be in an area at least somewhat related to theoretical AI safety research. For example, it could be in probability theory as applied to AI, or in theoretical CS (look for researchers who publish in <a href=\"https:\/\/www.learningtheory.org\/\">COLT<\/a> or <a href=\"http:\/\/ieee-focs.org\/\">FOCS<\/a>).<\/p>\n<p>Alternatively, one path is to become an empirical research lead before moving into theoretical research.<\/p>\n<p>Compared to empirical research, you&#8217;ll need to know relatively less about engineering, and relatively more about AI safety as a field.<\/p>\n<p>Once you&#8217;ve done <a href=\"#learn-the-basics\">the basics<\/a>, one possible next step you could try is reading papers from a particular researcher, or on a particular topic, and summarising what you&#8217;ve found.<\/p>\n<p>You could also try spending some time (maybe 10&#8211;100 hours) reading about a topic and then some more time (maybe another 10&#8211;100 hours) trying to come up with some new ideas on that topic. For example, you could try coming up with proposals to solve the problem of <a href=\"https:\/\/www.alignmentforum.org\/posts\/QEYWkRoCn4fZxXQAY\/prizes-for-elk-proposals\">eliciting latent knowledge<\/a>. Alternatively, if you wanted to focus on the more mathematical side, you could try having a go at the <a href=\"https:\/\/drive.google.com\/file\/d\/1VQiy9Nl2VqdtzsJYUWhSuKSATwmXE_T3\/view\">assignment at the end of this lecture by Michael Cohen, a grad student at the University of Oxford<\/a>.<\/p>\n<p>If you want to enter academia, reading a <em>ton<\/em> of papers seems particularly important. Maybe try writing a survey paper on a certain topic in your spare time. It&#8217;s a great way to master a topic, spark new ideas, spot gaps, and come up with research ideas. When applying to grad school or jobs, your paper is a fantastic way to show you love research so much you do it for fun.<\/p>\n<p>Other ways to get more concrete experience include doing research internships, working as a research assistant, or doing a PhD, all of which we&#8217;ve written about above, in the section on <a href=\"#should-you-do-a-phd\">whether and how you can get into a PhD programme<\/a>.<\/p>\n<p>One note is that a lot of people we talk to try to learn independently. This can be a great idea for some people, but is fairly tough for many, because there&#8217;s substantially less structure and mentorship.<\/p>\n<h3><span id=\"key-organisations\" class=\"toc-anchor\"><\/span>Key organisations<\/h3>\n<p>AI companies that have empirical technical safety teams, or are focused entirely on safety:<\/p>\n<p>AI organisations that have empirical technical safety teams, or are focused entirely on safety:<\/p>\n<ul>\n<li><a href=\"https:\/\/jobs.80000hours.org\/organisations\/anthropic\">Anthropic<\/a> is a safety-focused AI company working on building interpretable and safe AI systems. They focus on empirical AI safety research. Anthropic cofounders Daniela and Dario Amodei gave an <a href=\"https:\/\/futureoflife.org\/2022\/03\/04\/daniela-and-dario-amodei-on-anthropic\/\">interview about the lab on the Future of Life Institute podcast<\/a>. On our podcast, we spoke to <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/chris-olah-interpretability-research\/\">Chris Olah<\/a>, who leads Anthropic&#8217;s research into <a href=\"https:\/\/transformer-circuits.pub\/\">interpretability<\/a>, and <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/nova-dassarma-information-security-and-ai-systems\/\">Nova DasSarma<\/a>, who works on systems infrastructure at Anthropic.<\/li>\n<li><a href=\"https:\/\/metr.org\/\">METR<\/a> works on assessing whether cutting-edge AI systems could pose catastrophic risks to civilization, including early-stage, experimental work to develop techniques, and evaluating systems produced by Anthropic and OpenAI.<\/li>\n<li>The UK government&#8217;s <a href=\"https:\/\/www.aisi.gov.uk\/\">AI Safety Institute<\/a> is conducting research to assess the risks posed by advanced AI systems. It also coordinates with various companies, governments, and other actors, and it works to inform policymakers and shape safety practices around AI development globally.<\/li>\n<li>The <a href=\"https:\/\/jobs.80000hours.org\/organisations\/center-for-ai-safety\">Center for AI Safety<\/a> is a nonprofit that does technical research and promotion of safety in the wider machine learning community.<\/li>\n<li><a href=\"https:\/\/jobs.80000hours.org\/organisations\/far-ai\">FAR AI<\/a> is a research nonprofit that incubates and accelerates research agendas that are too resource-intensive for academia but not yet ready for commercialisation by industry, including research in adversarial robustness, interpretability and preference learning.<\/li>\n<li><a href=\"https:\/\/www.apolloresearch.ai\/\">Apollo Research<\/a> is a non-profit that aims to develop AI model evaluation processes to detect signs of misalignment and deception. It works on interpretability, behaviour tests, and fine-tuning, and it aims to provide technical support to lawmakers looking to govern advanced AI.<\/li>\n<li><a href=\"https:\/\/jobs.80000hours.org\/organisations\/deepmind\">Google DeepMind<\/a> is probably the largest and most well-known research group developing general artificial machine intelligence, and is famous for its work creating <a href=\"https:\/\/en.wikipedia.org\/wiki\/AlphaGo\">AlphaGo<\/a>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/AlphaZero\">AlphaZero<\/a>, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/AlphaFold\">AlphaFold<\/a>. It is not principally focused on safety, but has <a href=\"https:\/\/www.alignmentforum.org\/posts\/nzmCvRvPm4xJuqztv\/deepmind-is-hiring-for-the-scalable-alignment-and-alignment\">two teams focused on AI safety<\/a>, with the <a href=\"https:\/\/www.alignmentforum.org\/posts\/nzmCvRvPm4xJuqztv\/deepmind-is-hiring-for-the-scalable-alignment-and-alignment#Scalable_Alignment_s_current_approach__make_AI_critique_itself\">Scalable Alignment Team<\/a> focusing on aligning existing state-of-the-art systems, and the <a href=\"https:\/\/www.alignmentforum.org\/posts\/nzmCvRvPm4xJuqztv\/deepmind-is-hiring-for-the-scalable-alignment-and-alignment#Alignment_Team_s_portfolio_of_projects\">Alignment Team<\/a> focused on research bets for aligning future systems.<\/li>\n<li><a href=\"https:\/\/jobs.80000hours.org\/organisations\/openai\">OpenAI<\/a>, founded in 2015, is a company that is trying to build artificial general intelligence that is safe and benefits all of humanity. OpenAI is well known for its language models like <a href=\"https:\/\/en.wikipedia.org\/wiki\/GPT-4\">GPT-4<\/a>. Like DeepMind, it is not principally focused on safety, but has a preparedness team and a governance team.<\/li>\n<li><a href=\"https:\/\/ought.org\/\">Ought<\/a> is a machine learning lab building <a href=\"https:\/\/jobs.80000hours.org\/organisations\/elicit\">Elicit<\/a>, an AI research assistant. Their aim is to align open-ended reasoning by <a href=\"https:\/\/ought.org\/updates\/2022-04-06-process\">learning human reasoning steps<\/a>, and to direct AI progress towards helping with <a href=\"https:\/\/arxiv.org\/abs\/2301.01751\">evaluating evidence and arguments<\/a>.<\/li>\n<li><a href=\"https:\/\/jobs.80000hours.org\/organisations\/redwood-research\">Redwood Research<\/a> is an AI safety research organisation, whose <a href=\"https:\/\/www.alignmentforum.org\/posts\/k7oxdbNaGATZbtEg3\/redwood-research-s-current-project\">first big project<\/a> attempted to make sure language models (like GPT-3) produce output following certain rules with very high probability, in order to address failure modes too rare to show up in standard training. <\/li>\n<\/ul>\n<p>Theoretical \/ conceptual AI safety labs:<\/p>\n<ul>\n<li>The <a href=\"https:\/\/jobs.80000hours.org\/organisations\/alignment-research-center\">Alignment Research Center<\/a> (ARC) is attempting to produce alignment strategies that could be adopted in industry today while also being able to scale to future systems. They focus on conceptual work, developing strategies that could work for alignment and which may be promising directions for empirical work, rather than doing empirical AI work themselves. Their first project was releasing a report on <a href=\"https:\/\/www.alignmentforum.org\/posts\/qHCDysDnvhteW7kRd\/arc-s-first-technical-report-eliciting-latent-knowledge\">Eliciting Latent Knowledge<\/a>, the problem of getting advanced AI systems to honestly tell you what they believe (or &#8216;believe&#8217;) about the world. On our podcast, we interviewed <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/paul-christiano-a-message-for-the-future\/\">ARC founder Paul Christiano about his research<\/a> (before he founded ARC).<\/li>\n<li>The <a href=\"https:\/\/jobs.80000hours.org\/organisations\/center-on-long-term-risk\">Center on Long-Term Risk<\/a> works to address worst-case risks from advanced AI. They focus on conflict between AI systems.<\/li>\n<li>The <a href=\"https:\/\/jobs.80000hours.org\/organisations\/machine-intelligence-research-institute\">Machine Intelligence Research Institute<\/a> was one of the first groups to become concerned about the risks from machine intelligence in the early 2000s, and its team has published a <a href=\"https:\/\/intelligence.org\/research\/\">number of papers<\/a> on safety issues and how to resolve them.<\/li>\n<li>Some teams in commercial labs also do some more theoretical and conceptual work on alignment, such as Anthropic&#8217;s work on <a href=\"https:\/\/arxiv.org\/abs\/2302.00805\">conditioning predictive models<\/a> and the <a href=\"https:\/\/causalincentives.com\/\">Causal Incentives Working Group<\/a> at Google DeepMind.<\/li>\n<\/ul>\n<p>AI safety in academia (a very non-comprehensive list; while the number of academics explicitly and publicly focused on AI safety is small, it&#8217;s possible to do relevant work at a much wider set of places):<\/p>\n<ul>\n<li>The Algorithmic Alignment Group in the Computer Science and Artificial Intelligence Laboratory at MIT, led by <a href=\"https:\/\/people.csail.mit.edu\/dhm\/\">Dylan Hadfield-Menell<\/a><\/li>\n<li>The <a href=\"https:\/\/jobs.80000hours.org\/organisations\/university-of-california-center-for-human-compatible-artificial\">Center for Human-Compatible AI<\/a> at UC Berkeley, led by Stuart Russell, focuses on academic research to ensure AI is safe and beneficial to humans. (Our <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/stuart-russell-human-compatible-ai\/\">podcast with Stuart Russell<\/a> examines his approach to provably beneficial AI.)<\/li>\n<li><a href=\"https:\/\/jsteinhardt.stat.berkeley.edu\/\">Jacob Steinhardt&#8217;s research group<\/a> in the Department of Statistics at UC Berkeley<\/li>\n<li>The <a href=\"https:\/\/wp.nyu.edu\/ml2\/\">NYU Alignment research Group<\/a> led by Sam Bowman<\/li>\n<li>The <a href=\"https:\/\/www.kasl.ai\/\">Krueger AI Safety Lab<\/a> at the University of Cambridge led by David Krueger<\/li>\n<li>The <a href=\"https:\/\/tegmark.org\/\">Tegmark AI Safety Group<\/a> led by Max Tegmark of MIT<\/li>\n<li>The <a href=\"https:\/\/www.cs.cmu.edu\/~focal\/\">Foundations of Cooperative AI Lab<\/a> at Carnegie Mellon University<\/li>\n<li>The <a href=\"https:\/\/acsresearch.org\/\">Alignment of Complex Systems<\/a> research group at Charles University, Prague<\/li>\n<\/ul>\n<p>There are also some AI safety offices that can host independent researchers. These include <a href=\"https:\/\/www.safeai.org.uk\/\">LISA<\/a> in London and <a href=\"https:\/\/far.ai\/labs\/\">FAR Labs<\/a> in Berkeley, California.<\/p>\n<div class=\"well bg-gray-lighter margin-bottom margin-top padding-top-small padding-bottom-small\">\n<h2><span id=\"want-one-on-one-advice-on-pursuing-this-path\" class=\"toc-anchor\"><\/span>Want one-on-one advice on pursuing this path?<\/h2>\n<p>We think that the risks posed by the development of AI may be the most pressing problem the world currently faces. If you think you might be a good fit for any of the above career paths that contribute to solving this problem, we&#8217;d be <em>especially<\/em> excited to advise you on next steps, one-on-one.<\/p>\n<p>We can help you consider your options, make connections with others working on reducing risks from AI, and possibly even help you find jobs or funding opportunities \u2014 all for free.<\/p>\n<p><a href=\"\/speak-with-us\/?int_campaign=2022-08__ai-research-career-review\" title=\"\" class=\"btn btn-primary\">APPLY TO SPEAK WITH OUR TEAM<\/a><\/p>\n<\/div>\n<h2><span id=\"find-a-job-in-this-path\" class=\"toc-anchor\"><\/span>Find a job in this path<\/h2>\n<p>If you think you might be a good fit for this path and you&#8217;re ready to start looking at job opportunities that are currently accepting applications, see our <strong>curated<\/strong> list of opportunities for this path:<\/p>\n<p><script>\n    function getLocationString(arr) {\n      if (arr.length <= 3) { \n        return arr.join(\"<br \/>\");\n      }\n      return arr.slice(0, 3).join(\"<br \/>\") + \"...\";\n    }\n  <\/script><script>\n    function getUniqueCompanyJobs(jobs, limit) {\n      const uniqueCompanies = new Set();\n      const uniqueJobs = [];\n      const additionalJobs = [];\n      for (const job of jobs) {\n          const company = job.company_name;\n          if (!uniqueCompanies.has(company)) {\n              uniqueCompanies.add(company);\n              uniqueJobs.push(job);\n          } else {\n              additionalJobs.push(job);\n          }\n      }\n      return uniqueJobs.concat(additionalJobs).slice(0, limit);\n    }\n  <\/script><script>\n    window.addEventListener(\"load\", function() {\n        const container = document.querySelector(\"#vacancies-1\");\n        if (container) {\n          const searchClient = algoliasearch(\"W6KM1UDIB3\", \"d1d7f2c8696e7b36837d5ed337c4a319\");\n          searchClient.initIndex(\"jobs_prod\"); \n          const search = instantsearch({\n            indexName: \"jobs_prod\",\n            searchClient,\n          });\n          search.addWidget(\n            instantsearch.widgets.configure({\n              facetFilters: [[\"tags_area:AI safety & policy\"],[\"tags_skill:Research\"]],\n              hitsPerPage: 6,\n            })\n          );\n          search.addWidget({\n            render(options) {\n              const results = getUniqueCompanyJobs(options.results.hits, 3);\n              results.forEach(item => {\n                item.post_pk = DOMPurify.sanitize(item.post_pk);\n                item.company.logo_url = DOMPurify.sanitize(item.company.logo_url);\n                item.title = DOMPurify.sanitize(item.title);\n                item.company.name = DOMPurify.sanitize(item.company.name);\n                item.card_locations = DOMPurify.sanitize(getLocationString(item.card_locations));\n                item.posted_at_relative = DOMPurify.sanitize(item.posted_at_relative);\n              });\n              container.innerHTML = results.map(item => {\n                return `<\/p>\n<li class=\"vacancy border\">\n                    <a href=\"https:\/\/jobs.80000hours.org\/?jobPk=${item.post_pk}\" target=\"_blank\" rel=\"noopener noreferrer\" class=\"vacancy-summary pt-2 pb-2\"><\/p>\n<div class=\"col-12\">\n<div class=\"row\" style=\"position: relative;\">\n<div class=\"col-sm-8\" style=\"overflow: hidden;\">\n<div class=\"vacancy__org-logo\">\n                              <img decoding=\"async\" src=\"${item.company.logo_url}\">\n                            <\/div>\n<div class=\"vacancy__job-title-and-org-name\">\n<h5 class=\"vacancy__job-title tw--line-clamp-2\">${item.title}<\/h5>\n<p class=\"vacancy__org-name tw--line-clamp-2\">${item.company.name}<\/p><\/div><\/div>\n<div class=\"col-sm-4 text-right hidden-xs vacancy__location-and-date-listed\">\n<p class=\"pr-1\">${item.card_locations}<br \/>${item.posted_at_relative}<\/p><\/div><\/div><\/div>\n<p>                    <\/a>\n                  <\/li>\n<p>`;\n              }).join(\"\");\n            }\n          });\n          search.start();\n        }\n      });\n    <\/script><\/p>\n<ul id=\"vacancies-1\" class=\"!tw--p-0 no-visited-styling\"><\/ul>\n<p><a href=https:\/\/jobs.80000hours.org\/?refinementList%5Btags_area%5D%5B0%5D=AI%20safety%20%26%20policy&refinementList%5Btags_skill%5D%5B0%5D=Research class=\"btn btn-primary\" target=\"_blank\">View all opportunities<\/a><\/p>\n<h2><span id=\"learn-more-about-ai-safety-technical-research\" class=\"toc-anchor\"><\/span>Learn more about AI safety technical research<\/h2>\n<h3><span id=\"top-recommendations\" class=\"toc-anchor\"><\/span>Top recommendations<\/h3>\n<ul>\n<li><a href=\"https:\/\/course.aisafetyfundamentals.com\/alignment\">The AGI safety fundamentals technical alignment curriculum<\/a><\/li>\n<li><a href=\"https:\/\/80000hours.org\/podcast\/on-artificial-intelligence\/\"><em>The 80,000 Hours Podcast on Artificial Intelligence<\/em><\/a> (a collection of 10 key AI episodes from our podcast)<\/li>\n<li><a href=\"https:\/\/forum.effectivealtruism.org\/posts\/7WXPkpqKGKewAymJf\/how-to-pursue-a-career-in-technical-ai-alignment\">Charlie Rogers-Smith&#8217;s step-by-step guide to AI safety careers<\/a> (which this article is in large part based on) provides some helpful concrete advice, including <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/7WXPkpqKGKewAymJf\/how-to-pursue-a-career-in-technical-ai-alignment#Funding\">ways you might get some funding to help you move into an AI safety technical research career<\/a>.<\/li>\n<\/ul>\n<h3><span id=\"further-recommendations\" class=\"toc-anchor\"><\/span>Further recommendations<\/h3>\n<h4 class=\"no-toc\">Other articles and resources<\/h4>\n<p>Here are some suggestions about where you could learn more:<\/p>\n<ul>\n<li>To help you get oriented in the field, we recommend the <a href=\"https:\/\/forum.effectivealtruism.org\/posts\/pbiGHk6AjRxdBPoD8\/ai-safety-starter-pack\">AI safety starter pack<\/a>. <\/li>\n<li><a href=\"https:\/\/docs.google.com\/document\/d\/1RFo7_9JVmt0z8RPwUjB-mUMgCMoUQmsaj2CM5aHvxCw\/edit#heading=h.hfj52k67ycog\">Careers in Beneficial AI Research<\/a> by Adam Gleave, CEO of <a href=\"https:\/\/far.ai\/\">FAR AI<\/a><\/li>\n<li>Our <a href=\"https:\/\/80000hours.org\/problem-profiles\/positively-shaping-artificial-intelligence\/\">problem profile on AI risk<\/a> <\/li>\n<li>This <a href=\"https:\/\/www.alignmentforum.org\/s\/mzgtmmTKKn5MuCzFJ\">sequence of posts on AI safety technical alignment<\/a> by Richard Ngo)<\/li>\n<li>Our career review of <a href=\"https:\/\/80000hours.org\/career-reviews\/machine-learning-phd\/\">machine learning PhDs<\/a> <\/li>\n<li>Our career review of <a href=\"\/career-reviews\/software-engineering\/\">software engineering<\/a><\/li>\n<li>Our career review of <a href=\"\/career-reviews\/working-at-an-AI-lab\/\">working at a leading AI lab<\/a><\/li>\n<\/ul>\n<div class=\"panel panel-default panel-collapse\">\n<div class=\"panel-heading\">\n<h4 class=\"panel-title\"><a class=\"no-visited-styling collapsed\" data-toggle=\"collapse\" data-target=\"#-0\">Podcast episodes<\/a><\/h4>\n<\/p><\/div>\n<div id=\"-0\" class=\"panel-body-collapse collapse\" data-80k-event-label=\"Podcast episodes\">\n<div class=\"panel-body\">\n<p>If you prefer podcasts, there are some relevant episodes of <em>The 80,000 Hours Podcast<\/em> you might find helpful:<\/p>\n<ul>\n<li><a href=\"https:\/\/80000hours.org\/podcast\/episodes\/paul-christiano-ai-alignment-solutions\">Dr Paul Christiano on how OpenAI is developing real solutions to the &#8216;AI alignment problem,&#8217; and his vision of how humanity will progressively hand over decision-making to AI systems<\/a><\/li>\n<li><a href=\"https:\/\/80000hours.org\/podcast\/episodes\/olsson-and-ziegler-ml-engineering-and-safety\">Machine learning engineering for AI safety and robustness: a Google Brain engineer&#8217;s guide to entering the field<\/a><\/li>\n<li><a href=\"https:\/\/80000hours.org\/podcast\/episodes\/nick-joseph-anthropic-safety-approach-responsible-scaling\/\">Nick Joseph on whether Anthropic&#8217;s AI safety policy is up to the task<\/a><\/li>\n<li><a href=\"https:\/\/80000hours.org\/2017\/07\/podcast-the-world-needs-ai-researchers-heres-how-to-become-one\/\">The world needs AI researchers. Here&#8217;s how to become one<\/a><\/li>\n<li><a href=\"https:\/\/80000hours.org\/podcast\/episodes\/chris-olah-unconventional-career-path\/\">Chris Olah on working at top AI labs without an undergrad degree<\/a> and <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/chris-olah-interpretability-research\/\">What the hell is going on inside neural networks<\/a><\/li>\n<li><a href=\"https:\/\/80000hours.org\/podcast\/episodes\/richard-ngo-large-language-models\/\">Richard Ngo on large language models, OpenAI, and striving to make the future go well<\/a><\/li>\n<li>Nathan Labenz on <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/nathan-labenz-openai-red-team-safety\/\">the final push for AGI, understanding OpenAI&#8217;s leadership drama, and red-teaming frontier models<\/a> and <a href=\"https:\/\/80000hours.org\/podcast\/episodes\/nathan-labenz-ai-breakthroughs-controversies\/\">recent AI breakthroughs and navigating the growing rift between AI safety and accelerationist camps<\/a><\/li>\n<\/ul>\n<\/div><\/div><\/div>\n<div class=\"tw--mt-6 tw--p-3 tw--pt-2 tw--bg-gray-lighter tw--rounded-md \">\n<h3 class=\"no-toc\">\t\t<a class=\"no-visited-styling tw--text-off-black hover:tw--text-off-black hover:tw--no-underline focus:tw--text-off-black\" href=\"https:\/\/80000hours.org\/career-reviews\/\">\t\t\t<small>Read next:&nbsp;<\/small>\t\t\tLearn about other high-impact careers\t\t<\/a>\t<\/h3>\n<div class=\"tw--grid xs:tw--grid-flow-col tw--gap-3\">\n<div class=\"xs:tw--order-last tw--pt-1\">\t\t\t<a href=\"https:\/\/80000hours.org\/career-reviews\/\">\t\t\t\t<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/11\/HomepageB-3-720x448.jpg\" alt=\"Decorative post preview\" width=\"720\" height=\"448\">\t\t\t<\/a>\t\t<\/div>\n<div class=\"\">\n<div class=\"tw--pb-3\">\n<p>Want to consider more paths? See our list of the highest-impact career paths according to our research.<\/p>\n<\/div>\n<div class=\"\">\t\t\t\t<a href=\"https:\/\/80000hours.org\/career-reviews\/\" class=\"btn btn-primary\">Continue &rarr;<\/a>\t\t\t<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"well visible-if-not-newsletter-subscriber margin-bottom margin-top padding-top-small padding-bottom-small\">\n<h3 class=\"no-toc\">Plus, join our newsletter and we&#8217;ll mail you a free book<\/h3>\n<p>Join our newsletter and we&#8217;ll send you a free copy of <em>The Precipice<\/em> \u2014 a book by philosopher Toby Ord about how to tackle the greatest threats facing humanity. <a href=\"https:\/\/80000hours.org\/free-book\/#giveaway-terms\">T&#038;Cs here<\/a>.<\/p>\n<form data-80k-object-id=\"\" data-80k-form-action=\"newsletter__subscribe\" action=\"\/\" method=\"post\" class=\"form-newsletter-signup form-newsletter-signup-step-1 margin-bottom-smaller\">\n<div class=\"mc-field-group input-group compact-input-group \"> <input type=\"email\" value=\"\" name=\"email\" required class=\"form-control email\" placeholder=\"Email address\" id=\"input_email\"> <span class=\"submit input-group-btn input-group-btn-right\"> <input type=\"submit\" id=\"mc-embedded-subscribe\" value=\"GET THE BOOK\" class=\"btn btn-primary \" \/> <\/span> <\/div>\n<div> <input name=\"_eightyk_action\" value=\"mailchimp_add_subscriber\" type=\"hidden\"> <input name=\"redirect_path_after_step_2\" value=\"\/newsletter\/welcome\/\" type=\"hidden\"> <\/div>\n<div style=\"position: absolute; left: -5000px;\"> <input type=\"text\" name=\"b_abc12f58bbe8075560abdc5b7_43bc1ae55c\" tabindex=\"-1\" value=\"\"> <\/div>\n<\/form>\n<\/div>\n","protected":false},"author":423,"featured_media":82349,"parent":0,"menu_order":0,"template":"template-standard-article.php","format":"standard","meta":{"_acf_changed":false,"footnotes":"[fn neglectedness] Estimating this number is very difficult. Ideally we want to estimate the number of FTE (\"[full-time equivalent](https:\/\/en.wikipedia.org\/wiki\/Full-time_equivalent)\") working on the problem of reducing existential risks from AI using technical methods. After making a number of assumptions, I estimated that there were 76 to 536 FTE working on technical AI safety (90% confidence). To learn more, read the [section on neglectedness in our problem profile on AI](\/problem-profiles\/artificial-intelligence\/#neglectedness), alongside [footnote 3](https:\/\/80000hours.org\/problem-profiles\/artificial-intelligence\/#fn-3). [\/fn]\r\n\r\n[fn bayareasalaries]Data from [Levels.fyi](https:\/\/www.levels.fyi\/t\/software-engineer\/locations\/san-francisco-bay-area) (visited Jan 27, 2022).[\/fn]\r\n\r\n[fn JacobCOI]Jacob is my brother.[\/fn]\r\n\r\n[fn KarnofskyCOI]Holden Karnofsky is the co-founder of Open Philanthropy, 80000 Hours' largest funder.[\/fn]\r\n"},"categories":[1182,1181,1113,1321,1240,24,362,1241],"class_list":["post-74400","career_profile","type-career_profile","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-ai-safety-research","category-artificial-intelligence","category-computer-science-phd","category-machine-learning","category-promising-interventions","category-in-research","category-software-engineering","category-top-recommended-organisations"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v23.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>AI safety technical research - Career review<\/title>\n<meta name=\"description\" content=\"AI safety research \u2014 research on ways to prevent unwanted behaviour from AI systems \u2014 generally involves working as a scientist or engineer at major AI labs, in academia, or in independent nonprofits.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"AI safety technical research - Career review\" \/>\n<meta property=\"og:description\" content=\"AI safety research \u2014 research on ways to prevent unwanted behaviour from AI systems \u2014 generally involves working as a scientist or engineer at major AI labs, in academia, or in independent nonprofits.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/\" \/>\n<meta property=\"og:site_name\" content=\"80,000 Hours\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/80000Hours\" \/>\n<meta property=\"article:modified_time\" content=\"2024-11-25T18:32:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/ai_origami-scaled.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1810\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:description\" content=\"AI safety research \u2014 research on ways to prevent unwanted behaviour from AI systems \u2014 generally involves working as a scientist or engineer at major AI labs, in academia, or in independent nonprofits.\" \/>\n<meta name=\"twitter:site\" content=\"@80000hours\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"47 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/\",\"url\":\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/\",\"name\":\"AI safety technical research - Career review\",\"isPartOf\":{\"@id\":\"https:\/\/80000hours.org\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/ai_origami-scaled.jpg\",\"datePublished\":\"2023-06-19T10:28:33+00:00\",\"dateModified\":\"2024-11-25T18:32:18+00:00\",\"description\":\"AI safety research \u2014 research on ways to prevent unwanted behaviour from AI systems \u2014 generally involves working as a scientist or engineer at major AI labs, in academia, or in independent nonprofits.\",\"breadcrumb\":{\"@id\":\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#primaryimage\",\"url\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/ai_origami-scaled.jpg\",\"contentUrl\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/ai_origami-scaled.jpg\",\"width\":2560,\"height\":1810},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/80000hours.org\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AI safety technical research\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/80000hours.org\/#website\",\"url\":\"https:\/\/80000hours.org\/\",\"name\":\"80,000 Hours\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/80000hours.org\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/80000hours.org\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/80000hours.org\/#organization\",\"name\":\"80,000 Hours\",\"url\":\"https:\/\/80000hours.org\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/80000hours.org\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png\",\"contentUrl\":\"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png\",\"width\":1500,\"height\":785,\"caption\":\"80,000 Hours\"},\"image\":{\"@id\":\"https:\/\/80000hours.org\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/80000Hours\",\"https:\/\/x.com\/80000hours\",\"https:\/\/www.youtube.com\/user\/eightythousandhours\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"AI safety technical research - Career review","description":"AI safety research \u2014 research on ways to prevent unwanted behaviour from AI systems \u2014 generally involves working as a scientist or engineer at major AI labs, in academia, or in independent nonprofits.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/","og_locale":"en_US","og_type":"article","og_title":"AI safety technical research - Career review","og_description":"AI safety research \u2014 research on ways to prevent unwanted behaviour from AI systems \u2014 generally involves working as a scientist or engineer at major AI labs, in academia, or in independent nonprofits.","og_url":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/","og_site_name":"80,000 Hours","article_publisher":"https:\/\/www.facebook.com\/80000Hours","article_modified_time":"2024-11-25T18:32:18+00:00","og_image":[{"width":2560,"height":1810,"url":"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/ai_origami-scaled.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_description":"AI safety research \u2014 research on ways to prevent unwanted behaviour from AI systems \u2014 generally involves working as a scientist or engineer at major AI labs, in academia, or in independent nonprofits.","twitter_site":"@80000hours","twitter_misc":{"Est. reading time":"47 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/","url":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/","name":"AI safety technical research - Career review","isPartOf":{"@id":"https:\/\/80000hours.org\/#website"},"primaryImageOfPage":{"@id":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#primaryimage"},"image":{"@id":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#primaryimage"},"thumbnailUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/ai_origami-scaled.jpg","datePublished":"2023-06-19T10:28:33+00:00","dateModified":"2024-11-25T18:32:18+00:00","description":"AI safety research \u2014 research on ways to prevent unwanted behaviour from AI systems \u2014 generally involves working as a scientist or engineer at major AI labs, in academia, or in independent nonprofits.","breadcrumb":{"@id":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#primaryimage","url":"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/ai_origami-scaled.jpg","contentUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2023\/06\/ai_origami-scaled.jpg","width":2560,"height":1810},{"@type":"BreadcrumbList","@id":"https:\/\/80000hours.org\/career-reviews\/ai-safety-researcher\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/80000hours.org\/"},{"@type":"ListItem","position":2,"name":"AI safety technical research"}]},{"@type":"WebSite","@id":"https:\/\/80000hours.org\/#website","url":"https:\/\/80000hours.org\/","name":"80,000 Hours","description":"","publisher":{"@id":"https:\/\/80000hours.org\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/80000hours.org\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/80000hours.org\/#organization","name":"80,000 Hours","url":"https:\/\/80000hours.org\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/80000hours.org\/#\/schema\/logo\/image\/","url":"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png","contentUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png","width":1500,"height":785,"caption":"80,000 Hours"},"image":{"@id":"https:\/\/80000hours.org\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/80000Hours","https:\/\/x.com\/80000hours","https:\/\/www.youtube.com\/user\/eightythousandhours"]}]}},"_links":{"self":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/career_profile\/74400"}],"collection":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/career_profile"}],"about":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/types\/career_profile"}],"author":[{"embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/users\/423"}],"version-history":[{"count":4,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/career_profile\/74400\/revisions"}],"predecessor-version":[{"id":88180,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/career_profile\/74400\/revisions\/88180"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/media\/82349"}],"wp:attachment":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/media?parent=74400"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/categories?post=74400"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}