{"id":44313,"date":"2016-08-14T09:03:20","date_gmt":"2016-08-14T09:03:20","guid":{"rendered":"https:\/\/80000hours.org\/?post_type=article&#038;p=44313"},"modified":"2025-10-13T15:30:20","modified_gmt":"2025-10-13T15:30:20","slug":"ai-safety-syllabus","status":"publish","type":"article","link":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/","title":{"rendered":"AI safety&nbsp;syllabus"},"content":{"rendered":"<div class=\"well bg-gray-lighter margin-bottom margin-top padding-top-small padding-bottom-small\">\n<p><strong>Update April 2024:<\/strong> This syllabus was written in August 2016. The field of AI safety has progressed substantially since then. If you&#8217;re looking for up-to-date resources, we recommend:<\/p>\n<ul>\n<li>The curriculum from the <a href=\"https:\/\/course.aisafetyfundamentals.com\/alignment\">alignment course<\/a> or the <a href=\"https:\/\/course.aisafetyfundamentals.com\/governance\">governance-focused course<\/a>, both from <a href=\"https:\/\/aisafetyfundamentals.com\/\">AI Safety Fundamentals<\/a><\/li>\n<li>Our <a href=\"https:\/\/80000hours.org\/problem-profiles\/artificial-intelligence\/#top-resources-to-learn-more\">list of resources to learn more<\/a> in our full problem profile on <a href=\"https:\/\/80000hours.org\/problem-profiles\/artificial-intelligence\/\">preventing AI-related catastrophe<\/a>  <\/li>\n<li>These <a href=\"https:\/\/www.aisafetysupport.org\/lots-of-links\">resources<\/a> collected by the team at AI Safety Support<\/li>\n<\/ul>\n<\/div>\n<p><em>This page was written by Jan Leike, with contributions and comments by David Krueger, Jelena Luketina, Victoria Krakovna, Daniel Dewey, Laurent Orseau, and others. It is intended as a guide to working on <a href=\"\/career-reviews\/artificial-intelligence-risk-research\/\">technical aspects of AI safety<\/a>. See our <a href=\"\/articles\/ai-policy-guide\/\">guide to working in AI policy and strategy<\/a> for another approach.<\/em><\/p>\n<p>This is a syllabus of relevant background reading material and courses related to <a href=\"\/problem-profiles\/positively-shaping-artificial-intelligence\">AI safety<\/a>. It is intended as a guide for undergraduates in mathematics and computer science planning their degree, as well as people from other disciplines who are thinking about moving into AI safety. It includes tips how to design your degree, how to transition into research, and the relevant conferences. This is not intended as a general guide of how to become a researcher.<\/p>\n<div id=\"toc_container\" class=\"toc_white no_bullets\"><p class=\"toc_title\">Table of Contents<\/p><ul class=\"toc_list\"><li><a href=\"#reading-list\"><span class=\"toc_number toc_depth_1\">1<\/span> Reading List<\/a><\/li><li><a href=\"#degree\"><span class=\"toc_number toc_depth_1\">2<\/span> Degree<\/a><ul><li><a href=\"#undergraduate-degree\"><span class=\"toc_number toc_depth_2\">2.1<\/span> Undergraduate Degree<\/a><\/li><li><a href=\"#phd\"><span class=\"toc_number toc_depth_2\">2.2<\/span> PhD<\/a><\/li><li><a href=\"#google-ai-residency-program\"><span class=\"toc_number toc_depth_2\">2.3<\/span> Google AI residency program<\/a><\/li><li><a href=\"#research-groups\"><span class=\"toc_number toc_depth_2\">2.4<\/span> Research Groups<\/a><\/li><\/ul><\/li><li><a href=\"#conferences\"><span class=\"toc_number toc_depth_1\">3<\/span> Conferences<\/a><\/li><li><a href=\"#want-to-work-on-ai-safety-we-want-to-help\"><span class=\"toc_number toc_depth_1\">4<\/span> Want to work on AI safety? We want to help.<\/a><\/li><\/ul><\/div>\n<div class=\"well bg-gray-lighter margin-bottom margin-top padding-top-small padding-bottom-small\">\n<h2 class=\"no_toc\">Want to work on AI safety? We want to help.<\/h2>\n<p>We&#8217;ve helped dozens of people formulate their plans, and put them in touch with academic mentors. If you want to work on AI safety:<\/p>\n<p><a href=\"https:\/\/80000hours.org\/speak-with-us\/?int_campaign=article\" title=\"\" class=\"btn btn-primary\">Get in touch<\/a><\/p>\n<\/div>\n<h2><span id=\"reading-list\" class=\"toc-anchor\"><\/span>Reading List<\/h2>\n<p><em>We now recommend using the <a href=\"http:\/\/humancompatible.ai\/bibliography\">bibliography<\/a> from the Center for Human-Compatible AI at UC Berkeley. Their list is more comprehensive and up-to-date than the one below.<\/em><\/p>\n<p>This is a list of the most relevant reading topics and the appropriate material. The chapter recommendations are indicative of what you should know. If you find the topic interesting, read more! As an undergraduate student, you can plan these courses into your degree. As a graduate student, you can use the provided material to extend your knowledge into areas that you do not have much background in. Focus on the textbooks and lecture notes and use the video lectures as supplementary materials. Doing plenty of exercises is usually a good idea to make sure that you actually understand a topic instead of just thinking you understand it.<\/p>\n<p>Some of the relevant areas might not be offered as courses by your university. You can always read the listed books in your free time, or try to find a MOOC on <a href=\"https:\/\/www.coursera.org\/\">Coursera<\/a>.<\/p>\n<ul>\n<li><strong>Machine Learning<\/strong> is the modern probabilistic approach to artificial intelligence. It studies algorithms that learn to predict from (usually independent and identically distributed) data.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"http:\/\/www.springer.com\/us\/book\/9780387310732\">Bishop: Pattern Recognition and Machine Learning<\/a><\/li>\n<li><a href=\"https:\/\/mitpress.mit.edu\/9780262018029\/machine-learning\/\">Murphy: Machine Learning<\/a> (alternative to Bishop)<\/li>\n<li><a href=\"http:\/\/www.stat.columbia.edu\/~gelman\/book\/\">Gelman and Rubin: Bayesian Data Analysis<\/a> Chapters 2,4,5<\/li>\n<li><a href=\"http:\/\/www.inference.phy.cam.ac.uk\/itila\/\">McKay: Information Theory, Inference, and Learning Algorithms<\/a> Parts I-III<\/li>\n<\/ul>\n<\/li>\n<li>Video lectures:\n<ul>\n<li><a href=\"https:\/\/www.coursera.org\/learn\/machine-learning\">Andrew Ng: Machine Learning<\/a><\/li>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=w2OtwL5T1ow&amp;list=PLE6Wd9FR--EdyJ5lbFl8UuGjecvVw66F6\">Nando de Freitas: Machine Learning<\/a><\/li>\n<\/ul>\n<\/li>\n<li>Lecture Notes:\n<ul>\n<li><a href=\"http:\/\/ocw.mit.edu\/courses\/electrical-engineering-and-computer-science\/6-867-machine-learning-fall-2006\/lecture-notes\/\">MIT OpenCourseware: Machine Learning<\/a><\/li>\n<li><a href=\"http:\/\/ocw.mit.edu\/courses\/mathematics\/18-657-mathematics-of-machine-learning-fall-2015\/lecture-notes\/MIT18_657F15_LecNote.pdf\">Philippe Rigollet: Mathematics of Machine Learning<\/a><\/li>\n<li><a href=\"https:\/\/cs229.stanford.edu\/\">Stanford: Machine Learning<\/a> <\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><strong>Statistics<\/strong> is the related mathematical discipline. Having a solid understanding of the underlying mathematics is very useful when doing theoretical work.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"http:\/\/link.springer.com\/book\/10.1007%2F978-0-387-21736-9\">Wasserman: All of Statistics<\/a>, Chapters 1-12 or more (an easy-to-read overview of the field). <em>Editor&#8217;s note: Others have noted that Chapters 1-4, 9 and 13 are the most relevant.<\/em><\/li>\n<li><a href=\"https:\/\/services.math.duke.edu\/~rtd\/PTE\/pte.html\">Durrett: Probability: Theory and Examples<\/a>, Chapters 1-6 (for the more mathematically inclined reader)<\/li>\n<\/ul>\n<\/li>\n<li>Video Lectures:\n<ul>\n<li><a href=\"https:\/\/www.edx.org\/course\/introduction-probability-science-mitx-6-041x-1\">Introduction to Probability<\/a> (MITx)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><strong>Reinforcement Learning<\/strong> is a subdiscipline of machine learning that studies algorithms that learn to act in an unknown environment through trial and error: the algorithm only receives a reward signal in form of a numeric feedback how well it is currently doing and tries to maximize this signal. Reinforcement learning is currently the most promising approach for general intelligence.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"https:\/\/mitpress.mit.edu\/9780262039246\/reinforcement-learning\/\">Sutton and Barto: Reinforcement Learning<\/a>, Chapters 1-6 &amp; 8<\/li>\n<li><a href=\"http:\/\/www.hutter1.net\/ai\/uaibook.htm\">Hutter: Universal Artificial Intelligence<\/a>, Chapters 2-5 (high-level theoretical perspective)<\/li>\n<\/ul>\n<\/li>\n<li>Video lectures:\n<ul>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=2pWv7GOvuf0&amp;list=PL5X3mDkKaJrL42i_jhE4N-p6E2Ol62Ofa\">Silver: Reinforcement Learning<\/a><\/li>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=aUrX-rP_ss4&amp;list=PLjKEIQlKCTZYN3CYBlj8r58SbNorobqcp\">Schulman: Deep Reinforcement Learning<\/a> <\/li>\n<\/ul>\n<\/li>\n<li>Exercises:\n<ul>\n<li><a href=\"https:\/\/spinningup.openai.com\/en\/latest\/\">OpenAI: Deep Reinforcement Learning<\/a><\/li>\n<\/ul>\n<\/li>\n<li>Relevant papers:\n<ul>\n<li><a href=\"http:\/\/www.nature.com\/nature\/journal\/v518\/n7540\/full\/nature14236.html\">Mnih et al.: Human-level control through deep reinforcement learning<\/a><\/li>\n<li><a href=\"http:\/\/www.nature.com\/nature\/journal\/v529\/n7587\/full\/nature16961.html\">Silver et al: Mastering the game of Go with deep neural networks and tree search<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><strong>Deep Learning<\/strong> is a recently successful approach to machine learning based on neural networks with many layers. Convolutional and recurrent neural networks have enabled recent breakthroughs in computer vision, speech recognition, and other domains. At the moment the field is moving extremely fast and textbooks are likely to become obsolete quickly.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"http:\/\/www.deeplearningbook.org\/\">Goodfellow, Bengio, and Courville: Deep Learning<\/a>, Chapters 1-12<\/li>\n<li><a href=\"http:\/\/neuralnetworksanddeeplearning.com\/\">Nielsen: Neural Networks and Deep Learning<\/a>, Chapters 1-6 (more entry-level)<\/li>\n<\/ul>\n<\/li>\n<li>Video lectures:\n<ul>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=dV80NAlEins&amp;list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu\">Nando de Freitas: Deep Learning<\/a><\/li>\n<li><a href=\"https:\/\/www.youtube.com\/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH\">Hugo Larochelle: Neural Networks<\/a><\/li>\n<\/ul>\n<\/li>\n<li>Survey papers:\n<ul>\n<li><a href=\"http:\/\/www.nature.com\/nature\/journal\/v521\/n7553\/abs\/nature14539.html\">LeCun, Bengio, Hinton: Deep Learning<\/a><\/li>\n<li><a href=\"https:\/\/ieeexplore.ieee.org\/document\/6472238\">Bengio, Courville, Vincent: Representation Learning: A Review and New Perspectives<\/a><\/li>\n<li><a href=\"http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0893608014002135\">Schmidhuber: Deep learning in neural networks: An overview<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><strong>Artificial Intelligence<\/strong> encompasses the classical approaches: logic, planning, knowledge representation and reasoning. Probabilistic approaches to AI are covered in the other topics above.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"http:\/\/aima.cs.berkeley.edu\/\">Russell and Norvig: Artificial Intelligence: A Modern Approach<\/a>, Chapters 1-17 (very comprehensive, has been the standard textbook for many years)<\/li>\n<\/ul>\n<\/li>\n<li>Video lectures:\n<ul>\n<li><a href=\"https:\/\/www.udacity.com\/course\/intro-to-artificial-intelligence--cs271\">Norvig and Thrun: Introduction to Artificial Intelligence<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><strong>Game Theory<\/strong> provides the foundations to environments with multiple agents.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"https:\/\/global.oup.com\/ushe\/product\/an-introduction-to-game-theory-9780195128956?cc=gb&amp;lang=en&amp;\">Osborne: An Introduction to Game Theory<\/a>, Chapters 1-7,14,15<\/li>\n<\/ul>\n<\/li>\n<li>Video lectures:\n<ul>\n<li>Jackson, Leyton-Brown, Shoham: <a href=\"https:\/\/www.coursera.org\/learn\/game-theory-1\">Game theory I<\/a> and <a href=\"https:\/\/www.coursera.org\/learn\/game-theory-2\">II<\/a><\/li>\n<\/ul>\n<\/li>\n<li>Survey papers:\n<ul>\n<li><a href=\"https:\/\/ieeexplore.ieee.org\/document\/4445757\">Bu\u015foniu, Babu\u0161ka, De Schutter: A Comprehensive Survey of Multiagent Reinforcement Learning<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><strong>Philosophy<\/strong> originated some of the mental tools that are useful when thinking about superintelligent agents and originated the discussion around AI safety.\n<ul>\n<li>Books:\n<ul>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence:_Paths,_Dangers,_Strategies\">Bostrom: Superintelligence: Paths, Dangers, Strategies<\/a>, Chapters 1-10,12<\/li>\n<\/ul>\n<\/li>\n<li>Relevant papers:\n<ul>\n<li><a href=\"https:\/\/selfawaresystems.files.wordpress.com\/2008\/01\/ai_drives_final.pdf\">Omohundro: The Basic AI Drives<\/a><\/li>\n<li><a href=\"https:\/\/intelligence.org\/files\/AIPosNegFactor.pdf\">Yudkowsky: Artificial Intelligence as a Positive and Negative Factor in Global Risk<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>More remotely related are the following areas.<\/p>\n<ul>\n<li><strong>Automata and Complexity Theory<\/strong> sits at the core of theoretical computer science.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"http:\/\/infolab.stanford.edu\/~ullman\/ialc.html\">Ullman and Hopcroft: Introduction to Automata Theory, Languages, and Computation<\/a>, Chapters 1-10<\/li>\n<li><a href=\"http:\/\/www-math.mit.edu\/~sipser\/book.html\">Sipser: Introduction to the Theory of Computation<\/a>, Chapters 1-5,7 (alternatively to Ullman and Hopcroft)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><strong>Formal Logic<\/strong> is a branch of mathematics that studies formal proofs and the limits of mathematics.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"http:\/\/www.cambridge.org\/us\/academic\/subjects\/philosophy\/logic\/computability-and-logic-5th-edition\">Boolos and Burgess: Computability and Logic<\/a>, Chapters 1-4,8-20,23,25,27<\/li>\n<li><a href=\"http:\/\/store.elsevier.com\/A-Mathematical-Introduction-to-Logic\/Herbert-Enderton\/isbn-9780122384523\/\">Enderton: A Mathematical Introduction to Logic<\/a>, Chapters 0-3 (alternatively to Boolos and Burgess)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><strong>Formal Methods<\/strong> are mathematical techniques for the specification and verification of hardware and software systems.\n<ul>\n<li>Textbooks:\n<ul>\n<li><a href=\"https:\/\/mitpress.mit.edu\/9780262548922\/principles-of-cyber-physical-systems\/\">Alur: Principles of Cyber-Physical Systems<\/a>, Chapters 1-7,9<\/li>\n<li><a href=\"https:\/\/books.google.com.gi\/books?id=7GsKBwAAQBAJ&#038;printsec=copyright#v=onepage&#038;q&#038;f=false\">Monin: Understanding Formal Methods<\/a>, Chapters 1-10<\/li>\n<li><a href=\"https:\/\/mitpress.mit.edu\/9780262026499\/principles-of-model-checking\/\">Baier, Katoen, Larsen: Principles of Model Checking<\/a>, Chapter 1-7,10<\/li>\n<\/ul>\n<\/li>\n<li>Surveys:\n<ul>\n<li><a href=\"http:\/\/www.springer.com\/us\/book\/9783319105741\">Clarke, Henzinger, Veith: Handbook of Model Checking<\/a> (to appear soon)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2><span id=\"degree\" class=\"toc-anchor\"><\/span>Degree<\/h2>\n<h3><span id=\"undergraduate-degree\" class=\"toc-anchor\"><\/span>Undergraduate Degree<\/h3>\n<p>Ideally your undergraduate degree would be mathematics and computer science (for example, a bachelor&#8217;s degree in math and a master in computer science). But this does not mean that an undergraduate degree in a different related discipline like neuroscience or physics would be wasted. Make sure you have a solid handle on the relevant mathematics (linear algebra, calculus, statistics, \u2026)!<\/p>\n<p>For your undergraduate thesis, find someone who supervises well and who has time for you (not the most famous\/cool professor). Work on a topic that your supervisor finds interesting (to get lots of feedback). Pursuing your own ideas at this point is risky and usually means that you don&#8217;t get much supervision. Do something theoretical, preferably in computer science. Find an interesting research group and start doing research early in your degree (it helps a lot if you have clever things to say about their research). Ideally, you should get out of a Master&#8217;s degree with at least one publication at an international conference. It&#8217;s not a big deal if this delays your degree.<\/p>\n<p>Other tips:<\/p>\n<ul>\n<li>If you find a topic interesting, take more classes even if they don&#8217;t seem related<\/li>\n<li>Choose harder classes over easier ones (favor math courses and theoretical computer science courses over applied computer science courses)<\/li>\n<li>Choose your thesis by supervisor, and not necessarily by topic<\/li>\n<li>Publications are great, they are a considered a good predictor of your academic potential (even if you are not the first author). As such, they are very helpful when applying for PhD programs<\/li>\n<li>Read general advice on whether a PhD is for you and how to approach it<\/li>\n<li>Attend <a href=\"https:\/\/intelligence.org\/mirix\/\">MIRIx workshops<\/a> if they exist in your area<\/li>\n<\/ul>\n<h3><span id=\"phd\" class=\"toc-anchor\"><\/span>PhD<\/h3>\n<p>Getting a PhD is generally an excellent idea and usually a prerequisite for someone to hire you as a researcher. A PhD will not only put you at the cutting edge of research, but also teach you the relevant soft skills (how to write papers, communicate complex ideas, etc.).<\/p>\n<p>Your PhD should be in machine learning, reinforcement learning, statistics, or another discipline related to artificial intelligence. Focus on getting the required expertise first. If you feel comfortable in your area, shift your focus on to AI safety (e.g., in your final 1-2 years). Read <a href=\"https:\/\/80000hours.org\/career-reviews\/machine-learning-phd\/\">our profile on machine learning PhDs<\/a> for more information.<\/p>\n<p>For relevant problems, see:<\/p>\n<ul>\n<li><a href=\"https:\/\/arxiv.org\/abs\/1606.06565\">Amodei et al: Concrete Problems in AI Safety<\/a><\/li>\n<li><a href=\"https:\/\/intelligence.org\/files\/TechnicalAgenda.pdf\">Soares and Fallenstein: Aligning Superintelligence with Human Interests<\/a><\/li>\n<li><a href=\"http:\/\/arxiv.org\/abs\/1602.03506\">Russell et al: Research Priorities for Robust and Beneficial Artificial Intelligence<\/a><\/li>\n<li><a href=\"https:\/\/intelligence.org\/files\/AlignmentMachineLearning.pdf\">Taylor et al: Alignment for Advanced Machine Learning Systems<\/a><\/li>\n<\/ul>\n<h3><span id=\"google-ai-residency-program\" class=\"toc-anchor\"><\/span>Google AI residency program<\/h3>\n<p>The Google AI residency program is a year-long role, similar to spending a year in a master&#8217;s or PhD program in deep learning.<\/p>\n<p>It&#8217;s designed to quickly get you up to speed with deep learning research and is open to people with degrees in a STEM field (bachelor&#8217;s, master&#8217;s, or PhD). It&#8217;s more prestigious than a master&#8217;s degree and gives you access to Google&#8217;s computational resources and experts in deep learning. That said, it&#8217;s extremely competitive \u2013 you&#8217;re more likely to get accepted into a top graduate school programme.<\/p>\n<p>It&#8217;s worth applying to it after both undergraduate and master&#8217;s. If you&#8217;re choosing between the residency and a master&#8217;s, the residency will usually be better because of the advantages mentioned above, as well as the fact that you&#8217;ll be spending all your time on research.<\/p>\n<p>When choosing between the residency and a PhD you&#8217;ll need to consider how good your PhD offers are \u2013 if you&#8217;ve got offers from top places then it may not be worth postponing, especially if you can&#8217;t defer your PhD.<\/p>\n<h3><span id=\"research-groups\" class=\"toc-anchor\"><\/span>Research Groups<\/h3>\n<p>The following is an non-exhaustive list of research groups where you could apply for internships and PhD candidacy. Make sure you look at their research and see how it relates to your interests. Needless to say, it is not a good idea to mass-email everyone on this list.<\/p>\n<ul>\n<li>Berkeley (<a href=\"https:\/\/people.eecs.berkeley.edu\/~russell\/\">Stuart Russell<\/a>)<\/li>\n<li>Cambridge (<a href=\"http:\/\/mlg.eng.cam.ac.uk\/zoubin\/\">Zoubin Ghahramani<\/a>)<\/li>\n<li>University of Montreal (<a href=\"https:\/\/mila.quebec\/en\">MILA<\/a>)<\/li>\n<li>University of Alberta (<a href=\"http:\/\/incompleteideas.net\/rlai.cs.ualberta.ca\/RLAI\/ualberta.html\">RLAI group<\/a>)<\/li>\n<li>Imperial College London (<a href=\"https:\/\/www.doc.ic.ac.uk\/~mpsha\/\">Murray Shanahan<\/a>)<\/li>\n<li>University of Oxford (<a href=\"http:\/\/www.robots.ox.ac.uk\/~mosb\/\">Michael Osborne<\/a>)<\/li>\n<li>McGill (<a href=\"http:\/\/rl.cs.mcgill.ca\/\">RLLAB<\/a>)<\/li>\n<li>Australian National University (<a href=\"http:\/\/hutter1.net\/\">Marcus Hutter<\/a>)<\/li>\n<li>University of Amsterdam (<a href=\"https:\/\/amlab.science.uva.nl\/people\/MaxWelling\/\">Max Welling<\/a>)<\/li>\n<li>Stanford (<a href=\"https:\/\/cs.stanford.edu\/~pliang\/\">Percy Liang<\/a>, <a href=\"http:\/\/www.cs.cmu.edu\/~ebrun\/\">Emma Brunskill<\/a>)<\/li>\n<li>CMU (<a href=\"http:\/\/zicokolter.com\/\">Zico Kolter<\/a>)<\/li>\n<li>University of Toronto<\/li>\n<li>IDSIA (<a href=\"http:\/\/people.idsia.ch\/~juergen\/\">J\u00fcrgen Schmidhuber<\/a>)<\/li>\n<li>Princeton (<a href=\"http:\/\/hips.seas.harvard.edu\/\">HIPS<\/a>)<\/li>\n<li>MIT (<a href=\"https:\/\/www.csail.mit.edu\/user\/815\">Joshua Tenenbaum<\/a>)<\/li>\n<li><a href=\"https:\/\/deepmind.com\/\">Google DeepMind<\/a><\/li>\n<li><a href=\"https:\/\/research.google.com\/teams\/brain\/\">Google Brain<\/a><\/li>\n<li><a href=\"https:\/\/research.facebook.com\/ai\/\">FAIR<\/a><\/li>\n<li><a href=\"https:\/\/openai.com\/\">OpenAI<\/a><\/li>\n<\/ul>\n<h2><span id=\"conferences\" class=\"toc-anchor\"><\/span>Conferences<\/h2>\n<p>For your publications, always aim for the best conferences, even if you think your work will be rejected. Even if it is rejected, you will likely get more valuable feedback than in other places.<\/p>\n<p>Attend major conferences even if you don&#8217;t have a paper there. You will get a sense of what researchers are interested in, and you can connect to potential supervisors and collaborators related to your interests. Read some of their papers beforehand so that you have a good conversation starter.<\/p>\n<p>Major: <a href=\"http:\/\/icml.cc\/2016\/\">ICML<\/a>, <a href=\"http:\/\/nips.cc\/\">NIPS<\/a>, COLT, <a href=\"http:\/\/www.aaai.org\/home.html\">AAAI<\/a>, UAI, IJCAI, AAMAS, <a href=\"https:\/\/www.iclr.cc\/\">ICLR<\/a><br \/>\nMinor: <a href=\"http:\/\/www.aistats.org\/\">AISTATS<\/a>, ECAI, ECML, ALT<br \/>\nApplications: ICCV, CVPR (Computer vision), ICASSP (Speech), ICRA (Robotics), EMNLP, ACL (NLP)<\/p>\n<div class=\"well bg-gray-lighter margin-bottom margin-top padding-top-small padding-bottom-small\">\n<h2><span id=\"want-to-work-on-ai-safety-we-want-to-help\" class=\"toc-anchor\"><\/span>Want to work on AI safety? We want to help.<\/h2>\n<p>We&#8217;ve helped dozens of people formulate their plans, and put them in touch with academic mentors. If you want to work on AI safety:<\/p>\n<p><a href=\"https:\/\/80000hours.org\/speak-with-us\/?int_campaign=article\" title=\"\" class=\"btn btn-primary\">Get in touch<\/a><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":391,"featured_media":44705,"parent":0,"menu_order":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"categories":[1353,1387,1418,315,1369,1359,1385,291],"class_list":["post-44313","article","type-article","status-publish","has-post-thumbnail","hentry","category-ai","category-ai-policy","category-ai-safety-technical-research","category-career-advice-strategy","category-machine-learning-skill-building-and-career-capital","category-phds-career-advice-strategy","category-research","category-world-problems"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>AI safety syllabus | 80,000 Hours<\/title>\n<meta name=\"description\" content=\"This is a syllabus of relevant background reading material and courses related to AI safety. It is intended as a guide for undergraduates in mathematics and computer science.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"AI safety syllabus | 80,000 Hours\" \/>\n<meta property=\"og:description\" content=\"This is a syllabus of relevant background reading material and courses related to AI safety. It is intended as a guide for undergraduates in mathematics and computer science planning their degree, as well as people from other disciplines who are thinking about moving into AI safety. It includes tips how to design your degree, how to transition into research, and the relevant conferences. This is not intended as a general guide of how to become a researcher.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/\" \/>\n<meta property=\"og:site_name\" content=\"80,000 Hours\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/80000Hours\" \/>\n<meta property=\"article:modified_time\" content=\"2025-10-13T15:30:20+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/80000hours.org\/wp-content\/uploads\/2019\/05\/browse-archive-library.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"660\" \/>\n\t<meta property=\"og:image:height\" content=\"412\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@80000hours\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"10 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/\"},\"author\":{\"name\":\"The 80,000 Hours team\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/#\\\/schema\\\/person\\\/d2e16879764e2877972efe48038b297b\"},\"headline\":\"AI safety&nbsp;syllabus\",\"datePublished\":\"2016-08-14T09:03:20+00:00\",\"dateModified\":\"2025-10-13T15:30:20+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/\"},\"wordCount\":1978,\"publisher\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/80000hours.org\\\/wp-content\\\/uploads\\\/2019\\\/05\\\/browse-archive-library.jpg\",\"articleSection\":[\"AI\",\"AI policy\",\"AI safety technical research\",\"Career advice &amp; strategy\",\"Machine learning\",\"PhDs\",\"Research\",\"World problems\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/\",\"url\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/\",\"name\":\"AI safety syllabus | 80,000 Hours\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/80000hours.org\\\/wp-content\\\/uploads\\\/2019\\\/05\\\/browse-archive-library.jpg\",\"datePublished\":\"2016-08-14T09:03:20+00:00\",\"dateModified\":\"2025-10-13T15:30:20+00:00\",\"description\":\"This is a syllabus of relevant background reading material and courses related to AI safety. It is intended as a guide for undergraduates in mathematics and computer science.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/#primaryimage\",\"url\":\"https:\\\/\\\/80000hours.org\\\/wp-content\\\/uploads\\\/2019\\\/05\\\/browse-archive-library.jpg\",\"contentUrl\":\"https:\\\/\\\/80000hours.org\\\/wp-content\\\/uploads\\\/2019\\\/05\\\/browse-archive-library.jpg\",\"width\":660,\"height\":412},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/articles\\\/ai-safety-syllabus\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/80000hours.org\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Wp json\",\"item\":\"https:\\\/\\\/80000hours.org\\\/wp-json\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"AI safety&nbsp;syllabus\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/#website\",\"url\":\"https:\\\/\\\/80000hours.org\\\/\",\"name\":\"80,000 Hours\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/80000hours.org\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/#organization\",\"name\":\"80,000 Hours\",\"url\":\"https:\\\/\\\/80000hours.org\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/80000hours.org\\\/wp-content\\\/uploads\\\/2018\\\/07\\\/og-logo_0.png\",\"contentUrl\":\"https:\\\/\\\/80000hours.org\\\/wp-content\\\/uploads\\\/2018\\\/07\\\/og-logo_0.png\",\"width\":1500,\"height\":785,\"caption\":\"80,000 Hours\"},\"image\":{\"@id\":\"https:\\\/\\\/80000hours.org\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/80000Hours\",\"https:\\\/\\\/x.com\\\/80000hours\",\"https:\\\/\\\/www.youtube.com\\\/user\\\/eightythousandhours\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/80000hours.org\\\/#\\\/schema\\\/person\\\/d2e16879764e2877972efe48038b297b\",\"name\":\"The 80,000 Hours team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/20d69d91f87c06dc73dc845b0798b11da6d428034c22efa4fb5a43c29c3607a1?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/20d69d91f87c06dc73dc845b0798b11da6d428034c22efa4fb5a43c29c3607a1?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/20d69d91f87c06dc73dc845b0798b11da6d428034c22efa4fb5a43c29c3607a1?s=96&d=mm&r=g\",\"caption\":\"The 80,000 Hours team\"},\"sameAs\":[\"http:\\\/\\\/80000hours.org\"],\"url\":\"https:\\\/\\\/80000hours.org\\\/author\\\/80000hours\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"AI safety syllabus | 80,000 Hours","description":"This is a syllabus of relevant background reading material and courses related to AI safety. It is intended as a guide for undergraduates in mathematics and computer science.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/","og_locale":"en_US","og_type":"article","og_title":"AI safety syllabus | 80,000 Hours","og_description":"This is a syllabus of relevant background reading material and courses related to AI safety. It is intended as a guide for undergraduates in mathematics and computer science planning their degree, as well as people from other disciplines who are thinking about moving into AI safety. It includes tips how to design your degree, how to transition into research, and the relevant conferences. This is not intended as a general guide of how to become a researcher.","og_url":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/","og_site_name":"80,000 Hours","article_publisher":"https:\/\/www.facebook.com\/80000Hours","article_modified_time":"2025-10-13T15:30:20+00:00","og_image":[{"width":660,"height":412,"url":"https:\/\/80000hours.org\/wp-content\/uploads\/2019\/05\/browse-archive-library.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_site":"@80000hours","twitter_misc":{"Est. reading time":"10 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/#article","isPartOf":{"@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/"},"author":{"name":"The 80,000 Hours team","@id":"https:\/\/80000hours.org\/#\/schema\/person\/d2e16879764e2877972efe48038b297b"},"headline":"AI safety&nbsp;syllabus","datePublished":"2016-08-14T09:03:20+00:00","dateModified":"2025-10-13T15:30:20+00:00","mainEntityOfPage":{"@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/"},"wordCount":1978,"publisher":{"@id":"https:\/\/80000hours.org\/#organization"},"image":{"@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/#primaryimage"},"thumbnailUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2019\/05\/browse-archive-library.jpg","articleSection":["AI","AI policy","AI safety technical research","Career advice &amp; strategy","Machine learning","PhDs","Research","World problems"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/","url":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/","name":"AI safety syllabus | 80,000 Hours","isPartOf":{"@id":"https:\/\/80000hours.org\/#website"},"primaryImageOfPage":{"@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/#primaryimage"},"image":{"@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/#primaryimage"},"thumbnailUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2019\/05\/browse-archive-library.jpg","datePublished":"2016-08-14T09:03:20+00:00","dateModified":"2025-10-13T15:30:20+00:00","description":"This is a syllabus of relevant background reading material and courses related to AI safety. It is intended as a guide for undergraduates in mathematics and computer science.","breadcrumb":{"@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/#primaryimage","url":"https:\/\/80000hours.org\/wp-content\/uploads\/2019\/05\/browse-archive-library.jpg","contentUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2019\/05\/browse-archive-library.jpg","width":660,"height":412},{"@type":"BreadcrumbList","@id":"https:\/\/80000hours.org\/articles\/ai-safety-syllabus\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/80000hours.org\/"},{"@type":"ListItem","position":2,"name":"Wp json","item":"https:\/\/80000hours.org\/wp-json\/"},{"@type":"ListItem","position":3,"name":"AI safety&nbsp;syllabus"}]},{"@type":"WebSite","@id":"https:\/\/80000hours.org\/#website","url":"https:\/\/80000hours.org\/","name":"80,000 Hours","description":"","publisher":{"@id":"https:\/\/80000hours.org\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/80000hours.org\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/80000hours.org\/#organization","name":"80,000 Hours","url":"https:\/\/80000hours.org\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/80000hours.org\/#\/schema\/logo\/image\/","url":"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png","contentUrl":"https:\/\/80000hours.org\/wp-content\/uploads\/2018\/07\/og-logo_0.png","width":1500,"height":785,"caption":"80,000 Hours"},"image":{"@id":"https:\/\/80000hours.org\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/80000Hours","https:\/\/x.com\/80000hours","https:\/\/www.youtube.com\/user\/eightythousandhours"]},{"@type":"Person","@id":"https:\/\/80000hours.org\/#\/schema\/person\/d2e16879764e2877972efe48038b297b","name":"The 80,000 Hours team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/20d69d91f87c06dc73dc845b0798b11da6d428034c22efa4fb5a43c29c3607a1?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/20d69d91f87c06dc73dc845b0798b11da6d428034c22efa4fb5a43c29c3607a1?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/20d69d91f87c06dc73dc845b0798b11da6d428034c22efa4fb5a43c29c3607a1?s=96&d=mm&r=g","caption":"The 80,000 Hours team"},"sameAs":["http:\/\/80000hours.org"],"url":"https:\/\/80000hours.org\/author\/80000hours\/"}]}},"_links":{"self":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/article\/44313","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/article"}],"about":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/types\/article"}],"author":[{"embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/users\/391"}],"version-history":[{"count":0,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/article\/44313\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/media\/44705"}],"wp:attachment":[{"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/media?parent=44313"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/80000hours.org\/wp-json\/wp\/v2\/categories?post=44313"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}