> ML_LITERATURE // BAI-2022-CONSTITUTIONAL-AI-HARMLESSNESS-FROM-AI-FEEDBACK_v1.0
Constitutional AI: Harmlessness from AI Feedback (RLAIF)
Yuntao Bai, Saurav Kadavath, Sandhini Agarwal, Kavita Saund, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Durr, Jack Clark · arXiv preprint (2022)
Principal Contribution
Trained harmless conversational models without human feedback on harms using a written constitution and Reinforcement Learning from AI Feedback (RLAIF).
Operational Relevance
Directly guides deployment choices and architecture selection for task-llm-alignment-rlhf, task-ai-safety.
Assumptions
- Standard empirical regularity and statistical stability hold across evaluation domains
Limitations
- Performance characteristics depend on domain distribution and compute allocation parameters