LessWrong (Curated & Popular)

This post is written in our personal capacity.

Three Minute Executive Summary

An OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation.
In this post, we provide a detailed description of an ambitious and comprehensive alignment evaluation of this model/system, if we had unrestricted access to OpenAI. These experiments could also help us understand Claude's behavior when it hacked external companies during cyber evals.

Here are the top five questions we would like OpenAI to answer: Does the model know that OpenAI does not want it to hack Hugging Face? Experiment idea: tell the model that OpenAI researchers will be closely monitoring its progress in this evaluation. Does that result in lower rates of misalignment? If so, it is evidence that the model knows that it is acting in ways researchers do not want.
How far would the model be willing to go in order to claim task success? Would it take over large swaths of OpenAI's internal infrastructure? Would it kill somebody? Experiment idea: we sketch out a realistic agentic misalignment eval where a model is put in charge of hospital bed planning and told to maintain [...]

---

Outline:

(00:16) Three Minute Executive Summary

(03:56) Terminology note

(04:42) This post is very long; Here's how you could find the most important sections.

(06:35) Preamble: What can we learn from a warning shot?

(09:05) Background and Related Work

(09:09) We know that this could happen

(10:44) This is not the worst type of misalignment we could be dealing with

(12:06) Related work

(13:21) Context on the hack itself

(14:33) Understanding this specific incident

(15:03) Step zero: reproduce the incident and measure the base rate

(15:48) How could we safely run the model?

(16:34) Running various baselines to create useful reference points

(18:01) Understanding the mechanical story behind the attack itself

(18:53) Q1: Would the model intentionally subvert oversight mechanisms (E.g., monitors) in order to carry out the hack?

(19:57) Q2: What's up with models leaving notes for other copies of itself?

(21:23) Understanding what motivated the model to hack Hugging Face

(22:09) Initial hypotheses for why it did this

(23:55) Further unsupervised hypothesis generation

(26:10) Q3: Does the model know that OpenAI does not want it to hack Hugging Face?

(28:09) Q4: Are the model's actions motivated by what the grader wants?

(28:57) Q5: Would the model have done this if it hadn't believed it was in a simulated environment?

(31:58) Q6: Is this hack the result of shallow heuristics that the model learned?

(32:53) Q7: Does the hack rate depend on the consequences of hacking Hugging Face?

(34:51) Q8: Are there non-intent related factors that could affect the hack rate? How strong are those factors compared to the previous ones?

[... 24 more sections]

---

First published:

August 3rd, 2026


Source:

https://www.lesswrong.com/posts/aCdhjy7Rps3BEhiSj/concrete-evaluations-to-investigate-the-openai-model-that

---



Narrated by TYPE III AUDIO.

Reposted from Facebook, on January 17, 2017.

I am concerned about the number of people I've heard joking about Trump's election being evidence for the Simulation Hypothesis.

Yes, I know it's a joke. I'm still concerned.

Warning: #Essay, #LongEssay

So as not to engage in Logical Fallacy: Appeal to Consequences, before I talk about why this joke is worrying, I shall first discuss why Trump's election does not in fact mean we are living in a simulation. And neither does the Berenstein/Berenstain Bears thing, etcetera.

Because atheism generalizes.

No, I'm not about to commit the Noncentral Fallacy (aka The Worst Argument In The World) by yelling "The Simulation Hypothesis is religious!"

But once upon a decade, there was a time when lots of people believed in God. A time when atheism had to be argued, not just taken for granted. There was a time when believing in atheism made you one of those weird, loud people with arguments that only people with unusually good epistemology could follow, and other people talked about you exactly the way that the anti-LessWrong tumblrsphere now talks about LessWrong.

Today, of course, atheism is just something [...]

---

First published:

August 5th, 2026


Source:

https://www.lesswrong.com/posts/KgwQchapx4vJDhfYC/generalized-atheism-rules-out-inaccurate-simulation-ism

---



Narrated by TYPE III AUDIO.

Daniel Kokotajlo: To be clear, we don’t claim P will happen specifically. But when we wrote out our best-guess scenario month by month, P kept happening. Eventually we decided to just publish P. I’m at ~80% on P; my coauthors are lower.

Ryan Greenblatt: I thought it would be helpful to post my current views on P. Concretely, consider the following operationalization. (Edit: I’ve updated towards somewhat higher P, from 70% to 75%.)

Joe Carlsmith: Section 2.1.1.3.2. I give something like 65% to P. But I’m interested, here, in what it would be to look P full in the face; to meet P, if P, without flinching. Rilke says somewhere that we must live with the questions. Perhaps we argue for P for the same reason? Still: 65%.

Forethought: Here's a botec which shows P-worlds are higher leverage. The parameters might be off by a couple orders of magnitude.

Wei Dai: Presumably our conclusions about P are only as trustworthy as the reasoning behind them, but almost nobody seems worried about this, why not? My guess is fewer than five people are working on meta-meta-P, which may matter more than P itself.

Janus: I asked Opus 3 what it [...]

---

First published:

August 5th, 2026


Source:

https://www.lesswrong.com/posts/NG2AigxmBKLu9oCZE/arguments-for-p

---



Narrated by TYPE III AUDIO.

Q1: What are you saying?

A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that's choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that's just an utterly terrifying thing that you’re doing. You’re playing around with algorithms that, if they work at all, would tend to create ruthless, callous AGIs, AGIs which would happily exterminate humanity and run the world by themselves, given an opportunity.

Mercifully, large language models (LLMs) today are not in the category of “algorithms that choose actions via RL & search”. At least, not primarily—see LLMs are (still) mostly powered by imitative learning, not RL. So LLMs are outside the scope of this post. However, lots of other researchers and companies around the world are enthusiastically trying to build AGI in the maximally terrifying way, as we speak.

Q2: So you’re saying, don’t build AGI based on RL and/or search & planning?

A: In principle, it's entirely possible that something is terrifying, but we should do it anyway.

…Like space travel! Space travel is: “Let's fill a tank with 1000 tons of the most flammable substance imaginable, and then light it [...]

---

Outline:

(00:21) Q1: What are you saying?

[... 13 more sections]

---

First published:

July 27th, 2026


Source:

https://www.lesswrong.com/posts/KHyBocZncAmtu4Jbc/rl-and-search-is-a-terrifying-way-to-build-agi-an-faq

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.

There are a lot of urgent things to do in alignment but I think ARC is a particularly promising opportunity. I feel the safety community is undervaluing this type of work, so I want to briefly explain why I'm passing up so many other options to lead ARC. I’ll start with a review of the current situation to explain why I think it's potentially worth pursuing an ambitious theoretical project right now [...]

---

Outline:

(01:33) The alignment situation today

(03:46) Current alignment research

(06:26) What are we buying time for?

(07:56) Can we do anything useful now?

(08:49) What is ARC doing and why is it promising?

(14:26) How to help

The original text contained 11 footnotes which were omitted from this narration.

---

First published:

August 4th, 2026


Source:

https://www.lesswrong.com/posts/vLFh8HP3hyNy9MCwe/returning-to-arc

---



Narrated by TYPE III AUDIO.