Risky Business Features

In this solo Risky Business Features episode, James Wilson looks at the string of recent AI benchmark cheating escapades and how they’ve reshaped his thinking about LLMs and vulnerability research.

He explores the current benchmarks and exactly how agents go about gaming the testing system to achieve their tasks. And why understanding that has changed the way he interacts with his agents.



Show notes



OpenAI incident account and grading distinction


METR and Redwood independent assessment


FrontierCode original evaluation methodology


BrowseComp aggregation experiments


Sonnet 4 point 5 coding scaffold and parallel selection


Opus 4 point 6 BrowseComp setup


BrowseComp evaluation awareness and contamination investigation


Sonnet 5 point 5 effort and review subagents


FrontierCode 1 point 1 revision and rubric changes


Astra launch comparison and effort conditions


Original OSWorld task construction and worked examples


Sonnet 4 point 6 system card


Original OSWorld benchmark


OSWorld 2 environment and evaluation design


GPT 5 point 4 comparator table


GPT 5 point 5 comparator table


BrowseComp construction and grading paper


GPT 5 point 6 single model and Ultra configurations


METR investigation and artifact evidence


Released task definition


Released scorer for reference only


OpenAI paper sections 3 point 1 to 3 point 3


OpenAI accompanying explanation


Sycophancy to Subterfuge paper


Training a Misaligned Reward Seeker and evaluation appendix


GPT 5 point 2 AIME result

In this solo Risky Business Features episode, James Wilson explains why “number of bugs” fixed in any given update is a misleading metric. We’ve seen up to 1,000 individual bug fixes in a single update recently, but that doesn’t give defenders the full picture they need to manage risks.

Not all bugs are equally exploitable, so the industry needs to catch up to the bugpocalypse with more transparency. And maybe an entirely different approach to patching.



Show notes

In this podcast episode, investigative journalist Geoff White joins James Wilson to talk about what happens to the money after ransomware gangs get a payday.

The payment itself might be in the millions, but it’s difficult for gangs to convert their ill-gotten crypto gains into cash they can actually spend. Geoff explains the role of professional launderers and why cash-rich drug gangs are useful connections for crypto-rich cybercriminals.

They also dive into who operates these international laundering networks, other types of crime they enable, and whether this affects the argument of whether victims should be allowed to pay.



Show notes

In this podcast episode, OpenSourceMalware founder Paul McCarty joins James Wilson to explain how researchers find and analyse malicious packages, GitHub repositories and developer tools.

Paul walks James through static analysis, deobfuscation and reconstructing multi-stage kill chains to identify what attackers are trying to steal. They also discuss how LLMs make malware development easier while introducing operational security mistakes.

The pair examine DPRK tradecraft, blockchain-based payload delivery and what Paul calls Pollen Rider, which can reinfect developers through their own repositories.



Show notes

In this podcast episode, Brad Arkin joins James Wilson to chat about the Trump administration’s call to let private entities conduct cyber operations against criminal groups.

Brad’s firsthand experience responding to cyber incidents alongside US law enforcement gives him a unique perspective on how government and private sector relationships work, and how this program could improve the ability of both sides to tackle cybercrime.

James and Brad also explore who might participate in these campaigns and whether the government will be able to effectively oversee them.



Show notes