Ram

@PothamRam

Research Engineer @ DepthFirst reducing cyber risk. Prev AI Safety Fellow @ Redwood catching misaligned models. Previously exited an AI startup.

Joined April 2022

525

Following

153

Followers

Replies Reposts Media

Ram
@PothamRam
Sep 23
At depthfirst, we evaluated Opus 5.5 on our vulnerability detection eval. From my investigation, it's more selective and doesn't search as hard as OpenAI models. It is definitely an improvement on Opus-5, being more token efficient while having overall better performance!
Francesco Piccoli
@francescpicc
Sep 23
Opus 5.5 is one of the few models we have seen gain recall and precision together as reasoning effort rises. Of eight model families we ran at more than one effort setting on dfbench, six traded precision for recall. Opus 5 lost both. Opus 5.5 gained 6.8 points of recall and 4.3
1
Ram
@PothamRam
Sep 22
As agents keep pushing out more and more code, the security of that code is super important! It's important that we focus on security especially as autonomous hacking is getting cheaper.
depthfirst
@depthfirstlabs
Sep 22
Our co-founder and CEO @qasimmith sat down with @SaraIttelson at Accel, one of our earliest investors, to discuss the forces reshaping security: Starting depthfirst. Building secure software at speed. Finding the TikTok vulnerability with our flagship model dfs-large1, a flaw
00:00
Ram
@PothamRam
Jul 24
I think it's important that we can measure how good models are at automating AI safety research because we want to predict when this will happen as this will make the future a lot safer. I discuss a methodology of automating AI control research here:
Evaluating Red Team and Blue Team Capability for AI Control Research
From dearfutureais.substack.com
Ram
@PothamRam
Apr 20
We've been working on a testbed for monitors. I think this is probably the best place to make AI monitors to detect if your agent is malicious
Tyler Tracy
@tylertracy321
Apr 20
Introducing LinuxArena: a new control setting where AI agents work directly on live, multi-service production software. 20 environments, 1,671 main tasks, 184 side tasks. The largest and most diverse control setting for software engineering to date.
Ram
@PothamRam
Apr 7
My team did some safety evaluations on the Kimi k2.5 model, I found that it is better at sabotage than Opus 4.6
Yong Zheng-Xin
@yong_zhengxin
Apr 6
🚨New paper! How safe and aligned is Kimi K2.5? We found concerning dual-use capabilities, sabotage and self-replication tendencies, political censorship on Chinese-language queries, and potential agentic misuse risks. (1/N)
1