Ai2 (@allen_ai) / X

Ai2

3,792 posts

Ai2

@allen_ai

Breakthrough AI to solve the world's biggest problems. › Join us: allenai.org/careers › Newsletter: tinyurl.com/3vc2r2m8

Seattle, WA

allenai.org

Joined September 2015

Affiliates Replies Media

Pinned
Ai2
@allen_ai
May 7
Today we’re bringing new NSF OMAI compute online with NVIDIA Blackwell Ultra-powered systems, turning a $152M national investment from @NSF & @nvidia into a foundation for truly open AI research. 🧵
Ai2
@allen_ai
9m
We built SciArena to test how well AI models handle scientific literature questions, as judged by researchers. It's retiring July 15, and the results are in: ~1,700 users cast ~3,900 votes. Here's what they told us. 🧵
00:00
Ai2
@allen_ai
9m
Replying to @allen_ai
Overall, we’re pleased with SciArena's contributions: a dataset of real scientific questions for deep research agents, expert preferences, & verified feedback for building better evaluations. Thank you to the researchers who contributed questions, votes, & rationales!
Ai2
@allen_ai
9m
For a deeper look at what SciArena revealed about how scientists evaluate AI-generated answers to Qs about the scientific literature, read our NeurIPS 2025 Spotlight paper—which includes findings beyond the leaderboard. ⬇️
arxiv.org
SciArena: An Open Evaluation Platform for Non-Verifiable...
We present SciArena, an open and collaborative platform for evaluating foundation models on scientific literature-grounded tasks. Unlike traditional benchmarks for scientific literature...
Ai2
@allen_ai
Jul 13
Our @SkylightMarine team is rolling out Shippy, an AI agent for the people protecting our ocean in real time. A wrong answer at these stakes can send a patrol vessel miles off course. Here's the architecture we built so analysts can trust it. 🧵
Ai2
@allen_ai
Jul 13
Replying to @allen_ai and @SkylightMarine
Custom evals test each Shippy build for accuracy & safety. Experts write scenarios with weighted criteria: data accuracy, boundary handling, and source attribution. Each runs in a real session against live data; an LLM judge scores it. Regressing builds don’t ship.
Ai2
@allen_ai
Jul 13
Shippy is currently in private preview for select partners. For more on how we built it, what’s next, and how we’re ensuring users can trust and fully trace its outputs, read our new engineering blog:
What building Shippy taught us about building agents | Ai2
From allenai.org
Ai2
@allen_ai
Jul 10
olmOCR 2 is now in the Ai2 Playground—our home for our fully open text, video, and image understanding models. 🧵
Ai2
@allen_ai
Jul 10
olmOCR 2 is our compact vision-language model that reads challenging documents in a single pass. It can handle samples that usually break OCR, including handwriting, equations, tables, & multi-column layouts.
00:00
Ai2
@allen_ai
Jul 10
Try olmOCR 2 in the Ai2 Playground, check out our blog for more info, & download the weights and data from Hugging Face: ▶️ Playground: playground.allenai.org/model/olmocr-2… 📝 Blog: allenai.org/blog/olmocr-2 🤗 Model & data: huggingface.co/allenai/olmOCR…
Ai2
@allen_ai
Jul 10
OlmoEarth turns Earth data into insights—in hours, not years. But it's only as powerful as the people using it. So, we brought them into one room. 🧵
00:00
Ai2
@allen_ai
Jul 10
Replying to @allen_ai
We’re grateful to teams who brought their missions into the room: @mercycorps, @sig_gis, SIG-NAL, @SERVIRGlobal, @developmentseed, BAI Group, & researchers from @UW @UCBerkeley, & @Northeastern.
Ai2
@allen_ai
Jul 10
The fastest way to make AI useful is to build it with the people who'll use it—fully in the open for shared, real-world impact. 💻 Learn more:
OlmoEarth | Ai2
From allenai.org