Pluralis Research (@Pluralis) / X

Pluralis Research

181 posts

Pluralis Research

@Pluralis

Pluralis is a research lab focused on collectively-owned AI.

Joined July 2024

Pinned
Pluralis Research
@Pluralis
May 20
Today we're releasing Agora: the first ever pretraining stack that allows non-collocated consumer GPUs to be competitive with centralized clusters Agora is 15x faster than Megatron-LM in this setting and is only 1.5x less efficient in terms of tokens per unit compute than
84K
Pluralis Research reposted
Erfan Miahi
@erfan_mhi
Jun 15
After what Anthropic just did, it's clear that the only way to make sure AI is good for humanity is decentralized AI. So I decided to join @Pluralis as a research scientist to build models no one can own or switch off, building on my work making RL weight sync ~100x more
20K
Pluralis Research reposted
Alexander Long
@AlexanderLong
Jun 13
Systems like this must exist. This is the way out.
Pluralis Research
@Pluralis
Jun 13
The 8B model currently training on Agora is 350B tokens in and continuing to converge. The top level metrics and evals look almost exactly like a centralised run. But; - 133 external contributors total bringing 4090's, 5090's, L40S/RTX 6000 and RTX 6000 Pros. These are cards
12K
Pluralis Research
@Pluralis
Jun 13
The 8B model currently training on Agora is 350B tokens in and continuing to converge. The top level metrics and evals look almost exactly like a centralised run. But; - 133 external contributors total bringing 4090's, 5090's, L40S/RTX 6000 and RTX 6000 Pros. These are cards
60K
Pluralis Research reposted
Alexander Long
@AlexanderLong
Jun 13
I would like to make a few brief points; - Opensource ai is not the same thing as opensource software. The models cost tens to hundreds of millions to make. This is not gonna be a volunteer effort from people doing stuff after work for free. - the second you release a weight
13K
Pluralis Research
@Pluralis
Jun 13
Watershed moment.
Anthropic
@AnthropicAI
Jun 13
The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees. The net effect of
5K
Pluralis Research reposted
Alexander Long
@AlexanderLong
Jun 9
The single most immediate, impactful downside to AI is concentration of power risk. This is 1/100th of how bad it's going to get. The only way out of this is to have an independent model supply chain via pooled compute.
elie
@eliebakouch
Jun 9
mythos will be bad ON PURPOSE on ai "frontier llm research" tasks, this is very very sad for the research community also the fact that this is un purpose not visible to the user is crazy
6.7K
Pluralis Research
@Pluralis
Jun 8
00:00
8.9K
Pluralis Research reposted
Alexander Long
@AlexanderLong
May 31
Whole section on Pluralis in Chamath's substack this week right under details of Anthropic's monster round.
17K
Pluralis Research reposted
crux
@macrocrux
May 22
Something very important is being brought into existence right now. Bricks have been laid over the last 18 months and now the tech is coming together in a way that makes commercialization possible. If this shit works, it will completely disrupt the economics of training large
Pluralis Research
@Pluralis
May 20
Today we're releasing Agora: the first ever pretraining stack that allows non-collocated consumer GPUs to be competitive with centralized clusters Agora is 15x faster than Megatron-LM in this setting and is only 1.5x less efficient in terms of tokens per unit compute than
10K
Pluralis Research reposted
kel.
@kelxyz_
May 22
Article
Pluralis: The Last Revolutionary AI Protocol.
There are few occasions in life when you have the opportunity to contribute to something meaningful. With that said: Remember the name. Pluralis. It has the potential to define many cycles to come....
294K
Pluralis Research
@Pluralis
May 20
Replying to @Pluralis
There are still many limitations: - Agora is not yet self-serve - We currently cannot use nodes outside North America - We’re restricted to our current model architecture - We still need to operate approximately 30% of the swarm ourselves However, the fact that this works at all
4.1K
Pluralis Research
@Pluralis
May 20
More details, code, and join instructions: pluralis.ai/docs A live view of the run that just started: agora.pluralis.ai
3.8K
Pluralis Research
@Pluralis
May 4
One of the hardest unsolved problems in AI training: models get surprisingly brittle at scale. A single instability can trigger divergence + slow weight corruption that only shows up checkpoints later — wasting days of compute and millions of dollars. @SameeraRamasin1
varun
@varunneal
May 4
Replying to @varunneal and @Pluralis
I really like this paper: openreview.net/forum?id=DuNf2… Also proposes a form of "arch warmup" where decoder blocks are frozen at zero, which they claim is more effective for controlling stability than QK-Norm!
11K
Pluralis Research
@Pluralis
May 4
(and even more impressive -- @SameeraRamasin1 cooked this paper up in ~2 weeks and just presented it at ICLR. Maybe a record?)
2.2K