As a part of our effort to replicate LLaMA in an open-source manner, we are pleased to announce the release of preview of the 7B OpenLLaMA model that has been trained with 200 billion tokens on the RedPajama dataset.
- We are excited to share Large World Model (LWM), a general-purpose 1M context multimodal autoregressive model. It is trained on a large dataset of diverse long videos and books using RingAttention, and can perform language, image, and video understanding and generation.
- New paper w/ @matei_zaharia @pabbeel on transformers with large context size. We propose RingAttention, which allows training sequences that are device count times longer than those of prior state-of-the-arts, without attention approximations or incurring additional overhead.
- Humans learn from rich feedback in the form of language. Why not turning all feedback into a sentence to train models? We propose CoH: Just tell models which ones are not good and which ones are better. Better than SFT and RLHF on summary and dialogue tasks.
- 1/ Excited to share our new paper with @pabbeel on long context models! 📚✍️ Check it out here: arxiv.org/abs/2305.19370 Training 7B models with over 130K or 13B models with over 64K context windows on just 8 A100 GPUs! 😮🖥️ Curious how we did it?
- Excited to share our new work that explores the relationship between contrastive learning, discriminative modeling & generative modeling, through the lens of energy-based models. 🎓 arxiv.org/abs/2007.09070 💻 github.com/HDGE w/ @pabbeel summary thread: [1/N]
- We introduce an unsupervised method to align text and image. Language Quantized AutoEncoders (LQAE) enables few-shot image classification with GPT3 and linear classification of images based on RoBERTa text features. paper: arxiv.org/abs/2302.00902 code: github.com/lhao499/lqae
- Can language model pretraining be even better? Our paper shows that by randomly masking input tokens during pretraining, the zero-shot, few-shot, and fine-tuning performance can be significantly improved. arxiv.org/abs/2210.13432 🧵
- Excited to share M3AE, a simple but effective model for multimodal representation learning. TLDR: M3AE learns a unified encoder for both vision and language from both paired image-text data as well as unpaired data. arxiv.org/abs/2205.14204 w/ @younggeng Summary thread: [1/N]
- A new preprint “Behavior From the Void: Unsupervised Active Pre-Training”. arxiv.org/abs/2103.04551 w/ @pabbeel TLDR: A simple yet effective method for reward-free unsupervised pre-training in RL via particle-based entropy maximization. Here is a summary thread👇
- RingAttention's Jax code is available at github.com/lhao499/llm_la… In end-to-end FSDP training on GPU (7B params, 8x A100 80G), context expands from 32K to 256K tokens and can reach 16M tokens with 512x A100. On TPU (7B params, 1024x TPUv4, FSDP), context can reach 8M tokens.New paper w/ @matei_zaharia @pabbeel on transformers with large context size. We propose RingAttention, which allows training sequences that are device count times longer than those of prior state-of-the-arts, without attention approximations or incurring additional overhead.
- The code of blockwise parallel transformer is now available. github.com/lhao499/blockw…1/ Excited to share our new paper with @pabbeel on long context models! 📚✍️ Check it out here: arxiv.org/abs/2305.19370 Training 7B models with over 130K or 13B models with over 64K context windows on just 8 A100 GPUs! 😮🖥️ Curious how we did it?
- Replying to @haoliuhlThe possibility of very large context introduces exciting opportunities, such as video-audio-language model, learning from extended feedback or trial-and-error, and AI for science data like gene sequence. Paper link: arxiv.org/abs/2310.01889 Code link: coming soon
- Replying to @haoliuhlPaper: arxiv.org/abs/2402.08268 Models: huggingface.co/LargeWorldModel Code: github.com/LargeWorldMode… Website: largeworldmodel.github.io This is a joint work with amazing people @wilson1yan, @MateiZaharia, @PieterAbbeel
