Lecture 16: Parallelism and Scaling
youtu.be/Mpg1YJfAEH0
- Basics of training on one device
- Parallelization on multiple devices (e.g., data, tensor, pipeline parallel)
- Combining and comparing strategies
What do nucleus sampling, tree-of-thought, and PagedAttention have in common?
They're all part of our new survey: "From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models"
arxiv.org/abs/2406.16838
Announcing the L3 Lab at CMU!
cmu-l3.github.io
We focus on Learning, Language, and Logic, including:
- Principles of ML for language
- ML in high-trust areas, such as verifying math and programs
- ML systems that improve over time
Recruiting PhD students for fall 2024!
Interested in LLMs and Lean?
Check out LLMLean, a tool for using LLMs to suggest proof steps and complete proofs in Lean:
github.com/cmu-l3/llmlean
Here's an example of using LLMLean with GPT-4o to solve problems from Mathematics in Lean:
