From the course: Create Your Own Code Assistant with Llama 2, Node.js, and React.js

Llama training basics

From the course: Create Your Own Code Assistant with Llama 2, Node.js, and React.js

Start my 1-month free trial Buy for my team

Llama training basics

“

- [Instructor] Let's talk about training our LLM. Here, I'm logged into HO Face with a spaces set up for AutoTrain. This is a program that will allow me to upload a data set and train my model on it to give me a new custom model based off of that training data set. In our assistant that we've built, we're using Code Llama, which is a pre-trained Llama 2 variant. Theoretically, we could create the same thing using Meta's Llama 3. We should select the Instruct, since it already has pre-trained on instruction sets. Then you would upload your training data, or you could get some from the Hugging Face Hub. Once you've done that, you can come over and set your parameters based off the JSON. You can give it some sort of chat template. We'll go with a tokenizer. A fixed precision of 16 works well. We're going to go with the SGD optimizer. The PEFT and LoRA, we want to keep it true, which will help to cut down on the amount of effort our LLM needs to put into training. You can leave your scheduler at linear. We're not going to be using Unsloth, though we could. Unsloth is another program we could use to help aid in our training. You can adjust batch size based off of how much training data you have. We're going to leave ours at two. We'll leave our block size at 1024 and our epics at three. Gradient accumulation is another variable you can play with to try and get better results. We'll leave it at our default four. Learning rate, we can leave at the default. And max model length, we can leave at default. Once you've set all of these, you can click start training and your LLM will go through its training data set and give you results that you can then verify.

Contents