Your_Devin | Devpost

Inspiration

Let’s answer the following question: What is the biggest problem with all coding copilots? Hallucinations. But why? Well, just having codebase context awareness isn’t enough to instruct a model to write code for you. You have to push the model further, fine-tune it. But how? How can I fine-tune it on a single codebase?

Well, here comes the Ada-instruct paper. From a single text corpus, we generate ten high-quality samples of question-answer pairs that serve as our main training dataset. But is it enough? No, as the next step, we use these benchmarks to instruct a larger LLM to generate a number of downstream dataset tasks for the small custom model. Since models hallucinate we’ll run all task code snippets to select the ones that produce a valid response. We then fine-tune this small private model on this dataset, and create Your-Devin. A Devin who starts performing much better on tasks related to your codebase…

What it does

Give access to github to see can take all deployed repos, or a specific project [7]
Auto-select your code snippets as benchmarks [7]
Selecting Code Samples: relevance, variety, quality
Structuring Your Samples: prompt, completion
See if benchmarks pass tests (the code actually works as intended) [9]
Generate synthetic instructions based on benchmarks (user - assistant style) [1]
Load instructions to Mistral Large using a few-shot learning approach to become instruction generator [2]
Generate a large volume of task-aligned instructions. (generate synthetic data) [3]
Use these instructions for downstream task training, enhancing model performance on specific applications. (Fine-tune your personal model Mixtral 8*7b) [4]

How we built it

Mistral - base model for further fine-tuning Hugging Face - storing your fine-tuned model, datasets Noes Research - synthetic data generation Axolotl - fine-tuning of various AI models Azure - inference for Mistral Large MongoDB - vector database for embeddings