Life-Learn: SDFT Chat Correction Pipeline

A backend that demonstrates Self-Distillation Fine-Tuning (SDFT) triggered by user corrections in chat. When a user corrects the model, the system detects the correction, generates expert demonstrations, runs on-policy SDFT, and the model permanently learns from the correction.

Based on the paper "Self-Distillation Enables Continual Learning" by Shenfeld et al.

User corrects the model in chat
    -> Correction Detector (local LLM classifier)
    -> Prompt Augmenter (local LLM generates diverse question variations)
    -> Expert Demo Generator (GPT-4o-mini produces correct demonstrations)
    -> Data Formatter (HuggingFace Dataset with prompt + teacher_prompt)
    -> SDFT Trainer (on-policy distillation via DistilTrainer)
    -> Verification (fresh context, analogous question -- model should get it right)

During SDFT training (handled internally by DistilTrainer):

• Student generates on-policy rollouts from the prompt alone
• Teacher (same model, EMA weights) is conditioned on prompt + expert demo
• Loss: KL divergence between student and teacher distributions
• Teacher weights track student via exponential moving average

git clone https://github.com/Honyant/life-learn.git
cd life-learn

conda env create -f environment.yml
conda activate sdft

Or manually:

conda create -n sdft python=3.10
conda activate sdft
pip install -r requirements.txt
pip install pytest openai

export OPENAI_API_KEY="sk-..."

Or place it in ~/Documents/temp_dir/.env:

export OPENAI_API_KEY="sk-..."

This repo now includes a full-stack workspace with:

• secure authentication (session cookies + CSRF)
• multi-organization tenancy with role-based access
• invite-based member management per org
• multi-chat threads per organization
• per-organization model versioning and rollback
• correction-triggered continual learning jobs (SDFT)

Run it with:

uvicorn sdft_platform.main:app --host 0.0.0.0 --port 8080 --reload

Then open http://localhost:8080.

Environment variables (optional):

python -m pytest sdft_correction/test_pipeline.py -v -k 'not gpu'

python -m pytest sdft_correction/test_pipeline.py -v -m gpu -s

python -m sdft_correction.chat

Chat with the model. When you correct it, the pipeline triggers automatically:

1. Detects the correction
2. Generates 8 prompt variations via the local model
3. Gets 4 expert demos per prompt from GPT-4o-mini (~36 training pairs)
4. Runs SDFT training
5. Loads the trained model and verifies on an analogous question

.
├── distil_config.py          # SDFT config (patched: added full_logit_distillation field)
├── distil_trainer.py         # DistilTrainer from the Self-Distillation paper
├── main.py                   # Original paper's training script (reference)
├── sdft_platform/            # New multi-tenant web workspace (FastAPI + UI)
├── environment.yml           # Conda environment file
├── requirements.txt          # Pip dependencies
└── sdft_correction/          # Chat correction pipeline
    ├── config.py             # PipelineConfig (model, paths, hyperparams)
    ├── inference.py          # Local model wrapper for generation
    ├── correction_detector.py # LLM-based correction classifier
    ├── augmenter.py          # Generates diverse prompt variations
    ├── expert_demos.py       # GPT-4o-mini expert demonstration generator
    ├── data_formatter.py     # Formats data for DistilTrainer
    ├── trainer.py            # SDFT training wrapper (full fine-tuning)
    ├── chat.py               # Interactive chat loop
    ├── conftest.py           # Pytest config
    └── test_pipeline.py      # 22 unit tests + 1 GPU integration test

Edit sdft_correction/config.py to change defaults:

• On-policy student rollouts: generate_from_teacher=False -- the student generates its own completions during training, matching the paper's Algorithm 1
• External expert demos: GPT-4o-mini provides the demonstration context c for the teacher, analogous to how the paper uses GPT-4o for SciKnowEval
• Full fine-tuning: No LoRA, matching the paper's experimental setup. Use larger GPUs for 7B+ models.
• Forward KL (alpha=0.0): Matches the reference main.py implementation
• EMA teacher (sync_ref_model=True, ref_model_mixup_alpha=0.01): Teacher tracks student progress while smoothing updates

This project builds on the Self-Distillation repo by Shenfeld et al.:

Self-Distillation Enables Continual Learning (ICLR 2025) Idan Shenfeld, Mehul Damani, Jonas Hubotter, Pulkit Agrawal https://arxiv.org/abs/2601.19897