Project Setup & Run Documentation

Unsupervised Dense Reward Generation for RL Fine-Tuning (PPO) of BC Policies

1. Server & Hardware Info

Component	Details
GPU	NVIDIA GeForce RTX 3090 Ti
VRAM	24 GB
RAM	60 GB
CUDA Version	12.8
Driver Version	570.195.03
Python	3.8.13 (conda env `libero`)

2. Repo Structure

DensePPO/
├── .gitignore
├── requirements.txt
├── DOCUMENTATION.md
│
├── scripts/
│   ├── eval_ppo.py                  # Evaluate residual (dense) PPO models
│   ├── eval_bc_vanilla_ppo.py       # Evaluate BC+Vanilla PPO ablation models
│   ├── evaluate_waypoints.py        # Compute silhouette scores, timeline plots
│   ├── residual_ppo_single.py       # Residual PPO (single task, --task arg)
│   ├── bc_vanilla_ppo_single.py     # BC+Vanilla PPO ablation (sparse reward, --task arg)
│   └── visualization/
│       ├── extract_frames.py
│       ├── extract_waypoints.py     # Run BC encoder on demos, K-means cluster
│       ├── plot_combined.py         # BC training curves
│       ├── plot_loss.py
│       └── visualize_waypoints.py   # t-SNE + waypoint timeline plots
│
└── outputs/
    ├── eval_20260305_214058.txt          # Saved eval run (residual/dense PPO)
    ├── eval_bc_vanilla_20260309_145353.txt  # Saved eval run (BC+Vanilla PPO ablation)
    ├── ppo_logs/
    │   └── task0.log – task9.log        # Residual (dense) PPO training logs
    ├── ppo_vanilla_logs/
    │   └── bc_vanilla_task0.log – bc_vanilla_task9.log  # BC+Vanilla PPO training logs
    └── waypoints/
        └── task0_waypoints.pt – task9_waypoints.pt  # K-means waypoint centers

Large files excluded from repo via .gitignore: .pth, .zip, .npy, .hdf5, datasets, checkpoints

3. Environment Setup

Step 1: Install Miniconda

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p /234/miniconda
eval "$(/234/miniconda/bin/conda shell.bash hook)"

Step 2: Accept Conda Terms of Service

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r

Step 3: Create Python 3.8 Environment

conda create -n libero python=3.8.13 -y
source /234/miniconda/bin/activate libero

Step 4: Install PyTorch (CUDA 11.8)

pip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118

The default torch==1.11 in LIBERO's requirements.txt is too old for the server kernel. Must upgrade to 2.0.1.

Step 5: Clone and Install LIBERO

cd /234
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -r requirements.txt
pip install -e .

Step 6: Install Additional Dependencies

pip install scikit-learn
pip install stable-baselines3==2.3.2
pip install gymnasium==0.29.1 shimmy>=0.2.1
pip install tqdm rich

Step 7: Apply Bug Fix to LIBERO Source

# Fix: persistent_workers=True crashes when num_workers=0
# File: /234/LIBERO/libero/lifelong/algos/base.py, line ~145
sed -i 's/persistent_workers=True/persistent_workers=False/' /234/LIBERO/libero/lifelong/algos/base.py

Step 8: Download LIBERO-Spatial Dataset

cd /234/LIBERO
python benchmark_scripts/download_libero_datasets.py \
  --datasets libero_spatial \
  --download-dir /234/data/datasets

Activate Environment (every new session)

source /234/miniconda/bin/activate libero

4. Phase 1: BC Baseline Training

Run Training

cd /234/LIBERO
python libero/lifelong/main.py \
  seed=42 \
  benchmark_name=LIBERO_SPATIAL \
  policy=bc_transformer_policy \
  lifelong=base \
  train.num_workers=0 \
  eval.num_workers=0 \
  eval.use_mp=false

Key Hyperparameters

Parameter	Value
Policy	BCTransformerPolicy (ResNet-18 + Transformer)
Optimizer	AdamW
Learning Rate	1e-4 cosine decay to 1e-5
Batch Size	32
Epochs	50 per task
Eval Frequency	Every 5 epochs, 20 rollouts
Demos	50 per task (500 total)
Image Size	128×128
Sequence Length	10 frames
Action Head	GMM (5 modes)
Seed	42

Model Checkpoints

/234/LIBERO/experiments/LIBERO_SPATIAL/Sequential/BCTransformerPolicy_seed42/run_003/
└── task{0-9}_model.pth

BC Baseline Results (run_003)

Task	Description	BC Success
0	Between plate & ramekin	80%
1	Next to ramekin	80%
2	Table center	100%
3	On cookie box	90%
4	Top drawer	75%
5	On ramekin	70%
6	Next to cookie box	85%
7	On stove	90%
8	Next to plate	80%
9	On wooden cabinet	65%
AVG		81.5%

5. Phase 2: Latent Waypoint Extraction

Run

source /234/miniconda/bin/activate libero
python3 /234/scripts/visualization/extract_waypoints.py

What It Does

Loads each task's frozen BC encoder
Runs all 50 demos through spatial_encode + temporal_encode
Extracts 64-dim latent vectors per timestep
K-means clusters (k=3, 5, 10) to find waypoints
Re-orders clusters by mean temporal position

Outputs

/234/outputs/waypoints/
├── task{0-9}_latents.npy       # Raw latents (N, 64) — not in repo (large)
└── task{0-9}_waypoints.pt      # K-means centers for k=3,5,10

6. Phase 3: Waypoint Visualization & Quality Evaluation

Run

python3 /234/scripts/visualization/visualize_waypoints.py   # t-SNE + timeline
python3 /234/scripts/evaluate_waypoints.py                  # silhouette scores

Silhouette Scores (k=5)

Metric	Value
Mean	0.227
Best	Task 6: 0.292
Worst	Task 8: 0.179

7. Phase 4: Residual PPO Fine-Tuning

Architecture

Frozen BC policy provides base 7-dim actions
PPO learns 7-dim residual correction
Final action = clip(bc_action + 0.05 * residual, -1, 1)
Observation space: 64-dim latent from frozen BC encoder
Policy/value network: MLP [256, 256]

Dense Reward Function

reward = (prev_dist - curr_dist) * 2.0    # waypoint progress (every step)
       + 0.5  if dist < 0.15              # waypoint reached bonus
       + 1.0  if task_success             # task completion bonus

Key Hyperparameters

Parameter	Value
RESIDUAL_SCALE	0.05
K (waypoints)	5
MAX_STEPS	200
PPO_STEPS	500,000
learning_rate	3e-4
n_steps	2048
batch_size	64
n_epochs	10
gamma	0.99
gae_lambda	0.95
clip_range	0.2
ent_coef	0.01

Run Single Task

source /234/miniconda/bin/activate libero
tmux new -s ppo_task4
python3 /234/scripts/residual_ppo_single.py --task 4

Run Multiple Tasks in Parallel

tmux new -s ppo_batch
source /234/miniconda/bin/activate libero
python3 /234/scripts/residual_ppo_single.py --task 0 > /234/outputs/ppo_logs/task0.log 2>&1 &
python3 /234/scripts/residual_ppo_single.py --task 1 > /234/outputs/ppo_logs/task1.log 2>&1 &
python3 /234/scripts/residual_ppo_single.py --task 5 > /234/outputs/ppo_logs/task5.log 2>&1 &
python3 /234/scripts/residual_ppo_single.py --task 9 > /234/outputs/ppo_logs/task9.log 2>&1 &
wait && echo "Done!"
# Detach: Ctrl+B then D
# Reattach: tmux attach -t ppo_batch

Monitor Training

# Progress across all tasks
grep "episode_reward" /234/outputs/ppo_logs/task*.log | tail -10

# Live tail specific task
tail -f /234/outputs/ppo_logs/task4.log

# Check which processes are running
ps aux | grep residual_ppo_single | grep -v grep

# GPU / RAM usage
nvidia-smi --query-gpu=memory.used,memory.free,utilization.gpu --format=csv && free -h

Kill All PPO Runs

# Kill residual (dense) PPO runs
kill $(ps aux | grep residual_ppo_single | grep -v grep | awk '{print $2}')
# Kill BC+Vanilla PPO runs
kill $(ps aux | grep bc_vanilla_ppo_single | grep -v grep | awk '{print $2}')

Model Output Location

/234/outputs/ppo/task{id}/
├── best_model/best_model.zip    # Best checkpoint by eval reward
├── final_model.zip
├── checkpoints/
├── eval_logs/
├── tb_logs/
└── monitor.monitor.csv

8. Phase 5: Evaluation

Evaluate Residual PPO (all tasks, save output)

source /234/miniconda/bin/activate libero
python3 /234/scripts/eval_ppo.py 2>&1 | tee /234/outputs/eval_$(date +%Y%m%d_%H%M%S).txt

Evaluate BC+Vanilla PPO Ablation

python3 /234/scripts/eval_bc_vanilla_ppo.py 2>&1 | tee /234/outputs/eval_bc_vanilla_$(date +%Y%m%d_%H%M%S).txt

View All Saved Eval Results

ls /234/outputs/eval_*.txt
grep "Task\|Average" /234/outputs/eval_*.txt

9. BC + Vanilla PPO Ablation

BC+Vanilla PPO uses the same residual architecture as the main method (frozen BC policy + PPO residual) but with sparse reward only (no waypoints, no dense shaping) — used to isolate the contribution of the dense reward signal.

Architecture

Same as Residual PPO: frozen BC policy provides base 7-dim actions, PPO learns 7-dim residual correction
Reward: +1.0 on task success only (no waypoint progress signal)
Observation space: 64-dim latent from frozen BC encoder

Run All 10 Tasks in Parallel

tmux new -s ppo_vanilla
source /234/miniconda/bin/activate libero
for i in 0 1 2 3 4 5 6 7 8 9; do
  python3 /234/scripts/bc_vanilla_ppo_single.py --task $i > /234/outputs/ppo_vanilla_logs/bc_vanilla_task${i}.log 2>&1 &
done
wait && echo "Done!"

Results (BC + Vanilla PPO vs BC + Dense PPO)

Task	Description	BC	BC+Vanilla PPO	BC+Dense PPO
0	Between plate & ramekin	80%	80.0%	75%
1	Next to ramekin	80%	95.0%	95%
2	Table center	100%	100.0%	95%
3	On cookie box	90%	85.0%	90%
4	Top drawer	75%	75.0%	100%
5	On ramekin	70%	90.0%	85%
6	Next to cookie box	85%	85.0%	90%
7	On stove	90%	95.0%	95%
8	Next to plate	80%	70.0%	85%
9	On wooden cabinet	65%	80.0%	75%
AVG		81.5%	85.5%	88.5%

10. Links

Resource	URL
GitHub Repo	https://github.com/leejason2025/DensePPO
HuggingFace Models & Data	https://huggingface.co/datasets/leejason2025/DensePPO

11. Common Issues & Fixes

Issue	Fix
`persistent_workers` crash	Set to `False` in `libero/lifelong/algos/base.py`
h5py pickle error during eval	Add `eval.use_mp=false` to training command
PyTorch too old for server kernel	Upgrade to `torch==2.0.1+cu118`
`get_task_embs` crashes	Pass `[task.language]` not `[task]`
BC action shape (10,7) not (7,)	Use `dist.sample()[0, -1]` not `[0]`
Process killed when terminal closes	Always use `tmux` for long-running jobs

Name		Name	Last commit message	Last commit date
images		images
outputs		outputs
scripts		scripts
.gitignore		.gitignore
README.md		README.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

Project Setup & Run Documentation

Unsupervised Dense Reward Generation for RL Fine-Tuning (PPO) of BC Policies

1. Server & Hardware Info

2. Repo Structure

3. Environment Setup

Step 1: Install Miniconda

Step 2: Accept Conda Terms of Service

Step 3: Create Python 3.8 Environment

Step 4: Install PyTorch (CUDA 11.8)

Step 5: Clone and Install LIBERO

Step 6: Install Additional Dependencies

Step 7: Apply Bug Fix to LIBERO Source

Step 8: Download LIBERO-Spatial Dataset

Activate Environment (every new session)

4. Phase 1: BC Baseline Training

Run Training

Key Hyperparameters

Model Checkpoints

BC Baseline Results (run_003)

5. Phase 2: Latent Waypoint Extraction

Run

What It Does

Outputs

6. Phase 3: Waypoint Visualization & Quality Evaluation

Run

Silhouette Scores (k=5)

7. Phase 4: Residual PPO Fine-Tuning

Architecture

Dense Reward Function

Key Hyperparameters

Run Single Task

Run Multiple Tasks in Parallel

Monitor Training

Kill All PPO Runs

Model Output Location

8. Phase 5: Evaluation

Evaluate Residual PPO (all tasks, save output)

Evaluate BC+Vanilla PPO Ablation

View All Saved Eval Results

9. BC + Vanilla PPO Ablation

Architecture

Run All 10 Tasks in Parallel

Results (BC + Vanilla PPO vs BC + Dense PPO)

10. Links

11. Common Issues & Fixes

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages