conda create -yn vase python=3.11
conda activate vase
pip install torch==2.4.0
pip install -r requirements.txt
pip install flash-attn --no-build-isolation # we've tested on 2.7.3 and 2.8.3; don't use 2.7.4
cd eval/reasoning_tasksbash myscripts/{task_name}/eval_{method_name}.shNote: in the .sh files, you can change num_gpus for data parallelism.
The following script downloads per-problem test cases. We tested on medium examples of LiveCodeBench-v6 (383 in total).
pip install "datasets<4"
python data/livecodebench/download_tests.py # test data lands in data/livecodebench/livecodebench_testspython aggregate_results.py {task_name}/{model_name}See tables/ for our results reported in the paper.
Measure decode-phase throughput (tokens/s) and peak GPU memory for each KV-cache method on a single gpu:
bash benchmark/run_thpt.shThis sweeps token budgets (2048/4096/6144) and output lengths (16384/32768) across different methods.
- eval/reasoning_tasks/repo_walkthrough.md
- This repo is modified from: https://github.com/microsoft/SeerAttention/tree/main/eval/reasoning_tasks
